Learn Dutch: De mogelijkheid had nuttig kunnen zijn. - The possibility could have been useful.

This is because of the IPP (infinitive instead of participle) rule. When a modal verb like kunnen is used in the perfect tense together with another verb (here, zijn), the modal verb stays an infinitive rather than becoming a past participle. This is why you get a cluster of infinitives at the end of the sentence.

Why do we use had as the helper verb instead of was, since the final verb is zijn (to be)?

In these structures with a modal verb, the auxiliary helper verb is determined by the modal, not by the main action verb at the end. Because kunnen always takes hebben in the perfect tense, we use had, even though the verb zijn would normally use a form of zijn as its own helper.

What is the exact word order happening at the end of the sentence?

The conjugated verb had takes the second position in the sentence. The rest of the verbs stack up at the very end as a cluster of infinitives (kunnen zijn). Any other information, like the adjective nuttig, is placed between the second-position verb and that final verb cluster.

I see mogelijk (possible) inside mogelijkheid. How does this word ending work?