Learn Finnish: Onko tämä hattu sinun?

sinun is the genitive (possessive) form of sinä (you). By adding the -n ending, it becomes your or yours. Finnish doesn't use two different words for your and yours like English does, so sinun covers both depending on the sentence.

If we are talking about possession, why doesn't hattu get the -n ending?

Only the owner gets the genitive -n ending. Here, you are the owner, so sinä becomes sinun. The thing being owned (hattu) stays in its basic dictionary form.

Could I change the word order to Onko tämä sinun hattu?

Yes, you can! Just like in English, you can say Is this hat yours? (Onko tämä hattu sinun?) or Is this your hat? (Onko tämä sinun hattu?). Both are correct and mean basically the same thing.

Why is it onko instead of just on?

The suffix -ko is used to create a yes or no question. on means is, so attaching -ko makes it a question. Remember that the word with the question particle must always be moved to the very beginning of the sentence.