Learn Swedish: Vem tillhör pyjamasen i sovsalen? - Who do the pyjamas in the dormitory belong to?

You could absolutely ask Vems är pyjamasen? (Whose are the pyjamas?) and it would mean the same thing. However, this sentence uses the verb tillhör (belongs to). Because we are asking 'who' the pyjamas belong to, vem acts as the object of that verb, so we use the standard pronoun vem instead of the possessive vems.

Where is the Swedish word for 'to' in 'belong to'?

The Swedish verb att tillhöra (to belong to) already has the 'to' built into its meaning. It takes a direct object without needing an extra preposition, so you do not need to add a word like till here.

English uses the plural 'pyjamas', so why does pyjamasen act like a singular word here?

Unlike English, Swedish treats a set of sleepwear as a standard singular noun: en pyjamas. By adding the singular definite suffix -en, we get pyjamasen (the pyjamas). If you were talking about multiple pairs of pyjamas, you would use the plural form pyjamasarna.

What is the subject of this sentence, and why does it come after the verb?