Learn German: Der König traf den Historiker und dessen Bruder. - The king met the historian and his brother.

Using dessen clears up a classic third-person ambiguity. If the sentence said und seinen Bruder, it would mean the king met his own brother. By using the demonstrative possessive dessen (which translates to 'whose' or 'of that one'), it forces the reference to the most recently mentioned person, which is den Historiker.

Does the ending of dessen change to match Bruder?

No, it doesn't. Regular possessive adjectives like sein change their endings to match the noun they attach to. However, dessen is a fixed genitive pronoun. It gets its form entirely from the possessor (der Historiker, masculine singular) and stays as dessen regardless of the gender, number, or case of the noun being owned.

What case is Bruder in this sentence?

It is in the accusative case. The verb treffen (past tense traf) takes an accusative direct object. The king met two people: the historian and the brother. Both are direct objects. Even though dessen doesn't take an accusative ending like seinen would, Bruder is still grammatically accusative here.

Is dessen only used in formal writing?