Learn Portuguese (Portugal): Eles disseram que os bilhetes tinham esgotado.

Tinham esgotado is the Pretérito Mais-Que-Perfeito Composto (the "past before the past"). We use it because the tickets selling out happened before the action of them telling you about it (disseram). It is formed by combining the imperfect of ter (tinham) with the past participle (esgotado).

Since os bilhetes is plural, why isn't it tinham esgotados?

When you form compound tenses using the auxiliary verb ter, the past participle never changes its ending. It stays in its default masculine singular form (esgotado) regardless of the subject's gender or number.

Could I use estavam esgotados instead of tinham esgotado?

Yes, you could say Eles disseram que os bilhetes estavam esgotados. Tinham esgotado focuses on the event that happened before they spoke (the tickets had sold out), while estavam esgotados describes the resulting state at that moment (the tickets were sold out). Notice that with estar, esgotados acts as an adjective, so it does agree with the plural noun bilhetes.

I always mix up the past tense forms of . Why is it here?