Learn Portuguese (Portugal): O meu melhor amigo ficou chateado por não terem retribuído o gesto. - My best friend got annoyed for them not having reciprocated the gesture.

This is the Perfect Personal Infinitive. We use it after prepositions like por to talk about a completed past action. Because the people who didn't reciprocate ("they") are different from the person who got annoyed ("my best friend"), we have to conjugate the helping verb ter for "they", giving us terem. Then we add the past participle retribuído to mean "having reciprocated".

Could we use porque instead of por in this sentence?

Yes, but the grammar would change. Por is a preposition, which is why it triggers the use of the infinitive phrase terem retribuído. If you used the conjunction porque (because), you would use a regular conjugated past tense instead: porque não retribuíram o gesto (because they didn't reciprocate the gesture).

Why use ficou instead of foi or estava for "got annoyed"?

The verb ficar is used to show a change in state or emotion, which perfectly translates to "got" or "became" in English. If the sentence used estava chateado, it would just describe an ongoing feeling in the past ("was annoyed") rather than the exact moment he became annoyed.

Why is the word não placed before terem rather than between the verbs?