Learn Arabic (Modern Standard): أقام أخي حفلة خطوبة.

أقام is the Form IV version of قام (root ق-و-م). Form IV often makes a verb causative, meaning "to make something stand" or "to establish". In Arabic, you do not "do" or "make" a party; you "establish" or "host" it using أقام.

Why is it حفلة خطوبة without ال (al-) on the second word?

This is an indefinite Idafa (genitive construct). Because both words are indefinite, it translates to "an engagement party" rather than "the engagement party". If you wanted to say "My brother hosted the engagement party," you would add ال to the second word: حفلة الخطوبة.

أخي (my brother) is the subject of the sentence, so shouldn't it have a damma on the end for the nominative case?

Yes, grammatically it is in the nominative case, but you cannot hear or see the damma. The attached pronoun ي (my) forces the letter before it (the خ) to take a kasra (an "ee" sound) to make pronunciation smooth. The case ending is considered "hidden" or "assumed" because of this attached pronoun.

Could I use an adjective here instead, like حفلة خطوبية?

No, in Arabic, types of events are almost always expressed using an Idafa (noun followed by noun) rather than a noun and an adjective. You use the noun خطوبة (engagement) directly after حفلة (party) to literally say "a party of engagement".