Learn Russian: Мы приглашаем гостя домой.

In this sentence, the guest is the direct object—the person receiving the action of being invited. Because a guest is a living being (animate), Russian changes the ending of the masculine noun гость to show it is in the accusative case.

Since masculine animate nouns usually add -а in the accusative case, why does гостя end in -я?

The dictionary form гость ends in a soft sign (ь), meaning its stem is soft. For soft masculine nouns, the animate accusative ending is -я instead of the standard hard -а. You just drop the ь and add -я.

Why do we use домой for home here instead of дом or дома?

Домой is a special adverb used whenever there is a direction or motion involved, like inviting someone to your house (homewards). If you were talking about already being at home (a location with no motion), you would use дома.

What is the dictionary form of приглашаем and why does it have that ending?

The dictionary form (infinitive) is приглашать (to invite). It belongs to the standard first conjugation group of verbs. To say we invite (мы), you drop the -ть and add the standard ending -ем.