Learn Russian: Мы едем домой на метро.

Едем is a form of the verb ехать, which means "to go by vehicle". Because we are taking the metro in this sentence, we have to use this verb. If we were walking home, we would use идём (from идти, to go on foot).

I learned that "home" is дома. Why does this sentence use домой?

Дома is used for location, meaning "at home". Домой is used for direction, meaning "towards home" or "homewards". Because we are traveling to a destination in this sentence, we must use the directional word домой.

If we use the Prepositional case after на for transport, why didn't the ending of метро change?

You are exactly right that normally the noun would change (like автобус becoming на автобусе). However, метро is a borrowed foreign word. In Russian, many borrowed words ending in -о are "indeclinable." This means their endings never change, regardless of the grammatical case.

Why do we use the preposition на here instead of saying "by" metro?

While на often translates literally to "on", using на followed by a vehicle in the Prepositional case is the standard Russian formula to say "by [vehicle]". You will use this same pattern for traveling на автобусе (by bus) or на поезде (by train).