Learn Russian: Я живу в большом городе.

We use the Prepositional case to talk about location (where something is happening). The preposition в (in) requires the noun and its matching adjective to change. The masculine noun город takes the ending -е to become городе, and the adjective большой changes to its masculine Prepositional form, большом.

I learned a spelling rule for the letter ш. Shouldn't the adjective ending be -ем instead of -ом?

You are thinking of the 5-letter spelling rule! After the 'hushing' consonants (ж, ш, ч, щ) and ц, we write о if that vowel is stressed, and е if it is unstressed. Because the stress in большом falls on the last syllable, we use -ом. For a word where the stress is not on the ending, like хороший (good), you would use -ем (в хорошем).

Is the ending -ом only used for masculine adjectives in the Prepositional case?

It is used for both masculine and neuter adjectives. If the sentence was about a neuter noun, like окно (window), you would use the exact same ending: в большом окне (in the big window). Feminine adjectives get a completely different ending (-ой or -ей).