Learn Russian: Она выглянула из-за двери и улыбнулась. - She peeked out from behind the door and smiled.

You are right that из-за frequently means "because of" (as in из-за погоды - because of the weather). However, its literal, physical meaning is spatial: "out from behind". It combines из (out of) and за (behind). In this sentence, she is physically looking out from behind the door.

What case is двери in, and why?

It is in the Genitive case because the compound preposition из-за always requires the Genitive. The word дверь (door) is a feminine noun ending in a soft sign. Nouns of this type form their Genitive case by replacing the soft sign with -и.

What does the prefix вы- add to the verb выглянула?

The base verb глянуть means "to glance" or "to look" briefly. The prefix вы- adds the meaning of moving "out" or "outward". Together, выглянуть means "to look out" or "to peek out", which perfectly matches the outward movement described by из-за (out from behind).

Why does улыбнулась end in -сь instead of the usual reflexive ending -ся?

Reflexive verbs in Russian use -ся after consonants, but they shorten to -сь after vowels. Since the verb is in the feminine past tense and ends in the vowel -а (улыбнула), the reflexive ending must be -сь.