Learn Italian: Anna è stanca e sta zitta. - Anna is tired and keeps quiet.

Yes, stare literally means 'to stay'. So sta zitta translates literally to 'she stays quiet'. In English, it sounds more natural to say someone 'keeps quiet', but the Italian logic is that she is staying in a state of silence.

Why do we use è for stanca but sta for zitta? Could I say è zitta?

Essere (to be) is used to describe her physical feeling: è stanca (she is tired). Stare is used for a behavior or a condition you are actively maintaining. Stare zitto is a very common fixed phrase in Italian meaning 'to keep quiet'. You will almost always hear stare used with zitto, not essere.

Which form of the verb stare is sta here?

It is the third-person singular ('he/she/it') form of the irregular verb stare. Because we are talking about Anna, the verb matches her. Notice that Italian doesn't need to repeat the pronoun lei (she) before sta because the verb form already makes it clear who is doing the action.

Why do the words stanca and zitta both end in '-a'?

They are both adjectives describing Anna. In Italian, adjectives must match the gender and number of the person or thing they describe. Since Anna is one woman (feminine singular), both adjectives take the feminine singular ending -a.