Learn Italian: Che cosa è successo in biblioteca?

Because the verb succedere (to happen) is an event or state verb. In the passato prossimo (the past tense), verbs describing motion, states of being, or things happening use essere (to be) as their helper verb instead of avere (to have).

If verbs using essere need their ending to match the subject, why does successo end in -o?

When the subject is an unknown event or situation—like when you are asking "what" happened—Italian defaults to the masculine singular ending -o. So, the past participle becomes successo.

Does successo mean "success"?

It can be a noun meaning "success", but here it is the past participle of the verb succedere (to happen). So in this sentence, è successo simply means "happened".

Can I just say Cosa è successo instead of Che cosa è successo?

Yes! To ask "what" in Italian, you can use che cosa, just cosa, or just che. They are completely interchangeable, though cosa is very popular in everyday speech.