Learn Hardcore Russian: Ей не удалось уснуть, пока телефон звенел в комнате. - She couldn’t fall asleep while the phone was ringing in the room.

This is an impersonal construction: ей не удалось уснуть literally means to her it didn’t succeed to fall asleep.
So ей is the experiencer (the person affected), put in the dative. There is no grammatical subject in the main clause; the “event” (уснуть) is expressed by an infinitive.

What exactly does (не) удалось + infinitive mean, and how is it different from (не) смогла?

(Не) удалось + infinitive emphasizes success/failure of an attempt or circumstances: it “worked out / didn’t work out.”
(Не) смогла is more directly about ability (could/couldn’t), though in practice they often overlap.

Compare:

Ей не удалось уснуть = she failed to manage to fall asleep (something prevented it)
Она не смогла уснуть = she couldn’t fall asleep (focus on inability)

Why is уснуть used instead of спать?

Уснуть is perfective and means to fall asleep (reach the state of sleep).
Спать means to sleep (be asleep).

So не удалось уснуть = she couldn’t get to sleep, not that she couldn’t be sleeping.