Learn Spanish (Latin America): ¿Quién estaba durmiendo en el sótano?

Both can mean 'was sleeping', but the past continuous (estaba durmiendo) strongly emphasizes that the action was actively in progress at that exact moment. You usually use it to set the scene, often right before another event interrupts it.

Why is it spelled durmiendo with a 'u' instead of 'dormiendo'?

The verb dormir (to sleep) has an irregular gerund (the '-ing' form). Because it is an '-ir' verb that changes its stem in the present tense (duermo), its stem also changes in the gerund form, turning the 'o' into a 'u'.

Why is estaba conjugated in the singular form if we don't know who was in the basement?

The singular question word quién always takes a singular verb, much like 'who' does in English. If you already knew or strongly suspected there were multiple people sleeping down there, you would use the plural form of 'who' and say ¿Quiénes estaban durmiendo?.

Could I use the preterite tense here and say estuvo durmiendo?

It is possible, but it changes the feel of the sentence. Using the imperfect estaba treats the sleeping as an ongoing background event. If you used the preterite estuvo, you would be talking about a completed action that happened for a specific, enclosed amount of time in the past.