Learn Dutch: Zou u mij willen terugbellen?

This is the tentative simple past. Just like in English ("Would you..." instead of "Will you..."), using the past tense of zullen makes the request softer, more tentative, and much more polite.

What role does willen play here? Does it mean "want"?

In this polite context, willen translates closer to "to be willing to". While Zou u mij terugbellen? simply means "Would you call me back?", adding willen softens it further, roughly meaning "Would you be willing to call me back?"

Why are both willen and terugbellen stacked at the very end of the sentence?

Because Zou is the conjugated verb in this yes/no question, it takes the first position. All other verbs (willen and terugbellen) are pushed to the end to form a verbal cluster. When stacking infinitives, the modal verb (willen) usually comes before the main action (terugbellen).

The verb terugbellen is separable. Why isn't it split up in this sentence?

Separable verbs only split when they are the conjugated verb matching the subject. Because Zou takes on that conjugated role here, terugbellen remains a full infinitive at the end of the sentence, keeping its prefix (terug) attached to the root (bellen).