Learn Italian: Non capisco: che vuoi dire? - I do not understand: what do you mean?

Because this is a direct question! In the second part of the sentence (che vuoi dire?), 'what' is a question word. You only use quello che (or ciò che) in statements where 'what' means 'the thing that' or 'that which'.

How would I rewrite this without the colon to say 'I do not understand what you mean'?

That is exactly when you would use this lesson's grammar! As a single statement without a direct question, it becomes Non capisco quello che vuoi dire (literally: I do not understand that which you want to say).

Literally, vuoi dire means 'you want to say'. Is this always how Italians translate the English verb 'to mean'?

Yes, when talking about what a person means to express, Italian uses the phrase voler dire (to want to say). So che vuoi dire? translates perfectly to 'what do you mean?'.

Could I use cosa or che cosa instead of just che in this question?

Absolutely. When asking 'what?' as a question, che, cosa, and che cosa are generally interchangeable. Che cosa vuoi dire? or Cosa vuoi dire? would both be perfectly correct here.