Learn Spanish (Latin America): En otras palabras, no quiere venir. - In other words, he does not want to come.

Yes, it functions the exact same way. All of these are explicative discourse markers used to clarify or rephrase something that was just said. In this sentence, you could swap it for es decir, no quiere venir or o sea, no quiere venir without changing the core meaning.

Is there a difference in tone between en otras palabras and o sea?

Yes. O sea is heavily used in casual, everyday spoken Spanish. En otras palabras (literally 'in other words') and es decir (literally 'that is to say') are slightly more formal or structured, though they are still perfectly common in everyday conversation when you want to summarize a point clearly.

Is the comma after en otras palabras mandatory?

Yes. In Spanish, explicative discourse markers that introduce a rephrasing or summary, like en otras palabras, es decir, and o sea, must be separated from the rest of the sentence by a comma. If they appear in the middle of a sentence, they are enclosed in commas.

Could quiere refer to 'she' or 'you' instead of 'he'?