Learn French: J'ai entendu dire que tes parents se séparaient. - I heard that your parents were separating.

This is the sequence of tenses (backshifting) in action. Because the main verb ai entendu is in a past tense, the reported action shifts from the present (se séparent) to the imperfect (se séparaient). English does the exact same thing: you say 'I heard they were separating' instead of 'I heard they are separating'.

Why is the word dire (to say) in this sentence? Couldn't I just say J'ai entendu que...?

In French, to express that you heard news or gossip, you use the expression entendre dire que (literally, to hear say that). If you just say entendre que, it implies you physically heard a noise, like hearing that a dog is barking outside.

Why do we need the reflexive pronoun se in se séparaient?

The verb se séparer is used reflexively here to mean that the parents are separating from each other. Without the se, the verb séparer just means to separate other objects or people, like separating colors in the laundry.

Could I use the imperfect J'entendais instead of the passé composé J'ai entendu?

Usually, no. Hearing this piece of news was a single, completed event, which requires the passé composé (J'ai entendu). Using the imperfect (J'entendais) would mean you were in the middle of hearing it, or that you used to hear it repeatedly over a period of time.