Learn Russian: Этот текст редактируется кое-кем.

In Russian passive sentences, the person or thing doing the action (the agent) is put in the instrumental case. Since the text is being edited by someone, the pronoun кое-кто changes to its instrumental form, кое-кем.

How can we tell that редактируется is passive ("is being edited") rather than reflexive ("edits itself")?

The -ся ending on a verb can mean either, but since an inanimate object like текст cannot physically edit itself, it has to be a passive verb here. The presence of the instrumental agent кое-кем is also a guaranteed sign that this is a passive construction.

Could I use кем-то instead of кое-кем in this sentence?

You could, but it changes the nuance. Кое-кем implies that the speaker knows exactly who is doing the editing but chooses to keep it a secret. Кем-то would mean the text is being edited by a completely unknown person.

Is the hyphen in кое-кем mandatory?

Yes, it is absolutely mandatory. The prefix кое- is always attached to pronouns with a hyphen, just like the common suffixes -то and -нибудь.