Learn Russian: Твой успех кажется иллюзией.

The verb казаться (to seem) requires the noun or adjective that follows it to be in the Instrumental case. You are expressing what the success "seems to be." This is very similar to how verbs like быть (to be) and становиться (to become) also trigger the Instrumental case for their complements.

How do we get the ending -ей in иллюзией?

The dictionary form of the word is иллюзия (illusion). Feminine nouns ending in the soft vowel -я take the ending -ей (or sometimes -ёй) in the Instrumental case, replacing the final -я. If the noun ended in a hard -а (like мама), it would take -ой (мамой) instead.

I'm used to seeing кажется used with the Dative case, like "мне кажется" (it seems to me). Why is there no Dative here?

In expressions like мне кажется, the subject is an impersonal "it" and the Dative case shows who is having the thought. In this sentence, твой успех (your success) is the grammatical subject; the success itself is doing the "seeming." If you wanted to say "Your success seems like an illusion TO ME," you would just add the Dative: Мне твой успех кажется иллюзией.

Can I use the verb казать without the reflexive -ся ending to mean "to seem"?

No, in modern Russian, you must use the reflexive form казаться to mean "to seem." The non-reflexive verb казать is archaic and generally only survives in specific prefixed forms like показать (to show) or доказать (to prove).