Learn Hardcore Turkish: Güvenlik protokolünü tanımlayan mekanizmalar kuruldu. - Mechanisms defining the security protocol have been established.

Because it’s the definite direct object of the verb tanımlayan in the relative clause. In Turkish, definite objects take the accusative suffix (here -ü).

What is tanımlayan and how does it function in this sentence?

Tanımlayan is the active participle of tanımlamak (“to define”). The suffix -an turns the verb into a modifier, so güvenlik protokolünü tanımlayan means “that define the security protocol.”

Why is the main verb kuruldu in third-person singular when mekanizmalar (“mechanisms”) is plural?

Because kuruldu is in the passive voice. In Turkish passive constructions, the verb normally stays in third-person singular regardless of the subject’s number. Here kuruldu means “were established” or “were set up.”

How would you rewrite the sentence in active voice and specify who set up the mechanisms?

You could make the agent explicit and use an active verb form. For example:
Mühendisler güvenlik protokolünü tanımlayan mekanizmaları kurdular.
Here mühendisler is the subject and kurdular agrees in number (third-person plural).