Learn Dutch: Ik adviseer u om het losgeld niet te betalen.

This sentence uses an object control structure. The verb adviseren takes a direct object (u). This object then acts as the logical subject of the following infinitive clause. So, while Ik (I) am doing the advising, the object u (you) is the one who is supposed to not pay.

Is the word om strictly necessary in this sentence?

No, it is optional. With verbs of advising, ordering, or persuading, you can often leave out om. Saying Ik adviseer u het losgeld niet te betalen is also perfectly correct, though adding om is very common and helps clearly separate the infinitive clause from the main clause.

Why is niet placed right before te betalen instead of after adviseer?

If you put niet after adviseer, it would mean you are not giving any advice at all. Here, niet negates the action of paying, so it belongs inside the om ... te clause. In such clauses, niet is typically placed right before te and the infinitive verb.

Could I use jou instead of u as the object here?