Learn Swedish: Du borde inte prata så högt. - You should not talk so loudly.

Because borde is a modal verb (like "should" or "can" in English). In Swedish, modal verbs are followed directly by the infinitive (dictionary form) of the next verb, without using att (to). So it is borde inte prata and never borde inte att prata.

Why is inte placed right after borde?

In a standard Swedish main clause, the rule is that the negative word inte comes exactly after the first verb. Since borde is the first verb here, inte goes right after it, pushing the second verb (prata) further down the sentence.

Why is the word for "loudly" spelled högt with a "t" instead of just hög?

Because it is being used as an adverb to describe how someone talks, rather than an adjective describing a noun. In Swedish, you usually create an adverb from an adjective by adding a -t to the end, which looks exactly like the neuter form of the adjective.

Could I use måste instead of borde in this sentence?

You could, but it changes the tone completely! borde means "should" and is used to give a recommendation or advice. If you used måste ("must" or "have to"), the sentence becomes a strict command or rule.