Learn German: Mein Bruder hat in Deutschland studiert.

Verbs that end in -ieren, like studieren, do not get a ge- at the beginning of their past participle. You just take the stem (studier-) and add a -t at the end.

Can I use studiert to say my brother studied for a high school math test?

No, studieren specifically means to study at a university level or to major in an academic subject. If he was studying for a test, doing homework, or learning a skill, you would use the verb lernen (past participle: gelernt).

Why does studiert go at the very end of the sentence?

This is the standard word order for the German spoken past tense (Perfekt). The helping verb (hat) takes the second position in the sentence, and the past participle (studiert) is always pushed to the very end.

Why is it just in Deutschland and not im Deutschland?

Most countries in German, including Deutschland, are neuter and do not use an article (like "the") in normal sentences. When saying you are in one of these countries, you just use in followed directly by the country's name.