Learn Hardcore Japanese: tosyokan no sankai de nihongo wo benkyousimasu. - I study Japanese on the third floor of the library.

の is the genitive/possessive particle. Here it links 図書館 and 三階 to form the compound noun 図書館の三階, meaning “the third floor of the library.” Without の, you couldn’t show that the third floor belongs to the library.

What does the particle で after 三階 do in this sentence?

The particle で marks the location where an action takes place. In 三階で勉強します, 三階 is the place of the studying, so you use で to indicate “at/on (the third floor).”

Could I use に instead of で after 三階?

No. に marks:

a point in time (e.g., ３時に “at 3 o’clock”)
a destination or direction (e.g., 学校に行く “go to school”)
existence/location for verbs like いる/ある (e.g., 部屋にいる “be in the room”)
It doesn’t mark the location of an action—that’s what で is for.

Why is there an を after 日本語? I thought を was for direct objects of verbs.

Exactly right—を marks the direct object. Since 勉強する is a する-verb, the thing you study, 日本語, takes を: 日本語を勉強します “I study Japanese.”

Why does the verb 勉強します come at the end of the sentence?

Japanese is an SOV (Subject-Object-Verb) language. The verb always goes at or near the end, after its objects and any modifiers.

Why is there no subject like “I” or 私 in this sentence?

Subjects (and objects) are often omitted in Japanese when they’re clear from context. Here, it’s assumed the speaker is talking about themselves, so 私は is dropped.

What nuance does the –ます form in 勉強します add?

The –ます form is the polite present/future tense. It shows respect to the listener. In casual speech among friends you’d use the plain/dictionary form 勉強する instead.

Can I change the word order, for example to 日本語を図書館の三階で勉強します?

Yes. Because Japanese uses particles to show each word’s role, you can swap noun phrases around for emphasis:

図書館の三階で日本語を勉強します。 (default)
日本語を図書館の三階で勉強します。
Both are grammatically correct and mean the same, though putting 図書館の三階で first puts more emphasis on the location.