Learn Hardcore Japanese: zyugyougo ni sensei ni nooto wo watasimasu. - I hand the notebook to the teacher after class.

They’re marking two different things. In 授業後に, に indicates a point in time (“after class”). In 先生に, に marks the indirect object or recipient (“to the teacher”).

What is the function of を in ノートを渡します?

を marks ノート as the direct object of the verb 渡します. It shows what you are handing over (“notes”).

Why is there no subject in this sentence?

Japanese often omits the subject when it’s clear from context. Here, it’s implicitly “I” (or “we”), so you don’t need to say 私が.

Does the word order matter? For example, could I say ノートを先生に渡します授業後に?

Japanese word order is relatively flexible, but time expressions like 授業後に usually come before the verb phrase. A natural order is:

Time (授業後に)
Indirect object (先生に)
Direct object (ノートを)
Verb (渡します)

So 授業後に先生にノートを渡します or 授業後にノートを先生に渡します both work, but you wouldn’t put 授業後に at the very end.

Could I use へ instead of に for the teacher?

Yes, 先生へノートを渡します is grammatically correct. Both に and へ can mark the direction or recipient, but に is more neutral for “to someone,” while へ emphasizes direction a bit more.

What nuance does the verb 渡します carry compared to verbs like あげます or やります?

渡します literally means “to hand over” or “to pass.” It’s fairly neutral and formal.

あげます is “to give” more generally to someone of equal or lower status.
やります is casual/slangy and can be rude if used for someone higher than you.

Is 授業後に the only way to say “after class”?

No. You could also say:

授業が終わってから (after the class ends)
授業後で is less common for time. Usually 授業後に is preferred.

Why is 授業後 written as two kanji without の? Could I say 授業の後に?

Both are correct:

授業後に is the concise, compound form.
授業の後に uses the possessive の and is equally acceptable, just a bit more explicit.

Why is 渡します in the polite form? How would you say this casually?

渡します is the polite (ます) form. Casual/plain form is 渡す.
So casually you’d say: 授業後に先生にノートを渡す。