Learn Spanish (Latin America): En primer lugar, hay que estudiar.

Not quite. En primer lugar (in the first place / firstly) is used to list points in a sequential order, often followed by en segundo lugar (secondly). On the other hand, por un lado (on one hand) is specifically used to introduce contrasting ideas and is usually paired with por otro lado (on the other hand).

Why does the sentence use primer instead of primero?

The word primero drops its final -o when it comes directly before a masculine singular noun. Since lugar is a masculine singular noun, it becomes primer lugar. You will see this happen with a few other adjectives too, such as using buen instead of bueno before a masculine noun.

Could I say tienes que estudiar instead of hay que estudiar here?

Yes, but it changes the tone. Hay que is an impersonal expression meaning "one must" or "it is necessary to." It states a general rule or fact without pointing at anyone in particular. If you say tienes que estudiar, you are directly telling a specific person "you have to study."