Learn Russian: В столовой я купил большой кусок сыру. - In the cafeteria, I bought a large piece of cheese.

The word сыру is in the partitive genitive (often called the second genitive). Some masculine mass nouns, like сыр, чай, and сахар, can take an -у or -ю ending instead of -а or -я when talking about a specific quantity or a part of a whole (like a piece or a cup).

Do I have to say сыру here, or is сыра okay too?

You can absolutely use the regular genitive сыра. In fact, in modern Russian, кусок сыра is more common. The -у partitive ending is slowly fading from the language and mostly survives in set phrases like чашка чаю (a cup of tea) or to add a slightly traditional flavor when talking about portions.

Why does столовой look like an adjective instead of a normal noun?

Because столовая is an adjective that turned into a noun! It originally meant dining (from стол, table), as in a dining room, but now functions as the noun for cafeteria. Since it is structurally an adjective, it takes feminine adjective endings, which is why it becomes в столовой in the prepositional case.

Why didn't большой кусок change its ending since it is the direct object of what was bought?