Learn Croatian: Može li tvoj brat igrati nogomet? - Can your brother play football?

Because it follows the modal verb moći (can). Just like in English where we say 'can play' instead of 'can plays', in Croatian you conjugate the first verb (može for 'he can') and leave the second verb in its dictionary form, the infinitive (igrati).

How does the word li work in this question?

The word li is a small particle used to ask yes/no questions. To form a question, you start with the conjugated verb (Može), follow it immediately with li, and then add the subject (tvoj brat) and the rest of the sentence.

Why does the word nogomet not change its ending here?

Because nogomet (football) is the direct object being played, it must be in the accusative case. For masculine inanimate objects (things that are not alive), the accusative form looks exactly the same as the basic dictionary form.

Could we just say Tvoj brat može igrati nogomet? with a rising voice to ask the question?

In spoken, everyday Croatian, yes! People often just use the normal statement word order and raise their voice at the end to ask a question. However, using the verb + li structure (Može li...) is the standard, grammatically correct way to write a yes/no question.