Learn Portuguese (Portugal): Ele está a brincar, claro.

You are right that brincar means to play when talking about children or pets. However, in everyday conversation among adults, estar a brincar is the most common way to express that someone is joking, kidding, or not being serious.

I know claro means clear. How does it become of course in this sentence?

The word claro is indeed an adjective meaning clear or light. But when it is used as a standalone word or separated by a comma like in this sentence, it acts as a fixed expression that means of course, sure, or obviously.

Could I just say Ele brinca, claro?

That would change the meaning. Ele brinca is the simple present, which describes a general habit, meaning he is a person who jokes around. Because we want to say he is joking right now in this specific moment, we must use the continuous structure está a brincar.

I have heard the phrase está brincando before. Is that also correct here?

Using the gerund form brincando is the standard way to form this tense in Brazilian Portuguese. In European Portuguese, continuous actions are almost always formed with the verb estar plus the preposition a plus the infinitive verb, which gives us está a brincar.