Pensé que era una broma, pero en realidad hablaba en serio.

This is a very common Spanish tense pattern.

• Pensé is preterite because it refers to a completed mental action: I thought
• Era and hablaba are imperfect because they describe how something seemed / was at that time

So the sentence is structured like this:

• Pensé = one completed moment of thinking
• que era una broma = what I believed the situation was
• pero en realidad hablaba en serio = what was actually going on

Spanish often uses:

• preterite for the main event
• imperfect for background, description, or ongoing context

Because era focuses on how the speaker understood or interpreted the situation at the time.

• era una broma = it was a joke / I thought it was a joke
• fue una broma would sound more like identifying it as a completed fact or event

Here, the speaker is not saying what it truly was. They are saying what they believed it was. That is why era fits better.

Hablaba is used because it describes what the person was doing or the manner in which they were speaking at that moment.

• hablaba en serio = he/she was being serious / was speaking seriously
• habló en serio would sound more like a completed action: he/she spoke seriously

In this sentence, the idea is not really about a single finished act of speaking. It is about the real nature of the situation: the person was serious.

En realidad means actually, in fact, or in reality.

It is used to correct an impression or show what is true:

• Pensé que era una broma, pero en realidad... = I thought it was a joke, but actually...

It is very common in everyday Spanish.

En serio means seriously or for real.

In this sentence:

• hablaba en serio = he/she was serious / he/she meant it

It does not literally mean that the person was talking about serious topics. It means their words were not a joke.

You will also hear it in expressions like:

• ¿En serio? = Really?
• Te lo digo en serio. = I’m serious.

Because que introduces a subordinate clause after verbs of thinking, saying, believing, knowing, etc.

• Pensé que... = I thought that...

In English, that is often optional:

• I thought it was a joke

In Spanish, que is normally required:

• Pensé que era una broma

Spanish often leaves out subject pronouns when they are understood from context.

In this sentence, the subject of era and hablaba is not stated explicitly, but it is understood from the situation:

• Pensé que era una broma = I thought it was a joke
• pero en realidad hablaba en serio = but actually he/she was serious

Spanish does this all the time. If needed, a pronoun could be added for clarity:

• Pensé que era una broma, pero en realidad él hablaba en serio.
• Pensé que era una broma, pero en realidad ella hablaba en serio.

Without the pronoun, it sounds natural and normal.

Because in Spanish, singular countable nouns usually need an article or determiner.

So:

• una broma = a joke
• la broma = the joke

Saying just era broma is not the normal standard structure in this sentence.
The full form era una broma is the natural one.

It can mean both, depending on context.

• una broma = a joke
• sometimes also = a prank

In this sentence, it most likely means joke in the general sense: the speaker thought the other person was not being serious.

Yes. That would be very natural too.

• Pensé que... = I thought that...
• Creí que... = I believed / thought that...

In many contexts they are very similar.
Creí can sometimes sound a little stronger, more like I really believed.
But in everyday speech, both are common and correct here.

The accent mark shows the stress and also helps distinguish the form.

• pensé = I thought (preterite, first person singular)
• pense without an accent is not the standard form here

The stress falls on the last syllable:

• pen-SÉ

This is typical for many yo forms in the preterite:

• hablé
• miré
• pensé

A little, yes, but the original order is the most natural.

Original:

• Pensé que era una broma, pero en realidad hablaba en serio.

You could also hear:

• Pensé que era una broma, pero hablaba en serio en realidad.

But en realidad is very commonly placed before the part it corrects:

• pero en realidad...

That placement makes the contrast especially clear:

• I thought one thing, but actually...

In literal grammar terms, it is was speaking seriously.
But in natural English, it often means:

• was serious
• meant it
• wasn't joking

So although the verb is hablar, the expression hablar en serio often functions like an idiomatic way to say someone is being serious.