Это был настоящий кошмар.

Because in this kind of sentence, the past-tense form of быть (was) usually agrees with the noun that names what something was — here, кошмар.

• кошмар is a masculine singular noun
• so the verb is был (masculine singular past)

That is why we say:

• Это был настоящий кошмар.

Even though это often looks like a neuter word, in sentences like this it functions more like this/it/that in an identifying statement, and the verb agrees with the predicate noun:

• Это был мой брат. = This was my brother.
• Это была моя сестра. = This was my sister.
• Это было письмо. = This was a letter.

So the gender of был / была / было depends on the noun after it.

Here это means something like this, that, or it depending on context.

In Russian, это is very common in sentences that identify or characterize something:

• Это мой дом. = This is my house.
• Это проблема. = This is a problem.
• Это был кошмар. = It was a nightmare / That was a nightmare.

So in Это был настоящий кошмар, это points to some situation, experience, or event, and the rest of the sentence says what it was.

Russian usually leaves out to be only in the present tense.

So:

• Это кошмар. = This is a nightmare.
• Это настоящий кошмар. = This is a real nightmare.

But in the past tense, Russian does use forms of быть:

• Это был кошмар. = This was a nightmare.
• Это была ошибка. = This was a mistake.
• Это было трудно. = This was difficult.

So был is there because the sentence is in the past.

It is in the nominative case.

Both words are nominative singular masculine:

• настоящий = real, genuine
• кошмар = nightmare

So:

• настоящий agrees with кошмар
• both are masculine singular nominative

After это был / была / было, Russian often uses the nominative in this kind of identifying sentence:

• Это был хороший день.
• Это была странная история.
• Это было важное решение.

So настоящий кошмар is simply the noun phrase naming what это was.

Yes, literally настоящий means real, genuine, or actual.

But in this sentence it has the same force as English a real nightmare, meaning:

• truly awful
• extremely unpleasant
• a total nightmare

So it is not necessarily about an actual dream at night. It is often an emphatic description of a bad experience.

For example:

• Экзамен был настоящий кошмар. = The exam was a real nightmare.
• Поездка была настоящим кошмаром. = The trip was a real nightmare.

Because Russian has no articles.

English says:

• a nightmare
• the nightmare

Russian simply says:

• кошмар

Whether it means a nightmare, the nightmare, or just nightmare depends on context.

So Это был настоящий кошмар naturally translates as It was a real nightmare even though there is no separate word for a.

Yes, absolutely.

• Это был кошмар. = It was a nightmare.
• Это был настоящий кошмар. = It was a real nightmare.

Adding настоящий just makes it stronger and more expressive.

So the shorter sentence is perfectly normal; the longer one is more emphatic.

Yes, Russian word order is flexible, but the neutral, most natural order here is:

• Это был настоящий кошмар.

You may also see other orders for emphasis, such as:

• Настоящий кошмар это был.
• Кошмар это был настоящий.

But these sound more marked, dramatic, or stylistically special. For a learner, the standard version is the best one to use.

A careful pronunciation would be approximately:

EH-tuh byl nuh-stuh-YAH-shchiy kash-MAR

A few useful notes:

• Это sounds like EH-tuh
• был is roughly byl; the ы sound has no exact English equivalent
• stress in настоящий falls on -я-: настоя́щий
• stress in кошмар falls on the second syllable: кошма́р

If you want a more natural rhythm, the stress pattern is:

Э́то был настоя́щий кошма́р.

It can mean both.

1. Literal meaning: a nightmare during sleep
2. Figurative meaning: something horrible, chaotic, unbearable, or extremely unpleasant

In everyday speech, the figurative meaning is very common:

• Это просто кошмар! = This is just awful! / This is a nightmare!
• У нас был кошмарный день. = We had a terrible day.

So in your sentence, it very often means a terrible experience, not necessarily a dream.

The present-tense version would usually be:

• Это настоящий кошмар. = This is a real nightmare.

Notice that there is no word for is in normal present-tense Russian.

Compare:

• Это настоящий кошмар. = This is a real nightmare.
• Это был настоящий кошмар. = This was a real nightmare.

That contrast is a very useful pattern to remember.