Este libro cuesta mucho dinero.

Because este has to agree in gender with libro.

• libro is a masculine noun.
• The masculine singular form of “this” is este.
• The feminine singular form would be esta (e.g. esta mesa – this table).

So you say este libro (this book), not esta libro.

They all mean something like “this/that”, but they show distance:

• este libro – this book (near the speaker)
• ese libro – that book (near the listener or a bit farther away)
• aquel libro – that book over there (far from both speaker and listener)

In Spain, all three are common. The choice often depends on physical distance or sometimes emotional/mental distance.

Cuesta comes from the infinitive costar (to cost).

• It is 3rd person singular, present tense, indicative.
• The subject is este libro (it = the book).

Costar is a stem‑changing verb (o → ue):

• yo cuesto
• tú cuestas
• él / ella / usted cuesta
• nosotros / nosotras costamos
• vosotros / vosotras costáis
• ellos / ellas / ustedes cuestan

Yes, both are correct but they sound slightly different:

• Este libro es muy caro. – This book is very expensive.
  Focuses on the quality “expensive”.

• Este libro cuesta mucho dinero. – This book costs a lot of money.
  Emphasises the amount of money.

In everyday speech, Spaniards use both. Es muy caro is a bit shorter and very common.

Because dinero in Spanish is usually an uncountable noun, like money in English.

• You normally say el dinero, mucho dinero, poco dinero.
• You do not usually pluralise it (dineros) in modern everyday Spanish.

Mucho agrees with dinero:

• masculine singular noun → mucho dinero
• feminine singular noun → mucha agua
• masculine plural noun → muchos libros
• feminine plural noun → muchas casas

In Spanish, when you talk about an indefinite amount of an uncountable thing, you normally don’t use an article:

• cuesta mucho dinero – it costs a lot of money
• tiene mucha paciencia – she has a lot of patience

Using un here (un mucho dinero) is incorrect.
The pattern is: verb + mucho/mucha/muchos/muchas + noun, without an article in between.

Yes, that is very natural:

• Este libro cuesta mucho. – This book costs a lot / is very expensive.

When the context is clear (you’re talking about price), dinero can be omitted.
Adding dinero makes it more explicit and slightly more emphatic about the money.

With nouns, mucho normally goes before the noun:

• mucho dinero
• mucha gente
• muchos problemas
• muchas cosas

But mucho can also be an adverb, then it goes after the verb:

• Trabaja mucho. – He works a lot.
• Me gusta mucho. – I like it a lot.

So:

• mucho + noun → adjective, usually before the noun.
• verb + mucho → adverb, usually after the verb.

You need plural forms for este libro and cuesta:

• Estos libros cuestan mucho dinero.

Changes:

• este → estos (plural masculine)
• libro → libros (plural)
• cuesta → cuestan (3rd person plural of costar)

In Spanish, many nouns ending in -o are masculine by default:

• el libro – the book
• el vaso – the glass
• el perro – the dog (male, or generic dog)

So libro takes masculine articles and determiners:

• el libro
• un libro
• este libro
• mucho libro (in some contexts)

Cuesta has two syllables: cues‑ta.

• c
  • ue = sounds like “kwe”
• Stress is on the first syllable: CUESta.

Approximate pronunciation: KWEH-stah.

No, the infinitive is costar, not cuestar.

What happens is:

• Infinitive: costar (with o)
• Stem‑changing forms: cuesta, cuestan, etc. (the o changes to ue in many present‑tense forms)

This o → ue change is common in Spanish verbs, for example:

• poder → puede
• dormir → duerme
• volver → vuelve
• costar → cuesta