Learn Hardcore Spanish (Spain): Este libro antiguo es muy interesante.

este = this (masculine, singular)
esta = this (feminine, singular)
esto = this (neuter, not followed by a noun, used for abstract/unknown things)

Since libro is masculine and singular, you must use este.
If the noun were feminine, like mesa (table), you would say esta mesa antigua es muy interesante.

How do I know that libro is masculine?

Most nouns ending in -o are masculine, and libro follows that pattern.
You learn gender mainly by memorizing the noun with its article: el libro, la mesa, etc.
There are exceptions (like el día, la mano), but -o → masculine and -a → feminine is a useful general rule.

Why does antiguo come after libro? Can I say este antiguo libro?

Both este libro antiguo and este antiguo libro are grammatically correct, but they don’t mean exactly the same:

este libro antiguo = this old/ancient book (physically old)
este antiguo libro = this former book (for example, “this old/previous book of mine”, with more sense of “former” than of age)

Many adjectives in Spanish change nuance depending on whether they go before or after the noun, and antiguo is one of them.
For physical age, you normally put antiguo after the noun.

Why is antiguo in the masculine singular form?

Adjectives in Spanish agree in gender and number with the noun they describe.
Libro is masculine singular, so antiguo also has to be masculine singular:

libro antiguo (masc. sing.)
libros antiguos (masc. plural)
casa antigua (fem. sing.)
casas antiguas (fem. plural)

Why is es used here and not está?

Ser (es) is used for more permanent or essential characteristics, and estar (está) for temporary states or locations.
Being interesting is considered an inherent quality of the book, not a temporary condition, so you use es muy interesante.
If you were talking about where the book is, you’d use estar: El libro está en la mesa (The book is on the table).

Why do we say muy interesante and not mucho interesante?

Muy modifies adjectives and adverbs: muy interesante, muy alto, muy bien.
Mucho usually modifies nouns or verbs: mucho dinero, mucho trabajo, llueve mucho.
Since interesante is an adjective, you need muy: muy interesante.

Could I say Este libro es muy antiguo e interesante instead?

Yes, that sentence is correct and natural: Este libro es muy antiguo e interesante.
Now antiguo and interesante are both predicate adjectives after the verb es.
In your original sentence, antiguo is directly modifying libro, and interesante is in the predicate.
Both structures are fine; you just change the rhythm and emphasis a bit.

Why isn’t there an article before muy interesante? Why not es un muy interesante?

In Spanish, adjectives after ser usually appear without an article:

Es interesante
Es muy interesante
El libro es caro

Using un/una before an adjective in this position is unusual and often sounds wrong or very marked.
So es muy interesante is the normal, neutral form; es un muy interesante… is not idiomatic here.

Can interesante go before libro, like este interesante libro antiguo?

You can say este interesante libro antiguo, and it’s grammatically correct, but the normal and most natural order is:

Este libro antiguo es muy interesante
or Este libro antiguo e interesante es… (both adjectives after the noun)

Stacking several adjectives before the noun (este interesante libro antiguo) can sound literary or stylistically marked. In everyday Spanish, you usually keep most adjectives after the noun.

How would the sentence change in the plural: “These old books are very interesting”?

You need to make the demonstrative, noun, and adjective plural:

Estos libros antiguos son muy interesantes.

Changes:

este → estos (this → these, masculine plural)
libro → libros
antiguo → antiguos
es → son (3rd person singular → plural)
interesante → interesantes

Is there any difference in Spain between this sentence and how it would be said in Latin America?

For this particular sentence, no practical difference.
Este libro antiguo es muy interesante is standard and sounds natural in both Spain and Latin America.
Accent, speed, or intonation may vary by region, but the vocabulary and grammar are the same here.

How is antiguo pronounced, especially the g and the u?

Antiguo is pronounced approximately [an-TEE-gwo] in English terms.

Details:

an = like “an” in “ant” but without the final t.
ti = tee.
guo = gwo; the u is not silent here because it’s followed by o, not e or i. The stress is on ti: an-TI-guo.

What is the exact nuance of antiguo here: “old” or “ancient”?

With antiguo after the noun (libro antiguo), it usually means old in the sense of physically old or from long ago.
Depending on context, it can range from “old” to “ancient,” but it doesn’t automatically mean “ancient” in a technical/historical sense.
If you wanted to be very clear about a very long time ago, you might also use expressions like muy antiguo, antiquísimo, or specify the period: un libro del siglo XV.