Learn Spanish (Latin America): Esos limpiavidrios son muy caros. - Those glass cleaners are very expensive.

You actually don't change the word itself! Because limpiavidrios is formed by combining a verb (limpia) and a plural noun (vidrios), it is invariable. You only change the words around it, like using the plural esos and son, to show that there is more than one.

If I only wanted to talk about one glass cleaner, would I say ese limpiavidrio without the s?

No, you would still say ese limpiavidrios. The s is built into the base word because a cleaner handles multiple pieces of glass. The singular and plural forms look exactly the same.

Why is limpiavidrios considered masculine, as shown by esos and caros?

In Spanish, almost all compound nouns made from a verb plus a noun are masculine by default. It doesn't matter what gender the noun part is; the whole compound word takes masculine modifiers like esos and caros.

Does limpiavidrios refer to the cleaning liquid or the physical tool?

It can be both! Depending on the context, limpiavidrios can mean the glass-cleaning spray or the squeegee tool. Here, since they are very expensive (muy caros), it could refer to high-end tools or premium cleaning solutions.