Learn Russian: Что за свалка эта новая инсталляция!

In the fixed expression что за, the word за loses its normal prepositional function. The entire phrase acts as a single identifier meaning "what a" (in exclamations) or "what kind of" (in questions). Because it points out or identifies something, the noun that follows it (свалка) is always in the Nominative case.

If this ended in a question mark, would the meaning of что за change?

Yes! If it were a question (Что за свалка эта новая инсталляция?), it would mean "What kind of dump is this new installation?" Since it ends with an exclamation mark, что за takes on an exclamatory meaning: "What a dump...!"

Why is the order Что за свалка эта новая инсталляция instead of putting the subject first?

In Russian exclamations with что за, the noun being emphasized or evaluated (свалка) comes immediately after the phrase что за. The specific thing you are talking about (эта новая инсталляция) then follows. If you said Что за эта новая инсталляция..., you would be exclaiming "What a new installation...!" rather than calling the installation a dump.

Does свалка carry the exact same metaphorical meaning as "dump" in English?

Yes! Literally, a свалка is a landfill or garbage dump (from the verb свалить, meaning to dump or heap up). Just like in English, Russians use it casually to describe a very messy, disorganized, or trashy place.