Learn Dutch: Dit schilderij is mooi, behalve dat het te abstract is. - This painting is beautiful, except that it is too abstract.

The phrase behalve dat (except that) acts as a subordinating conjunction. This means it introduces a dependent clause, which forces the conjugated verb (is) to the end of that clause: ... dat het te abstract is.

Could I just say behalve het te abstract is without dat?

No, you need dat here. Behalve on its own is a preposition used before a noun or pronoun, like in behalve dit schilderij (except this painting). When you want to introduce a full clause that contains its own subject and verb, you must use behalve dat.

Could I use buiten dat instead of behalve dat in this sentence?

You could, but it changes the meaning slightly. Behalve dat introduces a direct exception or a negative condition (except that). Buiten dat usually introduces an addition (besides the fact that / apart from). If you used buiten dat, it would sound less like a flaw subtracting from its beauty, and more like an additional random fact about the painting.

Why do we use het in the second clause to refer to the painting?