Learn Icelandic: Flækjan gæti hugsanlega hafa verið minni. - The complication could possibly have been less.

This is the difference between a past guess and a past unfulfilled possibility. Gæti hafa verið means we are guessing about a past truth ('it might have been less, I am not sure'). Hefði getað verið means it definitely had the potential to be less, but we know it wasn't.

Normally geta takes the supine form. Why is hafa in the infinitive here instead of the supine haft?

You remembered the rule correctly! Normally, geta requires the supine. However, when used for this specific type of past guess, modern Icelandic uses the infinitive hafa followed by the supine (verið). It is a fixed exception you just have to memorize for 'might have' sentences.

Is flækjan related to physical tangles, or just abstract complications?

It works for both! The root noun flækja can mean a physical tangle, like in a rope or hair, as well as an abstract complication or plot twist. Here, with the definite article -n, it means 'the complication'.

Why does hugsanlega go between gæti and hafa? Could it go elsewhere?

In complex verb chains, adverbs of probability like hugsanlega (possibly) naturally sit right after the first conjugated verb (gæti). You could also put it at the very beginning or end of the sentence for slightly different emphasis, but nesting it inside the verb chain is the most natural word order.