Learn French: Ce problème est plus compliqué que je ne pensais. - This problem is more complicated than I thought.

This is the ne explétif. It doesn't make the sentence negative (which is why there is no pas). In written and formal French, it is often used after comparisons of inequality (plus... que, moins... que, mieux... que) when the main clause is affirmative. It simply highlights the gap between the reality of the problem and your previous expectation.

Is it mandatory to include the ne here? Can I just say "que je pensais"?

You can absolutely drop the ne, especially in everyday spoken French. Saying que je pensais is completely natural and correct in normal conversation. The ne explétif is optional and just adds a touch of formality or elegance to the sentence.

Would I still use this extra ne if the first part of the sentence was negative?

No. The ne explétif is only triggered in these comparisons when the main clause is affirmative. If you wanted to say "This problem isn't more complicated than I thought", you would drop the extra ne and say: Ce problème n'est pas plus compliqué que je pensais.

Why use the imperfect tense (pensais) here instead of the passé composé (j'ai pensé)?

The imperfect is used because it describes an ongoing state of mind—a belief or expectation you held over a period of time leading up to this discovery. The passé composé (j'ai pensé) would sound like one single, isolated thought you had at a specific moment in the past, which doesn't fit the context of a general expectation.