Learn Polish: Znam ten problem, ale jest za skomplikowany. - I know this problem, but it is too complicated.

Yes, you could absolutely say zbyt skomplikowany. Both za and zbyt mean "too" when used before an adjective or adverb. Za is very common in everyday spoken Polish, while zbyt can sound slightly more formal or emphatic, but they are fully interchangeable here.

I learned before that za means "behind" or "for". How do I know it means "too" here?

Context and word order give it away. When za is a preposition meaning "behind" or "for", it is followed by a noun or pronoun. But when you see za placed right in front of an adjective (like skomplikowany) or an adverb, it functions as a particle meaning "too".

Why is it ten problem? Shouldn't the object of znam take the accusative case?

It actually is in the accusative case! Problem is a masculine inanimate noun. For masculine inanimate nouns (things, not people or animals), the accusative case looks exactly the same as the basic nominative form. So both the noun problem and the word ten stay exactly the same.

Why do we use znam here instead of wiem for "I know"?