Learn Finnish: Onko tämä ongelma?

The -ko (or -kö) ending is a question particle. When you attach it to a word, it turns a regular statement into a yes/no question. Here, the verb on (is) gets the -ko ending and moves to the very beginning of the sentence to ask "Is...?"

The English translation says "Is this a problem?", but where is the word for "a"?

Finnish doesn't have articles like "a", "an", or "the". The word ongelma alone can mean "problem", "a problem", or "the problem" depending on the context.

Could I say Onko se ongelma? instead?

Yes, but the meaning changes slightly. Tämä specifically means "this" (referring to something close to the speaker or just mentioned). Se means "it" or "that". So Onko se ongelma? would mean "Is it a problem?" or "Is that a problem?"