Learn Russian: У меня есть сомнения, действительный ли это сертификат. - I have doubts whether this is a valid certificate.

The particle ли must come directly after the word that is the main focus of the question. Here, the doubt is specifically about whether the certificate is valid, so действительный is placed first in the indirect clause and ли attaches right to it.

Could I use если instead of ли here, since the English translation can use "if"?

No, you cannot use если for this type of sentence. если is only used for conditional statements (like "If it rains, I will stay home"). When "if" means "whether" in an indirect question, you must use the particle ли.

Why does the sentence use это instead of the masculine этот to match the masculine noun сертификат?

Here, это is acting as a standalone pronoun meaning "this is", rather than an adjective meaning "this". The phrase translates literally to "...whether valid, this [is a] certificate." If you used этот, it would mean "...whether this certificate [is] valid", which would usually require the short-form adjective: действителен ли этот сертификат.

Do I always need a comma before the indirect question?

Yes. In Russian, subordinate clauses must always be separated from the main clause by a comma. You place the comma before the first word of the new clause, which in this case is действительный.