Learn Finnish: Hänen muistikirjansa lienee kateissa. - His notebook is probably missing.

lienee is the potential mood of the verb olla (to be). The potential mood expresses probability or likelihood. While most verbs form this mood by adding -ne- to their consonant stem, olla is irregular and uses the special stem lie-. In this sentence, it translates to "is probably" or "may be".

Could I just say Hänen muistikirjansa on ehkä kateissa instead?

Yes, absolutely! Using the regular verb on with an adverb like ehkä (maybe) or luultavasti (probably) means the exact same thing. The potential mood (lienee) is simply a more concise and elegant way to express probability, and you will encounter it mostly in written Finnish or formal speech.

Why does the noun have the possessive suffix -nsa when the sentence already starts with Hänen (his)?

In standard Finnish grammar, the third-person pronouns hänen (his/her) and heidän (their) require the noun to also take a possessive suffix. So, Hänen muistikirja is grammatically incorrect in the standard written language; it must be Hänen muistikirjansa (literally "his notebook-his"). Keep in mind that in casual spoken Finnish, people often drop the suffix and just say Sen muistikirja.

What kind of word is kateissa? It looks like a noun in the inessive plural (in the...).