Learn Russian: Ты забыл запятую в вычислении. Результат — ноль целых пять десятых. - You forgot the comma in the calculation. The result is zero whole, five tenths.

In Russian math, a comma (запятая) is used instead of a period to separate the whole number from the fractional part (written as 0,5 rather than 0.5). That is why you will hear Russians talk about forgetting or moving the comma in calculations.

Why do we use the genitive plural forms целых and десятых here?

Decimal fractions imply the feminine noun часть (part), so you are literally saying 'zero whole parts, five tenth parts'. Both the number ноль (zero) and пять (five) govern the genitive plural, which forces the adjectives modifying these implied 'parts' into the genitive plural forms целых and десятых.

If запятая is a noun meaning 'comma', why does it take the accusative ending -ую like an adjective?

Запятая is a substantivized adjective. This means it acts as a noun in a sentence but still declines exactly like a standard feminine adjective. Because it is the direct object of the verb забыл, it takes the feminine accusative adjective ending -ую.

Why does the prepositional form в вычислении end in -ии instead of the standard -е?