Learn Farsi: آنها پرسیدند که آیا تابع این متغیر را برمی‌گرداند یا خیر. - They asked whether the function returns this variable or not.

خیر is a highly formal version of نه (no). Because this sentence discusses technical concepts like a function (تابع) and a variable (متغیر), it has an academic or professional tone. In such formal contexts, appending یا خیر to a reported yes/no question is much more natural and expected than the conversational یا نه.

Is it strictly necessary to use که آیا together, or could I just use آیا to mean 'whether'?

You can definitely just drop که and use آیا. The که acts as a conjunction connecting the main verb (پرسیدند) to the reported question, similar to saying 'They asked that whether...' in older English. While که is often dropped in spoken Farsi, keeping که آیا together is standard in formal writing and makes the boundary between the clauses very clear.

I know the verb for 'to return' is برگشتن. Why is برمی‌گرداند used here instead?

برگشتن is intransitive, meaning the subject itself goes back. Here, the function is returning something else (the variable). For this, you need the transitive verb برگرداندن (to return something or give something back), which takes the direct object marker را after این متغیر.

Can I use for any kind of 'function', like the function of a tool or a committee?