Learn Russian: У тебя есть лишняя пуговица?

In Russian, лишний means "extra" in the sense of "spare," "left over," or "superfluous" (like a spare button lying around). Дополнительный means "additional" (like an additional task or extra cheese). Since you are asking for a spare button the other person doesn't need, лишняя is the perfect word.

Could I use кнопка instead of пуговица for "button"?

No, пуговица specifically means a traditional clothing button (the kind that goes through a buttonhole). Кнопка refers to a push-button (like on a keyboard or elevator) or a metal snap fastener on a jacket.

Why does the adjective лишняя end in -яя instead of the usual -ая?

Лишняя is in the feminine nominative form to agree with пуговица. It takes the -яя ending instead of -ая because its dictionary form лишний is a "soft" adjective. Soft adjectives use endings with soft vowels (like я, е, и) rather than hard ones (like а, о, ы).

Can I drop the word есть and just ask У тебя лишняя пуговица?

Not if you want to ask "Do you have one?" You must include есть when asking about the existence or possession of an object. Dropping есть changes the focus to the description of the object, making it sound like you are pointing at a button and asking "Is that an extra button on you?"