Learn Italian: Lui deve finirla, è troppo fastidioso. - He must cut it out, he is too annoying.

It doesn't refer to a specific feminine noun. In the verb finirla, the -la is a fixed pronoun that just means 'it' (referring to the ongoing situation or behavior). Along with smetterla and piantarla, it's an idiomatic way to say 'stop it' or 'cut it out'.

Could I put the -la somewhere else in the sentence?

Yes. Because deve is a modal verb (a helping verb like must, can, or want), you have two choices for pronoun placement. You can attach it to the dropped-e infinitive as shown here (deve finirla), or place it before the conjugated modal verb (la deve finire). Both are perfectly natural.

Fastidioso looks like the English word fastidious. Is that what it means?

No, this is a classic false friend! Fastidioso means 'annoying', 'irritating', or 'bothersome'. If you want to say someone is fastidious (fussy or overly focused on details), you should use a word like pignolo or esigente.

What's the difference between using finirla versus smetterla here?

They are highly interchangeable and both mean 'to stop it'. Smetterla is the most common, general way to say 'quit it'. Finirla has a slightly stronger flavor of wanting to 'put an end' to something once and for all, which fits perfectly with how annoying he is being.