Learn Finnish: Ei kestä, ystävä! - Don't mention it, friend!

You are right! Kestää usually means 'to last' or 'to endure.' Here, it is short for ei kestä kiittää, which literally means 'it doesn't endure thanking' (meaning the favor was so small it doesn't require thanks). Finns use the short version ei kestä as a casual 'don't mention it' or 'no problem.'

What form of the verb is kestä here?

It is the negative present tense form. To make a verb negative in Finnish, you find the 'I' form and drop the -n. The 'I' form of kestää is kestän (I last). Drop the -n, and you get kestä. Combine it with the third-person negative word ei (it does not), and you get ei kestä.

Could I just say ole hyvä instead?

Yes! Ole hyvä is the most common way to say 'you're welcome.' However, ole hyvä is also used when handing something to someone (like 'here you go'). Ei kestä is used specifically to brush off thanks after you have done someone a favor, much like 'don't mention it' in English.

Why is ystävä in its basic dictionary form here?

When you are talking directly to someone and calling them by a name or a noun like ystävä (friend), Finnish just uses the basic dictionary form (the nominative case). There is no special ending added when addressing someone.