Learn Croatian: Dobar dan. Čaj, molim. Hvala! - Good day. Tea, please. Thank you!

Molim literally means 'I ask' and works perfectly on its own as 'please'. Adding Vas (the object form of the formal Vi) makes it molim Vas, which means 'I ask you (formally)'. It is a very polite and common way to address a waiter or shopkeeper.

Would I say Dobar dan to a friend I address with the informal ti?

You could, but it sounds a bit too formal. Dobar dan (Good day) is standard for cafes, shops, and people you address with Vi. For friends, you would normally just use Bok (Hi).

Why doesn't the word for tea (čaj) change its ending here if I am ordering it?

When ordering, the item is the object of your request (the accusative case). However, čaj is a masculine noun for a non-living thing. In Croatian, non-living masculine nouns keep their basic dictionary form when used as an object, which is why it stays exactly the same here.

Does the word for thank you (hvala) change depending on whether I use Vi or ti?