Learn Spanish (Latin America): Tu vestido es rojo.

Tu without an accent mark means "your" (a possessive adjective). Tú with an accent mark means "you" (the person). They sound exactly the same when spoken, but the accent mark in writing helps you tell the difference between "your" and "you".

If the person I am talking to is female, shouldn't "your" change to match her gender?

No, the possessive adjective tu does not change based on the gender of the owner or the object. Words like mi, tu, and su only change to match the number of things owned. If you wanted to say "your dresses", it would become tus vestidos.

Since a dress is usually worn by women, why is vestido masculine?

In Spanish, grammatical gender belongs to the word itself, not the person who uses or wears the item. The word vestido ends in -o, making it a masculine noun.

Why is the color rojo instead of roja?

Colors are adjectives that describe nouns, so they have to match the gender of the noun. Because vestido is masculine, the color must be the masculine form rojo. If you were talking about a feminine noun like a shirt (camisa), the color would be roja.