Learn Portuguese (Portugal): Esse vestido é teu?

esse is specifically used for an object near the person you are talking to. If the dress was near you (the speaker), you would use este (this). If it was far away from both of you, you would use aquele (that one over there).

Since vestido is masculine, why doesn't the word for "that" end in "o" like esso?

In Portuguese, the masculine singular form for this demonstrative word is just esse. The feminine form is essa. There is no word "esso"! Since vestido (dress) is masculine, we use esse.

Why is the word for "yours" teu instead of tua?

Possessive words in Portuguese always match the gender of the object being owned, not the person who owns it. Because vestido is a masculine word, you must use the masculine form teu, even if you are asking a woman if the dress belongs to her.

The word order here is exactly like the statement "That dress is yours." How do we know it's a question?

In Portuguese, you don't need to change the word order or add helper words like "is" or "do" to the front of a sentence to make it a question. You simply use a questioning tone of voice when speaking, or a question mark when writing.