Learn French: Un rectangle a quatre angles. - A rectangle has four angles.

You could, but it changes the nuance. Coins refers to physical "corners" (like the corner of a table or a room). In a descriptive or geometric context like this one, angles is the correct, precise term.

How do you pronounce the transition between quatre and angles?

In fluid speech, the final consonant sound of quatre (/tʁ/) links directly into the starting vowel of angles. This is called enchaînement (linking) and makes it sound a bit like "qua-trangles". Unlike a liaison, you are linking a consonant cluster that is normally pronounced anyway.

Are most geometric shapes masculine like un rectangle?

Yes, the vast majority of 2D shapes in French are masculine. Along with un rectangle, you have un carré (a square), un triangle (a triangle), and un cercle (a circle). A notable feminine exception is une étoile (a star).