You must match the gender of collègue to the person.
• If your colleague is male, you say mon collègue.
• If your colleague is female, you say ma collègue.
Note that collègue itself ends with the same spelling for masculine and feminine; it’s the possessive adjective that changes.