Learn Icelandic: Við töluðum saman í stað þess að rífast.

The phrase í stað (in place / instead) requires the pronoun following it to be in the genitive case. Here, it is followed by the pronoun það (it), which becomes þess in the genitive. The whole set phrase í stað þess að is used to mean "instead of [doing something]."

Could I use "í staðinn fyrir" here instead?

You generally use í stað þess að when an infinitive verb follows, like að rífast in this sentence. You use í staðinn fyrir when a noun or pronoun follows, such as "instead of me" (í staðinn fyrir mig).

The verb "rífast" ends in "-st". Is it a passive verb here?

No, this is a reciprocal -st verb, which describes an action two or more people do to each other. The active verb rífa means "to tear", so rífast paints a picture of people tearing at each other. Today, it simply means "to argue" or "to bicker."

Why do we need "saman" if "við töluðum" already means "we talked"?

In Icelandic, að tala saman is a very common fixed phrase meaning "to converse" or "to have a conversation." Without saman, við töluðum could just mean "we spoke" in a general sense, like giving a speech, rather than specifically talking with one another.