Learn Finnish: Ostin kalliita kirjoja, joita luin mielin määrin. - I bought expensive books, which I read to my heart's content.

It is a fixed idiom meaning "to your heart's content" or "in abundance". It consists of the words mieli (mind/desire) and määrä (amount), both put into the instructive plural case with the -in ending. Literally, it translates to something like "with minds, with amounts."

Is the instructive case still actively used to make new phrases like this?

Generally, no. In modern Finnish, the instructive case is rarely productive, meaning you shouldn't try to invent new combinations with it. It survives mostly in fixed adverbial idioms like mielin määrin, kaksin käsin (with two hands), or paljain jaloin (barefoot).

Why is the relative pronoun joita in the partitive plural?

A relative pronoun must match the noun it refers to in number (plural), but its case depends on its role in the new clause. Here, joita is the object of luin (I read). Since reading is viewed here as an ongoing or indefinite action, its object must be in the partitive case.

If the buying is a completed past action (ostin), shouldn't the object be a total object rather than partitive (kalliita kirjoja)?

Not in this case. Even though the action is complete, the object is an indefinite plural (meaning "some expensive books" rather than a specific, known set of books). An indefinite plural quantity always forces the partitive case, which overrides the total object rule.