Learn Russian: Эта категория представляет собой совокупность правил.

The phrase представлять собой is a formal way to say 'is', 'constitutes', or 'represents'. The word собой is the instrumental case of the reflexive pronoun себя (oneself). Literally, the phrase means 'represents by itself', but it functions as a single formal predicate. You will see it very often in academic and professional Russian to define concepts.

What case does the structure представлять собой require?

It requires the Accusative case for the object being defined or presented. In this sentence, the noun совокупность (a set or totality) is the direct object and is therefore in the Accusative case.

If совокупность is in the Accusative case, why does it look exactly like the dictionary form (Nominative)?

совокупность is an inanimate feminine noun ending in a soft sign, which places it in the 3rd declension group. For inanimate nouns in this group, the Accusative case form is always identical to the Nominative case form.

Why does the word for rules, правил, have no ending here?

It is in the Genitive plural because it follows совокупность to mean 'a set OF rules'. The dictionary form is the neuter noun правило. Neuter nouns that end in -о in the singular typically drop their vowel ending entirely (taking a 'zero ending') in the Genitive plural.