Learn Russian: Здесь была группа туристов.

Туристов is the genitive plural of турист (tourist). Because турист is a masculine noun ending in a hard consonant, it takes the -ов ending in the genitive plural. You use the genitive case here because you are saying "a group OF tourists".

Is there a Russian word for "of" in this sentence?

No, Russian rarely uses a separate word for "of" to show what a group is made of or who something belongs to. Instead, the second noun simply changes to the genitive case. The -ов ending on туристов does all the work of the English word "of".

Why is the verb была (was) in the feminine form if the tourists are a mixed or male group?

In Russian, the verb always agrees with the main grammatical subject of the sentence. The subject here is группа (group), which is a feminine singular noun. Even though the group contains туристов, the verb only cares about the core noun группа.

Why is the word order Здесь была группа instead of Группа была здесь?

Russian word order is flexible, but it generally puts the "new" or most important information at the end of the sentence. Здесь была группа туристов emphasizes WHAT was here (a group of tourists). If you started with Группа туристов была здесь, it would sound like you are answering the question "Where was the group?"