Learn Hardcore Finnish: Minä juoksen bussin perässä kadulla.

bussin takana = behind the bus (static position)
You are simply located behind it.
bussin perässä = behind/after the bus, often with a sense of following/chasing it
This fits nicely with juoksen (I’m running), so it suggests you’re trying to catch up with it.

So:

Both are correct; perässä just matches the idea of pursuit more naturally here.

What case is perässä, and what does the -ssä ending mean?

Perässä is in the inessive case (the -ssa/-ssä ending).

The inessive usually means in, inside something, but with these postpositions it’s part of a fixed system:

So even though perässä literally means in the rear, you translate it as after/behind in natural English.

Why is it kadulla and not kadussa or kadulle?

Katu = street
kadulla is the adessive case (ending -lla/-llä) and usually means on, at.

For places:

kadulla = on the street (surface / general area)
kadussa (inessive) would mean inside the street, which is not how we talk about streets
kadulle (allative) = onto the street / to the street
kadulta (ablative) = from the street

So:

Can I change the word order, like Minä juoksen kadulla bussin perässä?

Yes, you can change the word order. Finnish word order is relatively flexible because the case endings show the roles of the words.

All of these are grammatically fine:

The differences are about emphasis and information structure:

Elements earlier in the sentence are usually more topical / contrastive / emphasized.
For example, Kadulla juoksen bussin perässä emphasizes on the street more strongly.

In a neutral context, Minä juoksen bussin perässä kadulla or Juoksen bussin perässä kadulla are very natural.

Could I say Minä juoksen bussia to mean “I’m running after the bus / chasing the bus”?

Minä juoksen bussia is not idiomatic Finnish.

Common ways to say I’m running after the bus / trying to catch the bus:

Juoksen bussin perässä. = I’m running after the bus.
Juoksen bussia kiinni. = I’m running to catch the bus.
(Here bussia is partitive and kiinni adds the meaning of catching.)

So you cannot just put bussi into partitive (bussia) with juosta and get the meaning chase. You need a structure like:

Why isn’t minä capitalized, like English I?

In Finnish, minä is a normal pronoun and is not capitalized unless it is at the beginning of a sentence:

English I is a historical special case; Finnish doesn’t have such a rule. So:

middle of a sentence: … että minä juoksen …
start of a sentence: Minä juoksen … (capitalized only because it’s first in the sentence).

Where does juoksen come from? Why juokse- and not juosta-?

The dictionary form is juosta (to run), but in the present tense, many Finnish verbs change their stem.

For juosta:

Conjugation (present):

This is a regular pattern for this verb type; you just have to learn that the present stem is juokse-, not juosta-.

Is bussin perässä only physical, or can it be used metaphorically like after the bus (in time)?

Bussin perässä in this form is mainly spatial/physical: behind/after the bus in space.

For time/sequence, Finnish typically uses:

bussin jälkeen = after the bus (in order/sequence)
- Juna tulee bussin jälkeen. = The train comes after the bus.

So:

juoksen bussin perässä = I run behind/after the bus (physically)
junan vuoro on bussin jälkeen = the train’s turn is after the bus (in order)

In your sentence with juoksen, perässä is clearly about physical movement.