Learn Spanish (Spain): Ayer fui al castillo con mi hermano. - Yesterday I went to the castle with my brother.

You can tell from the context! Because it is followed by al castillo (to the castle), it indicates movement. If it meant 'I was,' it would usually be followed by a characteristic or identity, like fui estudiante (I was a student).

Where is the word 'I' (yo) in this sentence?

In Spanish, you can usually drop the subject pronoun because the verb itself tells you who is doing the action. The form fui specifically belongs to the 'I' form, so adding yo is optional and mostly used for extra emphasis.

Why does the sentence use al instead of a el for 'to the'?

Whenever the preposition a (to) is immediately followed by the masculine article el (the), they must combine to form the contraction al. You will never see a el written separately in Spanish unless the El is part of a capitalized proper name.

Does the word ayer (yesterday) always trigger this specific past tense?

Yes! Because ayer refers to a specific, completed time block in the past, it is a huge clue to use the Pretérito Indefinido (completed past) rather than an ongoing past tense.