The phrase Au petit-déjeuner has been moved to the front of the sentence for emphasis or to set the scene.
So the basic sentence is:
When the time expression comes first, a comma is often added:
It works like English sentences such as In the morning, I drink coffee.
Because French is using partitive articles here. They mean some or an unspecified amount of.
This is very common with food and drink when you are talking about an indefinite amount.
If you said le beurre or la confiture, you would usually mean a specific butter or a specific jam already known in the conversation.
Avec is a very natural way to say that the bread comes with butter and jam.
Other choices are possible, but they change the feel slightly:
So avec is simple, neutral, and very natural here.