Learn Russian: Он положил словарь около себя.

It is in the Genitive case. The preposition около (meaning "near" or "around") is one of the specific prepositions of location that always requires the Genitive case. Себя is the Genitive form of the reflexive pronoun.

Why do we use себя instead of него for "himself"?

In Russian, when an action is directed back at the subject of the sentence (in this case, Он), you must use the reflexive pronoun себя. If you said около него, it would mean he put the dictionary near some other male person, not himself.

Is словарь in the Nominative case here? It looks like the basic dictionary form.

It is actually in the Accusative case because it is the direct object (the thing being put). However, because словарь is a masculine, inanimate noun (a thing, not a living being), its Accusative form looks exactly the same as its Nominative form.

Could I use поставил instead of положил for putting down a book?

It depends on how you put it down! Положил comes from положить, which means "to lay (something) flat". If he stood the dictionary upright on a bookshelf, you would use поставил. Since the sentence uses положил, we know he laid it flat on a surface.