Learn Hardcore Icelandic: Vinnustaðurinn er nálægt bókabúðinni.

Because nálægt is a preposition that governs the dative. After nálægt, the following noun phrase must be in the dative, so the definite dative of bókabúð is bókabúðinni. Using bókabúðin (nominative) or bókabúðina (accusative) would be ungrammatical here.

Is nálægt a preposition or an adjective here?

Here it’s a preposition meaning near and it does not inflect: nálægt + dative. There is also an adjective nálægur meaning nearby that agrees with the subject and still takes a dative complement, e.g. Vinnustaðurinn er bókabúðinni nálægur. That adjective construction is correct but more formal; the prepositional nálægt is the neutral everyday choice.

Why do we see different endings: -urinn in vinnustaðurinn and -inni in bókabúðinni?

Icelandic marks definiteness with a suffix on the noun, and the suffix depends on gender and case:

vinnustaðurinn = masculine nominative singular definite of vinnustaður.
bókabúðinni = feminine dative singular definite of bókabúð. The double n in -inni is part of the feminine dative definite ending.

What would the sentence look like with indefinite nouns?