Knjiga koju tražim nije na polici u sobi.

Because koju is the accusative singular feminine form of the relative pronoun koji.

In Knjiga koju tražim..., the pronoun refers back to knjiga (book), so it must agree with knjiga in gender and number: feminine singular.

But its case is not chosen by knjiga. Its case depends on its role inside the relative clause:

• tražim koga/šta? = I am looking for what?
• answer: koju = which/that

So:

• knjiga = nominative, because it is the subject of the main sentence
• koju = accusative, because it is the object of tražim

That is why Serbian uses koju, not koja.

Koju introduces a relative clause, the part koju tražim = that I’m looking for.

So the structure is:

• Knjiga = the book
• koju tražim = that I’m looking for

Together:

• Knjiga koju tražim = The book that I’m looking for

English often uses that, which, or even no pronoun at all:

• The book that I’m looking for
• The book which I’m looking for
• The book I’m looking for

Serbian normally keeps the relative pronoun here: koju.

Because knjiga is the subject of the main clause, not the object of tražim directly.

The sentence is built like this:

• Knjiga ... nije na polici u sobi.
• The book ... is not on the shelf in the room.

Here, knjiga is the thing being talked about, so it is in the nominative.

Inside the relative clause, the same book has another role:

• koju tražim
• that I am looking for

There it is the object of tražim, so the relative pronoun takes the accusative: koju.

This is a very common pattern in Serbian:

• main clause role → one case
• relative clause role → another case

Because Serbian verbs usually already show the subject.

Tražim means I am looking for / I look for. The ending -im tells you it is 1st person singular: I.

So:

• tražim = I am looking for
• tražiš = you are looking for
• traži = he/she/it is looking for

You can add ja (I) for emphasis, but it is not necessary:

• Knjiga koju ja tražim... = The book that I am looking for...

That sounds more emphatic, as if contrasting with someone else.

In Serbian, the negative form of je (is) is normally written as the single word nije.

So:

• je = is
• nije = is not

This is standard:

• On je ovde. = He is here.
• On nije ovde. = He is not here.

In your sentence:

• Knjiga ... nije na polici = The book ... is not on the shelf

So nije is simply the normal negative form of je.

Because after na when it means on in a static location, Serbian uses the locative case.

The noun polica (shelf) changes in the locative singular:

• nominative: polica
• locative: polici

So:

• na polici = on the shelf

This is different from motion toward something. Compare:

• Knjiga je na polici. = The book is on the shelf.
  static location → locative
• Stavio sam knjigu na policu. = I put the book onto the shelf.
  movement/destination → accusative

So in your sentence, the book is not located there, which is why na polici is used.

For the same reason: u with a static location usually takes the locative.

• soba = room
• locative singular: sobi

So:

• u sobi = in the room

Compare:

• Knjiga je u sobi. = The book is in the room.
  location → locative
• Ulazim u sobu. = I am entering the room.
  motion into → accusative

In your sentence, u sobi describes where the shelf is, or more generally where the relevant location is.

Most naturally, it describes the shelf:

• nije na polici u sobi
• is not on the shelf in the room

So the usual interpretation is:

• there is a shelf in the room
• the book is not on that shelf

However, Serbian word order can sometimes allow slight ambiguity, just like English:

• not on the shelf in the room

In practice, people will usually understand it as:

• the shelf that is in the room

If you wanted to make the location of the book more clearly tied to the whole sentence, context or different wording could help.

Normally, no. Serbian usually keeps the relative pronoun.

English:

• The book that I’m looking for
• The book I’m looking for

Serbian:

• Knjiga koju tražim

You generally do not omit koju here. Leaving it out would sound ungrammatical or very unnatural.

So this is an important difference from English:

• English can often omit that/which
• Serbian usually cannot omit koji/koju/kojeg, etc.

Yes, Serbian word order is fairly flexible, but this is the most neutral and natural order.

The standard structure here is:

• Knjiga = topic / subject
• koju tražim = relative clause attached to knjiga
• nije = verb
• na polici u sobi = location

So:

• Knjiga koju tražim nije na polici u sobi.

Other orders are possible for emphasis, for example:

• Na polici u sobi nije knjiga koju tražim.

But that sounds more marked and contrastive, something like:

• The book I’m looking for is not on the shelf in the room.

For learners, the original order is the best one to use.

Yes, tražim is present tense.

Serbian present tense often covers both:

• a simple present meaning: I look for
• an ongoing present meaning: I am looking for

So:

• tražim can mean I look for or I’m looking for
• context tells you which is more natural

In this sentence, English normally translates it as:

• the book that I’m looking for

because that is the most natural meaning in context.

Yes. In Serbian, prepositions often work closely with case, and na and u are especially important because they can take different cases depending on whether you mean:

• location → usually locative
• movement toward/into → usually accusative

In your sentence both are about location:

• na polici = on the shelf → locative
• u sobi = in the room → locative

This location-vs-motion contrast is one of the key things English speakers need to learn in Serbian.

It can be translated as either that or which, depending on the style of English translation.

• Knjiga koju tražim
  = The book that I’m looking for = The book which I’m looking for

In everyday English, that is usually more natural here. But grammatically, koju is simply the relative pronoun referring to book.

So do not think of koju as matching only one English word. It corresponds to the function of that/which in this kind of clause.