Learn Farsi: فیلسوف باید خویشتن را خوب بشناسد.

Both خویشتن and خود mean himself or oneself, but خویشتن is a literary, formal word. Because this sentence is about a philosopher, the elevated tone fits the topic. In everyday conversation, you would just say خود را.

Could I use خویش instead of خویشتن in this sentence?

You could, but خویشتن is usually preferred when the pronoun stands completely on its own as a direct object, as it does here. خویش is more frequently used to show formal possession, like pointing out one's own book.

Why is the direct object marker را placed right after خویشتن?

The word خویشتن is the direct object of the sentence because it is who the philosopher must know. Since reflexive pronouns are always specific and definite, they must take the marker را when acting as the object.

I know the word باید means must, but why is the verb بشناسد in the subjunctive?

In Farsi, modal verbs like باید always require the following verb to be in the present subjunctive mood. That is why the present stem شناس (from the infinitive شناختن, meaning to know) takes the subjunctive prefix بـ and the third-person singular ending ـد.