Learn Russian: Капитаны купили новые катера.

Most masculine nouns form their plural with -ы or -и, which is why we have капитаны. However, a specific group of masculine nouns takes a stressed -а (or -я) in the plural instead. Катер (motorboat) belongs to this special group, so its plural is катера. You will also see this with common words like дом (дома) and город (города).

Since the captains bought the motorboats, shouldn't "new motorboats" be in the accusative case?

It actually is in the accusative case! Новые катера is the direct object here. Remember that for inanimate masculine nouns, the accusative plural is completely identical to the nominative plural. If the captains had bought something animate (like dogs or horses), the accusative plural would look like the genitive case instead.

Is there a trick to knowing which masculine words take this stressed -а plural like катера?

Unfortunately, there isn't a strict rule you can apply just by looking at a word. It is a historical exception that affects several dozen common masculine nouns. Often, these are everyday words (like глаза for eyes, or паспорта for passports), or specific professional and nautical terms. You simply have to memorize them as you learn the vocabulary.