Learn Russian: Учёный допускает, что атом — это очень маленькая частица. - The scientist admits that an atom is a very small particle.

Not quite. Допускать (imperfective) generally means "to allow," "to permit," or "to assume/concede." In academic contexts, it means to accept something as a possibility. If you wanted to talk about confessing to a mistake or a crime, you would use признавать or признаваться.

Why is there a dash and это before очень маленькая частица?

When you equate one noun to another in the present tense (here, the noun атом is equated to the noun частица), the verb "to be" is omitted. To act as a bridge, Russian writers use a dash followed by это. It acts like a grammatical equals sign between two nouns.

How would we apply this lesson's grammar to say the scientist thinks it is highly unlikely that an atom is a small particle?

You would change the main verb and drop the particle into the second clause: Учёный считает, что атом вряд ли очень маленькая частица. Notice that вряд ли (unlikely / hardly) simply drops in right before the doubted fact. It takes the normal indicative mood, meaning it reflects real facts rather than hypothetical ones, so you don't need to add the conditional particle бы.

Could we use едва ли instead of вряд ли to express this same doubt?