Learn Swedish: Glaset står på kanten av bordet. - The glass is standing on the edge of the table.

You actually can use the compound word bordskanten, which is very common! However, using kanten av bordet with the preposition av is another natural way to express a specific part of a whole physical object. You will often hear both forms used to mean the exact same thing.

Could I use till instead of av to mean 'of' in this sentence?

No. While till is used for 'of' when talking about relationships between people (like en vän till mig) or ownership, it doesn't work for physical parts. For non-proportional physical parts of an object, like the edge of a table or the roof of a house, av is the correct preposition.

Why does the sentence use står instead of är to say the glass 'is' on the table?

In Swedish, it is much more natural to use positional verbs like står (stands), ligger (lies), or sitter (sits) rather than just är (is) when describing where a physical object is located. Because a glass has a flat base and rests upright, it står.

Why are both kanten and bordet in the definite (the) form?

Just like in English ('the edge of the table'), we are talking about a specific edge of a specific table. When using av to connect a part to its whole, both nouns usually take their definite endings.