Unicode Forensics for Persian Editors

Two Persian strings can look identical and still behave differently in search, sorting, line breaking, cursor movement, or text-to-speech. The cause is often invisible or font-dependent: Arabic ك instead of Persian ک, a normal space instead of U+200C ZWNJ, an unexpected bidirectional control, or a pre-shaped presentation character instead of ordinary letters.

Unicode forensics means examining the stored sequence rather than trusting the glyphs. The goal is not to delete everything invisible. Some invisible characters, especially ZWNJ, encode real Persian word structure. A safe editor identifies each character's function, normalizes only the intended layer, and proves that cleanup has not changed the text's letter skeleton.

For the linguistic function of the most important invisible in this chapter, see Spaces, half-spaces, and the ZWNJ.

Start with code points, not appearance

The visual minimal pair ی / ى is especially treacherous: in many fonts both are dotless in final position. NFC does not equate them because they are distinct letters, not canonically equivalent spellings. A Persian-aware mapping must be explicit.

What Unicode normalization does—and does not do

NFC and NFD resolve canonical composition differences. NFKC and NFKD additionally fold compatibility forms. Arabic Presentation Forms-A and Forms-B contain isolated, initial, medial, final, and ligature glyphs created for compatibility with older encodings. They should not be used as the storage model for normal Persian text; fonts should shape base letters contextually.

Compatibility normalization can turn many presentation forms into base characters or sequences. That makes NFKC valuable in a diagnostic or controlled ingestion pipeline, but unsafe as an unreviewed universal rewrite: compatibility folding is broader than Persian letter shaping and may erase distinctions another domain intended to retain. NFC alone, meanwhile, will not replace Arabic ك/ي/ى with Persian ک/ی and will not decide whether a space ought to be a ZWNJ.

The rule is simple to state: normalize representation before applying language-specific mappings, but audit the result after each lossy step. The hard part is deciding which mappings are authorized for the document class.

ZWNJ is content

U+200C is default-ignorable in the sense that unsupported rendering need not show a glyph; it is not semantically disposable. In Persian it can mark the boundary after می‌/نمی‌, before plural ها, and inside compounds. It affects shaping, tokenization, matching, and sometimes line behavior.

Do not confuse ZWNJ with U+200B ZERO WIDTH SPACE, which supplies a potential break, or U+200D ZERO WIDTH JOINER, which requests joining. An editor that globally deletes characters whose width is zero destroys correct Persian morphology along with unwanted debris.

Bidirectional controls need an audit trail

Persian letters have right-to-left directionality, while Latin text and European numbers run left to right. Unicode's bidirectional algorithm ordinarily resolves mixed runs, but controls can override or isolate direction: U+061C ARABIC LETTER MARK, U+200E LRM, U+200F RLM, the older embeddings and overrides U+202A–U+202E, and the isolate controls U+2066–U+2069.

These controls are invisible yet can change display order and cursor behavior. Paired controls can become unbalanced during copy-and-paste. Overrides are especially suspicious in ordinary prose and in identifiers because displayed order may cease to reflect logical order.

For HTML, prefer structural direction controls such as properly scoped dir and isolation over sprinkling invisible marks through content. In plain-text interchange, preserve known legitimate controls and flag unexpected ones. Never silently delete a paired control from only one side.

Confusable punctuation and whitespace

Persian prose uses ، ؛ ؟, not the visually or functionally competing ASCII , ; ?. Also inspect U+00A0 NO-BREAK SPACE, tabs, repeated spaces, and line separators. A no-break space can look ordinary yet frustrate tokenization; an ASCII question mark can jump to an unexpected edge of an RTL run.

Punctuation normalization is context-sensitive. An English bibliographic title embedded in Persian may legitimately retain English punctuation, while the surrounding Persian sentence follows Persian convention. The editor must identify language runs rather than apply one global replacement table.

A skeleton-preserving cleanup pipeline

A defensible production pipeline has explicit stages:

1. Preserve the source bytes or original string and record its provenance.
2. Decode strictly; report replacement characters rather than hiding them.
3. Inventory characters by code point, including all default-ignorables.
4. Apply a declared Unicode normalization form to a working copy.
5. Decompose Arabic presentation forms to base sequences under review.
6. In Iranian Persian runs, map Arabic ك/ي/ى to Persian ک/ی; exempt genuine Arabic quotations.
7. Validate ZWNJ against morphology and the chosen orthographic standard.
8. Audit bidi controls, whitespace, digits, and punctuation by context.
9. Compare a skeleton before and after: base letters, Persian digits, word separators, and authorized ZWNJ boundaries must agree except for logged mappings.
10. Run search, rendering, copy-paste, and TTS tests on the result.

This guide applies the same principle to audio. A correct spokenForm adds vowel marks to exactly the letters, spaces, ZWNJ characters, punctuation, and Persian digits of its target. Removing U+064B–U+065F and U+0670 must recover the displayed line exactly. A vocalization that changes the skeleton is not annotation; it is a different utterance.

Register and scope

Canonical Persian encoding is required across (written-standard), (formal), (academic), and (literary) digital text. Legacy characters are evidence in archival data, not a prestigious register. Raw bidi controls may be legitimate in specialized plain text but should be exceptional and documented. The normalization policy for a diplomatic edition must be more conservative than that for a search index; one pipeline cannot safely serve both without separate layers.

Common Mistakes

1. Assuming NFC converts Arabic kaf and yeh

2. Using ZWJ as a half-space

3. Storing shaped presentation glyphs as letters

4. Using ASCII punctuation in a Persian question

5. Reversing letters to repair a display problem

Key Takeaways

Trust code points only after you inspect them, and trust glyphs only after you compare them with logical order. Unicode normalization, Persian-specific letter mapping, ZWNJ validation, bidi auditing, and punctuation cleanup solve different problems. Preserve the source, process a working layer, log lossy operations, and prove that every cleaned sentence still has the intended Persian skeleton.