Unicode, Copying, and Search

Two Arabic strings can look identical on screen yet fail an exact search. The usual cause is not Arabic grammar but text representation: one string may contain contextual presentation forms, decomposed combining marks, tatweel, an invisible joining control, or Persian lookalike letters. Unicode gives these characters different identities even when a font hides the difference.

The solution is to separate three layers: preserve the author's original text for display, derive a normalized form for comparison, and apply language-specific substitutions only when the intended language is known. “Remove everything unusual” is not a safe normalization strategy because ى/ي, ة/ه, and vowel marks can carry real distinctions.

Base letters versus displayed joining forms

Normal Arabic text stores base letters in logical reading order. A shaping engine chooses isolated, initial, medial, or final glyphs according to the surrounding letters.

Do not type visual joining forms manually. Type the base letters and let the renderer shape them.

The displayed lām–alif in لا may look like one glyph, but searchable text should contain two letters. Likewise, connected shapes throughout اللغة العربية are font output, not alternate spellings.

Combining tashkeel

Fatḥa, kasra, ḍamma, sukūn, shadda, tanwīn, superscript alif, and related signs are combining marks attached to a base character. A vocalized بَّ is therefore not a single decorated bāʾ in storage; it is a sequence.

Different input methods may emit shadda and a vowel in different orders. Unicode canonical normalization reorders marks according to standard combining classes, allowing canonically equivalent sequences to compare consistently. This does not mean the marks are meaningless: changing fatḥa to kasra still changes the reading.

Both targets use the same unvocalized skeleton, but their fully specified forms are not interchangeable. A diacritic-insensitive search may intentionally return both; an exact vocalized quotation search should not.

This yields two legitimate search keys:

• A diacritic-sensitive key distinguishes fully vocalized readings and is appropriate for proofreading or exact citation.
• A diacritic-insensitive key removes selected tashkeel after normalization and improves recall in ordinary unvocalized MSA.

Keep both purposes explicit. Stripping all marks from stored source text destroys information that cannot always be reconstructed.

NFC and NFKC do different work

Unicode normalization is not a single “clean Arabic” operation.

For an Arabic search index receiving legacy or copied text, NFKC is often useful for converting presentation forms before further processing. Preserve the original separately: compatibility normalization is broader than canonical normalization and should not silently rewrite archival or diplomatic source data.

The ordinary character آ and a decomposed alif-plus-madda sequence can render the same and become canonically equivalent under NFC. For the spelling value of آ itself, see the Madda page.

Presentation forms and PDF copying

Older software and some PDFs store the contextual glyph itself: an isolated, initial, medial, or final presentation form. What looks like ب may therefore not be base U+0628. NFC preserves many such compatibility characters; NFKC usually maps them back to base letters.

PDF extraction can cause a second problem: text may emerge in visual rather than logical order, or each glyph may be separated. Normalization cannot reverse a wrongly extracted line because that is an ordering error, not a canonical-equivalence issue. Compare a copied phrase with a known word, move through it with arrow keys, and use optical character recognition or a better PDF extraction path when logical order is broken.

Tatweel is visible but not a letter

Tatweel or kashida ـ (U+0640) lengthens a connection for layout: العــربية. It does not add a consonant or vowel. It is common in display typography and may appear in justified lines, headings, or informal emphasis.

NFC and NFKC do not remove tatweel. A search key that wants العــربية to match العربية must strip it explicitly. The display copy may preserve it when the elongation is genuinely part of the design.

Normal running text needs no manually inserted tatweel. Stretching a word to repair alignment creates a hidden search difference and does not fix paragraph direction.

Invisible characters can alter joining or direction

Zero-width joiner and zero-width non-joiner can request or block a connection; right-to-left and left-to-right marks can influence bidirectional layout around numbers and punctuation. These controls may be legitimate in specialized multilingual typesetting, but accidental copies can split a match while remaining invisible.

Unicode normalization normally preserves them. For a search key, remove or standardize them only under a documented policy. For source text, reveal them in an editor before deletion so that you do not damage a deliberate mixed-direction layout.

Set paragraph direction at the paragraph level. Do not scatter invisible direction marks or spaces through Arabic words as a visual patch. The Handwriting and Print Variants page develops the parallel principle for glyph variation: surface form must not replace letter identity.

Lookalikes that normalization must not merge

Arabic ي (U+064A) and Persian ی (U+06CC) are distinct letters; Arabic ك (U+0643) and Persian ک (U+06A9) are also distinct. Neither NFC nor NFKC maps those pairs together. An MSA application may apply an explicit Arabic-language mapping to imported data, but a multilingual application must not assume that Persian text is erroneous.

The same caution applies within Arabic:

Mapping ى → ي or ة → ه would merge real distinctions. These are spelling characters, not presentation variants.

A safe Arabic search pipeline

For ordinary MSA search, a robust system can derive—but not substitute for the display source—the following comparison key:

1. Decode text consistently and normalize canonical/compatibility variants according to the corpus policy.
2. Convert presentation forms to base letters when legacy input is in scope.
3. Remove tatweel and unintended invisible joining or direction controls.
4. Apply Arabic-versus-Persian character mapping only when the field is known to contain Arabic.
5. Optionally remove selected tashkeel for a diacritic-insensitive key.
6. Normalize whitespace and punctuation only as the search product requires.

Do not include lexical substitutions such as ى → ي or ة → ه. Index both the original and derived key when exact display, quoting, or auditability matters.

This is exactly the kind of failure that visual proofreading misses: the sentence can look right while the comparison key remains different.

Ordinary and specialist text

Ordinary (formal) MSA should use base Arabic letters, ordinary punctuation, and only the necessary tashkeel. (Quranic) text may contain superscript alif, recitation signs, and specialized orthographic characters whose distinctions must be preserved; stripping them turns a source edition into a different artifact. (academic) linguistic work may likewise require exact code-point or vocalization distinctions.

Choose the policy from the text's purpose. A news-site search key and a Quranic manuscript transcription should not use the same destructive cleanup routine.

Common Mistakes

1. Storing a visual lām–alif presentation ligature

2. Treating tatweel as harmless whitespace

3. Assuming normalization fixes Persian lookalikes

4. Merging alif maqṣūra and yāʾ

5. Leaving an invisible joiner inside a word

Key Takeaways

• Store base letters in logical order and let the font choose contextual glyphs and ligatures.
• NFC handles canonical composition and mark ordering; NFKC additionally folds many presentation forms, but neither removes tatweel nor maps Persian letters to Arabic.
• Preserve original text and derive separate exact and diacritic-insensitive search keys.
• Never normalize away real Arabic contrasts such as ى/ي or ة/ه.