Unicode, Normalization, and Arabic Search

Two Arabic strings can look identical on screen and still behave differently in search, sorting, copy-and-paste, or text-to-speech. Unicode stores letters, combining vowel marks, elongation characters, and historical display shapes as separate code points. A Lebanese reader may repair the text instantly; a search engine may not.

The practical goal is not to memorize hexadecimal numbers. It is to recognize the few character families that cause real problems, normalize text without erasing meaningful distinctions, and search across attested Lebanese spelling variants deliberately. The Arabic Keyboard page covers input methods; this page covers what happens after the characters enter a document.

The compact problem map

Combining marks are real characters

Arabic short vowels, sukūn, shadda, and related signs are combining marks layered on a base letter. The ordinary displayed word قهوة contains no tashkeel; the teaching form قَهْوِة contains additional characters. They share a consonant skeleton but are not necessarily equal in an exact search.

For broad search, try the unvocalized form قهوة. For pronunciation work, keep the pointed form in a separate field. Unicode normalization such as NFC can put canonically equivalent sequences into a consistent order, but it does not mean “remove all vowel marks.” Stripping marks is an additional search decision.

Shadda also carries meaning. حمام can represent ḥamām “pigeons” or ḥammām “bathroom” in unvocalized text; deleting shadda from a fully pointed teaching form loses a pronunciation distinction even though broad search may intentionally ignore it.

Tatweel and presentation forms are typography, not spelling

Tatweel ـ stretches a joining line for visual effect: قـهـوة. It is common in decorative posts, headlines, and emphatic chat writing (informal), but it is a character inside the string. Remove it before exact search, indexing, or comparison.

Presentation forms such as ﻻ، ﻲ، ﻛ are compatibility-era glyph code points. Modern text should store the underlying letters لا، ي، ك and let the font choose their shapes. NFC does not necessarily repair every presentation form; compatibility normalization such as NFKC usually does, but software must test the result rather than assume.

The target uses ordinary ل + ا, not the single legacy glyph ﻻ. The two may render almost identically while only the ordinary sequence behaves reliably across search, fonts, and screen readers.

Hamza and alif variants: broaden search, preserve words

The characters ا، أ، إ، آ are not freely interchangeable. In careful house spelling, use أنا، إنت، إذا. (informal) Lebanese messages often omit the hamza sign and write انا، انت، اذا, so a corpus search should try both marked and unmarked queries.

The predictable initial glottal onset is omitted in ordinary romanization here: iza, inte. Internal or contrastive hamza remains ʾ, as in سأل saʾal. Search expansion must therefore be position-sensitive: searching سال as an automatic replacement for سأل can retrieve a different verb, “flowed.”

هلق / هلأ hallaʾ “now” is a different situation. Both are established colloquial spellings, with هلأ (regional: Syria; also common in Lebanon) especially visible. Searching only this guide’s house form هلق misses legitimate material.

These are two correct search variants, not an incorrect/correct pair.

Ya, alif maqṣūra, and Persian yeh

Arabic ي U+064A, alif maqṣūra ى U+0649, and Persian ی U+06CC can converge visually in some fonts. Their identities still matter. Lebanese في “there is/in” ends in Arabic ya; على “on” ends in alif maqṣūra; Persian yeh belongs in neither.

Minimal contrast one: علي / على

The dots distinguish a name from a preposition. A normalization pipeline that maps both letters to one value improves recall only by destroying lexical precision.

Minimal contrast two: من / مين

Dropping Arabic ي changes mīn “who” into min “from.” Search-friendly normalization must not erase vowel letters just because Arabic also omits short vowels.

Search Lebanese morphology, not English spaces

Arabic conjunctions, one-letter prepositions, articles, and suffix pronouns attach. A literal query for و خبر ك will not match normal وخبرك, and ب القهوة will miss بالقهوة. Search a stable lexical core such as خبر alongside likely joined forms, or use a tokenizer that understands Arabic clitics.

For colloquial material, also search lexical variants: هيدا / هاد, هيدي / هاي, هلق / هلأ, and sometimes inherited versus more phonetic spellings. (regional: Lebanon, Syria, Palestine, Jordan) Variant sets must be built from the community and genre being searched; there is no responsible button labeled “normalize all Levantine.”

A realistic copied message

(informal) Suppose a copied message says:

To find related messages, try هلق and هلأ, pointed and unpointed text, and the lexical core قهوة with likely clitics القهوة، بالقهوة، عالقهوة. Do not replace the Arabic query with Latin halla2 or ahwe as your primary strategy; Arabizi is a separate corpus with its own variation.

Common Mistakes

Keeping a presentation-form glyph in searchable text

Importing Persian code points

Separating clitics to make search easier

Publishing transliteration instead of Arabic

Key Takeaways

Normalize typographic accidents without collapsing genuine letters. Remove tatweel, convert presentation forms to ordinary sequences, and decide explicitly whether a search should ignore combining marks. Preserve ي / ى and meaningful hamza distinctions in edited text, but query attested informal variants. Search clitic-bearing forms and documented Lebanese alternatives rather than forcing Arabic into English spacing or one imaginary pan-Levantine spelling.