Arabic text normalizer: the five rules that fix Arabic search

Arabic search breaks on differences too small to see at a glance. أحمد typed with أ will not match احمد typed with ا. على typed with ى will not match علي typed with ي. مدرسة typed with ة will not match مدرسه typed with ه. A word with tashkeel marks will not match the same word without them. Each of these is the same word to a reader and a different string to a computer, and together they make Arabic search fail in ways that look random to the user.

The free Cubbbix Arabic Text Normalizer applies the standard fix: the five-rule set published by Apache Lucene, the search library behind many Arabic search engines. Hamza-on-alef forms become a bare alef. Dotless yeh becomes yeh. Teh marbuta becomes heh. Diacritics are removed. Tatweel, the stretching character, is removed. Each rule is a switch on the page, so you can keep the rules your task needs and skip the rest, and the stats line counts every change by rule.

Figure 1: Five small rules fix the mismatches that make Arabic search look broken.

Why writers produce five versions of one word

Arabic writers drop hamzas, swap ى and ي, and write ة as ه without thinking, because in everyday writing those differences carry no meaning. Tashkeel marks are pronunciation guides: a Quranic quote or a children's textbook keeps them, and a chat message drops them, so the same word appears in both forms. Tatweel is a justification trick: a writer stretches السـلام to fill a line, and the stretching character stays in the text unless someone removes it. Search has to survive all of that, and normalization is how it does: reduce every variant to one form before comparing, on both sides of the search.

The rule set is deliberately lossy. مدرسة (school) and مدرسه (his school, in dialect spelling) become the same string after the marbuta rule runs, which is what you want for search and not what you want for grammar. That trade is the point of the tool, and the reason for its warning: normalize for search, matching and deduplication, and keep the original for anything people will read, sign or quote. Normalization does not add missing tashkeel, does not fix spelling, and does not turn dialect into standard Arabic. It removes the differences that block matching, and nothing else.

Figure 2: Alef forms to ا, dotless yeh to ي, teh marbuta to ه, tashkeel removed, tatweel removed.

Two jobs it does well

The first job is search: normalize the query and the documents with the same rules, and a search for احمد finds أحمد, احمد and أحمَد in one pass. The second job is deduplication: two customer lists, two product catalogues or two comment feeds match after normalization even when one side kept the hamzas and the other did not. The light preset on the page strips tashkeel only, for jobs where alef and marbuta differences should still count. The search preset turns every rule on, for the widest match.

Everything runs in the browser, with nothing uploaded or stored, so customer names and private text are safe to paste. If your team handles Arabic text at any volume, normalize at intake: one pass on the way in, and the matching problems stop appearing in every report downstream.

Figure 3: Normalize both sides before comparing, and the same word matches in every spelling.

Questions people ask