Language detector: identify the language and script of any text

You have a snippet of text in a script you cannot read, or a short paragraph you need filed under the right language, and you need an answer you can trust. The free Cubbbix Language Detector names the language and the script, shows the next best guesses, and labels its confidence. Paste a sentence or a paragraph, and the check runs in your browser, with nothing uploaded.

The check runs in two steps. Step one counts the script, using Unicode block ranges: Arabic is U+0600 to U+06FF with its supplement and extended blocks, Cyrillic is U+0400 to U+04FF plus its supplement, Greek is U+0370 to U+03FF, Hebrew is U+0590 to U+05FF, Devanagari is U+0900 to U+097F, Hiragana is U+3040 to U+309F, Katakana is U+30A0 to U+30FF, Hangul is U+AC00 to U+D7AF with its jamo blocks, and Han ideographs sit at U+4E00 to U+9FFF. When one script covers almost all the letters, the language is often settled at this step: Hangul means Korean, Thai script means Thai, Hebrew script means Hebrew.

Figure 1: Script first, then words. One script often settles the language before a single word is scored.

Common words, not rare ones

Step two only runs where it is needed, for scripts shared by several languages. Latin script covers English, French, Spanish, German, Italian, Portuguese, Dutch, Turkish, Polish, Swedish, Norwegian, Danish, Finnish, Czech, Romanian, Indonesian and Vietnamese, so the tool scores the text against a short list of common words for each: the, and, of for English, le, la, les, de for French, el, la, los, de for Spanish, der, die, das, und for German. Rare words are the ones a writer borrows from other languages, so they mislead. Common words appear in almost every sentence and are rarely borrowed, which is why a short list of them separates languages fast.

Distinctive letters add a small bonus: ñ for Spanish, ã and õ for Portuguese, ß for German, ğ and ş for Turkish, ą and ł for Polish, and ø and æ for Danish and Norwegian. Cyrillic text is scored the same way for Russian, Ukrainian, Bulgarian and Serbian, and Arabic-script text for Arabic, Persian and Urdu, where Persian uses پ, چ, ژ and گ and Urdu uses ٹ, ڈ, ڑ and ھ. The winner is the language with the highest score, the runner-up sits beside it, and the confidence label follows the gap between them and the length of the text.

Figure 2: Common words separate languages fast, because they appear in almost every sentence and are rarely borrowed.

The calls it refuses to make

A word-list check has weak spots, and the tool names them instead of hiding them. Short text is unreliable: three words can match several languages at once, and a single word such as bank exists unchanged in English, Dutch and other languages. Under about 10 words the tool reports Low confidence. Spanish and Portuguese share most of their common words, and Danish and Norwegian share most of theirs, so a short neutral sentence can go either way, and the runner-up score shows how close the call was.

Chinese characters are shared across Chinese varieties and Japanese kanji, so a text with Han characters and no kana is reported as Chinese, with a note that the variety is not distinguished. Japanese is identified when hiragana or katakana is present. Devanagari is shared by Hindi, Marathi and Nepali, and the tool scores Hindi words only, so a Marathi or Nepali text may still score as Hindi. A text that mixes two languages, a list of names, or a snippet of numbers and proper nouns has too little signal for any word list, and the tool says so instead of picking a winner with false confidence.

Figure 3: Short text, shared scripts and mixed text are the weak spots. The tool labels them instead of hiding them.

Questions people ask