Model · token classification · Hebrish
Hebrew-Latin Token Classifier: finds Hebrew written in Latin letters
Finds Hebrew words written in Latin letters inside English text, so text-to-speech can hand them to a Hebrew voice instead of reading makolet as English. There was no model for this before.
What it finds
I need to go to Bituach Leumi before the makolet closes.
My savta says the arnona went up again.
About the model
The model marks every span of romanized Hebrew in an English sentence, with character offsets and a confidence score. Multi-word terms come out as one span (Bituach Leumi, kupat holim). Labels: O, B-HE, I-HE.
It exists because nothing else does this. GlotLID, lid.176 and lid218e have no heb_Latn label, and a language detector sees Latin script inside an English sentence and calls it English. No token-level code-switching model covered Hebrew.
It was built for text-to-speech. English TTS reads makolet as "MACK-oh-let"; the fix is to find the Hebrew spans, convert them to Hebrew script and send each to a Hebrew voice. This model is pass 1 of that pipeline (detect, transliterate, add niqqud, tag, speak). The first user is My Weird Prompts, an AI-generated podcast whose episodes about life in Israel are full of these words.
The labelling test is pronunciation, not etymology. Words English already says acceptably (Shabbat, hummus, challah, kosher) are deliberately not flagged.
Results
v1, run xlm-roberta-base, 8 epochs, evaluated 15 October 2026. Scored on the term-disjoint test split: no Hebrew term in the test set appears in training, so this measures how well it generalises to words it has never seen. Higher is better.
| Test set | Span precision / recall | Span F1 |
|---|---|---|
| All test sentences (468) | 0.928 / 0.953 | 0.941 |
| Generated (430) | 0.928 / 0.948 | 0.938 |
| Dictated, human-chosen (38) | 0.929 / 1.000 | 0.963 |
| Validation, used for model selection (467) | 0.950 / 0.948 | 0.949 |
- Hard negatives: 0 false positives in 58 sentences built to trap it (substrings, English homographs such as dud, tor and salon, lookalikes, anglicised loanwords).
- Of the 41 test sentences it gets wrong, 14 are gaps in the gold labels rather than model errors: Hebrew words the blind annotator refused and no human has yet re-reviewed.
- For comparison, the closest published recipe (EnTaCs, English–Tamil, arXiv 2603.26587) reports F1 0.638 on its romanized class.
Training
- Fine-tune of xlm-roberta-base, 8 epochs, about 3 minutes on one A10G.
- 4,737 sentences over 1,235 seed terms. Generated sentences were each seeded with a term and annotated blind by a second model family; disagreements went to human review, along with a 10% random audit of agreements.
- 458 dictated sentences: the text of sentences I read aloud for the Hebrish speech fine-tunes. Spans nobody has ruled on, and Israeli person names, carry no loss and are not scored.
Training data
Checkpoints and roadmap
Where the model is now and the checkpoints still to come. Planned stages are the current plan, not promises.
- v1 Done 15 October 2026
First release as danielrosehill/hebrew-latin-token-classifier, DOI 10.57967/hf/10997. Demo Space and Replicate API.
- Moved to the Hebrish organisation Done 20 October 2026
Same weights, copied to Hebrish/Hebrew-Latin-Token-Classifier (DOI 10.57967/hf/11108). New versions are published there.
- Audio sibling In progress
Word-level English/Hebrew language ID in code-switched speech (Hebrish-ASR/Hebrew-English-Word-LID-Audio).
- Third label Planned
A possible later version: a label for "Hebrew that English already says", so the user chooses where the line falls.
Limitations
- English text with romanized Hebrew only. Hebrew script is not the target.
- The training data is largely synthetic; the dictated slice is the closest thing to real input.
- Under the default, conservative posture it does not flag place names (Dizengoff, Ramat Gan) or words English already says. That line is a design choice, not a fact.
- Take each word's label from its first sub-token. With the stock pipeline use aggregation_strategy="first"; "simple" cuts words short.
Related
- Replicate APICPU; inputs text and threshold
- White paper (PDF)