Dataset · benchmark test set · Hebrish
English–Hebrew Code-Switching ASR Benchmark
A frozen, versioned test set for scoring how well speech recognition writes the Hebrew words in English speech, "pick up some bamba from the makolet", with stock models and fine-tunes scored the same way on a public leaderboard.
The test set is gated on Hugging Face (manual approval): request access to download the audio. The runner, scorer, results and leaderboard are public.
The dataset in numbers
Test set v1, frozen at dataset revision 3cb0b2b3b3e4 (2026-10-15). Later commits change only the card. 24 english_control clips (3.3 min) sit alongside the clean subset; 17.3 min of audio in all.
About the dataset
English speakers in Israel drop Hebrew words into English, and English speech recognition writes the nearest English-sounding thing (Macaulay for makolet). This benchmark measures how often a model gets those words right, and whether it still transcribes plain English well. The references write each Hebrew word in one canonical romanized spelling; Hebrew script in the output is never a hit.
Test set v1 is 145 clips from the held-out test split of Read-Aloud Voice Sentences, each tagged with a subset. The headline is the clean subset: 84 phone recordings (11.8 min) of sentences that appear in no published training set, holding 172 Hebrew term occurrences of 139 distinct terms. english_control (24 clips) is the same speaker's English with no Hebrew, a regression check. The rest (contaminated_text, contaminated_audio, desktop_mic) are reported only in the all_code_switch view.
How it is scored. Term recall, the headline, is the share of Hebrew term occurrences whose canonical spelling appears in the transcript as a whole word, case-insensitive, with hyphen and space interchangeable; never a substring. Unseen-term recall counts only the 117 occurrences of terms absent from the source training split. Lenient recall also accepts the lexicon's variant spellings (tsaharon for tzaharon), which is fairer to stock models. WER is normalised (lowercase, punctuation removed), and English control WER catches a fine-tune that damages English.
Contamination is checked, not assumed. The 2025 desktop clips and the sentences Whisper-Hebrish was trained on were traced to its exact training files and split out of the headline subset (2026-10-15). Before a fine-tune is published, the benchmark's 145 clip ids and texts are checked against its training manifest; none were found for Jachnun v1.0 or the Nemotron fine-tune.
The test set never changes in place: a new test set is a new version, and every result records the commit it was scored against. Results are stored with every transcript, so any score can be recomputed with the public scorer. Stock models on the same clips: Whisper Large V3 45.9%, Whisper Large V3 Turbo 40.1%, Parakeet TDT 0.6B v2 32.6%, Parakeet v3 31.4%.
Checkpoints and roadmap
Each training run is pinned to a dataset revision, so every model checkpoint can be traced to the exact data it saw. Planned stages are the current plan, not promises.
- Test set v1 and round 1 Done 15 October 2026
145 clips from Read-Aloud-Voice-Sentences@3bdaa2a50d52, contamination-tagged; 4 references canonicalised. Runner, scorer and leaderboard Space published with seven models: four stock and three fine-tunes.
- Known issue flagged Done 15 October 2026
Marked early-stage: 22 of the 139 term labels are not Hebrew.
- Round 2 Done 18 October 2026
PoC-3 (published as v3): 67.4% term recall.
- Round 3 Done 19 October 2026
Jachnun v1.0 (73.3%, 3.3% WER) and the Nemotron streaming fine-tune (62.2%, 4.7%). No benchmark clip in either training manifest.
- Round 4 Done 20 October 2026
Sabich v1.0: 73.8% term recall, 3.3% WER; level with Jachnun within noise.
- Test set v1.1 Planned
Relabel the non-Hebrew terms and rescore every stored result.
What is being recorded next
- v1.1: relabel the 22 of 139 clean-subset terms that are English tech words or product names (JSON, grep, Proxmox, Tuya, ...), 26 of 172 occurrences, and rescore every model from its stored transcripts. Ranks are unaffected; term recall is 2 to 6 points high until then.
- More speakers, better labels and more models: contributions are welcome as pull requests with a result file.
Speaker and recording
- One speaker: Daniel Rosehill, the author, Irish-born, English-speaking, living in Jerusalem. Results rank models on this speaker's code-switching; they do not show how a model handles other accents or spontaneous speech.
- Read speech. The clean and english_control clips were recorded on a OnePlus Nord 3 5G internal microphone (2026-10-14/15); the desktop_mic and contaminated_audio clips are desktop-mic recordings from 2025-11-17.
- 16 kHz mono WAV.
Layout and columns
A single test split: data/test/audio/<id>.wav with one metadata.jsonl. The runner loads it at the pinned revision, so every result scores the same audio. load_dataset("Hebrish/English-Hebrew-Codeswitching-ASR-Benchmark") works directly once access is granted.
| Column | What it holds |
|---|---|
file_name | WAV, 16 kHz mono |
id | Same id as in Read-Aloud Voice Sentences |
text | Reference transcript: cased and punctuated, canonical romanized Hebrew |
subset | clean, english_control, contaminated_text, contaminated_audio or desktop_mic |
hebrew_terms | List of {term, variants, unseen}: the canonical spelling, accepted variants from the Hebrew-Latin token classifier lexicon, and whether the term is absent from the source train split |
source_text | Set on the 4 rows whose reference was corrected to the canonical term; the original transcript |
duration_s / categories / recording / source_dataset | Provenance |
Models scored on it
- Hebrish Sabich v1.0: 73.8% term recall, 3.3% WER (round 4) On Hugging Face
- Hebrish Jachnun v1.0: 73.3% term recall, 3.3% WER (round 3) On Hugging Face
- PoC-1 to PoC-3: 62.2%, 65.7%, 67.4% term recall On Hugging Face
- Shakshuka PoC-2 (Nemotron streaming, scored offline): 62.2% On Hugging Face
- Whisper-Hebrish: 54.1% (known training-data overlap) On Hugging Face
Related
- Benchmark write-up (blog)What the benchmark measures, what it caught, and the results
- Benchmarks on hebrish.appThis benchmark plus the held-out splits the Hebrish models are scored on
Licence: CC-BY-4.0, with sentence-text attribution per prompt_source as described on the parent dataset's card.