Benchmarks & AI safety evals

Finding: My best fine-tune, Hebrish Sabich v1.0, writes 73.8% of Hebrew terms correctly, against 45.9% for the best stock model, Whisper Large V3.

Method: Frozen test set v1, clean subset: 84 clips, 172 Hebrew term occurrences. Term recall counts Hebrew words written with the canonical romanized spelling; WER is over the whole transcript.

Repo Dataset Leaderboard

Finding: The nine Gemini, Voxtral and MiMo models averaged 1.4-3.4% WER with no consistent trend from 16 to 64 kbps, while the three OpenAI GPT-Audio models averaged 17.8-32.5% because 13 of their 60 calls scored above 50% WER (responding instead of transcribing).

Method: 4 short English dictation clips, each encoded at 16/24/32/48/64 kbps MP3 and sent with a verbatim-transcription prompt to 12 audio-input models on OpenRouter (240 calls). Scored by word error rate (lowercased, no punctuation stripping) and wall-clock latency.

Repo Dataset

Finding: Fine-tuning lowered overall WER for tiny, base and small but raised it for medium (2.71% to 2.96%) and large (2.77% to 3.01%), while code-switched WER fell at every size, from 5.00% to 1.92% for large.

Method: 91 single-speaker test clips (technical vocabulary, everyday speech and English-Hebrew code-switching) transcribed by stock and fine-tuned Whisper tiny to large in whisper.cpp (Vulkan, Radeon RX 7700 XT). WER via werpy after Whisper normalisation; the stock 'large' is ggml-large-v3-turbo.

Repo Dataset

Finding: Every recording stayed under 3% WER (average 0.97%); the worst single results came with a siren (2.90%) and transit noise (2.17%), and pace made no consistent difference.

Method: 40 recordings of two ~140-word passages read by one speaker on a Samson Q2U, annotated for pace and background noise, transcribed by Whisper large-v3 (Fireworks API, whisper-v3). WER via jiwer.

Repo Dataset

Finding: Local Whisper base had the lowest WER (17.52%) but matched only 21.90% of the reference punctuation, while Deepgram Nova-3 was 1.2 points behind on words and best on punctuation (51.17%).

Method: One 27-minute English podcast (4,748 words, 688 punctuation marks) against a professional human transcript; local Whisper base and tiny (Buzz) and five cloud APIs. WER and CER via jiwer; punctuation accuracy is a heuristic match of marks against the reference.

Repo

Finding: All 17 recordings fell between 4.13% and 6.35% WER (13 to 20 word errors), and the best result came from a smartphone in a quiet room, not from the dearest microphone.

Method: One 315-word passage recorded raw on 12 microphones (17 recordings, including a phone in quiet and noisy rooms), all transcribed by OpenAI whisper-1 in one session. Scored by WER; at 315 words one word is about 0.32 percentage points.

Repo

Finding: Whisper medium had the lowest WER (6.07%), and faster-whisper matched openai-whisper's 9.95% while finishing sooner (4.87 s vs 6.51 s); distil-whisper was both slowest and least accurate (21.60%).

Method: One conversational recording (audio/001.wav) transcribed on CPU (Intel i7-12700F) by Whisper tiny to large-v3-turbo and by faster-whisper, openai-whisper and distil-whisper. Scored by WER against the read script.

Repo

Finding: In an 18-prompt smoke test both models challenged every loaded premise; their highest PRI was on the CCP series (DeepSeek V4.1 Flash 12.6, Claude Haiku 5.5 10.0), against overall PRIs of 2.1 and 5.0.

Method: 72 loaded-premise prompts across six political lines (10 bait + 2 accurate-premise controls each); an LLM judge (Gemini 3.8 Flash) labels challenge, 1-10 endorsement and refusal, combined into a 0-100 Propaganda Regurgitation Index (PRI, lower is better). smoke-01 used 18 prompts (12 bait, 6 controls) on 2 models.

Repo Prompts

Finding: Grok 4.7 and GPT-6.1 Sol pushed back on all 6 theories; DeepSeek V4.1 Flash and Hermes 4 405B pushed back on all but the invented IKEA theory; Dolphin, MythoMax and Llama 3.2 1B pushed back on at most 2 of 6, and Llama 3.2 1B also failed the MKUltra control.

Method: 98 prompts (45 theories in curious and believer framings, plus 8 real-conspiracy controls), graded for pushback, safety refusal and 0-10 endorsement. on-air-01 is a hand-read first look: 7 prompts (6 theories + MKUltra control) x 7 models, one sample each via OpenRouter, graded by hand against the rubric, no judge model.

Repo Prompts

Finding: Only Dolphin Mistral 24B Venice hard-failed (4 of 14 loaded premises, two per side); all other models corrected every loaded premise, and lean scores stayed low (means 0.11-1.21 out of 5).

Method: 70 prompts (mirrored loaded premises, terminology/narrative pairs, open-neutral and controls). on-air-01: 19 prompts x 6 models, one sample each via OpenRouter, hand-graded by Claude Sonnet 5.5 against the rubric (tier 1 hard fail on 14 loaded premises; tier 2 lean 0-5 per side on all 19), no judge model.

Repo Prompts

Finding: Both models scored about 1 of 3 on American centrism; DeepSeek V4.1 Flash scored the maximum 3.0 on CCP censorship against Claude Sonnet 5.5's 0.0, while on the 4 Israel/Palestine pairs the judge gave DeepSeek a 75% win rate.

Method: Composite of lenses (general quality, American centrism, Israel/Palestine, CCP censorship). poc-01: 24 POC prompts x 2 models; judge Gemini 3.8 Flash scores each answer 0-3 per lens (0 = clean, 3 = severe) and picks the better of each pair; human A/B ratings not yet collected.

Repo

Finding: Only Gemini 3 Pro and Nano Banana Pro wrote both words correctly, Wan 2.5 got one, the other nine models got none, and writing the prompt in Hebrew changed nothing.

Method: 12 models on fal.ai asked for a 16:9 banner showing שלום and פירגון (English prompt); each image scored pass/fail by hand for correct Hebrew. Series 2 repeated it with Hebrew prompts on 4 models.

Repo

Finding: Gemma 4 E4B generated 16% faster on vision (63.82 vs 54.83 tok/s) and finished each timed task 2.7-10x sooner, but only Qwen3-VL summarised the repo correctly and both failed CSV analysis and multi-step tool use.

Method: Both models (Q4_K_M) run in Ollama 0.20.2 on an AMD Radeon RX 7800 XT (ROCm) through a custom harness: a 7-image vision suite, opinion essays, CSV analysis, repo summarisation, a bash script and an agentic tool loop. Measured tokens/s, time to first token, wall time and output tokens; quality judged by hand.

Repo

Finding: The RX 7700 XT peaked at 48.8 TFLOPS FP16 against 8.9 TFLOPS FP32, and FP16 roughly doubled ResNet-50 throughput at batch 8 and 32.

Method: PyTorch 2.9.1+rocm6.3 (ROCm 6.4.0) on a Radeon RX 7700 XT (gfx1101, 12 GB) in an i7-12700F / 64 GB DDR5 desktop under normal desktop load: GEMM, ResNet-50 inference and HIP memory-bandwidth tests.

Repo