Model · agent context management
Agent Task Break Detector: spots a new task in an AI agent session
On every user turn, scores whether the new message needs the conversation so far or is a new task a fresh session could handle. Runs locally.
About the model
A cross-encoder that reads the new user message together with the last four exchanges and outputs p(new_task). A harness hook can then start a fresh session automatically when an unrelated task is dropped into a long one, so the agent does not carry stale context into it.
It is a 150M-parameter encoder rather than an API call because it runs on every turn. The New-Task-Split hook for Claude Code uses it, with the recommended threshold shipped in new_task_split.json.
Trained only on synthetic sessions so far; it has not been evaluated on real, hand-labelled ones.
Which file should I download?
ONNX int8 151 MB
CPU inference. Use threshold 0.76: int8 probabilities sit about 0.14 lower than fp32.
Run with ONNX Runtime Download ONNX int8 →safetensors (Transformers) 598 MB
fp32 weights for AutoModelForSequenceClassification. Input format and code on the model card.
Run with Hugging Face Transformers Download safetensors (Transformers) →Results
v0.1, run atbd-r3-modernbert-full3ep, evaluated 24 October 2026. Synthetic test split: 608 rows, 115 matched pairs (the same message once as a new task and once as a continuation, so only the context separates them). Higher is better.
| Test set | TF-IDF, message + context | This model (fp32) |
|---|---|---|
| Accuracy | 0.760 | 0.951 |
| AUC-PR | 0.781 | 0.986 |
| new_task recall at precision 0.95 | 0.140 | 0.902 |
| Matched-pair accuracy | — | 0.791 |
- int8 ONNX on the test split: accuracy 0.929, AUC-PR 0.982, recall 0.888 at precision 0.95. At the validation threshold of 0.751, test precision is 0.941 and recall 0.893.
- Weakest cases: a message that looks self-contained but continues the work (0.896), and a different task in the same workspace (0.867).
- Session load takes about 1.2 s on CPU; per-call latency is not measured yet.
Training
- Fine-tune of answerdotai/ModernBERT-base on (message, context) pairs, context = last 4 exchanges newest first, max 512 tokens. 3 epochs, learning rate 5e-5, batch 16, bf16, best epoch by validation AUC-PR. One NVIDIA L4 on Modal, 6.9 minutes.
- Data: danielrosehill/Agent-Task-Break-Detector v0.2, the rows where generator and verifier agreed: 5,104 training rows, 1,025 of them matched pairs. Generated by Claude Haiku 5.5, verified by Claude Sonnet 5.5.
Training data
Checkpoints and roadmap
Where the model is now and the checkpoints still to come. Planned stages are the current plan, not promises.
- v0.1 Done 24 October 2026
First release, synthetic data only. int8 ONNX with a calibrated threshold.
- Real-session evaluation Planned
Score it on hand-labelled real agent sessions.
Limitations
- Synthetic data from one generator. Real sessions are longer and messier. English only.
- The training data is about 50/50, while most real turns are continuations. Run it at high precision on new_task: a wrong split throws away the agent's context, a missed one only costs extra context.