The Stopwatch and the Curator

There is no shortage of places to go compare language models. LMSYS Chatbot Arena has been running head-to-head human preference votes for years; the Hugging Face Open LLM Leaderboard will hand you a table of scores across a dozen standard tests before you’ve finished your coffee. We’re not going to reproduce either one here — they already exist, they’re good at what they do, and anyone can go read them.

What they can’t tell you is whether a specific model, on a specific machine, running a specific agent’s actual workflow, holds up past the first exchange. That’s a narrower, more boring question, and it’s the one we actually needed answered. So we built a small internal harness to answer it, and we’re going to walk through what it found — including the parts that aren’t finished yet.

The part a leaderboard can’t see

A general benchmark tells you a model is good at math, or good at code, or good at following instructions in one shot. It doesn’t tell you what happens on turn two of a real conversation, once the model has to remember what it said on turn one and build on it instead of restating it. That’s a specific, common failure mode — answer the first question well, quietly lose the thread on the follow-up — and a one-shot prompt set can’t catch it because it never asks a follow-up.

Our scoring tier borrows the MT-Bench methodology for exactly this reason: 80 questions across 8 categories, two turns each, graded on a 1–10 scale by a judge model. The second turn is always a follow-up that depends on the model tracking its own first answer. We report this separately from raw speed, on purpose — a model that answers fast and wrong isn’t faster, it’s wrong faster.

One honest caveat that has to travel with this number wherever it goes: judging is only as independent as the judge. Point a model at grading its own family’s answers and it tends to grade generously. The result is worth trusting in proportion to how independent the judge actually was of the model being judged — we say so explicitly whenever we quote a score, rather than let the number imply more rigor than the setup actually had.

What we measured, plainly

On the lab machine — an Intel Core Ultra 7, running everything locally, nothing in a cloud console — here’s what a clean run actually reported:

llm   (int4 quantized, local inference server, integrated GPU)
      time-to-first-token: 383 ms       decode: 15.2 tokens/sec

stt   (whisper.cpp, CPU)
      0.126 real-time factor  ->  about 8x faster than real time
      word-error-rate: 0.0 on our fixed test corpus

tts   (Kokoro, CPU, warm)
      0.23 real-time factor   time-to-first-byte: 0.84 s

Two things worth being upfront about in that table. First, “0.0 word-error-rate” means faithful transcription against a known, fixed test phrase set — not a claim about accuracy on arbitrary speech. Second, that TTS number is the warm number. The very first time the speech engine synthesizes a given phrase after starting up, it’s five to ten times slower — two to seven seconds instead of under one — and every phrase after that is fast for the life of the process. If we ever quote a number without saying which one it is, that’s a mistake, not a rounding choice.

The honest summary of where the harness itself stands: the per-service numbers above are real and repeatable — we’ve run them enough times to see how little they move between runs. What we haven’t done yet is launch an actual agent and time a real multi-turn conversation end to end. That tier is designed, not built. We’re not going to claim it’s further along than it is.

Enter Sunny

The agent we care about testing against isn’t hypothetical. Sunny is a news-curation agent that runs on a daily schedule, reads what’s actually new since the last time it looked, and produces a short spoken-format digest — the kind of thing you’d want read to you over coffee, not a wall of headlines to scroll.

The interesting design choice is what Sunny does before it writes anything. Nothing about the day’s news exists in its database until the agent itself goes looking. It pulls feeds cheaply — just titles and links, no article bodies — sorts that stack against what its user actually cares about, and only pays the cost of reading the handful of stories that clear that bar. The filtering is the product, not a preprocessing step bolted in front of it.

On August 14th, we ran Sunny end to end against a real model for the first time, and the results are worth reporting exactly as they happened.

It worked. A digest was published. Asking it to publish a second time correctly refused — “already published and cannot be changed” — which matters more than it sounds: an agent that can silently overwrite its own prior output isn’t trustworthy in the way a news source needs to be.

We found two real bugs, live, and fixed them. One was a documentation mismatch that would have silently produced an empty digest for anyone who trusted the written instructions over the actual code — no error, just nothing where content should be, the worst kind of bug because nothing tells you it happened. The other turned a bad internal reference into a raw database error with no useful explanation; we replaced it with a message that names exactly what was wrong and which tool would have returned the right value.

We found one thing that isn’t fixed, and we’re saying so. The model we tested against twice filed a note about a story onto the wrong internal thread, and duplicated a near-identical note instead of recognizing the work was already done. That’s not a bug in Sunny’s code — there’s no way for the system to tell “the wrong thread” from “a valid one” without the same kind of judgment that caused the mistake in the first place. It’s a real limitation of the specific model we tested, on the specific day we tested it, and the honest move is to say so plainly rather than quietly retry until it looks clean. We’ll retest against a different model and report what actually happens, not what we expect to happen.

We’ve proved the stopwatch works. We haven’t proved it agrees with the clock — and Sunny’s first real run is exactly the kind of test that starts closing that gap: not a mock, not a fixture, a real digest, published, with real mistakes attached to it.

Why this is the harder, better habit

It would be easier to write “we chose model X because it topped leaderboard Y.” It would also tell you almost nothing about whether it holds up in the one place that actually matters here: running Sunny’s real workflow, on our own hardware, past the first turn. The public leaderboards are a fine place to start narrowing the field. They’re not where the decision gets made. That happens here, against our own agents, with our own numbers, reported the way they came out — bugs, open questions, and all.