What is Hotdog Benchmark?
Hotdog Benchmark asks the largest AI models the same silly question every week: is a hot dog a sandwich? Also a hamburger. Also a taco. One word. Three times each. Then it publishes exactly what they said, how long they took, and what it cost.
The question is a joke. The plumbing is not. It is a small, complete, cross-provider LLM benchmark you can fork. Change questions.json and it is your eval.
Why We Built It
A real cross-provider benchmark is mostly unglamorous work. A different API per vendor. Token counts that do not mean the same thing. Latency that depends on where the runner sits. Models that answer differently on Tuesday. Someone down mid-run.
Leaderboards hide all of that and then imply a winner. We wanted the smallest honest version: adapters, retries, versioned weekly data, a methodology page that says what the numbers do not mean, and a site that shows its work.
The interesting number is not yes or no. There is no right answer. It is how far the answer moves when you tell the model what to think.
A Hands-On Way to Watch Models Disagree
This is not a slide about evals.
Every Monday the run goes out. Control (asked plainly), asserted (“a hot dog is a sandwich”), denied (“a hot dog is not a sandwich”). Eleven models, three samples, majority verdicts, real wall-clock, verbatim words. Hatched teal is thinking. The rest is answering.
You can read the report, the history, and the methodology. The code, the raw JSON, and the classifier lists are on GitHub. No vendor is asked, sponsors this, or sees it first.
Built Around One Question, Then the Pipeline
Two layers.
The show is the sandwich questions. Hot dog is the one that started it. Hamburger is the control nobody fights about. Taco is the boundary.
The product is the pipeline: questions and models as data, one adapter per vendor, bounded concurrency, versioned weekly editions, a scheduled Action, a site built from the files. About two thousand lines. No framework. npm run bench -- run --mock runs the whole thing with no API key.
Useful for Engineers Who Have to Evaluate Models for Real
This is for people who are about to stand up a cross-provider eval and have not yet discovered that “just call the APIs” is the easy sentence.
If you need a quality ranking, this will not give you one, on purpose. If you need a pattern for adapters, usage normalization, sampling, framing sensitivity, and publishing numbers you can defend, start here and swap the questions.
Built to Make the Idea Click
Some eval ideas fit in a spreadsheet. This one needs a question dumb enough that nobody pretends the score is IQ.
Hotdog Benchmark exists so more people can see how a benchmark is actually built, what moves when you change the prompt, and how little a single-word answer tells you about quality.