Career Hacker Alex
Blog

2026-10-23

I tested an AI that only judges — 75x cheaper, 16x faster, and 3 points short

OpenAI's Decisions API became generally available on October 22. I tested whether AutoKliq's thumbnail judge could move to it, on 5,740 public video pairs. It was 75x cheaper and 16x faster, and 3 points less accurate, so the judge stays where it is.

On October 22, 2026, OpenAI made its Decisions API generally available. It is a model that only judges. It reads images and text and answers closed questions: it picks one option from a list, or gives the probability that a statement is true. Output is free; input costs $0.10 per million tokens.

I had been waiting for it. AutoKliq, the service I build, has a judge that treats a thumbnail and its title as one package and picks which of two packages will earn more watch time. Claude Opus runs it today: $0.025 to judge a pair, about four seconds a call. A cheaper, faster judge would let the same budget compare far more candidates.

Three thumbnail-and-title sets for the same video, each showing Alex's face and a large headline about AI-made videos
The kind of packages the judge compares: three sets for a video I published on October 21, now competing for watch-time share in a YouTube Test & Compare.

On launch day I asked Claude Code to check three things against our current judge: whether Decisions could reach a similar level, how much cheaper it would be, and whether it could be faster and even better. I did not hide what I hoped. If it worked, it would be a big deal: far cheaper and far faster.

The short answer: the judge isn't moving. The Decisions API was 75x cheaper and 16x faster, and 3 points short on accuracy.

A judge is only compared on the same exam

I measure the judge on public videos. I collected 58,602 of them from 158 Korean and 205 English channels, then kept pairs of uploads from the same channel, at most 30 days apart, whose performance against the channel's usual differs by 1.4x or more. That gives 5,740 pairs. The judge sees only each video's thumbnail and title and has to say which one did better.

Because the two videos differ, topic and timing are mixed in as well. So the number that matters is not one score but the gap between two judges on the same pairs. Opus, the current judge, gets 59.7% of the 4,187 holdout pairs right. A coin flip gets 50%.

The Decisions API received exactly the instructions and images Opus receives. Only the answer format changed.

Six ways of asking the same question

Decisions answers in three formats. A choice picks one option and gives a probability for each. A predicate gives the probability that a statement is true. A score says which of several defined levels applies.

I built six designs from those formats:

  • Choice — show both packages and ask which one earns more watch time.
  • Predicate — the same view, asking how likely "package 1 earns more watch time" is to be true.
  • One overall score — show one package at a time and rate it on five levels against the channel's typical video.
  • Seven criteria scores — the overall score plus six criteria, combined: readable at a glance, curiosity, a concrete payoff, emotion and stakes, a title and thumbnail that complement each other, and fit with what the channel's audience comes for.
  • Two side-by-side layouts — the choice and predicate questions with each thumbnail placed right next to its number and title.

The rules came before the results

The experiment order: six designs, pick one on 1,553 dev pairs, test it once on 4,187 holdout pairs, move only if within 2 points of Opus
The order was written down before the first judgment: the six designs compete on dev pairs only, and only the winner is tested on the holdout.

Try enough designs and one of them will look good by chance, so the order was fixed first. The six designs competed on 1,553 dev pairs only. The best one got a single test on the 4,187 holdout pairs. If it could still be more than 2 points behind Opus, the judge would not move. Precisely: if the lower end of the 95% interval of the gap to Opus fell below −2 points.

The rule earned its keep. On dev, the side-by-side layout looked 1.9 points better than the other designs. On the holdout, that lead was gone. Picking on dev numbers alone would have meant deciding on an improvement that did not exist.

75x cheaper, 16x faster

Cost and speed. Judging one pair costs $0.0254 with Opus and $0.00034 with Decisions, 75x cheaper. A call takes 4.0 s with Opus and 0.25 s with Decisions, 16x faster
Cost comes from the input tokens of the actual calls. Speed: the same 20 pairs in both orders, 40 calls per judge, one at a time.

Judging one pair cost $0.0254 with Opus and $0.00034 with Decisions: 75x less. A call took a median of 4.0 seconds with Opus and 0.25 seconds with Decisions. The whole experiment called Decisions about 32,000 times for $5.90. Running the holdout once through Opus costs more than $100.

It also gave the same answer to the same question every time: 100 repeated calls returned identical probabilities. And the score format was honest about its own confidence. The probabilities from the seven criteria scores were off from the actual hit rate by 0.8 points on average; Opus was off by 3.8.

And 3 points short

Accuracy on 4,187 holdout pairs. Opus scores 59.7%; the six Decisions designs score 56.4% to 57.0%, all below the 57.7% line two points under Opus
Accuracy on the 4,187 holdout pairs. The dashed line sits 2 points under Opus; the actual rule used each design's 95% interval of the gap to Opus.

All six designs landed between 56.4% and 57.0%. Opus scored 59.7%. Both pre-registered tests failed. The predicate design was 2.8 points behind Opus (95% interval −4.4 to −1.2), and the side-by-side predicate was 3.0 behind (−4.6 to −1.4). Changing the question did not close the gap. The gap is the model.

Three points sound small, but not for a judge. Opus beats a coin flip by 9.7 points; Decisions by about 7. Decisions delivers roughly 70% of the judging skill at one seventy-fifth of the price.

Swap the order and the answer flips

Its biggest weakness was order. Asked to choose between two packages, Decisions picked the first one 69% of the time. Asked whether "package 1 is better" is true, it said yes only 31% of the time. The two formats leaned in opposite directions. Swapping only the order of the same two packages flipped the answer in 42% of pairs. For Opus it was 14%.

Placing each number and title right next to its thumbnail did not fix it. I also tried a mix on dev: Decisions where it was confident, Opus for the rest. Even on the pairs Decisions was surest about, Opus did as well or better, so no mix beat Opus alone.

I wanted a check on same-video A/B results too. Of the Test & Compare results uploaded to AutoKliq, only 7 tests, 14 pairs, had a title recorded for every variant. That is too few to say anything.

Keep the verdict with Opus; hand the screening to Decisions

The judge stays on Opus. When a creator acts on the verdict, 3 points matter.

That does not make Decisions useless. A 0.25-second answer at one seventy-fifth of the price opens other jobs: screening dozens of candidates so that Opus only judges the last few, scoring every upload's package as an input to performance forecasts, and reacting instantly while a thumbnail is being edited. Each of these is outside this experiment and gets its own test.

This experiment cost $6.65. At that price, every new model can sit the same exam. Cheap, fast models will keep coming. What lasts is not the model. It is the exam that tests each new one with the same questions.

Sources