We decreased our LLM costs with Opus
Article URL: https://www.mendral.com/blog/frontier-model-lower-costs Comments URL: https://news.ycombinator.com/item?id=47942903 Points: 1 # Comments: 0
Article URL: https://www.mendral.com/blog/frontier-model-lower-costs
Comments URL: https://news.ycombinator.com/item?id=47942903
Last week we wrote about feeding terabytes of CI logs to an LLM . Most of the questions on Hacker News weren't about the logs. They were about the agent: which models, how they coordinate, and how much it all costs.
Today we run Opus 4.6 and pay less than when we ran everything on Sonnet 4.0.
The reason is mostly what Opus doesn't do: 80% of failures never reach it, and when they do, it never reads a log line.
Let a cheap agent decide if the expensive one is needed
Last week we analyzed around 4,000 CI failures. 818 were new problems. The other 3,187 were a known issue surfacing again: a flaky test, an infrastructure hiccup, a network blip we'd already detected.
It makes no sense to wake up an expensive model when 80% of the time the answer is "it's a duplicate". Unfortunately, we can't deterministically detect duplicates: the same job can fail multiple times for completely different reasons, so you need to actually look at the logs to know if you've seen this before.