Exploring open models and model-routing.

A few weeks ago, “use an open model in production” was something I’d nod at in meetings and file under “future optimization.” The frontier models (the top closed models from labs like Anthropic and OpenAI) are better, the gap is wide enough to matter, and there wasn’t enough pressure to make the exploration worth the effort. As with most things these days, that changed quickly. The open models have gotten better, sure, but the reasons we started paying attention were more boring and practical.

Two things changed specifically: the price of running agents at scale, and access to the top models (which we can no longer take for granted). So we tested an open-weight model, GLM 5.2, inside our own code-review agent on real pull requests. (Open-weight means the model’s parameters are published: once you’ve downloaded them, no vendor can revoke your access.) As always, we kept the test to internal tools, not client engagements.

It’s early, but the value is showing up in the framework we built around it more than in the open model itself: letting the agent choose the model per task instead of hardcoding one or asking a human to decide. This is where we’re taking all our workflows.

01 · The contextWhat changed.

Pricing pressure rose across the industry in the first half of the year as agents and automation proliferated. Running one agent on one task is cheap. Running agents across many workflows and teams, 24/7, is not. Companies started capping usage, redirecting users to cheaper models, or both.

Then the US government restricted access to frontier models. A new federal framework gives the government up to 30 days to review frontier models before release. An export-control move cut off Anthropic’s newest models (Mythos 5 and Fable 5) for non-US nationals, forcing Anthropic to turn off its top model (Mythos 5) and the more restricted public version (Fable 5). Two weeks later, at the government’s request, OpenAI limited the release of GPT 5.6 to a small group of government-vetted US partners. It hasn’t reached the public. Around the same time, a Chinese lab, Z.ai, shipped an open-source model, GLM 5.2, that scored well on benchmarks and nearly matches Opus 4.6, the top model just months ago. The open-model discussion was everywhere, on X, Reddit, and Hacker News:

From the threads · X, Reddit, Hacker News

The discussion turned to whether relying entirely on frontier models was smart business.

If the best closed models can be switched off for political reasons, an open model you can hold onto starts to look like business insurance.

Put those together and the question stops being “are open models as good as the frontier?” The answer is still no, not at the top. The better question: for this change, is an open model good enough?

02 · The setupWhere we ran it, and why.

A bit of plumbing first. The routing only makes sense once you see it.

We run our agents inside Olympus, our agent control plane, where we define each agent, assign it a model, enforce human-in-the-loop checks, and log everything it does. Two agents matter here: Vulcan, a coding agent that picks up tasks from Habitat (our workflow app) and opens pull requests, and Argus, a code-review agent that reads a proposed change and pushes back the way a senior engineer would: this function does something different from what its name says, this query will fall over at scale, you dropped the null check, this is the third pattern we use for the same thing.

Argus is where we introduced GLM 5.2, and we picked code review on purpose. The task is bounded: Argus isn’t running off for an hour deciding what to build. It looks at a diff, has its say, and stops. A smaller, cheaper model has an easier time being reliable on a job with defined edges. We could also tell good work from bad. Code review passes through a quality gate that predates the model swap: the agent’s pushback, then human eyes, then evidence tracked against the change in Habitat. Reviews land back in that flow with humans in the loop, so swapping the model wasn’t a leap of faith. We could watch the new reviewer’s comments next to the old ones on real changes and see whether the feedback got worse.

That’s the part I’d underline for anyone tempted to copy this. The win wasn’t GLM. It was that we’d built the thing that let us trust GLM, or not, on the evidence.

03 · The mechanismWe don’t switch the model, we route.

Here’s the mechanism, because it’s the part worth stealing.

Every PR in our workflow carries metadata, scored on two axes: complexity and risk. A typo fix in a help string is low and low. A change to payments or auth is not. That score was already useful for deciding how much human attention a change deserved. It was also the right signal for choosing a model.

So Argus doesn’t run on one model. It routes:

for each pull request:
    read the change’s complexity and risk score

    if complexity is low and risk is low:
        review with the open model   # currently GLM 5.2
    else:                             # risk is moderate to high
        review with a frontier model

    always: human stays in the review loop

The open model only handles changes where a miss is cheap and easy to catch, which is most of them. The frontier model handles the changes where subtle, cross-cutting judgment earns its price. We’re not betting the codebase on an open model. We spend the frontier model where it pays and the open one where it doesn’t need to.

We’ll give Argus access to more models and more ownership over selection, including sub-agents with unique review jobs, each running on the right model, with Argus aggregating and reconciling results.

I expect this pattern to spread across our agents and workflows. Once you have a reliable read on how risky a unit of work is, “which model should handle this” stops being a decision you make once for the whole org and becomes a routing call you make per task. Vulcan, the doc agents, the rest of the bench: each has cheap work and expensive work mixed, and most of it is cheap.

04 · The resultsWhat held up, and what didn’t.

On the bread-and-butter of review, the open model was close enough that I couldn’t pick the closed one out of a lineup. Missing error handling, naming that lies about behavior, scaling traps, repeated patterns that should be consolidated: it caught these at a rate I was comfortable with on low-risk changes.

Where the frontier model pulled ahead was subtle, cross-cutting judgment: the catch that requires holding three files in your head and noticing the change breaks an assumption two modules away. The open model was more literal. It reviewed the diff well and reasoned across the codebase less. Which is why the routing exists: that weakness lines up with the high-risk, high-complexity changes, and those never reach the open model.

The real picture is narrower and more useful than “open models won”: on bounded, low-risk work with a gate around it, the open model did the job for a fraction of the cost, and we kept the frontier model and the human reviewer for the changes that earn them.

05 · The tradeoffThe cost of switching.

Putting an open model into rotation wasn’t free. The cost wasn’t the model usage. A self-hosted open model has no per-token API bill. It was everything around it.

Prompts tuned for one model don’t transfer cleanly. The review instructions we’d refined over months assumed a certain reasoning style, and the new model read some differently. We re-tuned.

Self-hosting an open model is its own project: somebody keeps it running, keeps it fast under load, and pins it to a known version, the operational reality a closed API was handling for us. We traded a vendor relationship for an ops responsibility. That’s a real trade, and you should price it in before assuming open means cheaper.

If you take one habit from this, take that one. The lesson of the export-control news has nothing to do with Chinese versus American models. It’s that any model you depend on can become unavailable, get more expensive, or get deprecated out from under you, and the teams that can move are the ones never tied to a specific vendor or model.

06 · ClosingThe calculation changed.

I won’t tell you open models have caught up, because at the top they haven’t, and the hype-merchant versions of this article that say “everything has changed” are lying to you. What changed is the calculation. Price and access pushed open-weight models from a someday into a this-week decision, and the right answer wasn’t “switch” but “route”: send each piece of work to the cheapest model that can do it well, and prove which one that is before you trust it.

If you want to try it, don’t start by picking a model. Start by picking a job you can already grade, get an honest read on how risky each unit of work is, and let that decide. The model choice gets easy once you can see what you’re choosing between.