Part 1 covered frontier models, open-weight models, and data sovereignty. Part 2 looked at the AI coding agent market and the operational constraints behind the tools. This time, I want to look at what happens after you choose your models: how to assign them work, where routing helps, and what it takes to run the whole setup in an enterprise.
First on the Leaderboard. First Choice for Your Team?
Every new model release brings another round of benchmarks. Comparing reasoning ability, coding accuracy, context windows, and token prices can become a job in itself. Picking a model with a solid reputation and getting on with your work is often the more comfortable option.
For small tasks on your own, I think that’s perfectly reasonable. But as the work you delegate gets longer and more people on your team start using agents, how you run those models has a much bigger effect on the outcome.
Finding a function and diagnosing a flaw in an authentication system require different levels of reasoning. Even with the same model, context size, reasoning tokens, and tool calls all affect the cost and time needed to finish. A fixed token price doesn’t give you a fixed cost per task.
Most of this is hard to see from the user’s screen. The agent reads files, edits code, and appears to keep making progress. Excessive reasoning on a simple step, or repeated attempts to fix a problem it hasn’t understood, may only become obvious when the bill arrives.
That’s why I want to focus on workload fit and operational efficiency. What quality can a model deliver on a particular job, and how much inference spend and verification work does it take to get there?
Fine for One Developer. A Bigger Bet on an Enterprise Contract.
There is some additional context to the Cursor and OpenAI example from Part 2.
On August 28, 2026, OpenAI announced plans to end its model supply agreement following SpaceX’s acquisition of Cursor, proposing November 12 as the termination date. As of October 29, OpenAI’s help page still said the final date would be announced once both companies had confirmed it. What has been established is the decision to end the agreement and a proposed transition date.
There are still ways to use OpenAI models within Cursor. OpenAI lists options including bring your own key (BYOK), the Codex IDE extension, and compatible gateways. The scope matters: BYOK applies to supported local Chat and Agent features, but does not extend to features such as Cursor’s Auto or Cloud Agents. API usage is also billed separately.
This is a practical example of vendor dependency. Your IDE or tool may stay exactly the same, while a change in how models are supplied forces you to revisit authentication, billing, and feature availability. If procurement approvals and internal cost allocation depend on the existing arrangement, the transition gets more involved.
The dependency extends across the model provider, API gateway, contract, and agent runtime. Registering several models does little for resilience if they all depend on the same delivery path. And having a replacement model available won’t prevent rework if it can’t pick up the existing session and task state: the context.
I count that revalidation and rework as part of the real switching cost. The time it takes to change a model name tells you very little about how dependent you are on a vendor.
I call this way of measuring cost tokenomics. It includes token charges, rebuilding context, revalidation, rework, and the operational effort needed to switch models. The unit that matters is one completed, verified piece of work.
Would You Fire a Missile at an Ant?
Suppose you ask a coding agent to fix a bug that intermittently logs users out. That short request contains several jobs: explore the code, diagnose the cause, implement a fix, test it, and check for regressions.
| Stage | Work involved | What to look for in a model |
|---|---|---|
| Exploration | Find authentication modules, configuration, dependencies, and relevant tests | Search accuracy, reliable tool use, response latency |
| Diagnosis | Trace the relationship between session expiry, token refresh, and concurrent requests | Hypothesis testing, complex reasoning, effective use of long context |
| Implementation | Apply the fix within the existing architecture and coding conventions | Code correctness, control of scope, consistency |
| Validation | Run reproduction and regression tests, then investigate failures | Test design, interpretation of results, defect detection |
| Review | Assess the effect on permissions and adjacent features | Security review, edge cases, requirements coverage |
A lightweight model may be sufficient when the filename or search criteria are clear. A race condition that appears only under concurrent requests is a different problem. The model has to reason about multiple execution paths and their timing. Both tasks concern the same login feature, but they demand different capabilities.
Assigning a fixed model to each stage isn’t enough either. Exploration can be difficult when it involves reconstructing dependencies in an unfamiliar, large codebase. Implementation may be straightforward if most of the changes follow an established pattern. Task type, difficulty, and the consequences of failure all need to inform the assignment.
It also helps to separate execution from judgment. A formatter or an existing test command runs as ordinary software. The model’s job is to choose the right command and interpret the result. Wrapping every execution step in another LLM call can add cost and latency without improving the outcome.
Profiling the work, assigning a suitable model, and adjusting that assignment as results come in is what I mean by task-based model routing. In practice, it is a policy for deciding where to spend your reasoning budget.
Changing Models Takes More Than Flicking a Switch
Task difficulty is hard to predict at the start. An apparent bug can turn out to be an architectural problem. Something that looks complicated may come down to one configuration value. During a long agent run, routing needs to take account of what the agent learns along the way.
NVIDIA’s NeMo Switchyard supports approaches including request classification, routing by work stage, and escalation to a more capable model. Stage-based routing uses signals from tool execution and error patterns. Escalation starts with a cheaper model and moves up when the work continues to give it trouble.
For the login bug, you could assign initial exploration to a lightweight model, then escalate if the same test keeps failing or the agent repeats a fix without producing new evidence. Once the cause is established and the change is well defined, the work can move to a model suited to implementation.
The policy also needs stopping conditions: retry limits, a token budget for the task, timeouts, and conditions that require human judgment. If the real problem is a missing permission or a broken test environment, changing models will simply spend more money on the same obstacle.
Routing has its own overhead. A classifier can add another model call and extra latency. Switching models can mean paying to send the context again. If the price difference is small, or almost every request needs the most capable model anyway, the economics may not justify the additional system.
74% Cheaper? Check the Scorecard, Too.
In August 2026, LangChain published an evaluation using NeMo Switchyard. It compared Claude Opus 4.8, Nemotron 3.5 Lightning, and a routed combination of the two across 145 multistep tasks involving customer support, incident investigation, and workflow automation. The costs below are from that experiment.
| Configuration | Evaluation accuracy | Cost of one full evaluation run |
|---|---|---|
| Opus 4.8 alone | 86.0% | $11.45 |
| Routing between Opus 4.8 and Nemotron 3.5 Lightning | 80.0% | $3.00 |
| Nemotron 3.5 Lightning alone | 77.7% | $0.72 |
Routing cost about 74% less than using Opus alone, with accuracy six percentage points lower. Compared with the cheaper model alone, the accuracy improvement was smaller than the variation between runs. The researchers therefore did not establish an advantage for routing on that comparison.
This is a cost-quality tradeoff on a specific workload. It is not evidence of a 74% saving at the same quality, and there is no basis for carrying those figures over to a different coding workload.
For an adoption decision, I find this kind of result useful. It lets you weigh a possible reduction in quality against the money saved. Classifying internal documents and changing authorization logic in a payment system need different acceptance criteria. The cost of finding and fixing mistakes differs considerably, too.
My preferred measure is cost per validated task. Add up all inference spend, including retries, along with the cost of running tests, human review, and rework. Then divide by the number of tasks that meet the agreed completion criteria.
This includes the verification cost I discussed in Part 2. A lower token price can still produce a higher total cost of ownership if people spend more time correcting the output.
OpenCode and LiteLLM: How About Building Our Own Model Router?
OpenCode and LiteLLM are worth considering if your team wants to build its own setup. Both give you more freedom in model selection and help reduce dependence on a single provider. They serve different parts of the system.
OpenCode – Assign Models to Suit Your Team
OpenCode is a coding agent that connects to multiple model providers and local models. Its documentation covers more than 75 providers, model assignments for individual agents, custom prompts, and tool permissions. You can separate exploration from implementation and give each agent a model suited to its role.
That flexibility is useful for evaluation and tuning. You can change the model assigned to one role while keeping the rest of the workflow intact, and adjust permissions and behavior for a particular project. For a team with the engineering capacity to build on it, OpenCode offers a strong foundation.
LiteLLM – Several Models, One Place to Manage Access
LiteLLM provides an SDK and AI gateway for access to multiple model providers. Its Router supports load balancing across deployments and routing strategies based on factors such as usage, latency, and cost. You can configure retries and timeouts, and fall back to another model group when a call fails.
It also provides enterprise management features: virtual keys for model access control, spend tracking by user and team, role-based access control, and integrations for logging and guardrails. Teams can choose the features and edition that fit their requirements.
For example, an agent could send exploration and implementation requests to different model configurations, while a LiteLLM gateway manages access, provider selection, and usage. Common operational functions move into a shared layer, reducing the amount each application has to implement independently.
OpenCode and LiteLLM make it easier to get started with multiple models. Expanding that setup into a standard development platform for the business still involves substantial integration and ongoing operations work.
Installed. Now Someone Has to Run It.
An agent on one developer’s project and a platform shared across an organization have very different operating requirements. In an enterprise, the model API, agent harness, governance, monitoring, and security controls have to work together. If you deploy open-weight models yourself, model serving and GPU infrastructure become part of the job as well.
OpenCode and LiteLLM provide useful building blocks. LiteLLM already includes access controls, spend tracking, logging integrations, and security guardrails. The team adopting them still has to configure those features around company policy, integrate them with existing systems, and verify that they continue to work as intended after changes. Someone needs to own the following work.
| Area | Implementation and ongoing responsibilities |
|---|---|
| Harness and runtime | Define agent roles and tool permissions, isolate workspaces, recover sessions, and verify state consistency when models change |
| Governance | Define approved models and providers, map organizational permissions, allocate budgets, and manage exceptions and policy changes |
| Monitoring and observability | Connect individual requests to tasks, build cost and latency dashboards, investigate failures, and maintain alerting and response procedures |
| Security | Integrate identity systems, manage credential lifecycles, control data egress, handle sensitive content, and manage log retention and access |
| Reliability and change management | Design gateway availability, exercise failover, run regression evaluations after model or API changes, and handle upgrades and vulnerabilities |
| Open-weight model operations | Review models and licenses, manage weights and versions, configure serving engines, evaluate quantization, deploy and roll back models, and test concurrent request handling |
| GPU operations | Plan capacity and availability, maintain driver and runtime compatibility, monitor memory and utilization, scale capacity, control idle spend, and respond to failures |
Serving open-weight models means managing the interaction between model versions, serving engines, quantization, GPU memory, and request concurrency. Provision generously and you pay for idle capacity. Provision too little and requests queue up. Any reduction in model costs needs to be assessed alongside the infrastructure spend and operational work it introduces.
Tracking cost by model is also different from calculating the total cost of fixing one bug. To do the latter, exploration, implementation, retries, and review requests need to share a task identifier. That lets you distinguish a higher model price from poor task assignment or repeated failures.
Security spans several layers. A gateway can restrict calls to approved models, while the agent’s access to files and shell commands needs its own controls. Audit requirements may cover approvals, file changes, and tool execution as well as prompts. You also need to prevent the logs themselves from becoming a store of sensitive data.
The data sovereignty question from Part 1 applies here, too. If an internal model fails while processing data that cannot leave the organization, the fallback must stay within the approved data boundary. A policy that holds during normal operation needs to hold during failover as well.
OpenCode and LiteLLM are good tools. Building and maintaining the harness, governance, monitoring, and security setup an enterprise requires still takes substantial engineering effort. How much depends on the maturity of the existing platform, regulatory requirements, available staff, and the commercial support you choose.
Building in-house can be a sensible decision when you have the expertise and clear requirements. The cost estimate needs to cover the years of operation that follow the initial installation.
Models Need Handoffs, Too.
One difficult part of using several models is keeping the work coherent when responsibility moves from one to another. The next model needs an accurate understanding of both the previous analysis and the current state of the files.
For the login bug, a summary saying “fixing an authentication issue” is nowhere near enough. The handoff should explain how the bug was reproduced, which findings are confirmed, which hypotheses remain untested, what code has changed, and which tests have passed. It should also preserve the reasoning behind rejected approaches so the next model doesn’t repeat them.
Task state includes more than conversation history. It includes the code version, changes made, test results, runtime environment, and approved scope of work. If the summary says a fix has been applied but the files say otherwise, the next model starts from a false premise.
Anthropic describes compaction, structured notes, and subagents as ways to manage context during long agent runs. It also notes that aggressive compaction can discard important information.
Compatibility between providers introduces another constraint. LiteLLM’s fallback documentation describes cases where a different provider or API key cannot decrypt reasoning data from a previous response. That information may need to be removed, with readable summaries retained where available.
Important decisions and verification results therefore need a durable record that another model, or a person, can understand. Agent roles, context management, and workspace state all need to be designed together.
Kimchi: Models Form Teams, and Teams Divide the Work
Kimchi, from CAST AI, where I work, combines multi-model orchestration with the operational features needed for enterprise AI coding. Its scope includes the harness, governance, monitoring, and security discussed above, along with open-weight model serving and GPU resource management.
Coding work is divided into exploration, planning, implementation, and review. An orchestrator manages progress, while the models assigned to each role handle their specialist work. The models operate much like a development team.
| Role | Responsibility |
|---|---|
| Orchestrator | Manage the execution flow, classify tasks, and delegate work |
| Explorer | Explore and investigate the codebase |
| Planner | Design the approach and plan the work |
| Builder | Implement code using models suited to the difficulty of the task |
| Reviewer | Review the implementation and check correctness |
You can configure model pools for each role, and subagents use models from those pools. Ferment, Kimchi’s project execution mode, breaks a goal into phases and individual steps. Independent phases can run in parallel across sessions. The harness records decisions and progress, evaluates results, and manages the handoffs needed to keep a longer project moving.
For enterprise adoption, the strength I want to emphasize is having execution, budget controls, access management, observability, and model infrastructure connected in one platform. The areas a team would otherwise have to integrate separately map to the following capabilities.
| Area | What Kimchi provides |
|---|---|
| Harness and execution | Model assignment by role, phased execution and evaluation, session persistence and recovery, and remote workspaces |
| Governance and budgets | Role-based access control, budgets by user, team, and API key, organization-wide limits, and policies for approved models and tools |
| Monitoring and observability | Request traces and reports covering models, tokens, latency, tags, and cost; usage by developer and team; cost attribution by work phase |
| Security and data sovereignty | SSO/SAML/OIDC integration, credential isolation, encryption in transit and at rest, deployment in the customer’s VPC, and policies for external model calls |
| Open-weight model operations | Kimchi-managed inference APIs, model deployment into customer Kubernetes environments, and access through compatible APIs |
| GPU capacity and cost | GPU infrastructure managed by Kimchi in Serverless mode, plus replica autoscaling and idle hibernation for supported Self-Hosted deployments |
| Operational history and change tracking | Model deployment status, scaling and hibernation events, and audit records for remote workspace and template changes |
Budget limits are enforced in the execution path. Teams can allocate spend and track it by project API key or user. Requests carry exploration, planning, implementation, and review phase information, so a cost increase can be traced back to the work that caused it. Engineering, platform, and finance teams can make decisions using the same usage data.
The connection to model serving and GPU operations is another practical advantage. In Serverless mode, Kimchi runs the models and GPU infrastructure, and developers use the inference API. Organizations that need direct control over where data is processed can deploy models into their own Kubernetes environment with Self-Hosted. Replica autoscaling for vLLM-based deployments and idle hibernation help match capacity to demand. The customer retains control of identity, networking, and the underlying nodes.
A team already using OpenCode can also connect a compatible endpoint to Kimchi and configure organizational policies for shared model access and cost management. That offers a way to introduce a common operational layer while retaining the existing development tool.
For me, this integration is Kimchi’s most practical strength. It connects model assignment and execution with usage measurement, budget enforcement, and the work of running open-weight models on GPUs. That can reduce how much infrastructure and integration code an organization has to build and maintain. From a tokenomics perspective, it gives you a way to assess model spend and platform operating effort together.
Great Demo. How Does It Handle Our Code?
If I were evaluating a multi-model setup, I would start with the team’s current single-model configuration as the baseline. I would build an evaluation set from recurring tasks, then compare the configurations using the same code version and completion criteria.
There is no need to distribute every task from day one. Start with work that has a clear scope and is easy to verify, then expand where the results justify it. A cheaper configuration that misses the quality threshold should have a limited role.
| Measure | What to include |
|---|---|
| Task completion rate | Whether the output meets the requirements and verification criteria agreed in advance |
| Cost per validated task | Inference, retries, test execution, human review, and rework |
| Time to completion | Model responses, tool execution, and time spent waiting for people |
| Latency distribution | The average and the slower end of the distribution, including p95 |
| Operational effort | Staffing needed for policy changes, upgrades, incident response, and model replacement |
If one primary model meets your quality and cost targets, keeping that setup is reasonable. As usage grows and the work becomes more varied, specialization may become valuable enough to justify investment in routing and shared operations.
What matters to me after model selection is having a reliable way to assign work and measure the results. With that in place, a new model can be evaluated against the existing task set and introduced first in the roles where it proves useful.
Next: It’s 6 p.m. Close the Laptop. Let the Agent Keep Working.
This article covered model assignment and the harness, governance, and observability needed to operate it. Next, I’ll look at what happens when the work moves to another environment.
Part 4 will cover Kimchi’s Teleport and Remote Workspaces: starting a job locally, handing it off to a remote workspace, and letting the agent continue after you close the laptop. I’ll look at how the session, code, and runtime carry over, and which settings and credentials need to be prepared separately.
답글 남기기