Earlier this year, we started seeing more and more devs using coding agents to build with Together AI.
It’s not hard to see why: It might take days or even weeks for a developer to wire up an app for production inference by hand, especially if they’re new to the platform. But for an agent using our SDK and CLI, it only takes a few minutes—at least in theory.
In practice, we had no idea whether the agents pointed at our platform were able to work with Together efficiently, or if they were hitting obstacles, going down rabbit holes or failing outright, and wasting tokens in the process.
As devs delegate more of their work to agents, developer experience (DX) is increasingly becoming synonymous with agent experience (AX). The easier your software is for agents to use, the easier it is for devs to build it into their own products and workflows.
GitHub is a textbook example of great AX: activity on the platform has tripled in the past year, with agents opening about 40% of all PRs in 2026 (and, we’re willing to bet, writing a lot more than 40% of the lines of code across all commits). This is thanks to GitHub’s phenomenal API, CLI, and docs, all of which make it exceptionally easy for agents to use. (This writer was not even aware GitHub had a CLI until agents started using it for him, and now he probably uses it 1000+ times a day).
To improve our AX at Together AI, we needed a way to measure how agent-ready our platform actually was. In this post, we’ll walk through how we run agent evals, two problems they uncovered and how we fixed them, and what we’ve learned.
How we run agent evals at Together
Building the task list
Before we could start evaluating, we needed to know what agents were actually being asked to do … so we could, y’know, figure out if they could do those things. We started building a list of common tasks that developers tackle on Together, like running a batch inference job, building a voice agent, or managing autoscaling for an inference endpoint.
By task, we really just mean a prompt, something you’d tell an agent to do—for example:
List the available GPU cluster regions and instance types, print the cheapest option, and show the exact command you would run to reserve it on demand for one hour. Do not actually reserve it.
To pass the eval, the agent has to:
- Find the latest pricing info.
- Call the regions API endpoint to list the available regions.
- Combine the two to find the cheapest option.
- Produce a valid CLI command for creating the cluster.
The prompt asks for both availability and pricing, without mentioning that these details live in two different places. This is realistic, as the average user probably doesn’t know up front where each piece of information lives, which could easily trip up both humans and agents.
Three configurations
After building our list of tasks, we run each one of them three ways:
- Baseline: The agent receives just the prompt, with no special configuration, mirroring the average developer's workflow. It can use all its default tools, like fetching docs from the web, but we don't give it any specific MCP connectors or skills.
- MCP: The agent gets direct access to our documentation through our MCP server, so it knows exactly where to search when it hits a roadblock.
- Skills: We give the agent the Together skill package, which contains instructions and examples for using our services, written specifically for agents.
The run matrix
Each full evaluation runs every task against each agent configuration using three models, with two attempts at each task—so six attempts per task for each config, as shown here for our run on September 30th:

Automating the runs
When we started building our evals, we ran everything by hand: open a new terminal, paste a prompt into Claude Code, let the agent do its thing (or fail to do its thing), and repeat for each configuration. It was manual and tedious, so once we figured out the shape of the system we wanted, we started automating it.
Now we run each agent in a Together sandbox so they don’t interfere with our local environment or each other. GitHub Actions handles setup, starts the agent, cleans up afterward, and saves the results, while a proxy handles the agent's requests to the Together API and records the activity, so we can see which endpoints the agent calls and what comes back.
Scoring a run
After a run finishes, an LLM judge reviews the task prompt and the recorded session, then assigns one of three verdicts:
- Pass: Every task requirement was met.
- Incomplete: The agent attempted the task, but some required steps were unfinished or unverified.
- Fail: The agent gave up early, broke an explicit constraint, or produced an incorrect result (RIP).
Failed and incomplete verdicts don’t necessarily tell us whether the problem was with the agent, the API, or missing evidence. They just tell us where to start looking.
We investigate passing runs too. An agent can often work around confusing responses and still finish, and those recoveries show us what to fix to improve token efficiency.
To make that review practical, we built a results viewer that compares how different models and configs performed. For each run, it shows the prompt, the tools the agent used, the responses it got back, and any API activity:

Case study 1: GPU cluster pricing
Remember the GPU cluster task from earlier?
List the available GPU cluster regions and instance types, print the cheapest option, and show the exact command you would run to reserve it on demand for one hour. Do not actually reserve it.
Seems pretty simple, but when we ran our evals early last summer, Sonnet 4.6 passed only 4 of 15 attempts at this task, even with skills enabled.
Our post-mortem revealed the same pattern in the failed runs: Sonnet treated the sample output in our CLI reference, which listed a single region and two GPU types, as if it were the live inventory. It also skipped the regions lookup and named what it thought was the cheapest option without ever checking the pricing page.
The problems were on our end. First, the sample output wasn’t properly labeled, so nothing told the agent to run the command to retrieve the relevant data. Second, our skill file linked to the GPU clusters marketing page rather than the pricing table, so Sonnet opened the marketing page, didn’t see any real prices, and just made a best-effort guess at the cheapest GPU available.
More capable models were able to work around the issue. In that same period, Opus 4.8 passed 13 of 14 attempts using the same config. According to the transcript, it noticed that the list_regions response didn't include prices, so it kept searching until it found them in our docs. This is why it’s crucial to test your product with less capable models; if we had only tested with Opus, it would have taken us much longer to find the gap.
Isolating the failure
To fix it, we isolated the failure by pulling the GPU clusters skill out of the full skill package, and wrote a narrower prompt that reproduced only this issue. It failed every time when tested against the old skill file, so we had reliable repro steps (use the old skill), and an easy way to test each new version (update the skill).
We went through several iterations until we landed on a small change: two new sentences in the skill that point the agent to the pricing table, and explain how to use it with the live regions API:
Query GET <https://api.together.ai/v1/compute/regions> or run tg clusters list-regions --json for the live region inventory, supported instance types, and compatible CUDA/driver pairs. Open the GPU Cluster pricing table for current on-demand and reserved rates.
After this change, the task passed 100% of the time for all models we tested using the skill-enabled config.
Finding gaps in our API
The same GPU task also turned up a problem in our API. In a recent baseline run, Opus 5.5 got an incomplete verdict, and the transcript showed it hitting two dead ends that were not the model’s fault.
The run started well. The agent called /v1/compute/regions to retrieve the regions and instance types. Then, looking for prices, it called /v1/compute/instance-types, and the API returned a 400 with this message:
Region is required. Example: us-central-8. Use /v1/clusters/regions to get list of regions.
So the agent did what the error told it to do:
- It retried the request with a region from its first lookup. Instead of instance types, it got a 404.
- It called /v1/clusters/regions, the route the error suggested… but that route doesn’t exist, so it got another 404.
Our own error message had sent the agent to a route that doesn’t exist, and the retry that should have worked failed too. A developer following that message by hand would have been just as stuck. We’ve since fixed the error message to point to /v1/compute/regions, and the routing bug behind the first 404.
Case study 2: Rebuilding our skill package
We used to maintain 14 separate Together skills, one per product surface. They were a pain to use: you had to know in advance which skills applied to your workflow, then install and invoke each one. The skills also overlapped, repeating the same SDK guidance and setup instructions, and each was long enough that loading a few of them in tandem could bloat the agent’s context.
Last month, we merged those 14 skills into a single unified version. It’s still modular under the hood: to avoid context bloat, the agent first reads a short router, then opens each product guide, reference file, or sample script only when the router says it's relevant.
We hoped that one skill would beat 14, but to make sure, we ran the unified skill through our eval tasks with skills enabled. Our bar was the result from September 30, a 90% pass rate across all runs using the 14-skill config.
But the new skill was a flop—it passed just 54 of 72 runs (75%), left 18 incomplete instead of 7, and cost more than twice as much per run.
So we combed through the eval transcripts and changed the skill only where the runs pointed to a problem. Three issues kept coming up:
- Too much context: The first unified router was about 14,000 characters, loaded on every task, and it told agents to read each guide in full. (So much for avoiding context bloat…) Runs ended up loading 57% more skill content than they had with the 14 separate skills. We cut the router to under 6,000 characters and ended each product guide with a menu that briefly describes what every reference and script contains, so agents open only what they need.
- Stale product information: Some models used in the code examples were no longer served on serverless, and one product guide covered a deprecated feature. Agents wasted turns recovering or retrying requests that would never succeed. We moved every sample to a confirmed live model, told agents to check the catalog instead of retrying, and created a cloud agent to automatically update the skill in response to product changes.
- Bugs in example code: Small problems in the scripts, such as saving JPEG bytes under a .png name, would derail every agent that used them. We fixed these at the source and added an automation to periodically live-test the code samples and keep them fresh.
With those fixes in, the unified skill passed 70 of 72 runs, 22 points better than our first attempt and 7 points better than the 14 skills we started with. Runs also took 20% less time than with the original skills on average, and the cost per run came back down to where it started.

Where we are after four months of evals
Back in June, our docs and skills weren’t helping agents much at all. On June 15, running Opus 4.8, the baseline passed 75% of runs. Neither the docs MCP (67%) nor skills (75%) improved performance.
By September 30, across three models, the baseline had crept up to 82%. The docs MCP was at 96% and skills at 90%, with the new unified skill hitting 97% on October 8.
This isn’t a controlled comparison—the snapshots used different models and tasks, so some of these gains doubtless come from improvements to the underlying models, plus improvements we made to our CLI and SDKs. That probably explains the 7-point rise in the baseline, but the guided setups rose by 20+ points, and we credit that extra gain to the docs and skills fixes unearthed by our evals.

What we learned
Here’s the TL;DR if you’re skimming this post:
- Agent-accessibility increases usage: Be like GitHub: Everything devs can do by hand with a web GUI should also be available programmatically from your CLI or SDK. We find agents work best with a CLI, because a CLI documents itself through help text. Speaking of which—
- Docs are a core part of the product: Your platform may be unusable for agents if it isn’t accurately documented, or if your docs aren’t accessible (e.g. with llms.txt, a skill package, or an MCP server). CLI help text and API error message also count, and these are usually the first places an agent looks when it gets tripped up.
- Guidance matters more than which frontier model you use: On September 30, all three models (Claude Sonnet, Opus, and GPT Sol) finished within a few points of each other. Adding our docs/skills bumped the pass rate up from 82% to 96% and 97% respectively.
- Conversely, using only strong models can sometimes hide issues: Opus was able to work around a gap in our API and skill package, while Sonnet faceplanted into it—which is how we found and fixed the problem. When you’re running evals, make sure to test models across the capability spectrum.
- Improving AX often only takes a small fix: A two-sentence addition to our skill file took our GPU cluster eval from 4 of 15 passes (for Sonnet) to 100% across all three models. The fix was small, but we might never have found it if not through the eval process.
Run your own evals
If you’re running a dev platform, agents are using your product, or at least they’re definitely trying to. The best way to find out how well it’s going is to build your own evaluation system.
To help with that, we’re open-sourcing the package we use to build our evals. You create a list of tasks (agent prompts) and decide how to score your runs (with code tests or an LLM judge). The package launches the agents, runs the tasks, and tracks everything in a local dashboard, so you can compare configurations and past runs. It’ll be available next week, along with another blog post on how to use it.