Newshot

Sandboxes for AI Agents

Original text(via HN AI)

Without a sandbox, your agent is guessing

Ask your agent to create a new feature - it will confidently put up PRs ready for review. But how do you know any of it works? You have to test it out. Send back the failures, iterate, continue back and forth until you have something usable.

Isn’t there a way to get agents to test and validate their own code? Is there a way for agents to prove to you their changes work. This is where your sandbox comes in.

What a sandbox is

A sandbox is a practice copy of your app. Think flight simulator: close enough to the real thing that practice counts, fake enough that a crash hurts nobody.

anatomy of a sandbox · wall, access, stack, data, receipts
anatomy of a sandbox · wall, access, stack, data, receipts

anatomy of a sandbox · wall, access, stack, data, receipts

The agent comes in through a dev-only login, works against the full stack and its own data, and leaves “receipts” on the PR - proof that a change was validated. Customer data and real credentials stay outside the wall.

The running stack

Your services, databases and queues, plus fakes or vendor sandboxes for third-party APIs like payments and email. Sounds easy. It rarely is. Every service drags in a few more, and a mature app can need dozens running before the home page loads.

Isolation

Isolation makes everything else safe. Keep customer data out. Keep shared data out too, like the staging database three other teams rely on. Lock down outbound traffic, and never put real third-party credentials inside.

A handy way to judge any setup is its blast radius: the worst thing an agent can reach while doing exactly what you asked. Once I asked an agent to validate a change to the “delete org” flow. It did, by deleting the org. In a staging or production environment, this could be a problem - but in a sandbox it’s fine.

Seven sandbox designs

Realistic, safe, fast: pick two

Every sandbox trades off realistic (it behaves like production), safe (mistakes stay inside) and fast and cheap (you can start lots of them in seconds). Nobody gets all three.

sandbox trade-offs · 7 designs on 3 goals
sandbox trade-offs · 7 designs on 3 goals

sandbox trade-offs · 7 designs on 3 goals

These are example designs I’ve seen in practice - I provided names.

0. Island

One service on its own, with every dependency faked: mocks, emulators like LocalStack, recorded HTTP responses. It’s the fastest and cheapest check you have, and nothing can escape it. It’s also only as true as your fakes, and it misses everything that happens between services, which is exactly where the agent in the intro got caught. Use it as a first check. I barely count it as a sandbox.

Island · one service, all fakes
Island · one service, all fakes

Island · one service, all fakes

1. Local Dev

The sandbox most people already have: the stack running on a developer’s laptop, with native processes or Docker Compose, maybe behind a CLI. Feedback is instant and it costs nothing.

Local Dev · the stack on a laptop
Local Dev · the stack on a laptop

Local Dev · the stack on a laptop

It also runs one agent at a time and drifts from machine to machine. The agent runs with the developer’s credentials, so the blast radius is the laptop plus anything those credentials can reach.

2. Devbox

A devbox is local dev moved to the cloud. A persistent machine built from a prebuilt image, either one per engineer or a few set aside for agents. It’s bigger than a laptop, identical for everyone, and keeps running when you close the lid. You pay for it while it sits idle, it drifts unless you rebuild it, and agents sharing a box share its mess.

Devbox · persistent, per engineer or per agent
Devbox · persistent, per engineer or per agent

Devbox · persistent, per engineer or per agent

3. Ephemeral

A fresh VM or container for every task, started from a pre-warmed snapshot and thrown away afterwards. Production and the open internet are out of reach.

Ephemeral · one box per task
Ephemeral · one box per task

Ephemeral · one box per task

This is where teams running agents at scale have landed. Stripe’s Minions run on pre-warmed devboxes that start in about 10 seconds. Ramp runs each session in a Modal sandbox booted from a snapshot rebuilt every 30 minutes. GitHub Copilot’s coding agent works on an Actions runner behind a firewall allowlist.

Snapshots need rebuilding, every task costs something, and you get a dev stack rather than production’s real infrastructure.

Pets vs. cattle

A pet has a name, and when it gets sick you nurse it back. Cattle get numbers and get replaced. Devboxes are pets. Ephemeral sandboxes are cattle.

An agent shouldn’t burn an hour debugging its own environment. If a sandbox gets into a weird state, throw it out and start a fresh one.

pets vs. cattle · Devbox vs. Ephemeral
pets vs. cattle · Devbox vs. Ephemeral

pets vs. cattle · Devbox vs. Ephemeral

4. Preview Environment

A production twin. You deploy production’s own config with sandbox settings, one isolated copy per environment, and agents reach in from outside. It’s the most realistic option, and there’s no second stack definition to keep in sync.

Preview Environment · production's config, copied
Preview Environment · production's config, copied

Preview Environment · production’s config, copied

It’s also slow to start, expensive and complicated, and some third-party services can’t be copied at all. Lyft ran full per-engineer environments until they stopped scaling, then moved to request routing. A small warm pool for integration and infrastructure work is usually enough.

5. Hybrid

Run only the services you’re changing and borrow the rest from a shared environment. Tagged requests reach your version; everything else hits the shared copies. Uber’s SLATE and Lyft’s staging overrides both work this way. It starts in seconds, and the dependencies are real.

Hybrid · your service in, the rest borrowed
Hybrid · your service in, the rest borrowed

Hybrid · your service in, the rest borrowed

The catch is that data and queues are shared, so there is always a risk your “sandbox” may break everyone else’s. This is trading cost for safety.

6. Shadow

The new version runs beside production on a copy of real traffic. Its responses are compared with production’s and then thrown away, so users never see them. Twitter open-sourced this as Diffy.

Shadow · mirrored traffic, compared and discarded
Shadow · mirrored traffic, compared and discarded

Shadow · mirrored traffic, compared and discarded

You get real inputs and catch regressions nobody wrote a test for. You also get customer data flowing through it, which clashes with the test-accounts rule, and every write and side effect has to be blocked or sent to test storage. This can act like a validation check.

Seven designs at a glance

Why sandboxes are hard

The idea is simple. Building it is where teams get stuck.

Too big for one machine

When the stack won’t fit on one machine, you can start trimming. Smaller data seeds helps. So do lighter stand-ins for the heaviest services. Bonus points if you have well defined service boundaries. If you don’t, you may have to start big.

It’s rarely just one service needed. A service will depend on other services, which depend on other databases, and on and on. Even a minimal viable sandbox may be hard to achieve.

Many repos

If you have a single monorepo, cloning may be large but more manageable. But often a companies codebase spans across multiple repos. You have some options:

  • Clone them all. One sandbox checks out every repo on the task’s branch, so the agent can change any of them. It is the most flexible and the slowest to start. Ona and GitHub Codespaces both support it.
  • Build one, pull the rest. Build the repo you are changing from source and run the others from the images they last published from main. Fast and cheap, but only one repo changes at a time.
  • Link sandboxes by branch. Each repo’s PR gets its own sandbox, a shared branch name groups them, and one routing key sends requests through all of them. Optionally consider the PR number from a common repo
multi-repo sandboxes · 3 patterns
multi-repo sandboxes · 3 patterns

multi-repo sandboxes · 3 patterns

Cost

A full copy of the stack for every open PR, idle most of the day, adds up to a second production bill. Agents make it worse. They open more PRs and leave them open longer.

So turn sandboxes off when nobody is using them: sleep after a quiet period, wake on the next push, tear down when the PR closes.

Pack services together too. You don’t need a database server per service when one server can hold all the databases.

How current does it need to be?

Your own change is always current. What matters is how far behind main everything else is.

Making a sandbox useful

Whichever design you pick, the same three things decide whether an agent can actually check its work in it.

The data

Without data, your app isn’t testable. Real bugs show up in specific states, like a huge account or an expired trial or one missing field, so each change needs its own scenario.

I’d combine two sources.

A nightly copy from a test account. Most companies already have QA, demo and internal orgs used for testing or development. Dump those from a read replica, following foreign keys so related rows come along. Then publish a dated snapshot that each sandbox restores copy-on-write.

daily data dump · test accounts to sandboxes
daily data dump · test accounts to sandboxes

daily data dump · test accounts to sandboxes

Customer data should never make it into the snapshot.

Seeded scenarios. Describe the required data in plain English, turn it into a reusable template, and create the data through the product’s own APIs so it’s valid.

seeding from a prompt · prompt, plan, API calls
seeding from a prompt · prompt, plan, API calls

seeding from a prompt · prompt, plan, API calls

Data processing. Sometimes rows aren’t enough. If your product runs on events, streams or batch jobs, the sandbox needs a small version of that pipeline to process that data as well. That may mean running a data pipeline on your seeded data to produce the write output values.

Access

An agent stuck at the login screen can’t check anything. Give sandboxes a dev-only login that skips SSO and captchas, and let the agent sign in as any seeded role: admin, viewer, a new user.

Passwords shouldn’t matter much in a sandbox - it’s a playground. Mock out your calls to password services.

Receipts

Every PR should come with receipts - evidence the changes work. Screenshots or a recording of the pages the agent visited, the API calls and their responses, server logs, the rows it read and wrote. A reviewer can skim that in a minute, or click through to a preview environment - which beats pulling the branch and running it.

Read the sessions themselves. Each place an agent couldn’t run something, or hacked around it, goes on a checklist ranked by how often it comes up. The platform team works down from the top, and every fix helps every session after it.

session review · gaps become a ranked checklist
session review · gaps become a ranked checklist

session review · gaps become a ranked checklist

Where to start

Humans likely already have ways to test and validate most changes - you just need to find a way to allow agents to run similar processes and view the results.

You don’t have to build all of it at once. Pick the design that fits your stack; if it fits on one machine, that’s probably Ephemeral. Wall it off from real data and production services. Load it with test-account data and give the agent a way to log in. Have the harness record what the agent actually did.

Then read the sessions. The first few will show you exactly where agents get stuck, and the most common blocker is your next piece of work.

Thanks for reading Shawn’s Substack! Subscribe for free to receive new posts and support my work.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by HN AI
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗