Newshot

Show HN: Stromboli Open-source AI SRE agent focused on fast incident mitigation

Original text(via HN AI)

stromboli

题图
题图

        (   )  (    )
                           (    )  (   )    )
                             (   ) (   )  (
                               (  ) (  )
                          *   .  \ | /  .   *
                       .    *  -=  *  =-  *    .
                               ' _/^\_ '
                               _/  ' \\_
                             _/  .    \\\_
                           _/       '  \\\\_
                         _/   .         \\\\\_
                       _/          .   ' \\\\\\_
                     _/    '              \\\\\\\_
                   _/            .       ' \\\\\\\\_
                 _/      .                  \\\\\\\\\_    ( )
               _/              '     .     ' \\\\\\\\\\_    ) (
   ~~~~~~~~~~~/~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\~~~~~~~~~~~~~
     ~~~   ~~~~     ~~~      ~~~~~      ~~~~     ~~~      ~~~~   ~~~
          ~~~~         ~~~         ~~~~        ~~~~          ~~~

                         s t r o m b o l i . d e v
      

An open-source AI SRE agent you host yourself. When an alert fires, stromboli gathers the context, tells you how to stop the bleeding, then finds the root cause, and posts all of it to Slack.

  • 🩹 Mitigation first. It proposes a concrete fix (rollback, restart, scale up…) within a few tool calls, before it starts digging for the root cause.
  • 🔍 Then the root cause. It tests hypotheses against your logs, metrics and deploys, and writes a structured report.
  • 🔒 Read-only by default. It suggests and never acts on its own. The one opt-in exception: on Fly.io it can run a machine action (restart, clone, cordon…) after a named human approves it (Machine actions).
  • 🧠 Any LLM. Claude by default; OpenAI, Gemini, DeepSeek, Ollama, OpenRouter or Vercel AI Gateway work too (Models).
  • 📦 No data ingestion. Logs and metrics are queried on demand where they already live. stromboli only stores the aggregated evidence each investigation used.

📚 Read the documentation: tutorials to get started, how-to guides for specific tasks, a complete reference (CLI, configuration, catalog, API), and explanations of how stromboli works. See docs/adr for the design decisions behind stromboli and what is planned next.

Contents

  • How an investigation works
  • The service catalog
  • Connecting your data: logs, metrics, Google Cloud, Fly.io
  • Machine actions and approving from Slack
  • Configuration
  • API and MCP server
  • Contributing

Quickstart

1. Start stromboli


        cp .env.example .env   # fill in STROMBOLI_SECRET_KEY, STROMBOLI_API_TOKEN and an LLM key (ANTHROPIC_API_KEY...)
docker compose up -d   # stromboli + Postgres on http://localhost:8080
      

Generate the secret key with openssl rand -base64 32. Slack is optional: create the app from its manifest to get reports there.

Already have Postgres? Run the image alone with STROMBOLI_DB_URL=postgres://.... Migrations run on startup. Binaries for Linux, macOS and Windows are on the releases page.

To install the CLI on macOS or Linux with Homebrew:


        brew install openstatushq/stromboli/stromboli
      

Running it for real? Read Self-hosting for the production checklist (TLS, tokens, backups) and how to upgrade.

Or deploy stromboli and Postgres to Railway in one command, with the secrets generated for you. The setup is code, in .railway/railway.go (details):


        railway login
ANTHROPIC_API_KEY=sk-ant-... deploy/railway/setup.sh
      

On Fly.io, fly.toml and a setup script do the same with Fly Managed Postgres (details):


        fly auth login
FLY_MPG_CLUSTER=<cluster-id> ANTHROPIC_API_KEY=sk-ant-... deploy/fly/setup.sh <app-name>
      

2. Connect your logs, metrics and deploys

stromboli is configured with its CLI (the same binary, so docker compose exec stromboli stromboli ... works too):


        export STROMBOLI_URL=http://localhost:8080 STROMBOLI_API_TOKEN=...

stromboli connector add loki loki url=http://loki:3100   # your log store
stromboli connector add github github token               # deploys and commits; prompts for the token
      

Loki is just an example: see Connecting your data for Axiom, CloudWatch, Google Cloud, Prometheus and Fly.io.

On Fly.io, a read-only org token is enough for metrics, the machine list and catalog discovery. It cannot deploy, restart or read secrets:


        fly tokens create readonly <org-slug>                    # prints FlyV1 fm2_...; or pass it inline:
stromboli connector add fly fly org=<org-slug> token="$(fly tokens create readonly <org-slug>)"
      

<org-slug> is in fly orgs list (personal for a personal org). Revoke it later with fly tokens list and fly tokens revoke <id>.

3. Describe your services

stromboli can build a catalog from what it finds in your logs or Fly metrics. It lists the services and asks whether to add them; then fill in owners and dependencies one service at a time:


        stromboli catalog discover loki   # or: discover fly, one service per Fly app
stromboli catalog set api --owner @backend --add-depends-on postgres
      

To review the entries first or keep them in git, redirect them to a file, add owners, dependencies and runbooks, then import it:


        stromboli catalog discover loki > services.yaml
stromboli catalog import services.yaml --runbooks runbooks/
      

4. Try it


        stromboli brief api --since 1h                                  # the context stromboli gathers (no LLM, free)
stromboli investigate api --description "5xx on /v1/monitors"   # a full investigation, streamed live
      

5. Send it your alerts

SourceEndpoint
openstatus (webhook notification with header Authorization: Bearer <token> )POST /webhooks/openstatus
AlertmanagerPOST /webhooks/alertmanager?token=<token>
Grafana alertingPOST /webhooks/grafana?token=<token>
Anything else, as {"service","title","fingerprint","details"}POST /webhooks/generic

That's it: the next alert starts an investigation on its own.

Slack

stromboli posts one message per investigation (the mitigation first, then the report) and replies in its thread. The app only needs the chat:write scope, and it is described in deploy/slack/manifest.json:

  • Create the app. Create it from the manifest and pick your workspace. Or go to api.slack.com/apps → Create New App → From a manifest and paste the file. Before creating it, replace stromboli.example.com in request_url with your stromboli's public URL (or remove the interactivity block if you won't use Apply/Reject buttons).
  • Install it. Install App → Install to Workspace, then copy the Bot User OAuth Token (xoxb-...).
  • Invite the bot to the channel: /invite @stromboli in it.
  • Configure stromboli and restart it: SLACK_BOT_TOKEN=xoxb-... SLACK_CHANNEL=#incidents # a name or a channel ID (C0123...) SLACK_SIGNING_SECRET=... # optional: Basic Information → Signing Secret, enables the buttons SLACK_APPROVERS=U0123,U0456 # optional: who may click Apply/Reject (profile → Copy member ID)

Configure stromboli and restart it:


        SLACK_BOT_TOKEN=xoxb-...
SLACK_CHANNEL=#incidents             # a name or a channel ID (C0123...)
SLACK_SIGNING_SECRET=...             # optional: Basic Information → Signing Secret, enables the buttons
SLACK_APPROVERS=U0123,U0456          # optional: who may click Apply/Reject (profile → Copy member ID)
      

Check it with stromboli investigate <service> --description "testing Slack": the message should appear in the channel. not_in_channel in the logs means step 3 is missing; invalid_auth, a wrong token.

How an investigation works


        alert ──▶ 1. match ──▶ 2. brief ──▶ 3. mitigate ──▶ 4. investigate ──▶ 5. report
                        (no LLM)    (posted now)    (read-only tools)   (Slack + API)
      
  • Match. The alert is matched to a catalog service (openstatus monitor ID, service/app labels). Repeats are merged into the running investigation, or into one that finished within STROMBOLI_DEDUPE_WINDOW, unless the alert resolved since (openstatus recovered, Alertmanager resolved) or one of its machine actions succeeded: the alert firing again is then a new incident. Merged alerts and webhooks with a wrong token are logged.
  • Brief (no LLM). stromboli collects what an on-call engineer would look at first: the service, its dependencies and blast radius log error patterns, compared with the preceding window of the same length per-instance metrics (memory, CPU, throttling, OOM kills, request rate, 5xx, p95 latency), against the same baseline recent deploys or commits of the service and its dependencies its runbooks and similar past investigations on Fly.io: the app's machines and the exact Fly command, with its undo, for each mitigation

Brief (no LLM). stromboli collects what an on-call engineer would look at first:

  • the service, its dependencies and blast radius
  • log error patterns, compared with the preceding window of the same length
  • per-instance metrics (memory, CPU, throttling, OOM kills, request rate, 5xx, p95 latency), against the same baseline
  • recent deploys or commits of the service and its dependencies
  • its runbooks and similar past investigations
  • on Fly.io: the app's machines and the exact Fly command, with its undo, for each mitigation
  • Mitigate. What matters most is how fast user impact stops, not how fast the bug is understood (as in Google's SRE practice). The model picks one mitigation from a fixed set: rollback, restart, scale_up, drain, disable_feature, block_traffic, failover, degrade or none. It gives the target, the exact command, how to check it worked, how to undo it, and the risk (safe, reversible, destructive). It is posted to Slack right away (with a thread reply so on-call is notified) and its time is stored as mitigated_at. After STROMBOLI_MITIGATION_TOOL_CALLS tool calls, the model is told to commit to one, and it cannot submit a report without one.
  • Investigate. The model looks for the root cause with read-only tools: Area Tools Logs log_patterns, query_logs, log_timeline Metrics get_metrics (a preset such as memory_used_percent, or any PromQL), list_metrics Deploys recent_deploys, get_commit Catalog get_service, list_services Knowledge read_runbook, search_past_investigations Tool output is aggregated, size-capped and redacted (tokens, passwords, emails) before the model sees it. If the evidence shows the mitigation won't help (a restart can't fix a bad config push), the model revises it.

Investigate. The model looks for the root cause with read-only tools:

AreaTools
Logslog_patterns , query_logs , log_timeline
Metricsget_metrics (a preset such as memory_used_percent , or any PromQL), list_metrics
Deploysrecent_deploys , get_commit
Catalogget_service , list_services
Knowledgeread_runbook , search_past_investigations

Tool output is aggregated, size-capped and redacted (tokens, passwords, emails) before the model sees it. If the evidence shows the mitigation won't help (a restart can't fix a bad config push), the model revises it.

  • Report. Likely cause, confidence, evidence, what was ruled out, the mitigation in light of the cause, the long-term fix and next steps, posted to Slack and the API. Every brief, tool call and model turn is kept as an audit trail.

The service catalog

The catalog tells stromboli what your services are and where their data lives:


        - name: api
  description: Public REST API
  owners: ["@team-backend"]
  tier: 1
  depends_on: [db-proxy]
  logs:
    source: loki                       # connector name
    selector: '{service_name="api"}'
  deploys:
    source: github                     # connector name
    repo: acme/api
    workflow: deploy.yml               # or environment: production; or neither = commits
  metrics:
    source: fly                        # connector name (fly or prometheus)
    selectors: ['app="api"']           # PromQL label matchers, ANDed
  runbooks: [runbooks/api-5xx.md]
  monitors: ["openstatus:1234"]        # alert → service mapping
  runtime: fly.io                      # or a fly: block, see Machine actions
      

The catalog lives in stromboli's database. YAML is its import/export format, so you can keep it in git and sync it from CI:


        stromboli catalog import services.yaml --runbooks runbooks/ --prune --yes   # --prune makes the file authoritative
stromboli catalog list                                                        # name, tier, owners, dependencies, sources
stromboli catalog rm api                                                      # delete one service
stromboli catalog export > services.yaml                                      # the full entries
      

A full example is in examples/catalog.

Connecting your data

WhatConnectors
Logsloki , axiom , cloudwatch , gcp_logging , vercel
Metricsfly , prometheus , gcp_monitoring
Deploys and commitsgithub , vercel ( Vercel )
Fly.io machinesfly , fly_machines ( Fly.io )

MCP first

stromboli calls a provider through the MCP server the provider hosts, when there is one. Otherwise it calls the provider's API directly, without adding a CLI or local process just to get MCP. stromboli is the MCP client: it calls the tools from code and turns their output into compact evidence. The model never talks to these servers directly.

  • Allow-list. Each connector names the few tools it may call (internal/connector/mcpconn). Any other tool the server offers is refused, including deploy, delete and exec.
  • Read-only check. For servers that annotate their tools (Google Cloud, GitHub), connecting fails unless every allowed tool is marked readOnlyHint.
  • Sessions are reused per connector and closed on shutdown.
ProviderThroughNot covered by MCP (REST)
Google Cloud: gcp_logging , gcp_monitoringGoogle's remote MCP servers ( logging.googleapis.com/mcp , monitoring.googleapis.com/mcp )none
GitHub: github with a tokenGitHub's remote MCP server, toolsets repos,actions , read-only modeDeployments ( deploys.environment ), and token-less use
Fly.io, Loki, Prometheus, CloudWatchtheir APIs: the Machines API, Fly's managed Prometheus, the Loki and Prometheus HTTP APIs, the AWS SDKno hosted MCP server: see Not on MCP
Axiom: axiomAxiom's remote MCP server ( mcp.axiom.co ), logging in with OAuthnone
Vercel: vercelVercel's remote MCP server ( mcp.vercel.com ), logging in with OAuthnone

Not on MCP

  • Fly.io offers MCP only through flyctl mcp server, a local process that runs flyctl commands and serves all of them (destroy, exec, secrets) without marking any read-only. stromboli calls the Machines API directly instead.
  • CloudWatch: AWS's MCP server (awslabs.cloudwatch-mcp-server) is a local Python process, so stromboli uses the AWS SDK.
  • Loki and Prometheus have no hosted MCP server. Grafana's mcp-grafana needs a Grafana instance in front of them.

Log connectors

KindAdd itlogs.selectorcatalog discover proposes
lokiconnector add loki loki url=http://loki:3100 (+ tenant_id , username , password , bearer_token )LogQL stream selector: {service_name="api"}one service per service_name / service / app / job / container label value
axiomconnector add axiom axiom logs in with OAuth through Axiom's MCP server (+ org_id , message_field , mcp_url=https://mcp.eu.axiom.co/mcp for the EU)Start of an APL query: ['logs'] or ['logs'] │ where ['service.name'] == "api"one service per service.name / service / app / fly.app.name … value in each log dataset, or one per dataset
cloudwatchconnector add cw cloudwatch region=eu-west-1 (+ access_key_id , secret_access_key , session_token , endpoint )Log group(s), optionally followed by Insights commands: /ecs/api or /ecs/prod, /ecs/worker │ filter service = "api"one service per log group, named after its last path segment
gcp_loggingconnector add gcp-logs gcp_logging project=<project-id> (+ [email protected] )Logging query language filter: resource.type="cloud_run_revision" AND resource.labels.service_name="api"one service per Cloud Run service or job, function, Cloud Scheduler job, Cloud Tasks queue, Pub/Sub subscription or topic seen in the last hour
vercelconnector add vercel vercel team_id=<slug> (logs in with OAuth in a browser)a project, optionally environment=production : web environment=productionone service per project

Error filters are RE2 regexes applied to the log line: Loki's line, Axiom's message field (detected from message, msg, body, log, ... or set with message_field), CloudWatch's @message, Cloud Logging's textPayload and jsonPayload.message, and Vercel's message prefixed with its level. On Cloud Logging, the error filter also matches severity>=ERROR and httpRequest.status>=500. Cloud Logging has no count tool, so error counts there stop at 5000.

Without access_key_id/secret_access_key, the CloudWatch connector uses the default AWS credential chain of the stromboli server (environment, shared config, IAM role). It needs logs:DescribeLogGroups, logs:StartQuery, logs:GetQueryResults and logs:StopQuery.

Metric connectors

KindAdd itmetrics.selectorsPresets
flyconnector add fly fly org=<org-slug> token="$(fly tokens create readonly <org-slug>)"app="checker"yes, see Fly.io
prometheusconnector add prom prometheus url=http://prometheus:9090 (+ tenant_id , username , password , bearer_token , service_label )job="api" , service="api"none: the agent uses list_metrics and writes PromQL
gcp_monitoringconnector add gcp-metrics gcp_monitoring project=<project-id> (+ [email protected] )subscription_id="checker-ams" , queue_id="alerting" , service_name="api"yes, see Google Cloud

prometheus works with anything speaking the Prometheus HTTP API (Prometheus, Mimir, Thanos, VictoriaMetrics, Grafana Cloud). In a PromQL query, $selector expands to the service's matchers, e.g. sum by (instance) (rate(http_requests_total{$selector, code=~"5.."}[1m])). Each series comes back as min, max, avg, last, its average over the baseline window and 12 points, capped at 10 series.

stromboli catalog discover <metrics-connector> proposes metrics for every app it finds (one per app label value on Fly; the first of service_name, service, app, job with values on Prometheus, or service_label). Services already in the catalog without metrics come back as their full entry with metrics filled in, so stromboli catalog import only adds the metrics.

Google Cloud

Both GCP connectors go through Google's remote MCP servers using PromQL and the Logging query language. Without credentials_json (a service account key), they use Application Default Credentials: GOOGLE_APPLICATION_CREDENTIALS, workload identity, or gcloud auth application-default login. The principal needs roles/mcp.toolUser, roles/logging.viewer and roles/monitoring.viewer. Calls are billed to project.

gcp_monitoring presets depend on the resource label in the service's selectors. A selector with subscription_id gets the Pub/Sub presets, queue_id gets Cloud Tasks, and service_name gets Cloud Run:

ProductPresets
Pub/Subpubsub_backlog_messages , pubsub_oldest_unacked_seconds , pubsub_acks_per_second , pubsub_sent_per_second , pubsub_expired_ack_deadlines_per_second , pubsub_dead_letter_per_second , pubsub_nacks
Cloud Taskstasks_queue_depth , tasks_attempts_per_second (by response code), tasks_attempt_delay_p95_seconds
Cloud Runhttp_requests_per_second , http_5xx_per_second , http_p95_seconds , instances , cpu_utilization_p95_percent , memory_utilization_p95_percent , startup_latency_p95_seconds (per revision)

list_metrics lists the project's active metrics under their PromQL names (pubsub_googleapis_com:subscription_num_undelivered_messages). catalog discover proposes one service per Pub/Sub subscription, Cloud Tasks queue and Cloud Run service with data in the last hour.

Fly.io

Machine metrics

Fly collects the metrics of every machine in the organisation into a managed Prometheus at https://api.fly.io/prometheus/<org-slug>, so there is nothing to deploy:


        stromboli connector add fly fly org=<org-slug> token="$(fly tokens create readonly <org-slug>)"
stromboli catalog discover fly
      

Tokens from fly tokens create are sent with the FlyV1 scheme, others (from fly auth token) as Bearer. The fly connector comes with presets, per machine (instance) and region:

PresetFrom
memory_used_percent , memory_used_bytes , swap_used_bytesfly_instance_memory_mem_total - mem_available , swap_total - swap_free
oom_killedfly_instance_exit_oom
cpu_percent , cpu_throttled_seconds_per_second , load_averagefly_instance_cpu (non-idle share), fly_instance_cpu_throttle , fly_instance_load_average
disk_used_percent , volume_used_percentfly_instance_filesystem_blocks* , fly_volume_used_pct
net_recv_bytes_per_second , net_sent_bytes_per_secondfly_instance_net_*_bytes
http_requests_per_second , http_5xx_per_second , http_p95_seconds , concurrencyfly_app_http_responses_count , fly_app_http_response_time_seconds , fly_app_concurrency
instances_upfly_instance_up

The brief shows memory, CPU, throttling, OOM kills, root filesystem and volume usage, request rate, 5xx and p95 latency; the agent can query the rest with get_metrics.

Logs

Fly has no API to query past logs. The whole organisation's logs are published on a NATS stream, and Fly's log shipper forwards them to a log store. Point it at Loki, Axiom or CloudWatch, then add that store as a connector:


        fly launch --image flyio/log-shipper:latest --no-public-ips --name log-shipper   # set internal_port = 8686
fly secrets set -a log-shipper ORG=<org-slug> ACCESS_TOKEN="$(fly tokens create readonly <org-slug>)"
# then one of:
fly secrets set -a log-shipper LOKI_URL=https://loki.example.com LOKI_USERNAME=... LOKI_PASSWORD=...
fly secrets set -a log-shipper AXIOM_TOKEN=xaat-... AXIOM_DATASET=fly-logs
fly secrets set -a log-shipper AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_REGION=eu-west-1 CLOUDWATCH_LOG_GROUP_NAME=/fly/logs
      
StoreConnectorDiscovered selector, one service per Fly app
Lokiconnector add loki loki url=...{fly_app_name="checker"}
Axiomconnector add axiom axiom['fly-logs'] │ where ['fly.app.name'] == "checker"
CloudWatchconnector add cw cloudwatch region=eu-west-1 log_group_prefix=/fly/ service_field=fly.app.name/fly/logs │ filter fly.app.name = "checker"

The shipper writes JSON events (message, log.level, fly.app.name, fly.region, host). stromboli shows the message with its level, so pattern grouping isn't split by region or instance. The CloudWatch shipper writes every app to one log group, so service_field makes discovery split it per app with one Insights query (log_group_prefix keeps that query to the Fly group).

Mitigation commands

For a service with runtime: fly.io or a fly: block, the brief gets a Fly.io section. It lists the app's machines from the Machines API, so the agent can name real machine IDs and regions (the instance label of the Fly metrics is the machine ID). It also gives the Fly command and its undo for each mitigation:

MitigationCommandUndo
restart one machinefly machine restart <id> -a <app>none needed
restart every machinefly apps restart <app>none needed
scale_up, one more machinefly machine clone <id> -a <app> --region <region>fly machine destroy <new-id> -a <app> --force
scale_up, more machines / memoryfly scale count <n> / fly scale memory <MB>the previous value
drainfly machine cordon <id> (no traffic, keeps running) or fly machine stop <id>uncordon / start
scale_up, more disk (full volume)fly volumes extend <vol-id> -s <GB> -a <app>none: volumes cannot shrink
free spacefly ssh console -a <app> --machine <id> , then delete or rotate datanone
rollbackfly releases --image , then fly deploy --image <previous image>redeploy the bad image

Machines with a volume are flagged and their volumes listed (ID, mount path, size), so a full volume can be extended by ID: a restart does not free space on a volume, and a clone would start with an empty one. The app name is fly.app, else the app="..." metrics selector, else the service name. The machines are listed with the fly.actions connector, else the fly metrics connector's token.

Health checks

Some failures never fire an alert until it is too late, like a volume filling up over days. With STROMBOLI_HEALTHCHECK_INTERVAL=1h, stromboli reads the metrics of every service with a fly metrics source once an hour (the first run is a minute after startup) and starts an investigation, with source healthcheck, when a machine or volume crosses a threshold over the last 15 minutes:

CheckThreshold
volume_used_percentlast value ≥ 85%
disk_used_percent (root filesystem)last value ≥ 90%
memory_used_percentlast value ≥ 90%
cpu_percent15-minute average ≥ 90%

When everything is fine, it only logs a line, unless STROMBOLI_HEALTHCHECK_SUMMARY=true: then it also posts ✅ All services operational to Slack. That message is only sent when every checked service is under its thresholds and no check failed; when some services could not be checked (a query failed), it posts ⚠️ Health check: could not check N services with the reason for each instead. Each unhealthy service gets one investigation, with the findings in its alert payload, posted to Slack like any other. A service with no metrics at all in the last 15 minutes (stopped machines, or selectors that match nothing) gets one too, so a stopped machine comes with a proposed start to approve. While the same checks keep failing, it is not investigated again for 6 hours. Without an LLM, findings are only logged.

Machine actions

By default, every mitigation is text for a human to apply. For Fly.io services you can let stromboli run some of them itself, always after a human approves.


        export STROMBOLI_ACTIONS_ENABLED=true          # off by default; requires STROMBOLI_API_TOKEN
stromboli connector add fly-deploy fly_machines org=<org-slug> token="$(fly tokens create deploy -a <app>)"
      

        - name: checker
  fly:
    app: openstatus-checker            # defaults as above
    actions: fly-deploy                # fly_machines connector; without it, mitigations stay text
    allow: [restart, cordon, uncordon] # default: restart, clone, cordon, uncordon (stop and start are opt-in)
      

What happens:

  • Proposed. When the brief says actions can run for the service, the agent may attach a machine_action (restart, clone, cordon, uncordon, stop or start, plus machine IDs and, for clone, a region) to its mitigation. stromboli checks it against the allow-list and the app's current machines (at most 10 per action; machines with a volume are never cloned) and stores it as proposed. A refused action is dropped, the mitigation stays a suggestion, and the agent is told why. Revising the mitigation supersedes the pending action.
  • Approved, or not. Nothing runs until a human approves it, before STROMBOLI_ACTION_TTL (30m) runs out: stromboli actions --status proposed stromboli actions approve <action-id> --as alice --note "ok from the incident call" # asks to confirm; --yes skips stromboli actions reject <action-id> --as alice --note "the db is the cause"

Approved, or not. Nothing runs until a human approves it, before STROMBOLI_ACTION_TTL (30m) runs out:


        stromboli actions --status proposed
stromboli actions approve <action-id> --as alice --note "ok from the incident call"   # asks to confirm; --yes skips
stromboli actions reject <action-id> --as alice --note "the db is the cause"
      
  • Run, safely. The approval is checked again against the catalog and the machines, then claimed atomically, so an action runs at most once. It runs machine by machine. A restart, start or clone waits for the machine to reach started (60s) and stops at the first failure, so a rolling restart never takes down more than one machine that did not come back. The result lists what was done and how to undo it (for a clone, the fly machine destroy of the new machine).
  • Recorded. Every action keeps who approved or rejected it, when, their note, the result and the error (stromboli actions show <id>, ActionService), and is posted in the investigation's Slack thread.

The approver name is recorded as given: anyone with the API token can approve. Use Slack approval with SLACK_APPROVERS to restrict who can.

Approve from Slack

With SLACK_SIGNING_SECRET set (and STROMBOLI_ACTIONS_ENABLED=true), a proposed action shows Apply and Reject buttons in the investigation's Slack message instead of the CLI command. Apply asks to confirm, showing the risk and the undo. The click goes through the same Approve/Reject as the CLI, with the Slack user recorded as the approver (@alice (Slack U0123)). The outcome is posted in the thread and replaces the buttons. Expired or superseded actions lose their buttons on the next redraw, and a click on one is answered with why it cannot run.

  • In your Slack app, turn on Interactivity & Shortcuts with Request URL https://<stromboli>/slack/interactions (already on if you created it from the manifest).
  • Set SLACK_SIGNING_SECRET (Basic Information → Signing Secret). Requests without a valid Slack signature, or older than 5 minutes, are refused.
  • Recommended: SLACK_APPROVERS=U0123,U0456 limits the buttons to those Slack user IDs. Without it, anyone who can see the channel can approve.

Only machine actions have buttons. A text-only mitigation (rollback, failover…) is still applied by a human outside stromboli.

Models

stromboli is not tied to one LLM vendor. As with the AI SDK, a model is provider:model, and each phase can use its own. A good split is a fast model for triage, where time to mitigation matters most, and a thorough one for the root cause:


        STROMBOLI_TRIAGE_MODEL=anthropic:claude-haiku-5-5     # until the mitigation is published
STROMBOLI_TRIAGE_EFFORT=low
STROMBOLI_INVESTIGATE_MODEL=openai:gpt-5              # root cause and report
STROMBOLI_INVESTIGATE_EFFORT=high
      

The investigate model takes over the triage conversation once the mitigation is published, so it sees everything triage saw.

The simplest setup is Vercel AI Gateway: one key, every model. As in the AI SDK, an id like openai/gpt-5 (no provider: prefix, a slash) goes through the gateway:


        AI_GATEWAY_API_KEY=...
STROMBOLI_TRIAGE_MODEL=anthropic/claude-haiku-5.5
STROMBOLI_INVESTIGATE_MODEL=openai/gpt-5
      
ProviderModel idConfiguration
Vercel AI Gatewayopenai/gpt-5 , anthropic/claude-opus-5.5 , deepseek/deepseek-v4-pro … (or gateway:<id> )AI_GATEWAY_API_KEY or VERCEL_OIDC_TOKEN , optional AI_GATEWAY_BASE_URL
Anthropicanthropic:claude-opus-5-5 (or a bare claude-… id)ANTHROPIC_API_KEY
OpenAIopenai:gpt-5OPENAI_API_KEY , optional OPENAI_BASE_URL (Azure, proxies)
Google Geminigoogle:gemini-2.5-proGEMINI_API_KEY
OpenRouteropenrouter:meta-llama/llama-3.3-70b-instructOPENROUTER_API_KEY
DeepSeekdeepseek:deepseek-v4-pro , deepseek:deepseek-flashDEEPSEEK_API_KEY , optional DEEPSEEK_BASE_URL
Ollamaollama:qwen3:32bOLLAMA_BASE_URL (default http://localhost:11434/v1 )
Any OpenAI-compatible API<name>:<model> , e.g. groq:llama-3.3-70b-versatileSTROMBOLI_PROVIDER_<NAME>_BASE_URL , STROMBOLI_PROVIDER_<NAME>_API_KEY

The model has to support tool calling.

Anthropic models use prompt caching, effort and server-side refusal fallbacks; through the gateway they use its Anthropic Messages API, so caching, effort and thinking blocks are kept (fallbacks are the gateway's own). Other providers go through the OpenAI Chat Completions format, where effort becomes reasoning_effort (only sent when set; DeepSeek accepts up to max). Reasoning that a provider returns (DeepSeek's reasoning_content, the gateway's reasoning_details) is sent back to that provider on later turns, as its tool loop requires. Switching models between phases resets the prompt cache once, at the switch.

Configuration

All settings are environment variables; .env.example has them all with comments.

VariableDefault
STROMBOLI_DB_URL(required)Postgres URL
STROMBOLI_SECRET_KEY(required)encrypts connector credentials (AES-256-GCM)
STROMBOLI_API_TOKENempty = no authbearer token for the API
STROMBOLI_WEBHOOK_TOKENAPI tokentoken for /webhooks/*
STROMBOLI_MODELanthropic:claude-opus-5-5default model for every phase, as provider:model ( Models )
STROMBOLI_EFFORTprovider defaultlow , medium , high , xhigh , max
STROMBOLI_TRIAGE_MODEL / STROMBOLI_TRIAGE_EFFORTSTROMBOLI_MODEL / STROMBOLI_EFFORTmodel until the mitigation is published
STROMBOLI_INVESTIGATE_MODEL / STROMBOLI_INVESTIGATE_EFFORTSTROMBOLI_MODEL / STROMBOLI_EFFORTmodel for the root cause and report
AI_GATEWAY_API_KEY , ANTHROPIC_API_KEY , OPENAI_API_KEY , GEMINI_API_KEY , OPENROUTER_API_KEY , DEEPSEEK_API_KEY , OLLAMA_BASE_URLcredentials of the providers you use
STROMBOLI_MAX_TOOL_CALLS30per investigation
STROMBOLI_MITIGATION_TOOL_CALLS6tool calls before the agent must propose a mitigation
STROMBOLI_INVESTIGATION_TIMEOUT8m
STROMBOLI_DEDUPE_WINDOW30m
STROMBOLI_ACTIONS_ENABLEDfalselet investigations propose machine actions a human can approve (requires STROMBOLI_API_TOKEN )
STROMBOLI_ACTION_TTL30mhow long a proposed action can be approved
STROMBOLI_HEALTHCHECK_INTERVALempty = offcheck Fly machine metrics on this schedule, e.g. 1h (at least 5m ), and investigate breaches ( Health checks )
STROMBOLI_HEALTHCHECK_SUMMARYfalsepost an all-clear to Slack after each health check run where every service is healthy, and a warning listing the services that could not be checked
SLACK_BOT_TOKEN , SLACK_CHANNELbot needs chat:write
SLACK_SIGNING_SECRETenables Apply/Reject buttons ( Approve from Slack )
SLACK_APPROVERSempty = anyone in the channelSlack user IDs allowed to click Apply/Reject
STROMBOLI_MCP_ENABLEDfalseserve the MCP tools at /mcp (requires STROMBOLI_API_TOKEN )
STROMBOLI_MCP_TOKENAPI tokenthe only token /mcp accepts when set; it reaches the MCP tools, not the whole API

CLI tips

Every command has --help with examples.

  • Secrets stay off the command line. For a secret connector key, pass token alone to be prompted (input is hidden), token=@file to read a file, or token=- to read stdin.
  • Ctrl-C cancels a prompt right away, and stops watching an investigation, but it keeps running on the server. Resume with stromboli show <id> --follow.
  • Destructive commands ask first: connector rm, catalog rm, actions approve, and catalog import --prune when it would delete something. Without a terminal, or with --no-input, they fail unless you pass --yes. actions approve and reject ask for your name when neither --as nor STROMBOLI_USER is set.
  • Scripting: data goes to stdout and messages to stderr. --json prints JSON (investigate --json prints one event per line), -q silences messages, and investigate --no-watch prints only the investigation ID. Exit codes: 0 success, 1 error, 2 bad usage, 130 interrupted.
  • Completions: stromboli completion bash|zsh|fish|powershell.

API

The API is ConnectRPC, defined in proto/stromboli/v1: CatalogService, ConnectorService, InvestigationService, ActionService. The same endpoints speak Connect, gRPC and gRPC-Web, plus plain JSON:


        curl -H "Authorization: Bearer $STROMBOLI_API_TOKEN" -H 'Content-Type: application/json' \
  -d '{"name":"api"}' localhost:8080/stromboli.v1.CatalogService/GetService
grpcurl -plaintext -H "Authorization: Bearer $STROMBOLI_API_TOKEN" localhost:8080 list   # reflection is on
      

InvestigationService/WatchInvestigation streams steps live.

MCP server

stromboli is also an MCP server, so you can use it from Claude Code, Claude Desktop, Cursor or a coding agent: read what stromboli sees on a service, the mitigation and report of an investigation, or start one (ADR 0021).

list_services , get_servicethe catalog
get_briefthe brief of a service, without the LLM (free)
list_investigations , get_investigationinvestigations, their mitigation and report; include_steps for the audit trail
wait_investigationwait until the mitigation is published (or the report written), with progress
start_investigationstart an investigation: spends LLM tokens and posts to Slack
list_actions , get_actionproposed machine actions and their outcome

Every tool is read-only except start_investigation, which only starts what a webhook or stromboli investigate would. MCP clients cannot approve machine actions or change the catalog or connectors: approval stays with a named human, in the CLI or Slack.

Over stdio, for clients that launch a command. stromboli mcp calls your server with the API token:


        claude mcp add stromboli -e STROMBOLI_URL=https://stromboli.example.com -e STROMBOLI_API_TOKEN=... -- stromboli mcp
      

Over HTTP, at /mcp on the server, with STROMBOLI_MCP_ENABLED=true (requires STROMBOLI_API_TOKEN). Clients send Authorization: Bearer <token>. Set STROMBOLI_MCP_TOKEN to give MCP clients a token that reaches only these tools, not the whole API:


        claude mcp add --transport http stromboli https://stromboli.example.com/mcp --header "Authorization: Bearer $STROMBOLI_MCP_TOKEN"
      

Investigations started from MCP have source mcp, with the client's name in their alert.

As a Claude Code skill, Claude drives the stromboli CLI for you: briefs, investigations, the catalog, connectors and action approvals, with secrets kept off the command line and approvals left to a named human. Copy skills/stromboli into your skills folder:


        cp -R skills/stromboli ~/.claude/skills/
      

Development

The dev stack includes a demo app that triggers a fake incident after 3 minutes, so you can watch a full investigation locally:


        docker compose -f docker-compose.dev.yml up -d     # Postgres :55432, Loki :3100, demo app
export STROMBOLI_DB_URL="postgres://stromboli:stromboli@localhost:55432/stromboli?sslmode=disable" STROMBOLI_SECRET_KEY=dev
go run ./cmd/stromboli serve
go run ./cmd/stromboli connector add loki loki url=http://localhost:3100
go run ./cmd/stromboli catalog import examples/catalog/services.yaml --runbooks examples/catalog/runbooks
go run ./cmd/stromboli brief api
      

        go test ./...                # add STROMBOLI_TEST_DB_URL=... to run the end-to-end agent test
buf lint && buf generate     # after editing proto/
sqlc generate                # after editing internal/store/queries or migrations
      

Contributing

Contributions are welcome: read CONTRIBUTING.md to get started, and SECURITY.md to report a vulnerability privately. Everyone taking part follows the code of conduct.

License

stromboli is released under the MIT license.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by HN AI
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗