Operations runbook

Environments and projects

Env GCP project URL Branch / trigger
sandbox snappy-agents-sandbox https://sandbox.agents.snappy.com every merge to main
prod snappy-agents-prod https://agents.snappy.com manual workflow_dispatch (env prod) with GitHub environment approval

Each project contains: VPC + Cloud NAT (static egress IP for Snappy allowlisting), Cloud SQL Postgres (private IP), Cloud Run service agents-api, Cloud Tasks queue, Cloud Scheduler (expire checkouts every 5 min), Secret Manager, Artifact Registry, global HTTPS load balancer with managed certificate, uptime check and alerting. See infra/terraform.

Credentials

Platform secrets (Terraform creates the containers, an operator adds the first version once):

Secret Source
auth-token-secret 32+ random bytes; rotating it signs every user and agent out
tasks-shared-secret Random; only used when TASKS_MODE=inline

Provider credentials are entered in the admin portal → Integrations and written to Secret Manager by the app (runtime identity has secretVersionAdder on those secrets only):

Integration Fields Where to get them
Snappy Public API API key (secret), account id, billing method id, webhook token (secret) Snappy dashboard → Company Settings → Sharing & Access → API Access; "Test connection" lists the active billing methods so you can pick the id
Stripe secret key, webhook signing secret Stripe dashboard (test keys on sandbox)
SendGrid API key, from email/name SendGrid
Twilio account SID, auth token (secret), Messaging Service SID (10DLC), fallback number Twilio console; register the brand + campaign for A2P 10DLC first

Until a provider is configured the platform degrades gracefully: fixture catalog and fake pay page on sandbox, sink messaging everywhere, and a clear *_not_configured error on prod.

One-time setup per environment

  1. tofu apply in infra/terraform/bootstrap (projects, state buckets, WIF). Then envs/<env>.
  2. Add the two platform secret versions (table above).
  3. Point DNS: agents.snappy.com / sandbox.agents.snappy.com A records → the load balancer IP from tofu output.
  4. Stripe: add webhook endpoint https://<host>/webhooks/stripe for checkout.session.completed, checkout.session.expired, checkout.session.async_payment_succeeded, refund.updated.
  5. Snappy dashboard: add webhook https://<host>/webhooks/snappy for order status, delivery status, order canceled, order out of stock. Snappy sends a verification token; the endpoint acknowledges it.
  6. Snappy: allowlist the Cloud NAT egress IP if the partner key is IP-restricted.
  7. Open https://<host>/admin (Identity-Aware Proxy asks for your Google Workspace account), go to Integrations and enter the Snappy key, account and billing method, then Stripe, SendGrid and Twilio.
  8. The first admin comes from ADMIN_EMAILS (bootstrap); manage the rest in the portal.

Admin portal

https://<host>/admin — behind Google Identity-Aware Proxy (@snappy.com accounts), allowlisted operators.

Page What you can do
Dashboard Users, orders, revenue and fees, checkout funnel
Users Search, detail (orders, checkouts, agent grants, messages), suspend, revoke agent tokens
Orders Filter by stage, detail with money breakdown and timeline, resync from Snappy, resend receipt, cancel and refund
Checkouts Funnel view incl. partially confirmed / failed / refunded with per-item reasons
Support Returns, damaged, wrong item, not received: take, refund (partial/full on the original payment), resolve, reject; the user is emailed
Integrations Snappy, Stripe, SendGrid, Twilio credentials (write-only, stored in Secret Manager) with "Test connection"
Agent clients Pre-provision or disable OAuth clients
Settings Service fee, ship-to countries, featured collections, order cap, maintenance mode, support contacts
Admins, Audit log, Webhooks Operators, every privileged action, raw delivery log

Spend cap

Each project carries a $100/month budget (infra/terraform/modules/budget, monthly_budget_usd) with email alerts at 50 %, 90 % and 100 % of actual spend and at 100 % of forecasted spend. GCP cannot stop spending by itself, so a kill switch is attached: budget notifications go to the billing-budget-alerts Pub/Sub topic and the billing-cap Cloud Run function detaches the project from the billing account when actual spend reaches the budget. Every paid resource then stops (Cloud Run, Cloud SQL, the load balancer) until billing is re-attached:

gcloud billing projects link snappy-agents-sandbox --billing-account 017C25-AFCFFA-B1502A

Raise the cap by changing monthly_budget_usd in envs/<env>/variables.tf; set hard_cap = false to keep alerts only.

Alerts

Cloud Monitoring policies (Terraform): 5xx ratio, uptime check on /readyz, Cloud Tasks queue depth, and a log-based alert on checkout.failed. Notification channel is an email group; add Slack via a Cloud Monitoring channel if desired.

Common procedures

Snappy Agents · agents.snappy.com · support@snappy.com