▋
platform/sst-ops/SKILL.md

name: sst-ops description: >- sst (Ion) deploy and state operations: bridging aws-login credentials, diagnosing stale locks, targeted-deploy/refresh hazards, and repairing a corrupted Pulumi snapshot. Use when running sst deploy/diff/refresh/unlock, when sst reports missing AWS credentials, a concurrent-update lock, or "Your app state is corrupted".

sst-ops

Operational gotchas for sst (Ion, v4+), learned the expensive way. sst embeds Pulumi with a DIY backend: state lives in the bootstrap S3 bucket at s3://<bootstrap-state-bucket>/app/<app>/<stage>.json, is pulled locally per command, and pushed back after.

Credentials: aws login sessions

The AWS CLI's newer aws login flow (~/.aws/config has login_session, no credentials file) works for the CLI but is invisible to sst/Pulumi's Go SDK — sst falls through to EC2 IMDS and reports "AWS credentials are not configured" even though aws sts get-caller-identity succeeds. Bridge it per-command:

eval "$(aws configure export-credentials --format env)" && npx sst deploy ...

Also check for an empty-string AWS_PROFILE in the environment; the CLI treats it as unset but Go SDK credential chains can choke on it.

Locks

Locked: A concurrent update was detected often means an earlier deploy died mid-flight, not that one is running. Before sst unlock, check ps for live sst/pulumi processes (and consider other machines/CI). A dead deploy may have half-applied cloud changes — e.g. a DynamoDB GSI still in CREATING — so after unlocking, expect the next deploy to hit ResourceInUseException until AWS settles. Poll the resource to ACTIVE first rather than retrying blind.

Targeted operations

  • sst deploy --target <Resource> works and is the right tool for converging one drifted resource without touching the rest of the stack (it passes --target-dependents to Pulumi).
  • Do not run sst refresh --target <Resource>. Observed on sst 4.17 / Pulumi 3.215: the targeted refresh rewrote the snapshot with duplicate live copies of dozens of resources ("duplicate resource … not marked for deletion"), after which every command failed with "Your app state is corrupted" and sst state repair no-oped. Refresh the whole stage or not at all.

Snapshot repair (when sst state repair no-ops)

Duplicate URNs whose extra copy has delete: true are legal pending-deletes; the integrity failure is URNs with two live (delete absent/false) copies.

  1. Back up, then pull the state: aws s3 cp s3://<state-bucket>/app/<app>/<stage>.json ./state.json (copy a timestamped backup object in S3 first).
  2. Walk checkpoint.latest.resources in order: keep every delete:true entry; keep the first live copy of each URN; drop later live duplicates. Order stays valid because the first block is the refreshed canonical ordering (parents before dependents).
  3. Verify zero live duplicates remain, upload back, re-run the command. A successful read (even if the command later fails for unrelated local reasons) confirms integrity is restored.

Misc

  • Treat sst diff output as sensitive. Components backed by Pulumi command:local:Command (including the Next.js builder) can include the current process environment in the raw property diff, which may expose temporary cloud credentials or application secrets. Inspect it only in a private terminal and filter/redact before sharing or recording it.
  • sst diff/deploy evaluates the whole program even with --target, so a Next.js site component fails on missing .open-next/**/resource.enc when the site was never built locally. That is a missing local artifact, not state corruption; a full deploy rebuilds it.
  • After repairing state or any manual S3 state surgery, run the cheapest command that reads state (sst diff --target <Resource>) to validate before attempting a real deploy.