Handling wizard incidents

When the wizard breaks, reproduce it, declare an incident, then find the part that broke and fix or roll it back. If you think it'll hit the whole top of the funnel, declare the incident.

Reproduce it

1. Run the wizard against an app known to work. The wizard workbench has a test app for every framework.

   Terminal

   PostHog AI

   npx -y @posthog/wizard@latest

2. Read the log at /tmp/posthog-wizard.log. It's verbose, so an agent can help you find the failing step.

3. Run an older release with npx @posthog/wizard@<version>. If it works, compare its log with the current one. This also confirms a safe version to roll back to.

Find where it starts

The wizard depends on a few services. Check each one before you blame the wizard.

Symptom	Likely cause	Where to look
Login fails, or a scope is missing	OAuth	The OAuth apps in Django admin for US and EU. If OAuth isn't down, check the app's scopes and permissions.
The agent can't reach a model	The AI gateway	Ask in #team-agent-infra. The Agent Infrastructure TeamAgent Infrastructure Team owns the gateway.
Skills fail to download	GitHub releases	The wizard falls back to the S3 mirror at context-mill.posthog.com. Check that both serve the latest release.
Runs complete, but the integration is wrong	A bad context mill release	Compare the latest context mill release with the one before it.
Everything upstream is healthy	A bug in the wizard	Compare the current npm release with the last good one.

Roll back a wizard release

We don't move npm's latest tag back. Instead, publish a new version that contains the last good code, so everyone running @latest gets the fix.

1. Open a PR that reverts the bad commits. The title needs a conventional commit prefix, like revert: roll back to 2.81.0.
2. Merge it. release-please opens a release PR with the version bump.
3. Merge the release PR. The publish workflow publishes the new version to npm.
4. Run npx -y @posthog/wizard@latest and confirm the fix.

Roll back a context mill release

A context mill rollback is simpler, because the wizard always reads the release marked latest.

1. On the releases page, mark the last good release as latest.
2. Point the S3 mirror back at the same release. The mirror keeps every release, and context-mill.posthog.com/latest is a pointer in a CloudFront key-value store. Ask someone with AWS access to move its latest key to the good version, then check that https://context-mill.posthog.com/latest/skill-menu.json serves it.
3. Delete the bad release.
4. Fix the problem and ship a new release as usual.

Cloud runs use their own pinned version

Cloud wizard runs don't use npm's latest. Each program pins an exact wizard version in the payload of the wizard-program-registry feature flag, and PostHog installs that version in the sandbox. A wizard rollback on npm doesn't reach cloud runs, and a fix doesn't either until the pin moves.

If cloud runs are broken:

1. Reach out to the Growth TeamGrowth Team in #team-growth. They own cloud runs.
2. Together, update the wizard_version for the affected programs in the wizard-program-registry payload to a known good version.

If the payload is missing or invalid, cloud runs fall back to the default version set in the PostHog backend.