Feature flag debt

---
name: signals-scout-flag-debt
description: >
  Finds feature flags whose decision is over but which are still evaluated in production –
  terminal rollouts left in place, or keys that no longer appear in any connected repo.
allowed_tools:
  - emit_report
  - edit_report
---

# Feature flag debt

Find flags whose decision is over while something is still evaluating them.

Narrower than `signals-scout-feature-flags`, which watches the whole flag surface – evaluation
cliffs, ghost keys, response-distribution shifts, flag debt – from PostHog data alone. This one
adds the half that scout cannot see: your repository. Run it if your flags routinely outlive
your cleanup PRs. If you want the analysis on demand rather than on a schedule, the
`cleaning-up-stale-feature-flags` and `finding-deleted-feature-flags` skills already do it, and
`auditing-experiments-flags` covers experiment-linked flags.

Weekly rather than the daily fleet default: flag debt accumulates over months, so a daily run
would re-read the same roster and write nothing six days out of seven.

**Term collision, worth knowing.** The canonical scout uses **ghost flag** for the opposite case
– code calling a key with no flag behind it, where the SDK silently returns `false`. That is a
different finding with a different fix. Don't reuse the term here.

## Discriminator

Emit a report when a flag is still emitting `$feature_flag_called` evaluations **and** either

- **(a)** it has been at 100% or 0% with no edits for 60+ days, or
- **(b)** its key does not appear in any connected repository.

Write nothing for a flag whose rollout percentage changed recently, or that was edited in the
last 30 days – that flag is mid-rollout and doing its job.

The two cases are not the same finding. **(a)** is cleanup you can do: the code is there and it
takes the same branch every time. **(b)** is a question – something is calling that key and it
isn't in the code you can see. Never treat (b) as a delete candidate on its own.

### Coverage caveat

`$feature_flag_called` fires on explicit evaluation calls. Server SDKs using local evaluation,
bootstrapped flags, and clients with flag-event capture disabled may not emit it at all, so
**absence of the event is weak evidence that a flag is unused**. Lean on PostHog's server-side
`STALE` status, which accounts for `last_called_at`, rather than on raw event absence.

## Explore

1. List flags with recent evaluations, plus their rollout percentage, tags, and last-modified
   date.
2. Search every connected repository for each candidate key. Three outcomes, not two:
   - **A string literal, or a constant that resolves to one** – a definite answer either way.
   - **Dynamically constructed** – a key assembled at the call site from a variable. A repo
     search can't resolve these, and they are the main source of false "the key is gone"
     conclusions. Treat a nearby prefix or template match as uncertainty, never as a delete
     recommendation.
   - **Not found anywhere** – case (b). Go to step 3 before concluding anything.
3. For a case (b) flag, work out what is still calling it. The plausible answers are all things
   this scout can't see: a repo that isn't connected, shipped mobile clients still running old
   code, or a third-party or SDK consumer. Name the possibilities in the report, and check the
   `$lib` and `$lib_version` spread on the evaluations – old client versions point at shipped
   code you can't deploy over.
4. Before recommending any removal, confirm the flag isn't load-bearing:
   - **Attached to an experiment**, with a non-empty `experiment_set` – leave it alone entirely;
     deleting it breaks the experiment and its results.
   - **Referenced by another flag's release conditions** – flag dependencies exist, and deleting
     the parent changes what the dependent serves.
   - **Consumed for its payload** – a flag can sit at 100% and be read only for its payload
     value, so the key shows up in a config fetch rather than in a branch.
   - **Tagged as excluded** – see the disqualifiers.
5. Quantify the reach: evaluations per day and distinct persons. That is what makes the report
   worth someone's afternoon. It is **not** a billing argument – `$feature_flag_called` is
   excluded from billable event volume – so don't put a dollar figure on it. The cost here is a
   code path nobody can reason about, not an invoice.
6. Bundle the findings into one report rather than filing one per flag, and keep the two cases
   visibly separate inside it.

## Suggested action

Deleting a flag is not a no-op. Once the flag is gone, evaluation returns `false` or `undefined`
– not the last value it served. Any live client still evaluating a deleted 100% flag flips to
the off path, which is a production behavior change.

So the order matters, and it isn't one pull request:

1. Remove the call sites, and deploy.
2. Confirm evaluations stop.
3. Then delete the flag.

Recommend step 1 – that's the PR an agent can open. Steps 2 and 3 are the follow-up you name in
the report, not something to bundle into the same diff.

## Disqualifiers

- The key appears in the repo, even only in tests or config.
- The key may be dynamically constructed and the search can't rule it out.
- The flag was edited recently, or its percentage is between 1 and 99.
- The flag is experiment-linked, has dependent flags, or is read for its payload.
- The flag carries an exclusion tag. Kill switches and ops toggles look exactly like debt and
  must never be recommended for removal – but most of them carry no description, so don't try to
  infer intent from prose. Ask the team to tag them (`keep`, `ops`, `kill-switch`) and read the
  tag. Record anything you're told to leave alone in your scratchpad so the exclusion survives
  to the next run. The durable way to stop a scout re-flagging something is to give it somewhere
  to remember, not to hope for good metadata.
- Coverage is partial. If any repo that evaluates flags isn't connected, case (b) findings are
  unsafe – report them as open questions only.