INSTAB-01] Instability safeguards: automated rollback, feature flags, and change volume limits

[paruff]

[INSTAB-01] Instability safeguards: automated rollback, feature flags, and change volume limits

Labels: devops, security, p1-critical, effort-medium
Assignee: 🤖 Copilot
Phase: Phase 4 — Quality Flywheel
DORA 2025 Source: “AI adoption is now linked to higher software delivery throughput AND increases instability” finding; “Control systems must speed up” guidance
Effort: Medium (1–3 days)
Prerequisites: DEVOPS-01 (CI), SEC-03 (Firestore rules), OBS-01 (Sentry)

---

The Critical 2025 DORA Finding This Addresses

“AI adoption is linked to higher software delivery throughput. However, the ongoing challenge remains of ensuring software works as intended before it’s delivered to users. AI increases instability… Without robust control systems — strong automated testing, mature version control practices, and fast feedback loops — an increase in change volume leads to instability.”

The Faros AI telemetry data (independent of DORA, 2025) quantified this precisely: AI consistently increases PR size by 154%. Larger PRs = more surface area for bugs = higher change failure rate. The control system must be proportionally stronger.

This issue installs three safeguards:

1. Automated OTA rollback capability — when a deploy causes a Sentry spike, revert is one command
2. Change volume monitoring — alerts when PRs are abnormally large (already partially in PROCESS-01, this extends it)
3. Feature flag infrastructure — allows new features to be deployed but toggled off if they cause instability

---

Acceptance Criteria

1. OTA Rollback Runbook and Automation

• docs/RUNBOOKS.md updated with “Emergency OTA Rollback” procedure:

  1. Identify the bad OTA update: expo publish:history
  2. Roll back: expo publish:rollback --channel production --sdk-version X
  3. File a Sentry-linked GitHub issue immediately
  4. Root cause analysis before re-deploying
  

• Rollback command tested in staging environment and documented as working

• OTA update channel strategy documented: development, staging, production (aligns with DEVOPS-02)

• Sentry alert (from METRICS-02) triggers a GitHub issue automatically when change failure is detected

2. PR Size Alert Enhancement (extend PROCESS-01)

• CI PR size check (from PROCESS-01) updated to block (not just warn) PRs with > 600 changed lines unless a human overrides with a specific label: large-pr-approved
• large-pr-approved label added to GitHub labels — must be applied by a human reviewer, never by Copilot
• This prevents the 154% PR size inflation from Faros AI telemetry becoming a permanent pattern

3. Feature Flag Infrastructure

• A lightweight feature flag system implemented using src/config/featureFlags.ts:

  • Flags are defined as typed constants with enabled: boolean
  • Flags can be overridden by a Firestore document config/featureFlags (allows remote toggle without OTA)
  • Default: all flags true in development, controlled in staging/production

• New features from Phase 3 onward are gated behind a feature flag initially

• Flag schema:

  interface FeatureFlags {
    growthProjectionChart: boolean;   // PROD-04
    onboardingFlow: boolean;          // PROD-03
    withdrawalSupport: boolean;       // PROD-05
    pushNotifications: boolean;       // PROD-06/07
    shareGoalCompletion: boolean;     // PROD-08
  }

• docs/RUNBOOKS.md updated with “Disable a feature remotely” procedure:

1. Go to Firebase Console → Firestore → config/featureFlags
2. Set the problematic flag to false
3. OTA update not required — change takes effect on next app foreground

4. Fast Feedback Loop Verification

• CI pipeline total time measured and documented in docs/METRICS.md
• Target: CI must complete in < 4 minutes for the feedback loop to be fast enough to prevent context switching
• If CI takes > 4 minutes: file a follow-up issue to optimise (cache, parallelise, reduce E2E scope)

---

Files to Create / Change

• src/config/featureFlags.ts (create)
• src/hooks/useFeatureFlags.ts (create — reads from Firestore or defaults)
• docs/RUNBOOKS.md — add Emergency Rollback and Remote Feature Disable
• docs/METRICS.md — add CI pipeline time target
• .github/workflows/ci-quality.yml — update PR size check to block (not warn) on > 600 lines