Update .kiro: remove SQLite hooks, add PostgreSQL migration hook, add workflow steering, sync specs

2026-05-12 14:45:58 -06:00
parent 3ee8487286
commit 1bb8ec1658
35 changed files with 4645 additions and 48 deletions
--- a/.kiro/specs/compliance-chart-replacements/design.md
+++ b/.kiro/specs/compliance-chart-replacements/design.md
@@ -0,0 +1,313 @@
+# Design Document: Compliance Chart Replacements
+
+## Overview
+
+This feature replaces two underperforming charts in the `ComplianceChartsPanel` with more actionable visualizations:
+
+| Slot | Old Chart | New Chart |
+|------|-----------|-----------|
+| Chart 4 | MTTR by Team (horizontal bar) | **Aging Findings Distribution** — stacked vertical bar showing active findings bucketed by `seen_count` with per-team color segments |
+| Chart 5 | Most Persistent Findings (horizontal bar) | **Net Change Waterfall** — waterfall bar chart showing per-cycle net movement (start → +new → +recurring → −resolved → end) |
+
+Both new charts derive data from columns already present in the SQLite schema (`compliance_items.seen_count`, `compliance_uploads.new_count / recurring_count / resolved_count`). No database migrations are required. The four other charts (Active Findings Over Time, Change per Cycle, Team Compliance Health, Archer Exception Pipeline) remain unchanged.
+
+The backend reuses the existing route paths (`/mttr` → aging, `/top-recurring` → waterfall) so no new API routes are introduced. The frontend replaces the two component functions and their corresponding state variables while preserving the shared infrastructure (`ChartCard`, `DarkTooltip`, `NoData`, `TEAM_COLORS`, `fmtDate`, style tokens).
+
+## Architecture
+
+The change is scoped to two files and follows the existing vertical-slice pattern:
+
+```mermaid
+graph LR
+    subgraph Backend ["Express Backend"]
+        R1["/compliance/mttr<br/>(was MTTR → now Aging)"]
+        R2["/compliance/top-recurring<br/>(was Persistent → now Waterfall)"]
+        DB[(SQLite<br/>compliance_items<br/>compliance_uploads)]
+        R1 -->|"SELECT … GROUP BY bucket, team"| DB
+        R2 -->|"SELECT … ORDER BY report_date"| DB
+    end
+
+    subgraph Frontend ["React Frontend"]
+        Panel["ComplianceChartsPanel"]
+        AC["AgingChart"]
+        WC["WaterfallChart"]
+        Panel --> AC
+        Panel --> WC
+    end
+
+    R1 -->|JSON| AC
+    R2 -->|JSON| WC
+```
+
+### Key Architectural Decisions
+
+1. **Route path reuse** — The old `/mttr` and `/top-recurring` paths are replaced in-place rather than creating new paths. This avoids route proliferation and means no proxy/CORS changes are needed.
+
+2. **No new dependencies** — Both charts are built with the existing Recharts `BarChart` / `Bar` components. The waterfall effect is achieved using a transparent "base" `Bar` with `fill="transparent"` to offset the visible segments, which is a standard Recharts stacking technique.
+
+3. **No database migration** — The aging chart queries `compliance_items.seen_count` (already populated by the upload pipeline). The waterfall chart reads `new_count`, `recurring_count`, `resolved_count` from `compliance_uploads` (already written by `persistUpload`). Starting/ending counts are computed in the route handler via a running accumulator.
+
+4. **Shared component reuse** — Both new chart components use the existing `ChartCard`, `DarkTooltip`, `NoData`, `TEAM_COLORS`, `AXIS_STYLE`, `GRID_STYLE`, and `LEGEND_STYLE` tokens already defined in `ComplianceChartsPanel.js`.
+
+## Components and Interfaces
+
+### Backend: Aging Endpoint (replaces `/mttr`)
+
+**Route:** `GET /compliance/mttr`
+
+**SQL Query:**
+```sql
+SELECT
+  CASE
+    WHEN seen_count = 1       THEN '1 cycle'
+    WHEN seen_count BETWEEN 2 AND 3 THEN '2–3 cycles'
+    WHEN seen_count BETWEEN 4 AND 6 THEN '4–6 cycles'
+    ELSE '7+ cycles'
+  END AS bucket,
+  team,
+  COUNT(*) AS count
+FROM compliance_items
+WHERE status = 'active'
+GROUP BY bucket, team
+ORDER BY
+  CASE bucket
+    WHEN '1 cycle'     THEN 1
+    WHEN '2–3 cycles'  THEN 2
+    WHEN '4–6 cycles'  THEN 3
+    WHEN '7+ cycles'   THEN 4
+  END
+```
+
+**Response shape:**
+```json
+{
+  "aging": [
+    { "bucket": "1 cycle",     "total": 12, "STEAM": 5, "ACCESS-ENG": 3, "ACCESS-OPS": 2, "INTELDEV": 2 },
+    { "bucket": "2–3 cycles",  "total": 8,  "STEAM": 2, "ACCESS-ENG": 4, "ACCESS-OPS": 1, "INTELDEV": 1 },
+    { "bucket": "4–6 cycles",  "total": 3,  "STEAM": 1, "ACCESS-ENG": 0, "ACCESS-OPS": 2, "INTELDEV": 0 },
+    { "bucket": "7+ cycles",   "total": 2,  "STEAM": 0, "ACCESS-ENG": 1, "ACCESS-OPS": 0, "INTELDEV": 1 }
+  ]
+}
+```
+
+**Handler logic:**
+1. Query active items grouped by bucket CASE expression and team.
+2. Pivot the flat rows into one object per bucket with per-team counts and a total.
+3. Return the array sorted by bucket order (ascending age).
+
+### Backend: Waterfall Endpoint (replaces `/top-recurring`)
+
+**Route:** `GET /compliance/top-recurring`
+
+**SQL Query:**
+```sql
+SELECT id, report_date,
+       COALESCE(new_count, 0)       AS new_count,
+       COALESCE(recurring_count, 0) AS recurring_count,
+       COALESCE(resolved_count, 0)  AS resolved_count
+FROM compliance_uploads
+ORDER BY report_date ASC
+```
+
+**Response shape:**
+```json
+{
+  "waterfall": [
+    {
+      "date": "2026-04-13",
+      "start": 0,
+      "new_count": 15,
+      "recurring_count": 0,
+      "resolved_count": 0,
+      "end": 15
+    },
+    {
+      "date": "2026-04-20",
+      "start": 15,
+      "new_count": 3,
+      "recurring_count": 12,
+      "resolved_count": 5,
+      "end": 25
+    }
+  ]
+}
+```
+
+**Handler logic:**
+1. Fetch all uploads ordered by `report_date ASC`.
+2. Iterate with a running accumulator: `start` for cycle N = `end` of cycle N−1 (first cycle starts at 0).
+3. Compute `end = start + new_count + recurring_count - resolved_count`.
+4. Return the array.
+
+### Frontend: `AgingChart` Component
+
+Replaces `MttrChart`. Renders a vertical stacked `BarChart`:
+- **X-axis:** bucket labels (`1 cycle`, `2–3 cycles`, `4–6 cycles`, `7+ cycles`)
+- **Y-axis:** finding count
+- **Stacked bars:** one `<Bar>` per team key from `TEAM_COLORS`, using `stackId="aging"`
+- **Tooltip:** `DarkTooltip` showing bucket label, per-team counts, and total
+- **Empty state:** `<NoData />` when `aging` array is empty
+- **Wrapper:** `<ChartCard title="Aging Findings Distribution" subtitle="Active findings by age bucket — stacked by team" />`
+
+### Frontend: `WaterfallChart` Component
+
+Replaces `RecurringChart`. Renders a vertical `BarChart` with stacked transparent base:
+- **X-axis:** cycle dates formatted via `fmtDate`
+- **Y-axis:** finding count
+- **Bars (stacked):**
+  1. `<Bar dataKey="start" stackId="w" fill="transparent" />` — invisible base
+  2. `<Bar dataKey="new_count" stackId="w" fill="#EF4444" />` — red new findings
+  3. `<Bar dataKey="recurring_count" stackId="w" fill="#F59E0B" />` — amber recurring
+- **Separate bar:** `<Bar dataKey="resolved_count" fill="#10B981" />` rendered as a standalone (not stacked) to visually represent the downward resolution. Since Recharts doesn't natively support negative waterfall segments, the resolved bar is rendered separately alongside the stacked group, matching the existing `DeltaChart` pattern already used in Chart 2.
+- **Tooltip:** Custom `WaterfallTooltip` (extends `DarkTooltip` pattern) showing date, start, new, recurring, resolved, and end values.
+- **Empty state:** `<NoData />` when `waterfall` array is empty
+- **Wrapper:** `<ChartCard title="Net Change Waterfall" subtitle="Per-cycle net movement: start → +new → +recurring → −resolved → end" />`
+
+### Frontend: State & Fetch Changes in `ComplianceChartsPanel`
+
+| Old | New |
+|-----|-----|
+| `const [mttr, setMttr] = useState([])` | `const [aging, setAging] = useState([])` |
+| `const [recurring, setRecurring] = useState([])` | `const [waterfall, setWaterfall] = useState([])` |
+| `fetch(…/compliance/mttr)` → `setMttr(d.mttr)` | `fetch(…/compliance/mttr)` → `setAging(d.aging)` |
+| `fetch(…/compliance/top-recurring)` → `setRecurring(d.items)` | `fetch(…/compliance/top-recurring)` → `setWaterfall(d.waterfall)` |
+
+The `Promise.all` pattern and error handling remain identical. The other two fetches (`/trends`, `/archer-tickets/status-trend`) are untouched.
+
+### Code Removal
+
+- Delete `MttrChart` function component
+- Delete `RecurringChart` function component
+- Remove `mttr` and `recurring` state variables and their setter calls
+- Remove old `/mttr` handler SQL and logic in `compliance.js`
+- Remove old `/top-recurring` handler SQL and logic in `compliance.js`
+
+## Data Models
+
+No schema changes are required. The feature reads from existing columns:
+
+### `compliance_items` (read by Aging endpoint)
+
+| Column | Type | Usage |
+|--------|------|-------|
+| `status` | TEXT (`'active'` / `'resolved'`) | Filter to active items only |
+| `seen_count` | INTEGER (default 1) | Bucketed into age groups via CASE expression |
+| `team` | TEXT | Group-by for per-team stacked segments |
+
+### `compliance_uploads` (read by Waterfall endpoint)
+
+| Column | Type | Usage |
+|--------|------|-------|
+| `report_date` | TEXT (YYYY-MM-DD) | X-axis label, ordering |
+| `new_count` | INTEGER (default 0) | New findings in this cycle |
+| `recurring_count` | INTEGER (default 0) | Recurring findings in this cycle |
+| `resolved_count` | INTEGER (default 0) | Resolved findings in this cycle |
+
+### Derived Waterfall Fields (computed in handler, not stored)
+
+| Field | Computation |
+|-------|-------------|
+| `start` | Previous cycle's `end` (0 for first cycle) |
+| `end` | `start + new_count + recurring_count - resolved_count` |
+
+
+## Correctness Properties
+
+*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
+
+### Property 1: Aging bucketing correctness and conservation
+
+*For any* array of active compliance items with arbitrary `seen_count` values (≥ 1) and arbitrary team assignments, applying the aging bucketing logic SHALL:
+- assign every item to exactly one of the four buckets (`1 cycle` for seen_count = 1, `2–3 cycles` for 2 ≤ seen_count ≤ 3, `4–6 cycles` for 4 ≤ seen_count ≤ 6, `7+ cycles` for seen_count ≥ 7),
+- produce per-team counts within each bucket that sum to the bucket's total count, and
+- produce bucket totals that sum to the total number of input items.
+
+**Validates: Requirements 1.1, 1.2**
+
+### Property 2: Waterfall chain linkage and arithmetic invariant
+
+*For any* ordered sequence of upload records with non-negative `new_count`, `recurring_count`, and `resolved_count` values, computing the waterfall SHALL produce a result where:
+- the first row has `start = 0`,
+- for every subsequent row N, `start[N] = end[N-1]` (chain linkage), and
+- for every row, `end = start + new_count + recurring_count - resolved_count` (arithmetic invariant).
+
+**Validates: Requirements 3.1, 3.2, 3.3**
+
+## Error Handling
+
+### Backend
+
+| Scenario | Behavior |
+|----------|----------|
+| No active findings (aging endpoint) | Return `{ aging: [] }` with HTTP 200 |
+| No uploads (waterfall endpoint) | Return `{ waterfall: [] }` with HTTP 200 |
+| SQLite query error (either endpoint) | Log error to console, return HTTP 500 with `{ error: "Database error" }` |
+| `seen_count` is NULL or missing | `COALESCE(seen_count, 1)` in the CASE expression treats NULL as 1 cycle |
+| `new_count` / `recurring_count` / `resolved_count` is NULL | `COALESCE(..., 0)` in the SELECT ensures zero-default (already present in the uploads table pattern) |
+
+### Frontend
+
+| Scenario | Behavior |
+|----------|----------|
+| Aging fetch fails (network error or non-200) | `aging` state remains `[]`, `AgingChart` renders `<NoData />` |
+| Waterfall fetch fails (network error or non-200) | `waterfall` state remains `[]`, `WaterfallChart` renders `<NoData />` |
+| One fetch fails, others succeed | Unaffected charts render normally — each fetch result is handled independently in the existing `try/catch` pattern |
+| API returns unexpected shape | Component guards (`data.length === 0`) trigger `<NoData />` gracefully |
+
+## Testing Strategy
+
+### Property-Based Tests (fast-check)
+
+The project backend already uses Jest. Property-based tests will use [fast-check](https://github.com/dubzzz/fast-check) (already available in the Node ecosystem, zero-config with Jest).
+
+Each property test runs a minimum of **100 iterations** with randomly generated inputs.
+
+**Test 1: Aging bucketing correctness and conservation**
+- **Tag:** `Feature: compliance-chart-replacements, Property 1: Aging bucketing correctness and conservation`
+- **Generator:** Array of objects `{ seen_count: fc.integer({ min: 1, max: 200 }), team: fc.constantFrom('STEAM', 'ACCESS-ENG', 'ACCESS-OPS', 'INTELDEV') }`
+- **Assertion:** Extract the pure bucketing/pivoting function from the route handler. For each generated array, verify:
+  1. Every item maps to exactly one of the four bucket labels
+  2. Per-team counts in each bucket sum to the bucket total
+  3. All bucket totals sum to the input array length
+
+**Test 2: Waterfall chain linkage and arithmetic invariant**
+- **Tag:** `Feature: compliance-chart-replacements, Property 2: Waterfall chain linkage and arithmetic invariant`
+- **Generator:** Array of objects `{ new_count: fc.nat({ max: 50 }), recurring_count: fc.nat({ max: 50 }), resolved_count: fc.nat({ max: 50 }), report_date: fc.date().map(d => d.toISOString().slice(0, 10)) }`
+- **Assertion:** Extract the pure waterfall computation function. For each generated sequence, verify:
+  1. `result[0].start === 0`
+  2. For all i > 0: `result[i].start === result[i-1].end`
+  3. For all i: `result[i].end === result[i].start + result[i].new_count + result[i].recurring_count - result[i].resolved_count`
+
+### Unit Tests (Jest, example-based)
+
+| Test | Validates |
+|------|-----------|
+| Aging endpoint returns correct JSON shape with known seed data | Req 1.3 |
+| Aging endpoint returns empty array when no active items exist | Req 1.4 |
+| Aging endpoint returns 500 on DB error | Req 1.5 |
+| Waterfall endpoint returns empty array when no uploads exist | Req 3.5 |
+| Waterfall endpoint returns 500 on DB error | Req 3.6 |
+| `AgingChart` renders `<NoData />` for empty data | Req 2.5 |
+| `AgingChart` renders ChartCard with correct title | Req 2.3 |
+| `WaterfallChart` renders `<NoData />` for empty data | Req 4.5 |
+| `WaterfallChart` renders ChartCard with correct title | Req 4.6 |
+
+### Integration Tests
+
+| Test | Validates |
+|------|-----------|
+| Dashboard fetches `/compliance/mttr` and `/compliance/top-recurring` on mount | Req 5.1, 5.2 |
+| Dashboard still fetches `/compliance/trends` and `/archer-tickets/status-trend` | Req 5.3 |
+| All four fetches run in parallel via `Promise.all` | Req 5.4 |
+| Single fetch failure does not break other charts | Req 5.5 |
+
+### Smoke Tests
+
+| Test | Validates |
+|------|-----------|
+| `/compliance/mttr` route responds and does NOT return old `{ mttr: [...] }` shape | Req 1.6 |
+| `/compliance/top-recurring` route responds and does NOT return old `{ items: [...] }` shape | Req 3.7 |
+
+### Testability Design Note
+
+To enable property-based testing, the bucketing logic and waterfall accumulator will be extracted as **pure exported functions** (e.g., `bucketAgingItems(items)` and `computeWaterfall(uploads)`) separate from the Express route handlers. This allows the property tests to call the pure functions directly without needing HTTP or database mocking.