# Design Document: Compliance Chart Replacements

## Overview

This feature replaces two underperforming charts in the `ComplianceChartsPanel` with more actionable visualizations:

| Slot | Old Chart | New Chart |
|------|-----------|-----------|
| Chart 4 | MTTR by Team (horizontal bar) | **Aging Findings Distribution** — stacked vertical bar showing active findings bucketed by `seen_count` with per-team color segments |
| Chart 5 | Most Persistent Findings (horizontal bar) | **Net Change Waterfall** — waterfall bar chart showing per-cycle net movement (start → +new → +recurring → −resolved → end) |

Both new charts derive data from columns already present in the SQLite schema (`compliance_items.seen_count`, `compliance_uploads.new_count / recurring_count / resolved_count`). No database migrations are required. The four other charts (Active Findings Over Time, Change per Cycle, Team Compliance Health, Archer Exception Pipeline) remain unchanged.

The backend reuses the existing route paths (`/mttr` → aging, `/top-recurring` → waterfall) so no new API routes are introduced. The frontend replaces the two component functions and their corresponding state variables while preserving the shared infrastructure (`ChartCard`, `DarkTooltip`, `NoData`, `TEAM_COLORS`, `fmtDate`, style tokens).

## Architecture

The change is scoped to two files and follows the existing vertical-slice pattern:

```mermaid
graph LR
    subgraph Backend ["Express Backend"]
        R1["/compliance/mttr<br/>(was MTTR → now Aging)"]
        R2["/compliance/top-recurring<br/>(was Persistent → now Waterfall)"]
        DB[(SQLite<br/>compliance_items<br/>compliance_uploads)]
        R1 -->|"SELECT … GROUP BY bucket, team"| DB
        R2 -->|"SELECT … ORDER BY report_date"| DB
    end

    subgraph Frontend ["React Frontend"]
        Panel["ComplianceChartsPanel"]
        AC["AgingChart"]
        WC["WaterfallChart"]
        Panel --> AC
        Panel --> WC
    end

    R1 -->|JSON| AC
    R2 -->|JSON| WC
```

### Key Architectural Decisions

1. **Route path reuse** — The old `/mttr` and `/top-recurring` paths are replaced in-place rather than creating new paths. This avoids route proliferation and means no proxy/CORS changes are needed.

2. **No new dependencies** — Both charts are built with the existing Recharts `BarChart` / `Bar` components. The waterfall effect is achieved using a transparent "base" `Bar` with `fill="transparent"` to offset the visible segments, which is a standard Recharts stacking technique.

3. **No database migration** — The aging chart queries `compliance_items.seen_count` (already populated by the upload pipeline). The waterfall chart reads `new_count`, `recurring_count`, `resolved_count` from `compliance_uploads` (already written by `persistUpload`). Starting/ending counts are computed in the route handler via a running accumulator.

4. **Shared component reuse** — Both new chart components use the existing `ChartCard`, `DarkTooltip`, `NoData`, `TEAM_COLORS`, `AXIS_STYLE`, `GRID_STYLE`, and `LEGEND_STYLE` tokens already defined in `ComplianceChartsPanel.js`.

## Components and Interfaces

### Backend: Aging Endpoint (replaces `/mttr`)

**Route:** `GET /compliance/mttr`

**SQL Query:**
```sql
SELECT
  CASE
    WHEN seen_count = 1       THEN '1 cycle'
    WHEN seen_count BETWEEN 2 AND 3 THEN '2–3 cycles'
    WHEN seen_count BETWEEN 4 AND 6 THEN '4–6 cycles'
    ELSE '7+ cycles'
  END AS bucket,
  team,
  COUNT(*) AS count
FROM compliance_items
WHERE status = 'active'
GROUP BY bucket, team
ORDER BY
  CASE bucket
    WHEN '1 cycle'     THEN 1
    WHEN '2–3 cycles'  THEN 2
    WHEN '4–6 cycles'  THEN 3
    WHEN '7+ cycles'   THEN 4
  END
```

**Response shape:**
```json
{
  "aging": [
    { "bucket": "1 cycle",     "total": 12, "STEAM": 5, "ACCESS-ENG": 3, "ACCESS-OPS": 2, "INTELDEV": 2 },
    { "bucket": "2–3 cycles",  "total": 8,  "STEAM": 2, "ACCESS-ENG": 4, "ACCESS-OPS": 1, "INTELDEV": 1 },
    { "bucket": "4–6 cycles",  "total": 3,  "STEAM": 1, "ACCESS-ENG": 0, "ACCESS-OPS": 2, "INTELDEV": 0 },
    { "bucket": "7+ cycles",   "total": 2,  "STEAM": 0, "ACCESS-ENG": 1, "ACCESS-OPS": 0, "INTELDEV": 1 }
  ]
}
```

**Handler logic:**
1. Query active items grouped by bucket CASE expression and team.
2. Pivot the flat rows into one object per bucket with per-team counts and a total.
3. Return the array sorted by bucket order (ascending age).

### Backend: Waterfall Endpoint (replaces `/top-recurring`)

**Route:** `GET /compliance/top-recurring`

**SQL Query:**
```sql
SELECT id, report_date,
       COALESCE(new_count, 0)       AS new_count,
       COALESCE(recurring_count, 0) AS recurring_count,
       COALESCE(resolved_count, 0)  AS resolved_count
FROM compliance_uploads
ORDER BY report_date ASC
```

**Response shape:**
```json
{
  "waterfall": [
    {
      "date": "2026-04-13",
      "start": 0,
      "new_count": 15,
      "recurring_count": 0,
      "resolved_count": 0,
      "end": 15
    },
    {
      "date": "2026-04-20",
      "start": 15,
      "new_count": 3,
      "recurring_count": 12,
      "resolved_count": 5,
      "end": 25
    }
  ]
}
```

**Handler logic:**
1. Fetch all uploads ordered by `report_date ASC`.
2. Iterate with a running accumulator: `start` for cycle N = `end` of cycle N−1 (first cycle starts at 0).
3. Compute `end = start + new_count + recurring_count - resolved_count`.
4. Return the array.

### Frontend: `AgingChart` Component

Replaces `MttrChart`. Renders a vertical stacked `BarChart`:
- **X-axis:** bucket labels (`1 cycle`, `2–3 cycles`, `4–6 cycles`, `7+ cycles`)
- **Y-axis:** finding count
- **Stacked bars:** one `<Bar>` per team key from `TEAM_COLORS`, using `stackId="aging"`
- **Tooltip:** `DarkTooltip` showing bucket label, per-team counts, and total
- **Empty state:** `<NoData />` when `aging` array is empty
- **Wrapper:** `<ChartCard title="Aging Findings Distribution" subtitle="Active findings by age bucket — stacked by team" />`

### Frontend: `WaterfallChart` Component

Replaces `RecurringChart`. Renders a vertical `BarChart` with stacked transparent base:
- **X-axis:** cycle dates formatted via `fmtDate`
- **Y-axis:** finding count
- **Bars (stacked):**
  1. `<Bar dataKey="start" stackId="w" fill="transparent" />` — invisible base
  2. `<Bar dataKey="new_count" stackId="w" fill="#EF4444" />` — red new findings
  3. `<Bar dataKey="recurring_count" stackId="w" fill="#F59E0B" />` — amber recurring
- **Separate bar:** `<Bar dataKey="resolved_count" fill="#10B981" />` rendered as a standalone (not stacked) to visually represent the downward resolution. Since Recharts doesn't natively support negative waterfall segments, the resolved bar is rendered separately alongside the stacked group, matching the existing `DeltaChart` pattern already used in Chart 2.
- **Tooltip:** Custom `WaterfallTooltip` (extends `DarkTooltip` pattern) showing date, start, new, recurring, resolved, and end values.
- **Empty state:** `<NoData />` when `waterfall` array is empty
- **Wrapper:** `<ChartCard title="Net Change Waterfall" subtitle="Per-cycle net movement: start → +new → +recurring → −resolved → end" />`

### Frontend: State & Fetch Changes in `ComplianceChartsPanel`

| Old | New |
|-----|-----|
| `const [mttr, setMttr] = useState([])` | `const [aging, setAging] = useState([])` |
| `const [recurring, setRecurring] = useState([])` | `const [waterfall, setWaterfall] = useState([])` |
| `fetch(…/compliance/mttr)` → `setMttr(d.mttr)` | `fetch(…/compliance/mttr)` → `setAging(d.aging)` |
| `fetch(…/compliance/top-recurring)` → `setRecurring(d.items)` | `fetch(…/compliance/top-recurring)` → `setWaterfall(d.waterfall)` |

The `Promise.all` pattern and error handling remain identical. The other two fetches (`/trends`, `/archer-tickets/status-trend`) are untouched.

### Code Removal

- Delete `MttrChart` function component
- Delete `RecurringChart` function component
- Remove `mttr` and `recurring` state variables and their setter calls
- Remove old `/mttr` handler SQL and logic in `compliance.js`
- Remove old `/top-recurring` handler SQL and logic in `compliance.js`

## Data Models

No schema changes are required. The feature reads from existing columns:

### `compliance_items` (read by Aging endpoint)

| Column | Type | Usage |
|--------|------|-------|
| `status` | TEXT (`'active'` / `'resolved'`) | Filter to active items only |
| `seen_count` | INTEGER (default 1) | Bucketed into age groups via CASE expression |
| `team` | TEXT | Group-by for per-team stacked segments |

### `compliance_uploads` (read by Waterfall endpoint)

| Column | Type | Usage |
|--------|------|-------|
| `report_date` | TEXT (YYYY-MM-DD) | X-axis label, ordering |
| `new_count` | INTEGER (default 0) | New findings in this cycle |
| `recurring_count` | INTEGER (default 0) | Recurring findings in this cycle |
| `resolved_count` | INTEGER (default 0) | Resolved findings in this cycle |

### Derived Waterfall Fields (computed in handler, not stored)

| Field | Computation |
|-------|-------------|
| `start` | Previous cycle's `end` (0 for first cycle) |
| `end` | `start + new_count + recurring_count - resolved_count` |


## Correctness Properties

*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*

### Property 1: Aging bucketing correctness and conservation

*For any* array of active compliance items with arbitrary `seen_count` values (≥ 1) and arbitrary team assignments, applying the aging bucketing logic SHALL:
- assign every item to exactly one of the four buckets (`1 cycle` for seen_count = 1, `2–3 cycles` for 2 ≤ seen_count ≤ 3, `4–6 cycles` for 4 ≤ seen_count ≤ 6, `7+ cycles` for seen_count ≥ 7),
- produce per-team counts within each bucket that sum to the bucket's total count, and
- produce bucket totals that sum to the total number of input items.

**Validates: Requirements 1.1, 1.2**

### Property 2: Waterfall chain linkage and arithmetic invariant

*For any* ordered sequence of upload records with non-negative `new_count`, `recurring_count`, and `resolved_count` values, computing the waterfall SHALL produce a result where:
- the first row has `start = 0`,
- for every subsequent row N, `start[N] = end[N-1]` (chain linkage), and
- for every row, `end = start + new_count + recurring_count - resolved_count` (arithmetic invariant).

**Validates: Requirements 3.1, 3.2, 3.3**

## Error Handling

### Backend

| Scenario | Behavior |
|----------|----------|
| No active findings (aging endpoint) | Return `{ aging: [] }` with HTTP 200 |
| No uploads (waterfall endpoint) | Return `{ waterfall: [] }` with HTTP 200 |
| SQLite query error (either endpoint) | Log error to console, return HTTP 500 with `{ error: "Database error" }` |
| `seen_count` is NULL or missing | `COALESCE(seen_count, 1)` in the CASE expression treats NULL as 1 cycle |
| `new_count` / `recurring_count` / `resolved_count` is NULL | `COALESCE(..., 0)` in the SELECT ensures zero-default (already present in the uploads table pattern) |

### Frontend

| Scenario | Behavior |
|----------|----------|
| Aging fetch fails (network error or non-200) | `aging` state remains `[]`, `AgingChart` renders `<NoData />` |
| Waterfall fetch fails (network error or non-200) | `waterfall` state remains `[]`, `WaterfallChart` renders `<NoData />` |
| One fetch fails, others succeed | Unaffected charts render normally — each fetch result is handled independently in the existing `try/catch` pattern |
| API returns unexpected shape | Component guards (`data.length === 0`) trigger `<NoData />` gracefully |

## Testing Strategy

### Property-Based Tests (fast-check)

The project backend already uses Jest. Property-based tests will use [fast-check](https://github.com/dubzzz/fast-check) (already available in the Node ecosystem, zero-config with Jest).

Each property test runs a minimum of **100 iterations** with randomly generated inputs.

**Test 1: Aging bucketing correctness and conservation**
- **Tag:** `Feature: compliance-chart-replacements, Property 1: Aging bucketing correctness and conservation`
- **Generator:** Array of objects `{ seen_count: fc.integer({ min: 1, max: 200 }), team: fc.constantFrom('STEAM', 'ACCESS-ENG', 'ACCESS-OPS', 'INTELDEV') }`
- **Assertion:** Extract the pure bucketing/pivoting function from the route handler. For each generated array, verify:
  1. Every item maps to exactly one of the four bucket labels
  2. Per-team counts in each bucket sum to the bucket total
  3. All bucket totals sum to the input array length

**Test 2: Waterfall chain linkage and arithmetic invariant**
- **Tag:** `Feature: compliance-chart-replacements, Property 2: Waterfall chain linkage and arithmetic invariant`
- **Generator:** Array of objects `{ new_count: fc.nat({ max: 50 }), recurring_count: fc.nat({ max: 50 }), resolved_count: fc.nat({ max: 50 }), report_date: fc.date().map(d => d.toISOString().slice(0, 10)) }`
- **Assertion:** Extract the pure waterfall computation function. For each generated sequence, verify:
  1. `result[0].start === 0`
  2. For all i > 0: `result[i].start === result[i-1].end`
  3. For all i: `result[i].end === result[i].start + result[i].new_count + result[i].recurring_count - result[i].resolved_count`

### Unit Tests (Jest, example-based)

| Test | Validates |
|------|-----------|
| Aging endpoint returns correct JSON shape with known seed data | Req 1.3 |
| Aging endpoint returns empty array when no active items exist | Req 1.4 |
| Aging endpoint returns 500 on DB error | Req 1.5 |
| Waterfall endpoint returns empty array when no uploads exist | Req 3.5 |
| Waterfall endpoint returns 500 on DB error | Req 3.6 |
| `AgingChart` renders `<NoData />` for empty data | Req 2.5 |
| `AgingChart` renders ChartCard with correct title | Req 2.3 |
| `WaterfallChart` renders `<NoData />` for empty data | Req 4.5 |
| `WaterfallChart` renders ChartCard with correct title | Req 4.6 |

### Integration Tests

| Test | Validates |
|------|-----------|
| Dashboard fetches `/compliance/mttr` and `/compliance/top-recurring` on mount | Req 5.1, 5.2 |
| Dashboard still fetches `/compliance/trends` and `/archer-tickets/status-trend` | Req 5.3 |
| All four fetches run in parallel via `Promise.all` | Req 5.4 |
| Single fetch failure does not break other charts | Req 5.5 |

### Smoke Tests

| Test | Validates |
|------|-----------|
| `/compliance/mttr` route responds and does NOT return old `{ mttr: [...] }` shape | Req 1.6 |
| `/compliance/top-recurring` route responds and does NOT return old `{ items: [...] }` shape | Req 3.7 |

### Testability Design Note

To enable property-based testing, the bucketing logic and waterfall accumulator will be extracted as **pure exported functions** (e.g., `bucketAgingItems(items)` and `computeWaterfall(uploads)`) separate from the Express route handlers. This allows the property tests to call the pure functions directly without needing HTTP or database mocking.