cve-dashboard/.kiro/specs/ccp-metrics-view-restructure/design.md

# Design Document

## Overview

This design restructures the CCP Metrics page from a vertical-first drill-down model to a metric-first model. The overview table changes from one row per vertical to one row per metric (aggregated across all verticals). Two new backend endpoints support the metric-centric view. The "By Vertical" table in the AggregatedBurndownChart is removed. Existing vertical-first endpoints are preserved for backward compatibility.

## Architecture

The restructure touches two layers:

1. **Backend** — Two new Express route handlers added to `backend/routes/vclMultiVertical.js`, querying `vcl_multi_vertical_summary` with aggregation across verticals.
2. **Frontend** — Replace `VerticalTable` and `VerticalDetailView` components with `MetricTable` and `MetricDetailView` in `frontend/src/components/pages/CCPMetricsPage.js`. Remove the "By Vertical" table JSX from `AggregatedBurndownChart`. Adjust drill-down state from `(selectedVertical, selectedMetric, selectedTeam)` to `(selectedMetric, selectedVertical, selectedTeam)`.

No database schema changes are required. No new tables or columns are needed — the existing `vcl_multi_vertical_summary` table already contains all data needed for metric-centric aggregation.

## Components and Interfaces

### Backend: New Endpoints

#### GET /api/compliance/vcl-multi/metrics

Aggregates all metrics across verticals using only `ALL:` rollup rows from the latest upload per vertical.

```javascript
// Query: get latest upload ID per vertical
const { rows: latestUploads } = await pool.query(`
    SELECT DISTINCT ON (vertical) id, vertical
    FROM compliance_uploads
    WHERE vertical IS NOT NULL
    ORDER BY vertical, id DESC
`);

const latestUploadIds = latestUploads.map(u => u.id);

// Aggregate metrics across verticals (ALL: rows only)
const { rows: metrics } = await pool.query(`
    SELECT metric_id,
           MAX(metric_desc) AS metric_desc,
           MAX(category) AS category,
           SUM(non_compliant)::int AS non_compliant,
           SUM(compliant)::int AS compliant,
           SUM(total)::int AS total,
           ROUND(AVG(target::numeric), 4) AS target
    FROM vcl_multi_vertical_summary
    WHERE upload_id = ANY($1) AND team LIKE 'ALL:%'
    GROUP BY metric_id
    ORDER BY non_compliant DESC
`, [latestUploadIds]);

// Compute compliance_pct for each metric
const result = metrics.map(m => ({
    ...m,
    compliance_pct: m.total > 0 ? m.compliant / m.total : 0,
}));
```

**Response shape:**
```json
{
  "metrics": [
    {
      "metric_id": "VM-001",
      "metric_desc": "Vulnerability Management - Patching",
      "category": "Vulnerability Management",
      "non_compliant": 450,
      "compliant": 3200,
      "total": 3650,
      "compliance_pct": 0.8767,
      "target": 0.95
    }
  ]
}
```

#### GET /api/compliance/vcl-multi/metric/:id/verticals

Returns per-vertical breakdown for a specific metric, including sub-team data within each vertical.

```javascript
const metricId = req.params.id;
if (!metricId || metricId.length > 50) {
    return res.status(400).json({ error: 'Invalid metric ID' });
}

// Get latest upload per vertical
const { rows: latestUploads } = await pool.query(`
    SELECT DISTINCT ON (vertical) id, vertical
    FROM compliance_uploads
    WHERE vertical IS NOT NULL
    ORDER BY vertical, id DESC
`);
const latestUploadIds = latestUploads.map(u => u.id);

// Get all rows for this metric from latest uploads
const { rows: allRows } = await pool.query(`
    SELECT vertical, metric_desc, category, team,
           non_compliant, compliant, total, compliance_pct, target
    FROM vcl_multi_vertical_summary
    WHERE upload_id = ANY($1) AND metric_id = $2
    ORDER BY vertical, team
`, [latestUploadIds, metricId]);

// Separate rollup rows (ALL:) from sub-team rows
// Build per-vertical entries with nested sub_teams
```

**Response shape:**
```json
{
  "metric_id": "VM-001",
  "metric_desc": "Vulnerability Management - Patching",
  "category": "Vulnerability Management",
  "verticals": [
    {
      "vertical": "NTS_AEO",
      "non_compliant": 200,
      "compliant": 1800,
      "total": 2000,
      "compliance_pct": 0.90,
      "target": 0.95,
      "sub_teams": [
        {
          "team": "STEAM",
          "non_compliant": 120,
          "compliant": 1080,
          "total": 1200,
          "compliance_pct": 0.90
        },
        {
          "team": "ACCESS-ENG",
          "non_compliant": 80,
          "compliant": 720,
          "total": 800,
          "compliance_pct": 0.90
        }
      ]
    }
  ]
}
```

### Frontend: Component Changes

#### AggregatedBurndownChart (Modified)

Remove the "By Vertical" contribution table JSX block. The component continues to receive `data.by_vertical` in its props (for backward compat with the `/burndown` API response) but no longer renders it.

**Before:** Summary header + bar chart + "By Vertical" table
**After:** Summary header + bar chart only

#### MetricTable (New — replaces VerticalTable)

Renders one row per metric from the `/metrics` endpoint response. Columns: Metric ID, Description, Category, Compliant, Non-Compliant, Total, Compliance %, Target %. Rows are clickable — clicking triggers `onSelectMetric(metricId)`.

```javascript
function MetricTable({ metrics, onSelectMetric }) {
    if (!metrics || metrics.length === 0) return null;
    // Render table with columns: metric_id, metric_desc, category,
    // compliant, non_compliant, total, compliance_pct, target
    // onClick row → onSelectMetric(metric.metric_id)
}
```

#### MetricDetailView (New — replaces VerticalDetailView)

Fetches data from `GET /metric/:id/verticals` and displays:
- Header with metric ID, description, category
- Aggregated stats cards (total, compliant, non-compliant, compliance %)
- Table of verticals with per-vertical compliance numbers
- Clicking a vertical row navigates to the sub-team view

```javascript
function MetricDetailView({ metricId, onBack, onSelectVertical }) {
    const [data, setData] = useState(null);
    const [loading, setLoading] = useState(true);

    useEffect(() => {
        fetch(`${API_BASE}/compliance/vcl-multi/metric/${encodeURIComponent(metricId)}/verticals`,
              { credentials: 'include' })
            .then(r => r.json())
            .then(d => { setData(d); setLoading(false); })
            .catch(() => setLoading(false));
    }, [metricId]);
    // Render: back button, header, stats, verticals table
}
```

#### MetricSubTeamView (Reused with minor adjustments)

The existing `MetricSubTeamView` component is reused. The only change is that it now receives its `metricData` (including `sub_teams`) from the `MetricDetailView`'s vertical entry rather than from the `VerticalDetailView`'s metric entry. The props interface remains the same: `{ vertical, metricId, metricData, onBack, onSelectTeam }`.

#### MetricDeviceList (Reused unchanged)

The existing `MetricDeviceList` component is reused without modification. It already accepts `{ vertical, metricId, team, onBack }` and calls the existing devices endpoint.

### Frontend: Drill-Down State Changes

**Current state model:**
```
selectedVertical → selectedMetric → selectedTeam
```

**New state model:**
```
selectedMetric → selectedVertical → selectedTeam
```

The main component's render logic changes from:

```javascript
// OLD
if (selectedTeam && selectedMetric && selectedVertical) → MetricDeviceList
if (selectedMetric && selectedVertical) → MetricSubTeamView
if (selectedVertical) → VerticalDetailView
else → Overview (VerticalTable)
```

To:

```javascript
// NEW
if (selectedTeam !== null && selectedMetric && selectedVertical) → MetricDeviceList
if (selectedVertical && selectedMetric) → MetricSubTeamView
if (selectedMetric) → MetricDetailView
else → Overview (MetricTable)
```

**State variables:**
- `selectedMetric` — string (metric_id) or null
- `selectedMetricData` — object with metric context (metric_desc, category) or null
- `selectedVertical` — string (vertical code) or null
- `selectedVerticalData` — object with vertical's sub_teams for the selected metric or null
- `selectedTeam` — string (team name) or null (null with selectedVertical set = "View All Devices")

## Data Flow

```
┌─────────────────────────────────────────────────────────────────┐
│ Overview                                                         │
│  GET /stats → StatsBar, DonutChart, TrendChart                  │
│  GET /burndown → AggregatedBurndownChart (no By Vertical table) │
│  GET /metrics → MetricTable                                      │
└──────────────────────────┬──────────────────────────────────────┘
                           │ click metric row
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ MetricDetailView                                                 │
│  GET /metric/:id/verticals → header + verticals table            │
└──────────────────────────┬──────────────────────────────────────┘
                           │ click vertical row
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ MetricSubTeamView                                                │
│  (data passed from parent — no additional fetch)                 │
└──────────────────────────┬──────────────────────────────────────┘
                           │ click team row
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ MetricDeviceList                                                 │
│  GET /vertical/:code/metric/:metricId/devices?team=X             │
└─────────────────────────────────────────────────────────────────┘
```

## Interfaces

### New API Endpoints

| Method | Path | Auth | Description |
|---|---|---|---|
| GET | `/api/compliance/vcl-multi/metrics` | requireAuth() | Metrics aggregated across all verticals |
| GET | `/api/compliance/vcl-multi/metric/:id/verticals` | requireAuth() | Per-vertical breakdown for a metric |

### Preserved API Endpoints (Backward Compatibility)

| Method | Path | Notes |
|---|---|---|
| GET | `/api/compliance/vcl-multi/stats` | Unchanged — still returns `vertical_breakdown` and `metric_breakdown` |
| GET | `/api/compliance/vcl-multi/vertical/:code/metrics` | Unchanged |
| GET | `/api/compliance/vcl-multi/vertical/:code/metric/:metricId/devices` | Unchanged |
| GET | `/api/compliance/vcl-multi/vertical/:code/burndown` | Unchanged |
| GET | `/api/compliance/vcl-multi/burndown` | Unchanged — still returns `by_vertical` in response |

### Component Props Interfaces

```javascript
// MetricTable
MetricTable.propTypes = {
    metrics: Array,       // from GET /metrics response
    onSelectMetric: Function, // (metricId: string) => void
};

// MetricDetailView
MetricDetailView.propTypes = {
    metricId: String,
    onBack: Function,         // () => void — returns to overview
    onSelectVertical: Function, // (vertical: string, verticalData: object) => void
};

// MetricSubTeamView (existing — props unchanged)
MetricSubTeamView.propTypes = {
    vertical: String,
    metricId: String,
    metricData: Object,       // { metric_desc, sub_teams, target, ... }
    onBack: Function,         // () => void — returns to metric-vertical view
    onSelectTeam: Function,   // (team: string|null) => void
};

// MetricDeviceList (existing — props unchanged)
MetricDeviceList.propTypes = {
    vertical: String,
    metricId: String,
    team: String,             // null = all teams
    onBack: Function,         // () => void — returns to sub-team view
};
```

## Data Models

No new database tables or columns. The feature uses existing data:

### vcl_multi_vertical_summary (existing)

| Column | Type | Notes |
|---|---|---|
| id | SERIAL | PK |
| upload_id | INTEGER | FK → compliance_uploads |
| vertical | TEXT | Vertical code (e.g., "NTS_AEO") |
| metric_id | TEXT | Metric identifier (e.g., "VM-001") |
| metric_desc | TEXT | Human-readable description |
| category | TEXT | Metric category |
| team | TEXT | "ALL: NTS-AEO" for rollup, "STEAM" for sub-team |
| non_compliant | INTEGER | Count |
| compliant | INTEGER | Count |
| total | INTEGER | Count |
| compliance_pct | NUMERIC(5,2) | Stored as decimal (0.95 = 95%) |
| target | NUMERIC(5,2) | Stored as decimal |

**Key convention:** Rows where `team LIKE 'ALL:%'` are rollup rows that include all sub-team totals. Sub-team rows are individual team breakdowns. Only rollup rows should be used for aggregation to avoid double-counting.

## Error Handling

### Backend

| Scenario | Response |
|---|---|
| No auth cookie / expired session | 401 `{ "error": "Authentication required" }` |
| Metric ID > 50 chars | 400 `{ "error": "Invalid metric ID" }` |
| Database query failure | 500 `{ "error": "Database error" }` |
| No data for metric | 200 with empty `verticals` array |
| No summary data at all | 200 with empty `metrics` array |

### Frontend

| Scenario | Behavior |
|---|---|
| Fetch failure on overview | Existing error state renders error message |
| Fetch failure on MetricDetailView | Show error message in component |
| Empty metrics array | Show "No metrics data" empty state |
| Empty verticals array for a metric | Show "No verticals found for this metric" message |

## Correctness Properties

*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*

### Property 1: Metrics aggregation uses only rollup rows from latest uploads

*For any* set of `vcl_multi_vertical_summary` rows across multiple verticals and uploads, the `GET /metrics` endpoint SHALL return exactly one entry per distinct `metric_id`, with `non_compliant`, `compliant`, and `total` equal to the sum of only `ALL:`-prefixed team rows from the single latest upload per vertical.

**Validates: Requirements 3.1, 3.3**

### Property 2: Metrics computed fields are mathematically correct

*For any* metric entry returned by the `GET /metrics` endpoint, `compliance_pct` SHALL equal `compliant / total` (or 0 when total is 0), and `target` SHALL equal the arithmetic mean of the `target` values across all verticals that have that metric in their latest upload.

**Validates: Requirements 3.4, 3.5**

### Property 3: Metrics response is sorted by non-compliant descending

*For any* response from `GET /metrics` containing two or more metric entries, each entry's `non_compliant` value SHALL be greater than or equal to the next entry's `non_compliant` value.

**Validates: Requirements 3.6**

### Property 4: Metric-verticals breakdown is complete and correct

*For any* metric ID that exists in the database, the `GET /metric/:id/verticals` endpoint SHALL return one vertical entry for each vertical that has that metric in its latest upload, with each entry's `sub_teams` array containing exactly the non-rollup, non-"(Other)" team rows for that metric in that vertical.

**Validates: Requirements 4.1, 4.3**

### Property 5: Metric-verticals response is sorted by non-compliant descending

*For any* response from `GET /metric/:id/verticals` containing two or more vertical entries, each entry's `non_compliant` value SHALL be greater than or equal to the next entry's `non_compliant` value.

**Validates: Requirements 4.4**

### Property 6: Drill-down state determines rendered view

*For any* combination of `(selectedMetric, selectedVertical, selectedTeam)` state values, the CCP_Metrics_Page SHALL render exactly one view: Overview when no metric is selected, MetricDetailView when only metric is selected, MetricSubTeamView when metric and vertical are selected (team is null or absent), and MetricDeviceList when all three are selected.

**Validates: Requirements 6.1, 6.5**

## Testing Strategy

### Property-Based Tests (Backend)

Properties 1–5 are testable via property-based tests against the backend endpoints. The test setup generates random `vcl_multi_vertical_summary` rows with varying numbers of verticals, metrics, uploads, and team configurations, then asserts the invariants hold.

### Example-Based Tests (Frontend)

Requirements 1.x, 2.x, 5.x, and 6.2–6.4 are UI rendering and interaction tests best covered by example-based component tests using React Testing Library.

### Integration Tests (Backward Compatibility)

Requirements 7.x are integration tests that verify existing endpoints continue to return the same response shape after the new endpoints are added.