# Design Document: VCL Compliance Reporting

## Overview

This feature adds an executive-level VCL (Vulnerability Compliance Level) reporting page to the existing Compliance module, extends device records with remediation tracking fields (resolution date, remediation plan), and introduces a bulk upload mechanism for updating device metadata in batch. The VCL Report Page mirrors the layout of the leadership's existing spreadsheet deck — summary statistics bar, trend chart with forecast, non-compliant asset donut chart, heavy hitters table, and vertical breakdown table with burndown projections.

The implementation builds on the existing `compliance.js` route module, `compliance_items` table, and `CompliancePage.js` frontend component. New backend endpoints compute VCL statistics from existing data plus the new `resolution_date` and `remediation_plan` columns. The frontend adds a new `VCLReportPage.js` component accessible from the Compliance module navigation.

## Architecture

```mermaid
sequenceDiagram
    participant U as User
    participant FE as React Frontend
    participant BE as Express Backend
    participant DB as PostgreSQL

    Note over FE,DB: Device Metadata Update (single device)
    U->>FE: Edit resolution_date / remediation_plan in DetailPanel
    FE->>BE: PATCH /api/compliance/items/:hostname/metadata
    BE->>DB: UPDATE compliance_items SET resolution_date, remediation_plan WHERE hostname = $1
    BE-->>FE: 200 OK { updated: count }

    Note over FE,DB: VCL Report Page Load
    FE->>BE: GET /api/compliance/vcl/stats
    BE->>DB: Aggregate compliance_items (counts, percentages, categorization)
    DB-->>BE: Raw counts
    BE->>BE: Compute stats, categorization, heavy hitters, vertical breakdown
    BE-->>FE: JSON { stats, donut, heavyHitters, verticalBreakdown }

    FE->>BE: GET /api/compliance/vcl/trend
    BE->>DB: Monthly aggregation from compliance_uploads + compliance_items history
    DB-->>BE: Monthly data points
    BE->>BE: Compute actuals + forecast
    BE-->>FE: JSON { months: [...] }

    Note over U,DB: Bulk Upload Flow
    U->>FE: Select xlsx file in bulk upload control
    FE->>FE: Parse xlsx with 'xlsx' library (client-side)
    FE->>FE: Map columns, validate fields, match hostnames
    FE->>BE: POST /api/compliance/vcl/bulk-preview { rows: [...] }
    BE->>DB: Match hostnames against compliance_items
    BE-->>FE: JSON { matched, unmatched, changes, invalid }
    FE->>FE: Display Diff_Preview
    U->>FE: Confirm changes
    FE->>BE: POST /api/compliance/vcl/bulk-commit { changes: [...] }
    BE->>DB: BEGIN; UPDATE compliance_items ...; COMMIT;
    BE-->>FE: 200 OK { committed: count }
```

### Data Flow Summary

1. **Device metadata** — stored directly on `compliance_items` rows. Updated via PATCH endpoint (single) or bulk commit (batch).
2. **VCL statistics** — computed on-demand from current `compliance_items` state. No separate materialized table needed since the dataset is small (~1000 devices).
3. **Trend data** — derived from `compliance_uploads` history (existing) plus monthly snapshots of compliance percentages stored in a new `compliance_snapshots` table.
4. **Burndown projections** — computed from `resolution_date` values on active non-compliant items, bucketed by month.

## Components and Interfaces

### Backend

#### New Endpoints (added to `backend/routes/compliance.js`)

**`PATCH /api/compliance/items/:hostname/metadata`**

Updates resolution_date and/or remediation_plan for all active items matching a hostname.

- Auth: `requireAuth()`, `requireGroup('Admin', 'Standard_User')`
- Body: `{ resolution_date?: string|null, remediation_plan?: string|null }`
- Validation: resolution_date must be a valid ISO date or null; remediation_plan must be <= 2000 chars
- Response: `{ updated: number }`

**`GET /api/compliance/vcl/stats`**

Returns computed VCL executive summary statistics.

- Auth: `requireAuth()`
- Response:
```json
{
  "stats": {
    "total_devices": 1200,
    "in_scope": 1100,
    "compliant": 950,
    "non_compliant": 150,
    "remediations_required": 150,
    "compliance_pct": 86,
    "target_pct": 95
  },
  "donut": {
    "blocked": { "count": 45, "pct": 30 },
    "in_progress": { "count": 105, "pct": 70 }
  },
  "heavy_hitters": [
    { "vertical": "Network Ops", "team": "STEAM", "non_compliant": 42, "compliance_date": "2026-06-30", "notes": "..." }
  ],
  "vertical_breakdown": [
    {
      "vertical": "Network Ops",
      "compliance_pct": 82,
      "team": "STEAM",
      "non_compliant": 42,
      "actual_burndown": { "2026-01": 5, "2026-02": 8 },
      "forecast_burndown": { "2026-03": 10, "2026-04": 12 },
      "blockers": 8,
      "risk_acceptances": 3,
      "notes": ""
    }
  ]
}
```

**`GET /api/compliance/vcl/trend`**

Returns monthly compliance trend data for the overview chart.

- Auth: `requireAuth()`
- Query params: none
- Response:
```json
{
  "months": [
    {
      "month": "2026-01",
      "compliant_count": 900,
      "compliance_pct": 82,
      "forecast_pct": null,
      "target_pct": 95
    }
  ]
}
```

Forecast is computed using linear regression on the last 3+ months of actual data, projected forward.

**`POST /api/compliance/vcl/bulk-preview`**

Accepts parsed bulk upload rows and returns a diff preview.

- Auth: `requireAuth()`, `requireGroup('Admin', 'Standard_User')`
- Body: `{ rows: [{ hostname, resolution_date?, remediation_plan?, notes? }] }`
- Response:
```json
{
  "matched": 850,
  "unmatched": 12,
  "changes": 200,
  "invalid": 5,
  "details": [
    {
      "hostname": "srv-001",
      "status": "changed",
      "fields": {
        "resolution_date": { "old": null, "new": "2026-06-15" },
        "remediation_plan": { "old": "", "new": "Patch in next window" }
      }
    }
  ],
  "unmatched_rows": ["unknown-host-1"],
  "invalid_rows": [{ "hostname": "srv-bad", "errors": ["resolution_date: invalid date format"] }]
}
```

**`POST /api/compliance/vcl/bulk-commit`**

Commits validated bulk changes in a single transaction.

- Auth: `requireAuth()`, `requireGroup('Admin', 'Standard_User')`
- Body: `{ changes: [{ hostname, resolution_date?, remediation_plan?, notes? }] }`
- Response: `{ committed: number }`
- Audit: logs `compliance_bulk_update` action

#### Pure Helper Functions (exported for testing)

```javascript
// Truncates text to maxLen chars with ellipsis
function truncateText(text, maxLen = 80) { ... }

// Validates remediation_plan length
function validateRemediationPlan(text) { ... }

// Validates a date string (ISO format)
function isValidDateString(str) { ... }

// Computes VCL summary stats from device rows
function computeVCLStats(items, targetPct) { ... }

// Categorizes non-compliant devices into blocked/in-progress
function categorizeNonCompliant(items) { ... }

// Ranks verticals by non-compliant count descending
function rankHeavyHitters(verticalData) { ... }

// Computes forecasted burndown from resolution_date values
function computeForecastBurndown(items) { ... }

// Matches uploaded rows to existing devices by hostname
function matchByHostname(uploadedRows, existingHostnames) { ... }

// Computes diff between uploaded values and current DB values
function computeBulkDiff(matchedRows, currentData) { ... }

// Maps column headers to known field names
function mapColumnHeaders(headers) { ... }

// Formats a decimal as a whole-number percentage string
function formatPct(decimal) { ... }
```

### Frontend

#### New Component: `VCLReportPage.js`

Located at `frontend/src/components/pages/VCLReportPage.js`. Accessible via a tab/button on the existing CompliancePage or as a separate nav entry.

**Sub-components:**

| Component | Purpose |
|-----------|---------|
| `VCLStatsBar` | Horizontal bar with 7 stat cards (Total, In-Scope, Compliant, Non-Compliant, Remediations, Current %, Target %) |
| `ComplianceOverviewChart` | Recharts ComposedChart — bars for compliant count, solid line for actual %, dashed line for forecast %, ReferenceLine for target |
| `NonCompliantDonutChart` | Recharts PieChart (donut) — Blocked vs In-Progress segments |
| `HeavyHittersTable` | Sorted table of top verticals by non-compliant count |
| `VerticalBreakdownTable` | Full breakdown table with burndown columns |
| `BulkUploadModal` | Modal with file picker, column mapping preview, diff display, confirm/cancel |

#### Modified Component: `ComplianceDetailPanel.js`

Add two new fields to the device detail panel:
- **Resolution Date** — `<input type="date">` with save on blur/enter
- **Remediation Plan** — `<textarea>` with character counter (max 2000) and save button

#### Modified Component: `CompliancePage.js`

- Add "VCL Report" tab/button in the page header that navigates to VCLReportPage
- Add `resolution_date` and `remediation_plan` columns to the device table

### Chart Specifications

#### Compliance Overview Chart (Recharts ComposedChart)

```javascript
<ComposedChart data={months}>
  <CartesianGrid stroke="rgba(255,255,255,0.05)" strokeDasharray="3 3" />
  <XAxis dataKey="month" tick={AXIS_STYLE} />
  <YAxis yAxisId="count" tick={AXIS_STYLE} />
  <YAxis yAxisId="pct" orientation="right" domain={[0, 100]} unit="%" tick={AXIS_STYLE} />
  <Bar yAxisId="count" dataKey="compliant_count" fill="#10B981" fillOpacity={0.7} />
  <Line yAxisId="pct" dataKey="compliance_pct" stroke={TEAL} strokeWidth={2} dot={{ r: 3 }} />
  <Line yAxisId="pct" dataKey="forecast_pct" stroke={TEAL} strokeWidth={2} strokeDasharray="5 3" dot={false} />
  <ReferenceLine yAxisId="pct" y={targetPct} stroke="#F59E0B" strokeDasharray="4 4" label="Target" />
</ComposedChart>
```

#### Non-Compliant Assets Donut (Recharts PieChart)

```javascript
<PieChart>
  <Pie data={donutData} innerRadius={60} outerRadius={90} dataKey="count" nameKey="name">
    <Cell fill="#EF4444" />  {/* Blocked */}
    <Cell fill="#F59E0B" />  {/* In-Progress */}
  </Pie>
  <Legend />
</PieChart>
```

## Data Models

### Schema Changes to `compliance_items`

Two new columns:

```sql
ALTER TABLE compliance_items ADD COLUMN IF NOT EXISTS resolution_date DATE DEFAULT NULL;
ALTER TABLE compliance_items ADD COLUMN IF NOT EXISTS remediation_plan TEXT DEFAULT NULL;
```

- `resolution_date` — target date for remediation completion. NULL means no date set.
- `remediation_plan` — free-text description of the fix approach. NULL or empty means no plan documented. Max 2000 characters enforced at application layer.

### New Table: `compliance_snapshots`

Stores monthly compliance percentage snapshots for trend charting. One row per vertical per month.

```sql
CREATE TABLE IF NOT EXISTS compliance_snapshots (
    id              SERIAL PRIMARY KEY,
    snapshot_month  TEXT NOT NULL,          -- 'YYYY-MM' format
    vertical        TEXT NOT NULL,
    total_devices   INTEGER NOT NULL DEFAULT 0,
    compliant       INTEGER NOT NULL DEFAULT 0,
    non_compliant   INTEGER NOT NULL DEFAULT 0,
    compliance_pct  NUMERIC(5,2) DEFAULT 0,
    created_at      TIMESTAMPTZ DEFAULT NOW(),
    UNIQUE(snapshot_month, vertical)
);

CREATE INDEX IF NOT EXISTS idx_compliance_snapshots_month
ON compliance_snapshots(snapshot_month);
```

Snapshots are created automatically when a new compliance upload is committed — the commit logic inserts/updates the snapshot for the current month.

### Migration Script: `backend/migrations/add_vcl_reporting_columns.js`

```javascript
const pool = require('../db');

async function run() {
    console.log('Starting VCL reporting migration...');
    try {
        await pool.query(`ALTER TABLE compliance_items ADD COLUMN IF NOT EXISTS resolution_date DATE DEFAULT NULL`);
        console.log('✓ resolution_date column added');

        await pool.query(`ALTER TABLE compliance_items ADD COLUMN IF NOT EXISTS remediation_plan TEXT DEFAULT NULL`);
        console.log('✓ remediation_plan column added');

        await pool.query(`
            CREATE TABLE IF NOT EXISTS compliance_snapshots (
                id              SERIAL PRIMARY KEY,
                snapshot_month  TEXT NOT NULL,
                vertical        TEXT NOT NULL,
                total_devices   INTEGER NOT NULL DEFAULT 0,
                compliant       INTEGER NOT NULL DEFAULT 0,
                non_compliant   INTEGER NOT NULL DEFAULT 0,
                compliance_pct  NUMERIC(5,2) DEFAULT 0,
                created_at      TIMESTAMPTZ DEFAULT NOW(),
                UNIQUE(snapshot_month, vertical)
            )
        `);
        console.log('✓ compliance_snapshots table created');

        await pool.query(`CREATE INDEX IF NOT EXISTS idx_compliance_snapshots_month ON compliance_snapshots(snapshot_month)`);
        console.log('✓ compliance_snapshots index created');
    } catch (err) {
        console.error('Migration error:', err.message);
        process.exit(1);
    }
    console.log('Migration complete.');
    process.exit(0);
}

run();
```

## Correctness Properties

*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*

### Property 1: Device Metadata Persistence Round-Trip

*For any* valid resolution_date (ISO date string or null) and any valid remediation_plan (string of 0–2000 characters or null), saving the metadata via the update endpoint and then fetching the device should return the same resolution_date and remediation_plan values.

**Validates: Requirements 1.3, 2.3**

### Property 2: Text Truncation

*For any* string, `truncateText(text, 80)` should return the original string if its length is <= 80, or the first 80 characters followed by "…" if its length exceeds 80. The output length should never exceed 81 characters (80 + ellipsis).

**Validates: Requirements 2.4**

### Property 3: Remediation Plan Length Validation

*For any* string, `validateRemediationPlan(text)` should return valid if and only if the string length is <= 2000 characters. Strings exceeding 2000 characters should be flagged as invalid.

**Validates: Requirements 2.5, 9.4**

### Property 4: Summary Statistics Computation Invariants

*For any* set of compliance items with total, compliant, and non-compliant counts where total >= compliant >= 0 and non_compliant = total - compliant, `computeVCLStats(items, target)` should produce: non_compliant + compliant = total, compliance_pct = Math.round((compliant / total) * 100) when total > 0, and compliance_pct = 0 when total = 0.

**Validates: Requirements 3.2, 7.3**

### Property 5: Percentage Formatting

*For any* decimal number between 0 and 1 (inclusive), `formatPct(decimal)` should return `Math.round(decimal * 100) + '%'`. The output should always match the regex pattern `/^\d{1,3}%$/`.

**Validates: Requirements 3.3**

### Property 6: Non-Compliant Device Categorization Partition

*For any* array of non-compliant device objects, `categorizeNonCompliant(items)` should produce two groups (blocked, in_progress) where: every input item appears in exactly one group, blocked.count + in_progress.count = items.length, and each group's percentage equals Math.round((group.count / items.length) * 100) when items.length > 0.

**Validates: Requirements 5.2, 5.3**

### Property 7: Heavy Hitters Descending Sort

*For any* array of vertical objects with non_compliant counts, `rankHeavyHitters(verticals)` should return the array sorted in strictly non-increasing order by non_compliant count. For all consecutive pairs (a, b) in the output, a.non_compliant >= b.non_compliant.

**Validates: Requirements 6.1, 6.3**

### Property 8: Forecasted Burndown Projection

*For any* set of non-compliant devices with resolution_date values (some null, some valid future dates), `computeForecastBurndown(items)` should produce monthly buckets where: the sum of all monthly forecast counts equals the number of items with non-null resolution_dates, and each item with a resolution_date appears in exactly the bucket corresponding to its resolution month.

**Validates: Requirements 7.5**

### Property 9: Hostname Matching with Unmatched Flagging

*For any* array of uploaded rows (each with a hostname) and a set of existing hostnames, `matchByHostname(rows, existing)` should produce: matched rows (hostname exists in the set) + unmatched rows (hostname not in set) = total input rows. Every matched row's hostname must be in the existing set, and every unmatched row's hostname must not be in the existing set.

**Validates: Requirements 8.2, 8.7**

### Property 10: Bulk Diff Change Detection

*For any* array of matched row pairs (uploaded value, current DB value) for fields resolution_date and remediation_plan, `computeBulkDiff(matched, current)` should flag a row as "changed" if and only if at least one field value differs between uploaded and current. Rows where all fields are identical should be flagged as "unchanged".

**Validates: Requirements 8.3, 8.4**

### Property 11: Column Header Mapping

*For any* array of column header strings, `mapColumnHeaders(headers)` should: return a mapping that includes "hostname" if any header case-insensitively matches "Hostname", include "resolution_date" if any header matches "Resolution Date", include "remediation_plan" if any header matches "Remediation Plan", and include "notes" if any header matches "Notes". Headers not matching any known field should be ignored.

**Validates: Requirements 9.2**

### Property 12: Date String Validation

*For any* string, `isValidDateString(str)` should return true if and only if the string can be parsed into a valid Date object representing a real calendar date (e.g., "2026-02-30" is invalid). Null and empty string should return false.

**Validates: Requirements 9.3**

### Property 13: Row Count Arithmetic Invariant

*For any* bulk upload preview result with matched, unmatched, and invalid counts, the sum matched + unmatched must equal the total number of input rows. Additionally, within matched rows, changed + unchanged must equal matched count.

**Validates: Requirements 9.6**

## Error Handling

### Device Metadata Update Errors

| Condition | HTTP Status | Response | Behavior |
|-----------|-------------|----------|----------|
| Hostname not found | 404 | `{ "error": "Device not found" }` | No state change |
| Invalid date format | 400 | `{ "error": "Invalid resolution_date format" }` | No state change |
| Remediation plan > 2000 chars | 400 | `{ "error": "Remediation plan exceeds 2000 characters" }` | No state change |
| Database error | 500 | `{ "error": "Failed to update device metadata" }` | No state change |

### VCL Stats Endpoint Errors

| Condition | HTTP Status | Response | Behavior |
|-----------|-------------|----------|----------|
| No compliance data | 200 | `{ "stats": { all zeros }, ... }` | Return empty/zero stats gracefully |
| Database error | 500 | `{ "error": "Database error" }` | Log error |

### Bulk Upload Errors

| Condition | HTTP Status | Response | Behavior |
|-----------|-------------|----------|----------|
| No rows in file | 400 | `{ "error": "File contains no data rows" }` | No state change |
| No Hostname column | 400 | `{ "error": "File must contain a Hostname column" }` | No state change |
| No updatable columns | 400 | `{ "error": "No updatable fields found (need Resolution Date, Remediation Plan, or Notes)" }` | No state change |
| File exceeds 2000 rows | 400 | `{ "error": "File exceeds maximum of 2000 rows" }` | No state change |
| Transaction failure on commit | 500 | `{ "error": "Failed to commit changes" }` | Full rollback, no partial updates |

### Frontend Error Handling

- API failures display inline error messages (red text, monospace, consistent with existing patterns)
- Bulk upload validation errors are shown per-row in the diff preview with red highlighting
- Network errors show a retry prompt
- File parsing errors (corrupt xlsx) show a user-friendly message suggesting re-export from the source

## Testing Strategy

### Property-Based Testing

Use `fast-check` as the property-based testing library (already used in this project). Each correctness property maps to a single property-based test with a minimum of 100 iterations.

Property tests focus on the pure helper functions exported from the compliance route module:
- `truncateText` — Property 2
- `validateRemediationPlan` — Property 3
- `computeVCLStats` — Property 4
- `formatPct` — Property 5
- `categorizeNonCompliant` — Property 6
- `rankHeavyHitters` — Property 7
- `computeForecastBurndown` — Property 8
- `matchByHostname` — Property 9
- `computeBulkDiff` — Property 10
- `mapColumnHeaders` — Property 11
- `isValidDateString` — Property 12

Tag format: **Feature: vcl-compliance-reporting, Property {number}: {title}**

Test file: `backend/__tests__/vcl-compliance-reporting.property.test.js`

### Unit Testing

Unit tests cover specific examples, edge cases, and integration points:

- **PATCH metadata endpoint** — happy path, invalid date, plan too long, hostname not found
- **VCL stats with no data** — verify zero/empty response
- **Bulk preview with all unmatched** — verify correct counts
- **Bulk preview with mixed valid/invalid** — verify row classification
- **Bulk commit transactional** — verify all-or-nothing behavior
- **Donut chart with single category** — verify full donut rendering
- **Trend chart with < 2 months** — verify no forecast line
- **Vertical with zero non-compliant** — verify zero display

Test file: `backend/__tests__/vcl-compliance-reporting.test.js`

### Integration Testing

- Full bulk upload flow: parse → preview → commit → verify DB state
- Device metadata update → verify VCL stats reflect the change
- Snapshot creation on upload commit → verify trend data includes new month