Update .kiro: remove SQLite hooks, add PostgreSQL migration hook, add workflow steering, sync specs

2026-05-12 14:45:58 -06:00
parent 3ee8487286
commit 1bb8ec1658
35 changed files with 4645 additions and 48 deletions
--- a/.kiro/specs/vcl-compliance-reporting/design.md
+++ b/.kiro/specs/vcl-compliance-reporting/design.md
@@ -0,0 +1,505 @@
+# Design Document: VCL Compliance Reporting
+
+## Overview
+
+This feature adds an executive-level VCL (Vulnerability Compliance Level) reporting page to the existing Compliance module, extends device records with remediation tracking fields (resolution date, remediation plan), and introduces a bulk upload mechanism for updating device metadata in batch. The VCL Report Page mirrors the layout of the leadership's existing spreadsheet deck — summary statistics bar, trend chart with forecast, non-compliant asset donut chart, heavy hitters table, and vertical breakdown table with burndown projections.
+
+The implementation builds on the existing `compliance.js` route module, `compliance_items` table, and `CompliancePage.js` frontend component. New backend endpoints compute VCL statistics from existing data plus the new `resolution_date` and `remediation_plan` columns. The frontend adds a new `VCLReportPage.js` component accessible from the Compliance module navigation.
+
+## Architecture
+
+```mermaid
+sequenceDiagram
+    participant U as User
+    participant FE as React Frontend
+    participant BE as Express Backend
+    participant DB as PostgreSQL
+
+    Note over FE,DB: Device Metadata Update (single device)
+    U->>FE: Edit resolution_date / remediation_plan in DetailPanel
+    FE->>BE: PATCH /api/compliance/items/:hostname/metadata
+    BE->>DB: UPDATE compliance_items SET resolution_date, remediation_plan WHERE hostname = $1
+    BE-->>FE: 200 OK { updated: count }
+
+    Note over FE,DB: VCL Report Page Load
+    FE->>BE: GET /api/compliance/vcl/stats
+    BE->>DB: Aggregate compliance_items (counts, percentages, categorization)
+    DB-->>BE: Raw counts
+    BE->>BE: Compute stats, categorization, heavy hitters, vertical breakdown
+    BE-->>FE: JSON { stats, donut, heavyHitters, verticalBreakdown }
+
+    FE->>BE: GET /api/compliance/vcl/trend
+    BE->>DB: Monthly aggregation from compliance_uploads + compliance_items history
+    DB-->>BE: Monthly data points
+    BE->>BE: Compute actuals + forecast
+    BE-->>FE: JSON { months: [...] }
+
+    Note over U,DB: Bulk Upload Flow
+    U->>FE: Select xlsx file in bulk upload control
+    FE->>FE: Parse xlsx with 'xlsx' library (client-side)
+    FE->>FE: Map columns, validate fields, match hostnames
+    FE->>BE: POST /api/compliance/vcl/bulk-preview { rows: [...] }
+    BE->>DB: Match hostnames against compliance_items
+    BE-->>FE: JSON { matched, unmatched, changes, invalid }
+    FE->>FE: Display Diff_Preview
+    U->>FE: Confirm changes
+    FE->>BE: POST /api/compliance/vcl/bulk-commit { changes: [...] }
+    BE->>DB: BEGIN; UPDATE compliance_items ...; COMMIT;
+    BE-->>FE: 200 OK { committed: count }
+```
+
+### Data Flow Summary
+
+1. **Device metadata** — stored directly on `compliance_items` rows. Updated via PATCH endpoint (single) or bulk commit (batch).
+2. **VCL statistics** — computed on-demand from current `compliance_items` state. No separate materialized table needed since the dataset is small (~1000 devices).
+3. **Trend data** — derived from `compliance_uploads` history (existing) plus monthly snapshots of compliance percentages stored in a new `compliance_snapshots` table.
+4. **Burndown projections** — computed from `resolution_date` values on active non-compliant items, bucketed by month.
+
+## Components and Interfaces
+
+### Backend
+
+#### New Endpoints (added to `backend/routes/compliance.js`)
+
+**`PATCH /api/compliance/items/:hostname/metadata`**
+
+Updates resolution_date and/or remediation_plan for all active items matching a hostname.
+
+- Auth: `requireAuth()`, `requireGroup('Admin', 'Standard_User')`
+- Body: `{ resolution_date?: string|null, remediation_plan?: string|null }`
+- Validation: resolution_date must be a valid ISO date or null; remediation_plan must be <= 2000 chars
+- Response: `{ updated: number }`
+
+**`GET /api/compliance/vcl/stats`**
+
+Returns computed VCL executive summary statistics.
+
+- Auth: `requireAuth()`
+- Response:
+```json
+{
+  "stats": {
+    "total_devices": 1200,
+    "in_scope": 1100,
+    "compliant": 950,
+    "non_compliant": 150,
+    "remediations_required": 150,
+    "compliance_pct": 86,
+    "target_pct": 95
+  },
+  "donut": {
+    "blocked": { "count": 45, "pct": 30 },
+    "in_progress": { "count": 105, "pct": 70 }
+  },
+  "heavy_hitters": [
+    { "vertical": "Network Ops", "team": "STEAM", "non_compliant": 42, "compliance_date": "2026-06-30", "notes": "..." }
+  ],
+  "vertical_breakdown": [
+    {
+      "vertical": "Network Ops",
+      "compliance_pct": 82,
+      "team": "STEAM",
+      "non_compliant": 42,
+      "actual_burndown": { "2026-01": 5, "2026-02": 8 },
+      "forecast_burndown": { "2026-03": 10, "2026-04": 12 },
+      "blockers": 8,
+      "risk_acceptances": 3,
+      "notes": ""
+    }
+  ]
+}
+```
+
+**`GET /api/compliance/vcl/trend`**
+
+Returns monthly compliance trend data for the overview chart.
+
+- Auth: `requireAuth()`
+- Query params: none
+- Response:
+```json
+{
+  "months": [
+    {
+      "month": "2026-01",
+      "compliant_count": 900,
+      "compliance_pct": 82,
+      "forecast_pct": null,
+      "target_pct": 95
+    }
+  ]
+}
+```
+
+Forecast is computed using linear regression on the last 3+ months of actual data, projected forward.
+
+**`POST /api/compliance/vcl/bulk-preview`**
+
+Accepts parsed bulk upload rows and returns a diff preview.
+
+- Auth: `requireAuth()`, `requireGroup('Admin', 'Standard_User')`
+- Body: `{ rows: [{ hostname, resolution_date?, remediation_plan?, notes? }] }`
+- Response:
+```json
+{
+  "matched": 850,
+  "unmatched": 12,
+  "changes": 200,
+  "invalid": 5,
+  "details": [
+    {
+      "hostname": "srv-001",
+      "status": "changed",
+      "fields": {
+        "resolution_date": { "old": null, "new": "2026-06-15" },
+        "remediation_plan": { "old": "", "new": "Patch in next window" }
+      }
+    }
+  ],
+  "unmatched_rows": ["unknown-host-1"],
+  "invalid_rows": [{ "hostname": "srv-bad", "errors": ["resolution_date: invalid date format"] }]
+}
+```
+
+**`POST /api/compliance/vcl/bulk-commit`**
+
+Commits validated bulk changes in a single transaction.
+
+- Auth: `requireAuth()`, `requireGroup('Admin', 'Standard_User')`
+- Body: `{ changes: [{ hostname, resolution_date?, remediation_plan?, notes? }] }`
+- Response: `{ committed: number }`
+- Audit: logs `compliance_bulk_update` action
+
+#### Pure Helper Functions (exported for testing)
+
+```javascript
+// Truncates text to maxLen chars with ellipsis
+function truncateText(text, maxLen = 80) { ... }
+
+// Validates remediation_plan length
+function validateRemediationPlan(text) { ... }
+
+// Validates a date string (ISO format)
+function isValidDateString(str) { ... }
+
+// Computes VCL summary stats from device rows
+function computeVCLStats(items, targetPct) { ... }
+
+// Categorizes non-compliant devices into blocked/in-progress
+function categorizeNonCompliant(items) { ... }
+
+// Ranks verticals by non-compliant count descending
+function rankHeavyHitters(verticalData) { ... }
+
+// Computes forecasted burndown from resolution_date values
+function computeForecastBurndown(items) { ... }
+
+// Matches uploaded rows to existing devices by hostname
+function matchByHostname(uploadedRows, existingHostnames) { ... }
+
+// Computes diff between uploaded values and current DB values
+function computeBulkDiff(matchedRows, currentData) { ... }
+
+// Maps column headers to known field names
+function mapColumnHeaders(headers) { ... }
+
+// Formats a decimal as a whole-number percentage string
+function formatPct(decimal) { ... }
+```
+
+### Frontend
+
+#### New Component: `VCLReportPage.js`
+
+Located at `frontend/src/components/pages/VCLReportPage.js`. Accessible via a tab/button on the existing CompliancePage or as a separate nav entry.
+
+**Sub-components:**
+
+| Component | Purpose |
+|-----------|---------|
+| `VCLStatsBar` | Horizontal bar with 7 stat cards (Total, In-Scope, Compliant, Non-Compliant, Remediations, Current %, Target %) |
+| `ComplianceOverviewChart` | Recharts ComposedChart — bars for compliant count, solid line for actual %, dashed line for forecast %, ReferenceLine for target |
+| `NonCompliantDonutChart` | Recharts PieChart (donut) — Blocked vs In-Progress segments |
+| `HeavyHittersTable` | Sorted table of top verticals by non-compliant count |
+| `VerticalBreakdownTable` | Full breakdown table with burndown columns |
+| `BulkUploadModal` | Modal with file picker, column mapping preview, diff display, confirm/cancel |
+
+#### Modified Component: `ComplianceDetailPanel.js`
+
+Add two new fields to the device detail panel:
+- **Resolution Date** — `<input type="date">` with save on blur/enter
+- **Remediation Plan** — `<textarea>` with character counter (max 2000) and save button
+
+#### Modified Component: `CompliancePage.js`
+
+- Add "VCL Report" tab/button in the page header that navigates to VCLReportPage
+- Add `resolution_date` and `remediation_plan` columns to the device table
+
+### Chart Specifications
+
+#### Compliance Overview Chart (Recharts ComposedChart)
+
+```javascript
+<ComposedChart data={months}>
+  <CartesianGrid stroke="rgba(255,255,255,0.05)" strokeDasharray="3 3" />
+  <XAxis dataKey="month" tick={AXIS_STYLE} />
+  <YAxis yAxisId="count" tick={AXIS_STYLE} />
+  <YAxis yAxisId="pct" orientation="right" domain={[0, 100]} unit="%" tick={AXIS_STYLE} />
+  <Bar yAxisId="count" dataKey="compliant_count" fill="#10B981" fillOpacity={0.7} />
+  <Line yAxisId="pct" dataKey="compliance_pct" stroke={TEAL} strokeWidth={2} dot={{ r: 3 }} />
+  <Line yAxisId="pct" dataKey="forecast_pct" stroke={TEAL} strokeWidth={2} strokeDasharray="5 3" dot={false} />
+  <ReferenceLine yAxisId="pct" y={targetPct} stroke="#F59E0B" strokeDasharray="4 4" label="Target" />
+</ComposedChart>
+```
+
+#### Non-Compliant Assets Donut (Recharts PieChart)
+
+```javascript
+<PieChart>
+  <Pie data={donutData} innerRadius={60} outerRadius={90} dataKey="count" nameKey="name">
+    <Cell fill="#EF4444" />  {/* Blocked */}
+    <Cell fill="#F59E0B" />  {/* In-Progress */}
+  </Pie>
+  <Legend />
+</PieChart>
+```
+
+## Data Models
+
+### Schema Changes to `compliance_items`
+
+Two new columns:
+
+```sql
+ALTER TABLE compliance_items ADD COLUMN IF NOT EXISTS resolution_date DATE DEFAULT NULL;
+ALTER TABLE compliance_items ADD COLUMN IF NOT EXISTS remediation_plan TEXT DEFAULT NULL;
+```
+
+- `resolution_date` — target date for remediation completion. NULL means no date set.
+- `remediation_plan` — free-text description of the fix approach. NULL or empty means no plan documented. Max 2000 characters enforced at application layer.
+
+### New Table: `compliance_snapshots`
+
+Stores monthly compliance percentage snapshots for trend charting. One row per vertical per month.
+
+```sql
+CREATE TABLE IF NOT EXISTS compliance_snapshots (
+    id              SERIAL PRIMARY KEY,
+    snapshot_month  TEXT NOT NULL,          -- 'YYYY-MM' format
+    vertical        TEXT NOT NULL,
+    total_devices   INTEGER NOT NULL DEFAULT 0,
+    compliant       INTEGER NOT NULL DEFAULT 0,
+    non_compliant   INTEGER NOT NULL DEFAULT 0,
+    compliance_pct  NUMERIC(5,2) DEFAULT 0,
+    created_at      TIMESTAMPTZ DEFAULT NOW(),
+    UNIQUE(snapshot_month, vertical)
+);
+
+CREATE INDEX IF NOT EXISTS idx_compliance_snapshots_month
+ON compliance_snapshots(snapshot_month);
+```
+
+Snapshots are created automatically when a new compliance upload is committed — the commit logic inserts/updates the snapshot for the current month.
+
+### Migration Script: `backend/migrations/add_vcl_reporting_columns.js`
+
+```javascript
+const pool = require('../db');
+
+async function run() {
+    console.log('Starting VCL reporting migration...');
+    try {
+        await pool.query(`ALTER TABLE compliance_items ADD COLUMN IF NOT EXISTS resolution_date DATE DEFAULT NULL`);
+        console.log('✓ resolution_date column added');
+
+        await pool.query(`ALTER TABLE compliance_items ADD COLUMN IF NOT EXISTS remediation_plan TEXT DEFAULT NULL`);
+        console.log('✓ remediation_plan column added');
+
+        await pool.query(`
+            CREATE TABLE IF NOT EXISTS compliance_snapshots (
+                id              SERIAL PRIMARY KEY,
+                snapshot_month  TEXT NOT NULL,
+                vertical        TEXT NOT NULL,
+                total_devices   INTEGER NOT NULL DEFAULT 0,
+                compliant       INTEGER NOT NULL DEFAULT 0,
+                non_compliant   INTEGER NOT NULL DEFAULT 0,
+                compliance_pct  NUMERIC(5,2) DEFAULT 0,
+                created_at      TIMESTAMPTZ DEFAULT NOW(),
+                UNIQUE(snapshot_month, vertical)
+            )
+        `);
+        console.log('✓ compliance_snapshots table created');
+
+        await pool.query(`CREATE INDEX IF NOT EXISTS idx_compliance_snapshots_month ON compliance_snapshots(snapshot_month)`);
+        console.log('✓ compliance_snapshots index created');
+    } catch (err) {
+        console.error('Migration error:', err.message);
+        process.exit(1);
+    }
+    console.log('Migration complete.');
+    process.exit(0);
+}
+
+run();
+```
+
+## Correctness Properties
+
+*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
+
+### Property 1: Device Metadata Persistence Round-Trip
+
+*For any* valid resolution_date (ISO date string or null) and any valid remediation_plan (string of 0–2000 characters or null), saving the metadata via the update endpoint and then fetching the device should return the same resolution_date and remediation_plan values.
+
+**Validates: Requirements 1.3, 2.3**
+
+### Property 2: Text Truncation
+
+*For any* string, `truncateText(text, 80)` should return the original string if its length is <= 80, or the first 80 characters followed by "…" if its length exceeds 80. The output length should never exceed 81 characters (80 + ellipsis).
+
+**Validates: Requirements 2.4**
+
+### Property 3: Remediation Plan Length Validation
+
+*For any* string, `validateRemediationPlan(text)` should return valid if and only if the string length is <= 2000 characters. Strings exceeding 2000 characters should be flagged as invalid.
+
+**Validates: Requirements 2.5, 9.4**
+
+### Property 4: Summary Statistics Computation Invariants
+
+*For any* set of compliance items with total, compliant, and non-compliant counts where total >= compliant >= 0 and non_compliant = total - compliant, `computeVCLStats(items, target)` should produce: non_compliant + compliant = total, compliance_pct = Math.round((compliant / total) * 100) when total > 0, and compliance_pct = 0 when total = 0.
+
+**Validates: Requirements 3.2, 7.3**
+
+### Property 5: Percentage Formatting
+
+*For any* decimal number between 0 and 1 (inclusive), `formatPct(decimal)` should return `Math.round(decimal * 100) + '%'`. The output should always match the regex pattern `/^\d{1,3}%$/`.
+
+**Validates: Requirements 3.3**
+
+### Property 6: Non-Compliant Device Categorization Partition
+
+*For any* array of non-compliant device objects, `categorizeNonCompliant(items)` should produce two groups (blocked, in_progress) where: every input item appears in exactly one group, blocked.count + in_progress.count = items.length, and each group's percentage equals Math.round((group.count / items.length) * 100) when items.length > 0.
+
+**Validates: Requirements 5.2, 5.3**
+
+### Property 7: Heavy Hitters Descending Sort
+
+*For any* array of vertical objects with non_compliant counts, `rankHeavyHitters(verticals)` should return the array sorted in strictly non-increasing order by non_compliant count. For all consecutive pairs (a, b) in the output, a.non_compliant >= b.non_compliant.
+
+**Validates: Requirements 6.1, 6.3**
+
+### Property 8: Forecasted Burndown Projection
+
+*For any* set of non-compliant devices with resolution_date values (some null, some valid future dates), `computeForecastBurndown(items)` should produce monthly buckets where: the sum of all monthly forecast counts equals the number of items with non-null resolution_dates, and each item with a resolution_date appears in exactly the bucket corresponding to its resolution month.
+
+**Validates: Requirements 7.5**
+
+### Property 9: Hostname Matching with Unmatched Flagging
+
+*For any* array of uploaded rows (each with a hostname) and a set of existing hostnames, `matchByHostname(rows, existing)` should produce: matched rows (hostname exists in the set) + unmatched rows (hostname not in set) = total input rows. Every matched row's hostname must be in the existing set, and every unmatched row's hostname must not be in the existing set.
+
+**Validates: Requirements 8.2, 8.7**
+
+### Property 10: Bulk Diff Change Detection
+
+*For any* array of matched row pairs (uploaded value, current DB value) for fields resolution_date and remediation_plan, `computeBulkDiff(matched, current)` should flag a row as "changed" if and only if at least one field value differs between uploaded and current. Rows where all fields are identical should be flagged as "unchanged".
+
+**Validates: Requirements 8.3, 8.4**
+
+### Property 11: Column Header Mapping
+
+*For any* array of column header strings, `mapColumnHeaders(headers)` should: return a mapping that includes "hostname" if any header case-insensitively matches "Hostname", include "resolution_date" if any header matches "Resolution Date", include "remediation_plan" if any header matches "Remediation Plan", and include "notes" if any header matches "Notes". Headers not matching any known field should be ignored.
+
+**Validates: Requirements 9.2**
+
+### Property 12: Date String Validation
+
+*For any* string, `isValidDateString(str)` should return true if and only if the string can be parsed into a valid Date object representing a real calendar date (e.g., "2026-02-30" is invalid). Null and empty string should return false.
+
+**Validates: Requirements 9.3**
+
+### Property 13: Row Count Arithmetic Invariant
+
+*For any* bulk upload preview result with matched, unmatched, and invalid counts, the sum matched + unmatched must equal the total number of input rows. Additionally, within matched rows, changed + unchanged must equal matched count.
+
+**Validates: Requirements 9.6**
+
+## Error Handling
+
+### Device Metadata Update Errors
+
+| Condition | HTTP Status | Response | Behavior |
+|-----------|-------------|----------|----------|
+| Hostname not found | 404 | `{ "error": "Device not found" }` | No state change |
+| Invalid date format | 400 | `{ "error": "Invalid resolution_date format" }` | No state change |
+| Remediation plan > 2000 chars | 400 | `{ "error": "Remediation plan exceeds 2000 characters" }` | No state change |
+| Database error | 500 | `{ "error": "Failed to update device metadata" }` | No state change |
+
+### VCL Stats Endpoint Errors
+
+| Condition | HTTP Status | Response | Behavior |
+|-----------|-------------|----------|----------|
+| No compliance data | 200 | `{ "stats": { all zeros }, ... }` | Return empty/zero stats gracefully |
+| Database error | 500 | `{ "error": "Database error" }` | Log error |
+
+### Bulk Upload Errors
+
+| Condition | HTTP Status | Response | Behavior |
+|-----------|-------------|----------|----------|
+| No rows in file | 400 | `{ "error": "File contains no data rows" }` | No state change |
+| No Hostname column | 400 | `{ "error": "File must contain a Hostname column" }` | No state change |
+| No updatable columns | 400 | `{ "error": "No updatable fields found (need Resolution Date, Remediation Plan, or Notes)" }` | No state change |
+| File exceeds 2000 rows | 400 | `{ "error": "File exceeds maximum of 2000 rows" }` | No state change |
+| Transaction failure on commit | 500 | `{ "error": "Failed to commit changes" }` | Full rollback, no partial updates |
+
+### Frontend Error Handling
+
+- API failures display inline error messages (red text, monospace, consistent with existing patterns)
+- Bulk upload validation errors are shown per-row in the diff preview with red highlighting
+- Network errors show a retry prompt
+- File parsing errors (corrupt xlsx) show a user-friendly message suggesting re-export from the source
+
+## Testing Strategy
+
+### Property-Based Testing
+
+Use `fast-check` as the property-based testing library (already used in this project). Each correctness property maps to a single property-based test with a minimum of 100 iterations.
+
+Property tests focus on the pure helper functions exported from the compliance route module:
+- `truncateText` — Property 2
+- `validateRemediationPlan` — Property 3
+- `computeVCLStats` — Property 4
+- `formatPct` — Property 5
+- `categorizeNonCompliant` — Property 6
+- `rankHeavyHitters` — Property 7
+- `computeForecastBurndown` — Property 8
+- `matchByHostname` — Property 9
+- `computeBulkDiff` — Property 10
+- `mapColumnHeaders` — Property 11
+- `isValidDateString` — Property 12
+
+Tag format: **Feature: vcl-compliance-reporting, Property {number}: {title}**
+
+Test file: `backend/__tests__/vcl-compliance-reporting.property.test.js`
+
+### Unit Testing
+
+Unit tests cover specific examples, edge cases, and integration points:
+
+- **PATCH metadata endpoint** — happy path, invalid date, plan too long, hostname not found
+- **VCL stats with no data** — verify zero/empty response
+- **Bulk preview with all unmatched** — verify correct counts
+- **Bulk preview with mixed valid/invalid** — verify row classification
+- **Bulk commit transactional** — verify all-or-nothing behavior
+- **Donut chart with single category** — verify full donut rendering
+- **Trend chart with < 2 months** — verify no forecast line
+- **Vertical with zero non-compliant** — verify zero display
+
+Test file: `backend/__tests__/vcl-compliance-reporting.test.js`
+
+### Integration Testing
+
+- Full bulk upload flow: parse → preview → commit → verify DB state
+- Device metadata update → verify VCL stats reflect the change
+- Snapshot creation on upload commit → verify trend data includes new month