jramos/cve-dashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update .kiro: remove SQLite hooks, add PostgreSQL migration hook, add workflow steering, sync specs

Browse Source
This commit is contained in:
Jordan Ramos
2026-05-12 14:45:58 -06:00
parent 3ee8487286
commit 1bb8ec1658
35 changed files with 4645 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.kiro/specs/compliance-chart-replacements/.config.kiro Normal file
View File

@@ -0,0 +1 @@
{"specId": "9f706e1f-2c1d-4eb0-aa80-b511a0be2bd3", "workflowType": "requirements-first", "specType": "feature"}

313
.kiro/specs/compliance-chart-replacements/design.md Normal file
View File

@@ -0,0 +1,313 @@
# Design Document: Compliance Chart Replacements
## Overview
This feature replaces two underperforming charts in the `ComplianceChartsPanel` with more actionable visualizations:
| Slot | Old Chart | New Chart |
|------|-----------|-----------|
| Chart 4 | MTTR by Team (horizontal bar) | **Aging Findings Distribution** — stacked vertical bar showing active findings bucketed by `seen_count` with per-team color segments |
| Chart 5 | Most Persistent Findings (horizontal bar) | **Net Change Waterfall** — waterfall bar chart showing per-cycle net movement (start → +new → +recurring → −resolved → end) |
Both new charts derive data from columns already present in the SQLite schema (`compliance_items.seen_count`, `compliance_uploads.new_count / recurring_count / resolved_count`). No database migrations are required. The four other charts (Active Findings Over Time, Change per Cycle, Team Compliance Health, Archer Exception Pipeline) remain unchanged.
The backend reuses the existing route paths (`/mttr` → aging, `/top-recurring` → waterfall) so no new API routes are introduced. The frontend replaces the two component functions and their corresponding state variables while preserving the shared infrastructure (`ChartCard`, `DarkTooltip`, `NoData`, `TEAM_COLORS`, `fmtDate`, style tokens).
## Architecture
The change is scoped to two files and follows the existing vertical-slice pattern:
```mermaid
graph LR
subgraph Backend ["Express Backend"]
R1["/compliance/mttr<br/>(was MTTR → now Aging)"]
R2["/compliance/top-recurring<br/>(was Persistent → now Waterfall)"]
DB[(SQLite<br/>compliance_items<br/>compliance_uploads)]
R1 -->|"SELECT … GROUP BY bucket, team"| DB
R2 -->|"SELECT … ORDER BY report_date"| DB
end
subgraph Frontend ["React Frontend"]
Panel["ComplianceChartsPanel"]
AC["AgingChart"]
WC["WaterfallChart"]
Panel --> AC
Panel --> WC
end
R1 -->|JSON| AC
R2 -->|JSON| WC
```
### Key Architectural Decisions
1. **Route path reuse** — The old `/mttr` and `/top-recurring` paths are replaced in-place rather than creating new paths. This avoids route proliferation and means no proxy/CORS changes are needed.
2. **No new dependencies** — Both charts are built with the existing Recharts `BarChart` / `Bar` components. The waterfall effect is achieved using a transparent "base" `Bar` with `fill="transparent"` to offset the visible segments, which is a standard Recharts stacking technique.
3. **No database migration** — The aging chart queries `compliance_items.seen_count` (already populated by the upload pipeline). The waterfall chart reads `new_count`, `recurring_count`, `resolved_count` from `compliance_uploads` (already written by `persistUpload`). Starting/ending counts are computed in the route handler via a running accumulator.
4. **Shared component reuse** — Both new chart components use the existing `ChartCard`, `DarkTooltip`, `NoData`, `TEAM_COLORS`, `AXIS_STYLE`, `GRID_STYLE`, and `LEGEND_STYLE` tokens already defined in `ComplianceChartsPanel.js`.
## Components and Interfaces
### Backend: Aging Endpoint (replaces `/mttr`)
**Route:** `GET /compliance/mttr`
**SQL Query:**
```sql
SELECT
CASE
WHEN seen_count = 1 THEN '1 cycle'
WHEN seen_count BETWEEN 2 AND 3 THEN '2–3 cycles'
WHEN seen_count BETWEEN 4 AND 6 THEN '4–6 cycles'
ELSE '7+ cycles'
END AS bucket,
team,
COUNT(*) AS count
FROM compliance_items
WHERE status = 'active'
GROUP BY bucket, team
ORDER BY
CASE bucket
WHEN '1 cycle' THEN 1
WHEN '2–3 cycles' THEN 2
WHEN '4–6 cycles' THEN 3
WHEN '7+ cycles' THEN 4
END
```
**Response shape:**
```json
{
"aging": [
{ "bucket": "1 cycle", "total": 12, "STEAM": 5, "ACCESS-ENG": 3, "ACCESS-OPS": 2, "INTELDEV": 2 },
{ "bucket": "2–3 cycles", "total": 8, "STEAM": 2, "ACCESS-ENG": 4, "ACCESS-OPS": 1, "INTELDEV": 1 },
{ "bucket": "4–6 cycles", "total": 3, "STEAM": 1, "ACCESS-ENG": 0, "ACCESS-OPS": 2, "INTELDEV": 0 },
{ "bucket": "7+ cycles", "total": 2, "STEAM": 0, "ACCESS-ENG": 1, "ACCESS-OPS": 0, "INTELDEV": 1 }
]
}
```
**Handler logic:**
1. Query active items grouped by bucket CASE expression and team.
2. Pivot the flat rows into one object per bucket with per-team counts and a total.
3. Return the array sorted by bucket order (ascending age).
### Backend: Waterfall Endpoint (replaces `/top-recurring`)
**Route:** `GET /compliance/top-recurring`
**SQL Query:**
```sql
SELECT id, report_date,
COALESCE(new_count, 0) AS new_count,
COALESCE(recurring_count, 0) AS recurring_count,
COALESCE(resolved_count, 0) AS resolved_count
FROM compliance_uploads
ORDER BY report_date ASC
```
**Response shape:**
```json
{
"waterfall": [
{
"date": "2026-04-13",
"start": 0,
"new_count": 15,
"recurring_count": 0,
"resolved_count": 0,
"end": 15
},
{
"date": "2026-04-20",
"start": 15,
"new_count": 3,
"recurring_count": 12,
"resolved_count": 5,
"end": 25
}
]
}
```
**Handler logic:**
1. Fetch all uploads ordered by `report_date ASC`.
2. Iterate with a running accumulator: `start` for cycle N = `end` of cycle N−1 (first cycle starts at 0).
3. Compute `end = start + new_count + recurring_count - resolved_count`.
4. Return the array.
### Frontend: `AgingChart` Component
Replaces `MttrChart`. Renders a vertical stacked `BarChart`:
- **X-axis:** bucket labels (`1 cycle`, `2–3 cycles`, `4–6 cycles`, `7+ cycles`)
- **Y-axis:** finding count
- **Stacked bars:** one `<Bar>` per team key from `TEAM_COLORS`, using `stackId="aging"`
- **Tooltip:** `DarkTooltip` showing bucket label, per-team counts, and total
- **Empty state:** `<NoData />` when `aging` array is empty
- **Wrapper:** `<ChartCard title="Aging Findings Distribution" subtitle="Active findings by age bucket — stacked by team" />`
### Frontend: `WaterfallChart` Component
Replaces `RecurringChart`. Renders a vertical `BarChart` with stacked transparent base:
- **X-axis:** cycle dates formatted via `fmtDate`
- **Y-axis:** finding count
- **Bars (stacked):**
1. `<Bar dataKey="start" stackId="w" fill="transparent" />` — invisible base
2. `<Bar dataKey="new_count" stackId="w" fill="#EF4444" />` — red new findings
3. `<Bar dataKey="recurring_count" stackId="w" fill="#F59E0B" />` — amber recurring
- **Separate bar:** `<Bar dataKey="resolved_count" fill="#10B981" />` rendered as a standalone (not stacked) to visually represent the downward resolution. Since Recharts doesn't natively support negative waterfall segments, the resolved bar is rendered separately alongside the stacked group, matching the existing `DeltaChart` pattern already used in Chart 2.
- **Tooltip:** Custom `WaterfallTooltip` (extends `DarkTooltip` pattern) showing date, start, new, recurring, resolved, and end values.
- **Empty state:** `<NoData />` when `waterfall` array is empty
- **Wrapper:** `<ChartCard title="Net Change Waterfall" subtitle="Per-cycle net movement: start → +new → +recurring → −resolved → end" />`
### Frontend: State & Fetch Changes in `ComplianceChartsPanel`
| Old | New |
|-----|-----|
| `const [mttr, setMttr] = useState([])` | `const [aging, setAging] = useState([])` |
| `const [recurring, setRecurring] = useState([])` | `const [waterfall, setWaterfall] = useState([])` |
| `fetch(…/compliance/mttr)` → `setMttr(d.mttr)` | `fetch(…/compliance/mttr)` → `setAging(d.aging)` |
| `fetch(…/compliance/top-recurring)` → `setRecurring(d.items)` | `fetch(…/compliance/top-recurring)` → `setWaterfall(d.waterfall)` |
The `Promise.all` pattern and error handling remain identical. The other two fetches (`/trends`, `/archer-tickets/status-trend`) are untouched.
### Code Removal
- Delete `MttrChart` function component
- Delete `RecurringChart` function component
- Remove `mttr` and `recurring` state variables and their setter calls
- Remove old `/mttr` handler SQL and logic in `compliance.js`
- Remove old `/top-recurring` handler SQL and logic in `compliance.js`
## Data Models
No schema changes are required. The feature reads from existing columns:
### `compliance_items` (read by Aging endpoint)
| Column | Type | Usage |
|--------|------|-------|
| `status` | TEXT (`'active'` / `'resolved'`) | Filter to active items only |
| `seen_count` | INTEGER (default 1) | Bucketed into age groups via CASE expression |
| `team` | TEXT | Group-by for per-team stacked segments |
### `compliance_uploads` (read by Waterfall endpoint)
| Column | Type | Usage |
|--------|------|-------|
| `report_date` | TEXT (YYYY-MM-DD) | X-axis label, ordering |
| `new_count` | INTEGER (default 0) | New findings in this cycle |
| `recurring_count` | INTEGER (default 0) | Recurring findings in this cycle |
| `resolved_count` | INTEGER (default 0) | Resolved findings in this cycle |
### Derived Waterfall Fields (computed in handler, not stored)
| Field | Computation |
|-------|-------------|
| `start` | Previous cycle's `end` (0 for first cycle) |
| `end` | `start + new_count + recurring_count - resolved_count` |
## Correctness Properties
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
### Property 1: Aging bucketing correctness and conservation
*For any* array of active compliance items with arbitrary `seen_count` values (≥ 1) and arbitrary team assignments, applying the aging bucketing logic SHALL:
- assign every item to exactly one of the four buckets (`1 cycle` for seen_count = 1, `2–3 cycles` for 2 ≤ seen_count ≤ 3, `4–6 cycles` for 4 ≤ seen_count ≤ 6, `7+ cycles` for seen_count ≥ 7),
- produce per-team counts within each bucket that sum to the bucket's total count, and
- produce bucket totals that sum to the total number of input items.
**Validates: Requirements 1.1, 1.2**
### Property 2: Waterfall chain linkage and arithmetic invariant
*For any* ordered sequence of upload records with non-negative `new_count`, `recurring_count`, and `resolved_count` values, computing the waterfall SHALL produce a result where:
- the first row has `start = 0`,
- for every subsequent row N, `start[N] = end[N-1]` (chain linkage), and
- for every row, `end = start + new_count + recurring_count - resolved_count` (arithmetic invariant).
**Validates: Requirements 3.1, 3.2, 3.3**
## Error Handling
### Backend
| Scenario | Behavior |
|----------|----------|
| No active findings (aging endpoint) | Return `{ aging: [] }` with HTTP 200 |
| No uploads (waterfall endpoint) | Return `{ waterfall: [] }` with HTTP 200 |
| SQLite query error (either endpoint) | Log error to console, return HTTP 500 with `{ error: "Database error" }` |
| `seen_count` is NULL or missing | `COALESCE(seen_count, 1)` in the CASE expression treats NULL as 1 cycle |
| `new_count` / `recurring_count` / `resolved_count` is NULL | `COALESCE(..., 0)` in the SELECT ensures zero-default (already present in the uploads table pattern) |
### Frontend
| Scenario | Behavior |
|----------|----------|
| Aging fetch fails (network error or non-200) | `aging` state remains `[]`, `AgingChart` renders `<NoData />` |
| Waterfall fetch fails (network error or non-200) | `waterfall` state remains `[]`, `WaterfallChart` renders `<NoData />` |
| One fetch fails, others succeed | Unaffected charts render normally — each fetch result is handled independently in the existing `try/catch` pattern |
| API returns unexpected shape | Component guards (`data.length === 0`) trigger `<NoData />` gracefully |
## Testing Strategy
### Property-Based Tests (fast-check)
The project backend already uses Jest. Property-based tests will use [fast-check](https://github.com/dubzzz/fast-check) (already available in the Node ecosystem, zero-config with Jest).
Each property test runs a minimum of **100 iterations** with randomly generated inputs.
**Test 1: Aging bucketing correctness and conservation**
- **Tag:** `Feature: compliance-chart-replacements, Property 1: Aging bucketing correctness and conservation`
- **Generator:** Array of objects `{ seen_count: fc.integer({ min: 1, max: 200 }), team: fc.constantFrom('STEAM', 'ACCESS-ENG', 'ACCESS-OPS', 'INTELDEV') }`
- **Assertion:** Extract the pure bucketing/pivoting function from the route handler. For each generated array, verify:
1. Every item maps to exactly one of the four bucket labels
2. Per-team counts in each bucket sum to the bucket total
3. All bucket totals sum to the input array length
**Test 2: Waterfall chain linkage and arithmetic invariant**
- **Tag:** `Feature: compliance-chart-replacements, Property 2: Waterfall chain linkage and arithmetic invariant`
- **Generator:** Array of objects `{ new_count: fc.nat({ max: 50 }), recurring_count: fc.nat({ max: 50 }), resolved_count: fc.nat({ max: 50 }), report_date: fc.date().map(d => d.toISOString().slice(0, 10)) }`
- **Assertion:** Extract the pure waterfall computation function. For each generated sequence, verify:
1. `result[0].start === 0`
2. For all i > 0: `result[i].start === result[i-1].end`
3. For all i: `result[i].end === result[i].start + result[i].new_count + result[i].recurring_count - result[i].resolved_count`
### Unit Tests (Jest, example-based)
| Test | Validates |
|------|-----------|
| Aging endpoint returns correct JSON shape with known seed data | Req 1.3 |
| Aging endpoint returns empty array when no active items exist | Req 1.4 |
| Aging endpoint returns 500 on DB error | Req 1.5 |
| Waterfall endpoint returns empty array when no uploads exist | Req 3.5 |
| Waterfall endpoint returns 500 on DB error | Req 3.6 |
| `AgingChart` renders `<NoData />` for empty data | Req 2.5 |
| `AgingChart` renders ChartCard with correct title | Req 2.3 |
| `WaterfallChart` renders `<NoData />` for empty data | Req 4.5 |
| `WaterfallChart` renders ChartCard with correct title | Req 4.6 |
### Integration Tests
| Test | Validates |
|------|-----------|
| Dashboard fetches `/compliance/mttr` and `/compliance/top-recurring` on mount | Req 5.1, 5.2 |
| Dashboard still fetches `/compliance/trends` and `/archer-tickets/status-trend` | Req 5.3 |
| All four fetches run in parallel via `Promise.all` | Req 5.4 |
| Single fetch failure does not break other charts | Req 5.5 |
### Smoke Tests
| Test | Validates |
|------|-----------|
| `/compliance/mttr` route responds and does NOT return old `{ mttr: [...] }` shape | Req 1.6 |
| `/compliance/top-recurring` route responds and does NOT return old `{ items: [...] }` shape | Req 3.7 |
### Testability Design Note
To enable property-based testing, the bucketing logic and waterfall accumulator will be extracted as **pure exported functions** (e.g., `bucketAgingItems(items)` and `computeWaterfall(uploads)`) separate from the Express route handlers. This allows the property tests to call the pure functions directly without needing HTTP or database mocking.

99
.kiro/specs/compliance-chart-replacements/requirements.md Normal file
View File

@@ -0,0 +1,99 @@
# Requirements Document
## Introduction
Replace two underperforming compliance dashboard charts with more actionable visualizations. Chart 4 (MTTR by Team) is replaced by an Aging Findings Distribution histogram, and Chart 5 (Most Persistent Findings) is replaced by a Net Change Waterfall chart. Both replacements use data already present in the existing SQLite schema (`compliance_items.seen_count` and `compliance_uploads` aggregate columns) and require no database migrations. The four other charts in `ComplianceChartsPanel` remain unchanged.
## Glossary
- **Dashboard**: The compliance charts panel rendered by `ComplianceChartsPanel.js`, containing six chart cards in a responsive grid.
- **Aging_Chart**: The new Chart 4 replacement — a stacked bar / histogram showing active findings distributed across age buckets derived from `seen_count`.
- **Waterfall_Chart**: The new Chart 5 replacement — a waterfall chart showing net movement per upload cycle (starting count → +new → +recurring → −resolved → ending count).
- **Cycle**: A single compliance report upload. Each upload represents one weekly reporting period.
- **Age_Bucket**: A grouping of active findings by how many cycles they have been continuously observed: 1 cycle, 2–3 cycles, 4–6 cycles, 7+ cycles.
- **Seen_Count**: The `seen_count` column on `compliance_items`, incremented each cycle an active finding persists. Starts at 1 on first appearance.
- **Compliance_API**: The Express router at `cve-dashboard/backend/routes/compliance.js` serving `/api/compliance/*` endpoints.
- **ChartCard**: The shared wrapper component providing title, subtitle, and dark-themed card styling for each chart in the Dashboard.
- **Recharts**: The React charting library used by the Dashboard for all visualizations.
- **Design_System**: The dark theme specification defined in `DESIGN_SYSTEM.md`, including color palette, monospace fonts, and card styling.
## Requirements
### Requirement 1: Aging Findings Distribution Backend Endpoint
**User Story:** As a compliance analyst, I want an API endpoint that returns active findings grouped by age bucket, so that the frontend can render the Aging Findings Distribution chart.
#### Acceptance Criteria
1. WHEN a GET request is made to the aging endpoint, THE Compliance_API SHALL return active findings from `compliance_items` grouped into four Age_Buckets: 1 cycle (`seen_count = 1`), 2–3 cycles (`seen_count` between 2 and 3), 4–6 cycles (`seen_count` between 4 and 6), and 7+ cycles (`seen_count >= 7`).
2. WHEN a GET request is made to the aging endpoint, THE Compliance_API SHALL include a per-team breakdown within each Age_Bucket so the frontend can render stacked segments.
3. THE Compliance_API SHALL return the response as a JSON object with an array of bucket objects, each containing the bucket label, total count, and a count per team.
4. WHEN no active findings exist, THE Compliance_API SHALL return an empty array with HTTP status 200.
5. IF a database error occurs, THEN THE Compliance_API SHALL return HTTP status 500 with a JSON error message.
6. THE Compliance_API SHALL replace the existing `/mttr` endpoint path so that no new route paths are introduced and the old MTTR endpoint is removed.
### Requirement 2: Aging Findings Distribution Frontend Chart
**User Story:** As a compliance analyst, I want to see a stacked bar chart showing how many active findings fall into each age bucket, so that I can understand the shape of the current backlog and identify stale findings.
#### Acceptance Criteria
1. THE Aging_Chart SHALL render a vertical stacked bar chart using Recharts with one bar group per Age_Bucket (1 cycle, 2–3 cycles, 4–6 cycles, 7+ cycles) on the X-axis and finding count on the Y-axis.
2. THE Aging_Chart SHALL use one stacked color segment per team, using the existing `TEAM_COLORS` mapping defined in the Dashboard.
3. THE Aging_Chart SHALL be wrapped in a ChartCard with the title "Aging Findings Distribution" and a subtitle describing the chart purpose.
4. THE Aging_Chart SHALL use the shared `DarkTooltip` component to display bucket label, per-team counts, and total on hover.
5. THE Aging_Chart SHALL render a `NoData` placeholder when the API returns an empty array.
6. THE Aging_Chart SHALL occupy the same grid position (Chart 4 slot) previously held by the MTTR chart in the Dashboard layout.
7. THE Aging_Chart SHALL conform to the Design_System by using monospace axis labels, the standard grid style, and the dark card background gradient.
### Requirement 3: Net Change Waterfall Backend Endpoint
**User Story:** As a compliance analyst, I want an API endpoint that returns per-cycle waterfall data (starting count, new, recurring, resolved, ending count), so that the frontend can render the Net Change Waterfall chart.
#### Acceptance Criteria
1. WHEN a GET request is made to the waterfall endpoint, THE Compliance_API SHALL return one row per upload cycle containing: the cycle report date, starting active count, new count, recurring count, resolved count, and ending active count.
2. THE Compliance_API SHALL compute the starting count for each cycle as the ending count of the previous cycle, with the first cycle starting at zero.
3. THE Compliance_API SHALL compute the ending count for each cycle as `starting_count + new_count + recurring_count - resolved_count`.
4. THE Compliance_API SHALL read `new_count`, `recurring_count`, and `resolved_count` directly from the `compliance_uploads` table without requiring additional aggregation queries.
5. WHEN no uploads exist, THE Compliance_API SHALL return an empty array with HTTP status 200.
6. IF a database error occurs, THEN THE Compliance_API SHALL return HTTP status 500 with a JSON error message.
7. THE Compliance_API SHALL replace the existing `/top-recurring` endpoint path so that no new route paths are introduced and the old persistent-findings endpoint is removed.
### Requirement 4: Net Change Waterfall Frontend Chart
**User Story:** As a compliance analyst, I want to see a waterfall chart showing net movement per cycle (starting count → +new → +recurring → −resolved → ending count), so that I can immediately tell whether the compliance posture is improving or degrading.
#### Acceptance Criteria
1. THE Waterfall_Chart SHALL render a bar chart using Recharts where each cycle shows four visual segments: a transparent base (starting count), a red segment for new findings, an amber segment for recurring findings, and a green downward segment for resolved findings, producing a waterfall effect.
2. WHEN a user hovers over any segment, THE Waterfall_Chart SHALL display a DarkTooltip showing the cycle date, starting count, new count, recurring count, resolved count, and ending count.
3. THE Waterfall_Chart SHALL label the X-axis with cycle dates formatted as MM/DD/YY using the existing `fmtDate` helper.
4. THE Waterfall_Chart SHALL label the Y-axis with finding counts using monospace axis styling consistent with the Design_System.
5. THE Waterfall_Chart SHALL render a `NoData` placeholder when the API returns an empty array.
6. THE Waterfall_Chart SHALL be wrapped in a ChartCard with the title "Net Change Waterfall" and a subtitle describing the chart purpose.
7. THE Waterfall_Chart SHALL occupy the same grid position (Chart 5 slot) previously held by the Most Persistent Findings chart in the Dashboard layout.
8. THE Waterfall_Chart SHALL use `#EF4444` for new findings, `#F59E0B` for recurring findings, and `#10B981` for resolved findings, matching the existing color conventions in the Dashboard.
### Requirement 5: Frontend Data Fetching Integration
**User Story:** As a developer, I want the Dashboard to fetch data from the two new endpoints in place of the old ones, so that the panel loads the correct data for the replacement charts without breaking the other four charts.
#### Acceptance Criteria
1. THE Dashboard SHALL replace the fetch call to `/compliance/mttr` with a fetch call to the aging endpoint and store the response in component state.
2. THE Dashboard SHALL replace the fetch call to `/compliance/top-recurring` with a fetch call to the waterfall endpoint and store the response in component state.
3. THE Dashboard SHALL continue to fetch `/compliance/trends` and `/archer-tickets/status-trend` without modification.
4. THE Dashboard SHALL load all four endpoints in parallel using `Promise.all` on component mount, matching the existing loading pattern.
5. IF any individual fetch fails, THEN THE Dashboard SHALL render the corresponding chart in its `NoData` state without affecting the other charts.
### Requirement 6: Removal of Obsolete Code
**User Story:** As a developer, I want the old MTTR and Persistent Findings chart components and endpoints removed, so that the codebase does not contain dead code.
#### Acceptance Criteria
1. THE Dashboard SHALL remove the `MttrChart` component function and all references to the `mttr` state variable.
2. THE Dashboard SHALL remove the `RecurringChart` component function and all references to the `recurring` state variable.
3. THE Compliance_API SHALL remove the old `/mttr` SQL query and handler logic, replacing it with the aging endpoint handler at the same route path.
4. THE Compliance_API SHALL remove the old `/top-recurring` SQL query and handler logic, replacing it with the waterfall endpoint handler at the same route path.

140
.kiro/specs/compliance-chart-replacements/tasks.md Normal file
View File

@@ -0,0 +1,140 @@
# Implementation Plan: Compliance Chart Replacements
## Overview
Replace Chart 4 (MTTR by Team) and Chart 5 (Most Persistent Findings) in the compliance dashboard with two new visualizations: Aging Findings Distribution (stacked bar by age bucket) and Net Change Waterfall (per-cycle net movement). The implementation reuses existing route paths, requires no database migrations, and extracts pure functions for testability.
## Tasks
- [x] 1. Extract pure backend functions and implement aging endpoint
- [x] 1.1 Create `bucketAgingItems(items)` pure function in `cve-dashboard/backend/routes/compliance.js`
- Export a pure function that accepts an array of `{ seen_count, team }` objects
- Bucket each item into one of four age groups: `1 cycle` (seen_count = 1), `2–3 cycles` (2–3), `4–6 cycles` (4–6), `7+ cycles` (≥ 7)
- Pivot into one object per bucket with per-team counts and a total, sorted by ascending age
- Return the array in the response shape defined in the design: `[{ bucket, total, STEAM, ACCESS-ENG, ACCESS-OPS, INTELDEV }]`
- _Requirements: 1.1, 1.2, 1.3_
- [x] 1.2 Replace the `/mttr` route handler with the aging endpoint
- Remove the old MTTR SQL query and handler logic
- Add a new handler at `GET /mttr` that queries active compliance items grouped by bucket CASE expression and team
- Use the SQL from the design: `SELECT CASE WHEN seen_count = 1 THEN '1 cycle' ... END AS bucket, team, COUNT(*) AS count FROM compliance_items WHERE status = 'active' GROUP BY bucket, team ORDER BY ...`
- Call `bucketAgingItems()` to pivot the flat rows, then return `{ aging: [...] }`
- Return `{ aging: [] }` with HTTP 200 when no active findings exist
- Return HTTP 500 with `{ error: "Database error" }` on SQLite errors
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 6.3_
- [ ]* 1.3 Write property test for aging bucketing (Property 1)
- **Property 1: Aging bucketing correctness and conservation**
- Create `cve-dashboard/backend/__tests__/compliance-aging-bucketing.property.test.js`
- Generator: array of `{ seen_count: fc.integer({ min: 1, max: 200 }), team: fc.constantFrom('STEAM', 'ACCESS-ENG', 'ACCESS-OPS', 'INTELDEV') }`
- Assert: every item maps to exactly one bucket, per-team counts sum to bucket total, all bucket totals sum to input length
- **Validates: Requirements 1.1, 1.2**
- [ ]* 1.4 Write unit tests for aging endpoint
- Create `cve-dashboard/backend/__tests__/compliance-aging-endpoint.test.js`
- Test correct JSON shape with known seed data (Req 1.3)
- Test empty array returned when no active items exist (Req 1.4)
- Test HTTP 500 returned on DB error (Req 1.5)
- _Requirements: 1.3, 1.4, 1.5_
- [x] 2. Implement waterfall endpoint with pure function
- [x] 2.1 Create `computeWaterfall(uploads)` pure function in `cve-dashboard/backend/routes/compliance.js`
- Export a pure function that accepts an ordered array of `{ report_date, new_count, recurring_count, resolved_count }`
- Iterate with a running accumulator: first row starts at 0, each subsequent row's start = previous row's end
- Compute `end = start + new_count + recurring_count - resolved_count` for each row
- Return array of `{ date, start, new_count, recurring_count, resolved_count, end }`
- _Requirements: 3.1, 3.2, 3.3_
- [x] 2.2 Replace the `/top-recurring` route handler with the waterfall endpoint
- Remove the old persistent-findings SQL query and handler logic
- Add a new handler at `GET /top-recurring` that queries `compliance_uploads` ordered by `report_date ASC`
- Use `COALESCE` for null safety on count columns
- Call `computeWaterfall()` to build the chain, then return `{ waterfall: [...] }`
- Return `{ waterfall: [] }` with HTTP 200 when no uploads exist
- Return HTTP 500 with `{ error: "Database error" }` on SQLite errors
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 6.4_
- [ ]* 2.3 Write property test for waterfall computation (Property 2)
- **Property 2: Waterfall chain linkage and arithmetic invariant**
- Create `cve-dashboard/backend/__tests__/compliance-waterfall-chain.property.test.js`
- Generator: array of `{ new_count: fc.nat({ max: 50 }), recurring_count: fc.nat({ max: 50 }), resolved_count: fc.nat({ max: 50 }), report_date: fc.date().map(d => d.toISOString().slice(0, 10)) }`
- Assert: first row start = 0, each row N start = row N-1 end, each row end = start + new_count + recurring_count - resolved_count
- **Validates: Requirements 3.1, 3.2, 3.3**
- [ ]* 2.4 Write unit tests for waterfall endpoint
- Create `cve-dashboard/backend/__tests__/compliance-waterfall-endpoint.test.js`
- Test empty array returned when no uploads exist (Req 3.5)
- Test HTTP 500 returned on DB error (Req 3.6)
- Test correct chain computation with known seed data (Req 3.1, 3.2, 3.3)
- _Requirements: 3.1, 3.2, 3.3, 3.5, 3.6_
- [x] 3. Checkpoint — Backend complete
- Ensure all tests pass, ask the user if questions arise.
- [x] 4. Implement frontend AgingChart component
- [x] 4.1 Replace `MttrChart` with `AgingChart` in `cve-dashboard/frontend/src/components/pages/ComplianceChartsPanel.js`
- Remove the `MttrChart` function component entirely
- Add `AgingChart({ data })` that renders a vertical stacked `BarChart` using Recharts
- X-axis: bucket labels (`1 cycle`, `2–3 cycles`, `4–6 cycles`, `7+ cycles`)
- Y-axis: finding count with monospace axis styling
- One `<Bar>` per team key from `TEAM_COLORS` using `stackId="aging"`
- Use `DarkTooltip` for hover showing bucket label, per-team counts, and total
- Render `<NoData />` when data array is empty
- Wrap in `<ChartCard title="Aging Findings Distribution" subtitle="Active findings by age bucket — stacked by team" />`
- Place in the Chart 4 grid slot previously held by MTTR
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7_
- [ ]* 4.2 Write unit tests for AgingChart
- Test `<NoData />` renders for empty data (Req 2.5)
- Test ChartCard renders with correct title (Req 2.3)
- _Requirements: 2.3, 2.5_
- [x] 5. Implement frontend WaterfallChart component
- [x] 5.1 Replace `RecurringChart` with `WaterfallChart` in `cve-dashboard/frontend/src/components/pages/ComplianceChartsPanel.js`
- Remove the `RecurringChart` function component entirely
- Add `WaterfallChart({ data })` that renders a vertical `BarChart` using Recharts
- Stacked bars: transparent base `<Bar dataKey="start" stackId="w" fill="transparent" />`, red new `<Bar dataKey="new_count" fill="#EF4444" />`, amber recurring `<Bar dataKey="recurring_count" fill="#F59E0B" />`
- Separate bar for resolved: `<Bar dataKey="resolved_count" fill="#10B981" />` (not stacked, matching existing DeltaChart pattern)
- X-axis: cycle dates formatted via `fmtDate`, Y-axis: finding count with monospace styling
- Custom `WaterfallTooltip` showing date, start, new, recurring, resolved, and end values
- Render `<NoData />` when data array is empty
- Wrap in `<ChartCard title="Net Change Waterfall" subtitle="Per-cycle net movement: start → +new → +recurring → −resolved → end" />`
- Place in the Chart 5 grid slot previously held by Most Persistent Findings
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8_
- [ ]* 5.2 Write unit tests for WaterfallChart
- Test `<NoData />` renders for empty data (Req 4.5)
- Test ChartCard renders with correct title (Req 4.6)
- _Requirements: 4.5, 4.6_
- [x] 6. Update state and fetch logic in ComplianceChartsPanel
- [x] 6.1 Replace state variables and fetch calls
- Replace `const [mttr, setMttr] = useState([])` with `const [aging, setAging] = useState([])`
- Replace `const [recurring, setRecurring] = useState([])` with `const [waterfall, setWaterfall] = useState([])`
- In the `Promise.all` fetch block, update the `/compliance/mttr` response handler: `setAging(d.aging || [])`
- Update the `/compliance/top-recurring` response handler: `setWaterfall(d.waterfall || [])`
- Keep `/compliance/trends` and `/archer-tickets/status-trend` fetches unchanged
- Pass `aging` to `<AgingChart data={aging} />` and `waterfall` to `<WaterfallChart data={waterfall} />`
- Ensure individual fetch failures render `<NoData />` without affecting other charts
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 6.1, 6.2_
- [ ]* 6.2 Write integration tests for dashboard fetch logic
- Test that dashboard fetches `/compliance/mttr` and `/compliance/top-recurring` on mount (Req 5.1, 5.2)
- Test that `/compliance/trends` and `/archer-tickets/status-trend` are still fetched (Req 5.3)
- Test that all four fetches run in parallel via `Promise.all` (Req 5.4)
- Test that a single fetch failure does not break other charts (Req 5.5)
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5_
- [x] 7. Final checkpoint — All tests pass
- Ensure all tests pass, ask the user if questions arise.
## Notes
- Tasks marked with `*` are optional and can be skipped for faster MVP
- Each task references specific requirements for traceability
- Checkpoints ensure incremental validation after backend and full integration
- Property tests validate universal correctness properties from the design (Properties 1 and 2)
- Unit tests validate specific examples and edge cases
- Pure functions `bucketAgingItems()` and `computeWaterfall()` are extracted first to enable property-based testing
- No new route paths are introduced — `/mttr` and `/top-recurring` are reused in-place
- No database migrations are needed — all data comes from existing columns

1
.kiro/specs/fp-submissions-cleanup/.config.kiro Normal file
View File

@@ -0,0 +1 @@
{"specId": "f0fd28b9-12b8-48bb-b0b2-3a0e5872a3e4", "workflowType": "requirements-first", "specType": "feature"}

222
.kiro/specs/fp-submissions-cleanup/design.md Normal file
View File

@@ -0,0 +1,222 @@
# Design Document: FP Submissions Cleanup
## Overview
This feature addresses three usability improvements to the FP Submissions section in the Queue Panel:
1. **Auto-clear approved submissions** — filter out approved FP submissions from the visible list so finalized items do not clutter the panel.
2. **Dismiss rejected submissions** — add an X button on rejected submissions allowing users to permanently hide them from view, persisted in the database.
3. **Collapsible Submissions section** — add a collapse/expand toggle on the SUBMISSIONS header, with state persisted in localStorage.
The design builds on the existing `ivantiFpWorkflow.js` route, the `QueuePanel` component's submissions section in `ReportingPage.js`, and the `ivanti_fp_submissions` table. Changes are minimal: one new database column (`dismissed_at`), one new API endpoint (`PATCH /submissions/:id/dismiss`), and frontend filtering/UI logic.
## Architecture
```mermaid
sequenceDiagram
participant U as User
participant FE as React Frontend
participant BE as Express Backend
participant DB as PostgreSQL
Note over FE,DB: Auto-Clear Approved (frontend filter only)
FE->>BE: GET /api/ivanti/fp-workflow/submissions
BE->>DB: SELECT * FROM ivanti_fp_submissions WHERE user_id = $1
DB-->>BE: All submissions (including approved)
BE-->>FE: JSON array of submissions
FE->>FE: Filter out approved + dismissed submissions for display
Note over U,DB: Dismiss Rejected Submission
U->>FE: Click X button on rejected submission
FE->>BE: PATCH /api/ivanti/fp-workflow/submissions/:id/dismiss
BE->>DB: UPDATE ivanti_fp_submissions SET dismissed_at = NOW() WHERE id = $1
BE->>DB: INSERT audit_logs (action: ivanti_fp_submission_dismissed)
BE-->>FE: 200 OK
FE->>FE: Remove submission from visible list
Note over U,FE: Collapse/Expand Toggle
U->>FE: Click collapse toggle on SUBMISSIONS header
FE->>FE: Toggle collapsed state
FE->>FE: Persist to localStorage (key: steam_submissions_collapsed)
```
## Components and Interfaces
### Backend
#### Extended Route: `backend/routes/ivantiFpWorkflow.js`
One new endpoint added to the existing `createIvantiFpWorkflowRouter`.
**Endpoint: `PATCH /api/ivanti/fp-workflow/submissions/:id/dismiss`**
Marks a rejected submission as dismissed.
- Auth: `requireAuth()`, `requireGroup('Admin', 'Standard_User')`
- Ownership: verified via `user_id` match (returns 403 otherwise)
- Lifecycle guard: only submissions with `lifecycle_status === 'rejected'` can be dismissed (returns 400 otherwise)
- On success: sets `dismissed_at = NOW()` on the submission record, logs audit entry
- Response: `{ success: true }`
Request: no body required.
Response:
```json
{ "success": true }
```
Error responses:
- 404: `{ "error": "Submission not found." }`
- 403: `{ "error": "You can only dismiss your own submissions." }`
- 400: `{ "error": "Only rejected submissions can be dismissed." }`
#### Modified Endpoint: `GET /api/ivanti/fp-workflow/submissions`
No changes to the backend query — it continues to return all submissions for the user. The frontend handles filtering. This preserves the ability to open approved submissions from workflow badges in read-only mode.
#### Pure Helper Function (exported for testing)
- `filterVisibleSubmissions(submissions)` — new, filters an array of submission objects to exclude those with `lifecycle_status === 'approved'` or `dismissed_at !== null`. Returns the filtered array.
### Frontend
#### QueuePanel Submissions Section Modifications
**Filtering logic:**
The `fpSubmissions` array passed to `QueuePanel` is filtered before rendering:
- Exclude submissions where `lifecycle_status === 'approved'`
- Exclude submissions where `dismissed_at` is not null
This filtering is applied in the `useMemo` that already enriches `fpSubmissionsRaw` with Ivanti workflow state.
**Dismiss button (X icon):**
For submissions with `lifecycle_status === 'rejected'`, render an X button (lucide `X` icon, 12px) on the right side of the submission row. On click:
1. Call `PATCH /api/ivanti/fp-workflow/submissions/:id/dismiss`
2. On success, re-fetch submissions (or optimistically remove from local state)
The X button uses `e.stopPropagation()` to prevent triggering the row's click-to-edit handler.
**Collapse/expand toggle:**
The SUBMISSIONS header row gets a clickable chevron icon:
- Expanded state: `ChevronDown` icon (already imported from lucide-react)
- Collapsed state: `ChevronUp` icon
State management:
- `const [submissionsCollapsed, setSubmissionsCollapsed] = useState(() => localStorage.getItem('steam_submissions_collapsed') === 'true')`
- On toggle: flip state and persist to localStorage
- When collapsed: hide the submission list, keep the header visible with the count badge
- Default: expanded (when localStorage has no stored value)
localStorage key: `steam_submissions_collapsed`
#### FpEditModal — No Changes
The `FpEditModal` already handles approved submissions in read-only mode. Since the modal can be opened from workflow badges in the Reporting Table (which bypass the Queue Panel filter), no changes are needed.
## Data Models
### Schema Change to `ivanti_fp_submissions`
One new column:
```sql
ALTER TABLE ivanti_fp_submissions ADD COLUMN dismissed_at TIMESTAMPTZ DEFAULT NULL;
```
- `NULL` means the submission is not dismissed (visible, subject to other filters)
- A timestamp value means the submission was dismissed at that time (hidden from the Submissions section)
### Migration Script: `backend/migrations/add_fp_submissions_dismissed.js`
Applies the schema change idempotently. Uses the same pattern as `add_fp_submission_editing.js`:
- PostgreSQL: `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`
- Wrapped in try/catch for safety
```javascript
const pool = require('../db');
async function run() {
console.log('Starting FP submissions dismissed migration...');
try {
await pool.query(`ALTER TABLE ivanti_fp_submissions ADD COLUMN IF NOT EXISTS dismissed_at TIMESTAMPTZ DEFAULT NULL`);
console.log('✓ dismissed_at column added (or already exists)');
} catch (err) {
console.error('Error adding dismissed_at column:', err.message);
process.exit(1);
}
console.log('Migration complete.');
process.exit(0);
}
run();
```
## Correctness Properties
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
### Property 1: Submission Visibility Filter
*For any* array of FP submission objects with arbitrary `lifecycle_status` values (from the set {submitted, approved, rejected, rework, resubmitted}) and arbitrary `dismissed_at` values (null or a timestamp string), `filterVisibleSubmissions(submissions)` should return only submissions where `lifecycle_status` is NOT "approved" AND `dismissed_at` is null. Additionally, every submission in the input that satisfies both conditions must appear in the output, and the output length must be less than or equal to the input length.
**Validates: Requirements 1.1, 2.2, 2.3**
### Property 2: Dismiss Button Visibility Predicate
*For any* FP submission object with a `lifecycle_status` value drawn from {submitted, approved, rejected, rework, resubmitted} and a `dismissed_at` value (null or timestamp), the dismiss button should be rendered if and only if `lifecycle_status === 'rejected'` AND `dismissed_at` is null.
**Validates: Requirements 2.1**
## Error Handling
### Dismiss Endpoint Errors
| Condition | HTTP Status | Response | Behavior |
|-----------|-------------|----------|----------|
| Submission not found | 404 | `{ "error": "Submission not found." }` | No state change |
| User does not own submission | 403 | `{ "error": "You can only dismiss your own submissions." }` | No state change |
| Submission not in "rejected" status | 400 | `{ "error": "Only rejected submissions can be dismissed." }` | No state change |
| Database error | 500 | `{ "error": "Internal server error." }` | Log error, no state change |
### Frontend Error Handling
- If the dismiss API call fails, show a brief error toast/message and leave the submission in the list
- If `localStorage` is unavailable (private browsing), the collapse state defaults to expanded and is not persisted (graceful degradation)
## Testing Strategy
### Property-Based Testing
Use `fast-check` as the property-based testing library. Each correctness property maps to a single property-based test with a minimum of 100 iterations.
Property tests focus on the pure filtering function:
- `filterVisibleSubmissions(submissions)` — Property 1
- Dismiss button visibility predicate — Property 2
Tag format: **Feature: fp-submissions-cleanup, Property {number}: {title}**
Test file: `backend/__tests__/fp-submissions-cleanup.property.test.js`
### Unit Testing
Unit tests cover specific examples, edge cases, and integration points:
- **Filter with all approved**: verify empty result when all submissions are approved
- **Filter with all dismissed**: verify empty result when all submissions are dismissed
- **Filter with mixed statuses**: verify correct subset returned
- **Dismiss endpoint — happy path**: verify `dismissed_at` is set and audit log created
- **Dismiss endpoint — wrong status**: verify 400 when attempting to dismiss a non-rejected submission
- **Dismiss endpoint — ownership check**: verify 403 when non-owner attempts dismiss
- **Collapse state persistence**: verify localStorage read/write with the correct key
- **Collapse default**: verify expanded state when localStorage has no value
Test file: `backend/__tests__/fp-submissions-cleanup.test.js`
### Integration Testing
- Dismiss a rejected submission via API, re-fetch submissions, verify it's excluded from the response when filtered
- Verify audit log entry is created with correct action and entity_id
- Verify approved submissions are still accessible via direct ID lookup (for the edit modal)

51
.kiro/specs/fp-submissions-cleanup/requirements.md Normal file
View File

@@ -0,0 +1,51 @@
# Requirements Document
## Introduction
This feature addresses three usability issues in the FP Submissions section of the Queue Panel. Approved submissions currently persist indefinitely in the list, rejected submissions cannot be dismissed, and the entire Submissions section cannot be collapsed. These enhancements improve the signal-to-noise ratio of the Queue Panel by giving users control over which submissions remain visible and allowing the section to be hidden when not needed.
## Glossary
- **Dashboard**: The STEAM Security Dashboard application
- **Queue_Panel**: The slide-out panel in the Reporting Page that displays the user's Ivanti todo queue items, FP workflow submissions, and action buttons
- **Submissions_Section**: The "SUBMISSIONS" area within the Queue_Panel that lists the user's FP workflow submissions with their lifecycle statuses
- **FP_Submission**: A local database record in the `ivanti_fp_submissions` table tracking a False Positive workflow submission, including its lifecycle status
- **Lifecycle_Status**: The current state of an FP submission: submitted, approved, rejected, rework, or resubmitted
- **Dismissed_Submission**: An FP_Submission that the user has explicitly removed from the Submissions_Section view without deleting the underlying database record
## Requirements
### Requirement 1: Auto-Clear Approved Submissions
**User Story:** As an editor or admin, I want approved FP submissions to be automatically removed from the Submissions list, so that finalized items do not clutter the Queue Panel indefinitely.
#### Acceptance Criteria
1. WHEN the Dashboard fetches FP_Submissions for display in the Submissions_Section, THE Dashboard SHALL exclude submissions with Lifecycle_Status "approved" from the visible list
2. WHEN an FP_Submission transitions to Lifecycle_Status "approved" while the Queue_Panel is open, THE Dashboard SHALL remove the submission from the Submissions_Section without requiring a page refresh
3. THE Dashboard SHALL retain approved FP_Submission records in the database for audit and history purposes
4. WHEN the user opens the FpEditModal from a Reporting Table workflow badge for an approved submission, THE Dashboard SHALL still display the submission data in read-only mode regardless of its visibility in the Submissions_Section
### Requirement 2: Dismiss Rejected Submissions
**User Story:** As an editor or admin, I want to dismiss rejected FP submissions from the Submissions list, so that I can remove items I no longer intend to rework.
#### Acceptance Criteria
1. WHEN an FP_Submission has Lifecycle_Status "rejected", THE Dashboard SHALL display a dismiss button (X icon) on the submission row in the Submissions_Section
2. WHEN the user clicks the dismiss button on a rejected submission, THE Dashboard SHALL mark the FP_Submission as dismissed and remove it from the Submissions_Section
3. THE Dashboard SHALL persist the dismissed state so that dismissed submissions remain hidden across page reloads and session changes
4. THE Dashboard SHALL retain dismissed FP_Submission records in the database for audit and history purposes
5. IF the user dismisses a rejected submission, THEN THE Dashboard SHALL log an audit entry with action "ivanti_fp_submission_dismissed" and the submission's workflow batch ID as entity ID
### Requirement 3: Collapsible Submissions Section
**User Story:** As a user, I want to collapse the Submissions section in the Queue Panel, so that I can hide it when I only need to work with queue items.
#### Acceptance Criteria
1. THE Dashboard SHALL display a collapse/expand toggle control on the Submissions_Section header row
2. WHEN the user clicks the collapse toggle, THE Dashboard SHALL hide the list of FP_Submissions while keeping the section header visible with the submission count
3. WHEN the user clicks the expand toggle, THE Dashboard SHALL reveal the full list of FP_Submissions
4. THE Dashboard SHALL persist the collapsed/expanded state in browser local storage so that the preference survives page reloads
5. THE Dashboard SHALL default the Submissions_Section to the expanded state when no stored preference exists

50
.kiro/specs/fp-submissions-cleanup/tasks.md Normal file
View File

@@ -0,0 +1,50 @@
# Tasks
## Task 1: Database Migration — Add `dismissed_at` Column
- [x] 1.1 Create migration file `backend/migrations/add_fp_submissions_dismissed.js` that adds a `dismissed_at TIMESTAMPTZ DEFAULT NULL` column to the `ivanti_fp_submissions` table using `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`
- [x] 1.2 Run the migration and verify the column exists in the database schema
## Task 2: Backend — Dismiss Endpoint
- [x] 2.1 Add `PATCH /submissions/:id/dismiss` endpoint to `backend/routes/ivantiFpWorkflow.js` with `requireAuth()` and `requireGroup('Admin', 'Standard_User')` middleware
- [x] 2.2 Implement ownership verification (user_id match, return 403 if mismatch)
- [x] 2.3 Implement lifecycle guard (only `lifecycle_status === 'rejected'` can be dismissed, return 400 otherwise)
- [x] 2.4 Set `dismissed_at = NOW()` on the submission record on success
- [x] 2.5 Log audit entry with action `ivanti_fp_submission_dismissed`, entity_type `ivanti_workflow`, and the workflow batch ID as entity_id
## Task 3: Backend — Pure Filter Function
- [x] 3.1 Create and export `filterVisibleSubmissions(submissions)` function in `backend/routes/ivantiFpWorkflow.js` that excludes submissions with `lifecycle_status === 'approved'` or `dismissed_at !== null`
- [x] 3.2 Create and export `shouldShowDismissButton(submission)` predicate that returns true only when `lifecycle_status === 'rejected'` and `dismissed_at` is null
## Task 4: Frontend — Filter Approved and Dismissed Submissions
- [x] 4.1 Modify the `fpSubmissions` useMemo in `ReportingPage.js` to filter out submissions where `lifecycle_status === 'approved'` or `dismissed_at` is not null before passing to the QueuePanel
- [x] 4.2 Verify that the FpEditModal can still be opened for approved submissions from Reporting Table workflow badges (no filtering on that path)
## Task 5: Frontend — Dismiss Button on Rejected Submissions
- [x] 5.1 Add an X button (lucide `X` icon, 12px) to the right side of submission rows where `lifecycle_status === 'rejected'` in the QueuePanel submissions section
- [x] 5.2 Implement click handler that calls `PATCH /api/ivanti/fp-workflow/submissions/:id/dismiss` and removes the submission from the visible list on success
- [x] 5.3 Use `e.stopPropagation()` on the X button to prevent triggering the row's click-to-edit handler
- [x] 5.4 Show error feedback if the dismiss API call fails
## Task 6: Frontend — Collapsible Submissions Section
- [x] 6.1 Add `submissionsCollapsed` state initialized from `localStorage.getItem('steam_submissions_collapsed')`, defaulting to `false` (expanded)
- [x] 6.2 Add a clickable chevron icon (ChevronDown when expanded, ChevronUp when collapsed) to the SUBMISSIONS header row
- [x] 6.3 Conditionally render the submissions list based on collapsed state — when collapsed, hide the list but keep the header with the count badge visible
- [x] 6.4 Persist collapsed/expanded state to localStorage on toggle using key `steam_submissions_collapsed`
## Task 7: Property-Based Tests
- [x] 7.1 Create `backend/__tests__/fp-submissions-cleanup.property.test.js` with fast-check
- [x] 7.2 Implement Property 1 test: for any array of submissions with random lifecycle_status and dismissed_at values, `filterVisibleSubmissions` returns only non-approved and non-dismissed submissions (min 100 iterations)
- [x] 7.3 Implement Property 2 test: for any submission with random lifecycle_status and dismissed_at, `shouldShowDismissButton` returns true iff status is 'rejected' and dismissed_at is null (min 100 iterations)
## Task 8: Unit and Integration Tests
- [x] 8.1 Write unit tests for the dismiss endpoint (happy path, wrong status, ownership check, not found)
- [x] 8.2 Write unit tests for filter edge cases (all approved, all dismissed, mixed, empty array)
- [x] 8.3 Write integration test verifying dismissed submissions remain in the database but are excluded from the filtered list

1
.kiro/specs/jira-api-compliance-cleanup/.config.kiro Normal file
View File

@@ -0,0 +1 @@
{"specId": "9b8872ac-5b3f-430b-ac55-c243cf049756", "workflowType": "requirements-first", "specType": "feature"}

133
.kiro/specs/jira-api-compliance-cleanup/design.md Normal file
View File

@@ -0,0 +1,133 @@
# Design Document
## Overview
This design addresses two compliance issues flagged during the Charter Jira Data Center service account approval review (ATLSUP ticket), plus the corresponding documentation update. The changes are minimal and surgical — removing one Express route, changing one string literal, and updating one documentation file.
## Architecture
### Current State
The Jira integration has three layers:
1. **Helper layer** (`backend/helpers/jiraApi.js`): Low-level HTTP functions (`jiraGet`, `jiraPost`, etc.) and high-level operations (`getIssue`, `searchIssuesByKeys`, `searchIssues`, `createIssue`, etc.). All Charter compliance enforcement (rate limits, delays, blocked paths, explicit fields) lives here.
2. **Route layer** (`backend/routes/jiraTickets.js`): Express routes mounted at `/api/jira`. Includes both Jira API integration routes (connection-test, lookup, search, create-in-jira, sync-all, single sync) and local CRUD routes.
3. **Documentation layer** (`docs/api/jira-api-use-cases.md`): Documents all Jira API use cases, compliance posture, and estimated daily usage for the ATLSUP reviewer.
### Changes Required
#### Change 1: Remove `POST /api/jira/search` route
**File:** `backend/routes/jiraTickets.js`
**Action:** Delete the entire `router.post('/search', ...)` handler (approximately lines 136–196). This route accepts arbitrary JQL from the frontend and passes it to `jiraApi.searchIssues()`. No frontend code references this endpoint (confirmed via codebase search). The underlying `searchIssues()` helper function in `jiraApi.js` is NOT removed — it is still called internally by `getIssue()` and `searchIssuesByKeys()`.
**Impact:** None on existing workflows. The four actual workflows (create, lookup, single sync, bulk sync) use dedicated routes that do not depend on the search route.
#### Change 2: Widen JQL window from `-24h` to `-72h`
**File:** `backend/helpers/jiraApi.js`
**Action:** In the `searchIssuesByKeys()` function, change the JQL string literal from:
```
`key in (${keyList}) AND updated >= -24h AND project = ${JIRA_PROJECT_KEY}`
```
to:
```
`key in (${keyList}) AND updated >= -72h AND project = ${JIRA_PROJECT_KEY}`
```
**Impact:** The bulk sync-all flow will now pick up tickets that were updated over the weekend. This increases the potential result set size for Monday morning syncs but remains well within the 1000-result cap (the team tracks dozens to low hundreds of tickets, not thousands).
#### Change 3: Update API documentation
**File:** `docs/api/jira-api-use-cases.md`
**Actions:**
- Remove the entire "Use Case 8: JQL Search (Bulk Sync)" section that references `POST /rest/api/2/search` with arbitrary JQL, and renumber it to describe only the scoped bulk sync via `GET /rest/api/2/search` with the predefined key-based JQL pattern
- Update the JQL pattern in the bulk sync use case from `updated >= -24h` to `updated >= -72h`
- Update the Compliance Summary Table: change the "Bulk reads via JQL" row to clarify that JQL is predefined/scoped (not arbitrary)
- Update the "JQL scoping" row to reflect the `-72h` window
- Remove the "JQL search (sync)" row from the Estimated Daily API Usage table or update it to reflect only the scoped bulk sync
- Recalculate the total estimated daily API call range
- Add `POST /rest/api/2/search` to the Blocked Endpoints section since arbitrary JQL search via POST is no longer used
## Correctness Properties
### Property 1: JQL window is always 72 hours in bulk sync (Invariant)
**Requirement:** 2.1, 2.3
**Property:** For any non-empty array of issue keys passed to `searchIssuesByKeys()`, the generated JQL string SHALL contain the substring `updated >= -72h` and SHALL contain the substring `project =`.
**Type:** Property-based test — the JQL structure must hold for any valid input array of keys.
**Test approach:** Mock `searchIssues()` to capture the JQL argument. Generate random arrays of valid-looking issue keys (e.g., `KEY-1` through `KEY-N`). Assert the JQL always contains `updated >= -72h` and `project =`.
### Property 2: Search route is absent from router (Example)
**Requirement:** 1.1, 1.2
**Property:** After the route removal, a `POST` request to `/api/jira/search` SHALL return HTTP 404.
**Type:** Example-based test — single request, single assertion.
**Test approach:** Mount the router in a test Express app, send `POST /api/jira/search` with a body, assert 404.
### Property 3: Existing routes remain functional after search route removal (Example)
**Requirement:** 1.3, 1.4, 1.5, 1.6
**Property:** The routes `GET /api/jira/lookup/:issueKey`, `POST /api/jira/sync-all`, `POST /api/jira/:id/sync`, and `POST /api/jira/create-in-jira` SHALL continue to respond with non-404 status codes.
**Type:** Example-based test — verify each route is still registered.
**Test approach:** Mount the router, send requests to each endpoint, assert none return 404 (they may return 503 if Jira is not configured, which is fine — the point is the route exists).
### Property 4: searchIssues helper remains exported (Example)
**Requirement:** 4.1
**Property:** The `jiraApi` module SHALL export `searchIssues` as a function.
**Type:** Example-based test — single assertion on module exports.
### Property 5: Documentation does not reference removed endpoint (Example)
**Requirement:** 3.1, 3.3, 3.4
**Property:** The file `docs/api/jira-api-use-cases.md` SHALL NOT contain the string `POST /api/jira/search` or `POST /rest/api/2/search` as an active use case.
**Type:** Example-based test — read file, assert absence of the string.
### Property 6: Documentation reflects 72-hour window (Example)
**Requirement:** 3.2
**Property:** The file `docs/api/jira-api-use-cases.md` SHALL contain `updated >= -72h` in the bulk sync use case and SHALL NOT contain `updated >= -24h`.
**Type:** Example-based test — read file, assert string presence/absence.
## File Changes
| File | Change Type | Description |
|---|---|---|
| `backend/routes/jiraTickets.js` | Modify | Remove the `router.post('/search', ...)` handler (~60 lines) |
| `backend/helpers/jiraApi.js` | Modify | Change `-24h` to `-72h` in `searchIssuesByKeys()` (1 line) |
| `docs/api/jira-api-use-cases.md` | Modify | Remove search use case, update JQL window, update compliance table, update usage estimates, add POST search to blocked endpoints |
## Dependencies
No new dependencies. No changes to `package.json`. No database migrations. No frontend changes required (no frontend code references the search endpoint).
## Risk Assessment
**Low risk.** All three changes are removals or single-line edits in well-understood code paths:
- The search route removal has zero callers in the frontend codebase
- The JQL window change is a single string literal with no behavioral side effects beyond returning a larger result set
- The documentation changes are purely textual
The only regression risk is if an undiscovered caller depends on `POST /api/jira/search`. The codebase search confirmed no such caller exists.

68
.kiro/specs/jira-api-compliance-cleanup/requirements.md Normal file
View File

@@ -0,0 +1,68 @@
# Requirements Document
## Introduction
The STEAM Security Dashboard integrates with Charter's Jira Data Center via a service account. A service account approval request (ATLSUP ticket) was submitted and the reviewer identified two compliance issues that must be resolved before production approval:
1. The `POST /api/jira/search` Express route accepts arbitrary JQL from the frontend and proxies it to `POST /rest/api/2/search`, which is not an approved Jira API pattern. This endpoint must be removed entirely — the dashboard's actual workflows (create, sync, lookup) are already served by other endpoints.
2. The `searchIssuesByKeys()` bulk-sync function uses `updated >= -24h` in its JQL, but the team works Monday–Friday, meaning the worst-case gap between syncs is Friday evening to Monday morning (~60 hours). The window must be widened to `-72h` to cover weekends.
3. The API use-cases documentation (`jira-api-use-cases.md`) must be updated to reflect both changes so the compliance summary remains accurate for the ATLSUP reviewer.
## Glossary
- **Dashboard**: The STEAM Security Dashboard Node.js/Express backend application
- **Search_Route**: The `POST /api/jira/search` Express route in `backend/routes/jiraTickets.js` that accepts arbitrary JQL from the frontend
- **SearchIssuesByKeys_Function**: The `searchIssuesByKeys()` helper function in `backend/helpers/jiraApi.js` used by the bulk sync-all flow
- **SearchIssues_Function**: The `searchIssues()` helper function in `backend/helpers/jiraApi.js` that executes JQL queries via `GET /rest/api/2/search`
- **Sync_All_Route**: The `POST /api/jira/sync-all` Express route that bulk-refreshes all tracked Jira tickets
- **JQL_Window**: The `updated >= -Xh` clause appended to JQL queries to limit results to recently-changed issues
- **API_Documentation**: The file `docs/api/jira-api-use-cases.md` that documents all Jira API use cases for the ATLSUP reviewer
- **Compliance_Summary_Table**: The table in the API_Documentation that summarizes Charter compliance posture
## Requirements
### Requirement 1: Remove the arbitrary JQL search route
**User Story:** As a service account reviewer, I want the dashboard to not expose an arbitrary JQL search endpoint, so that the integration only uses approved Jira API patterns.
#### Acceptance Criteria
1. WHEN the Dashboard starts, THE Dashboard SHALL NOT register a `POST /api/jira/search` route
2. WHEN a client sends a `POST` request to `/api/jira/search`, THE Dashboard SHALL respond with HTTP 404
3. WHILE the Search_Route is removed, THE Sync_All_Route SHALL continue to function by calling the SearchIssuesByKeys_Function
4. WHILE the Search_Route is removed, THE Dashboard SHALL continue to support single-ticket lookup via `GET /api/jira/lookup/:issueKey`
5. WHILE the Search_Route is removed, THE Dashboard SHALL continue to support single-ticket sync via `POST /api/jira/:id/sync`
6. WHILE the Search_Route is removed, THE Dashboard SHALL continue to support ticket creation via `POST /api/jira/create-in-jira`
### Requirement 2: Widen the bulk-sync JQL time window to 72 hours
**User Story:** As a dashboard administrator, I want the bulk sync to use a 72-hour JQL window, so that tickets updated over the weekend are captured on Monday morning.
#### Acceptance Criteria
1. WHEN the SearchIssuesByKeys_Function builds a JQL query, THE SearchIssuesByKeys_Function SHALL include the clause `updated >= -72h` instead of `updated >= -24h`
2. WHEN the Sync_All_Route processes a batch of ticket keys, THE Sync_All_Route SHALL receive results that include tickets updated within the last 72 hours
3. WHILE the JQL_Window is set to 72 hours, THE SearchIssuesByKeys_Function SHALL continue to include the `project = <KEY>` clause in the JQL query
4. WHILE the JQL_Window is set to 72 hours, THE SearchIssuesByKeys_Function SHALL continue to cap `maxResults` at 1000
### Requirement 3: Update the API use-cases documentation
**User Story:** As a service account reviewer, I want the API documentation to accurately reflect the dashboard's current Jira API usage, so that the ATLSUP approval can proceed without discrepancies.
#### Acceptance Criteria
1. WHEN the API_Documentation is updated, THE API_Documentation SHALL NOT contain a use case entry for `POST /api/jira/search` or arbitrary JQL search
2. WHEN the API_Documentation is updated, THE API_Documentation SHALL show the JQL pattern for the bulk sync use case as `updated >= -72h` instead of `updated >= -24h`
3. WHEN the API_Documentation is updated, THE Compliance_Summary_Table SHALL accurately reflect that all JQL queries use scoped, predefined patterns with no arbitrary JQL passthrough
4. WHEN the API_Documentation is updated, THE API_Documentation SHALL update the estimated daily API usage table to remove the row for the arbitrary JQL search endpoint
5. WHEN the API_Documentation is updated, THE API_Documentation SHALL update the total estimated daily API call range to reflect the removal of the search endpoint
### Requirement 4: Preserve the internal searchIssues helper for scoped callers
**User Story:** As a developer, I want the internal `searchIssues()` helper to remain available for use by `getIssue()` and `searchIssuesByKeys()`, so that existing scoped JQL operations continue to work.
#### Acceptance Criteria
1. WHILE the Search_Route is removed, THE SearchIssues_Function SHALL remain exported from `jiraApi.js` for use by internal callers
2. WHEN the `getIssue()` function is called, THE SearchIssues_Function SHALL execute the JQL query and return the matching issue
3. WHEN the SearchIssuesByKeys_Function is called, THE SearchIssues_Function SHALL execute the bulk JQL query and return matching issues

28
.kiro/specs/jira-api-compliance-cleanup/tasks.md Normal file
View File

@@ -0,0 +1,28 @@
# Tasks
## Task 1: Remove the POST /api/jira/search route
- [x] 1.1 Delete the `router.post('/search', ...)` handler from `backend/routes/jiraTickets.js` (the entire block from the JSDoc comment through the closing `});`)
- [x] 1.2 Verify no other code in the routes file references the removed handler
- [x] 1.3 Verify the server starts without errors after the route removal
## Task 2: Widen the JQL time window to 72 hours
- [x] 2.1 In `backend/helpers/jiraApi.js`, change the `searchIssuesByKeys()` JQL string from `updated >= -24h` to `updated >= -72h`
- [x] 2.2 Update the JSDoc comment on `searchIssuesByKeys()` if it references the 24-hour window
## Task 3: Update the API use-cases documentation
- [x] 3.1 In `docs/api/jira-api-use-cases.md`, update the bulk sync use case (Use Case 8) to reflect the `-72h` JQL window and remove any reference to arbitrary JQL or `POST /rest/api/2/search`
- [x] 3.2 Update the Compliance Summary Table to reflect that JQL queries use predefined scoped patterns (no arbitrary JQL passthrough) and the `-72h` window
- [x] 3.3 Update the Estimated Daily API Usage table to remove the arbitrary JQL search row and recalculate the total
- [x] 3.4 Add `POST /rest/api/2/search` to the Blocked Endpoints section with an explanation that arbitrary JQL search via POST is not used
## Task 4: Write property-based test for JQL window invariant
- [x] 4.1 Create a property-based test that verifies `searchIssuesByKeys()` always generates JQL containing `updated >= -72h` and `project =` for any non-empty array of issue keys
## Task 5: Write example-based tests for route removal and remaining routes
- [x] 5.1 Write a test that sends `POST /api/jira/search` and asserts HTTP 404
- [x] 5.2 Write tests that verify the remaining Jira routes (`GET /lookup/:issueKey`, `POST /sync-all`, `POST /:id/sync`, `POST /create-in-jira`) still respond with non-404 status codes

244
.kiro/specs/multi-bu-tenancy/design.md Normal file
View File

@@ -0,0 +1,244 @@
# Design Document: Multi-BU Tenancy (Option B)
## Overview
This design adds per-user BU team scoping to the existing single-tenant dashboard. The approach uses a single Ivanti API key with a broadened sync filter, stores all findings in the existing cache, and filters at query time based on the authenticated user's assigned teams. The compliance page leverages its existing team-partitioned data model — only the frontend team selector needs scoping.
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Frontend │
│ │
│ AuthContext.teams ──► ReportingPage (filter by teams) │
│ ──► CompliancePage (scope team selector) │
│ ──► ExportsPage (scope exports) │
│ ──► CVE HomePage (NO filtering) │
│ │
└───────────────────────────────┬─────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Backend API │
│ │
│ GET /api/auth/me ──► returns { ...user, teams: ['STEAM',...] } │
│ GET /api/ivanti/findings?teams=STEAM,ACCESS-ENG │
│ GET /api/ivanti/findings/counts?teams=STEAM │
│ GET /api/compliance/items?team=STEAM (unchanged) │
│ │
│ Ivanti Sync: uses IVANTI_BU_FILTER env var (broadened) │
│ │
└───────────────────────────────┬─────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Database │
│ │
│ users.bu_teams TEXT DEFAULT '' (comma-separated team IDs) │
│ ivanti_findings_cache.findings_json (all BUs, filtered on read) │
│ compliance_items.team (already team-partitioned) │
│ │
└─────────────────────────────────────────────────────────────────┘
```
## Database Changes
### Migration: `add_user_bu_teams.js`
```sql
ALTER TABLE users ADD COLUMN bu_teams TEXT NOT NULL DEFAULT '';
```
- Stores comma-separated team identifiers (e.g. `'STEAM,ACCESS-ENG'`)
- Empty string means no teams assigned (new users start unscoped)
- No foreign key — validated at application layer against known teams list
- Existing users get empty string (admin must assign teams post-migration)
### Known Teams Constant
```js
const KNOWN_TEAMS = ['STEAM', 'ACCESS-ENG', 'ACCESS-OPS', 'INTELDEV'];
```
Defined once in a shared location (e.g. `backend/helpers/teams.js`) and imported by routes that need validation.
## Backend Changes
### 1. Auth Middleware (`middleware/auth.js`)
Update the session query to SELECT `bu_teams` and parse it onto `req.user`:
```js
req.user = {
id: session.user_id,
username: session.username,
email: session.email,
group: session.user_group,
teams: session.bu_teams ? session.bu_teams.split(',').filter(Boolean) : []
};
```
### 2. Auth Routes (`routes/auth.js`)
- `POST /login` response includes `teams` array
- `GET /me` response includes `teams` array
### 3. User Management Routes (`routes/users.js`)
- `POST /` (create user): accepts optional `bu_teams` field, validates against KNOWN_TEAMS
- `PATCH /:id` (update user): accepts `bu_teams` field, validates, logs change to audit
- `GET /` and `GET /:id`: return `bu_teams` in user records
### 4. Ivanti Findings Routes (`routes/ivantiFindings.js`)
**Sync filter change:**
```js
// Before (hardcoded):
const FINDINGS_FILTERS = [
{ field: 'assetCustomAttributes.1550_host_1.value', value: 'NTS-AEO-ACCESS-ENG,NTS-AEO-STEAM', ... }
];
// After (configurable):
const BU_FILTER_VALUE = process.env.IVANTI_BU_FILTER || 'NTS-AEO-ACCESS-ENG,NTS-AEO-STEAM';
const FINDINGS_FILTERS = [
{ field: 'assetCustomAttributes.1550_host_1.value', value: BU_FILTER_VALUE, ... }
];
```
**Query-time filtering on GET /findings:**
```js
router.get('/findings', requireAuth(db), async (req, res) => {
const state = await readState(db);
let findings = state.findings || [];
// Filter by teams if provided
const teamsParam = req.query.teams;
if (teamsParam) {
const teams = teamsParam.split(',').map(t => t.trim().toUpperCase());
findings = findings.filter(f =>
teams.some(t => (f.buOwnership || '').toUpperCase().includes(t))
);
}
res.json({ findings, total: findings.length, ... });
});
```
Same pattern for `/findings/counts` — filter before counting.
### 5. BU Ownership Matching Strategy
The Ivanti `buOwnership` field contains full BU names like `"NTS-AEO-STEAM"` or `"NTS-AEO-ACCESS-ENG"`. The user's `bu_teams` stores short names like `"STEAM"` or `"ACCESS-ENG"`. Matching uses a contains/includes check:
```js
// Match: user team "STEAM" matches finding buOwnership "NTS-AEO-STEAM"
const matchesTeam = (buOwnership, userTeams) =>
userTeams.some(t => (buOwnership || '').toUpperCase().includes(t.toUpperCase()));
```
This is intentionally loose to handle variations in Ivanti's BU naming without requiring exact string equality.
## Frontend Changes
### 1. AuthContext
Add `teams` to the user state from `/api/auth/me`:
```js
const [user, setUser] = useState(null);
// user.teams = ['STEAM', 'ACCESS-ENG'] or []
```
Add helper:
```js
const hasTeams = () => (user?.teams?.length ?? 0) > 0;
const isTeamMember = (team) => user?.teams?.includes(team) || isAdmin();
// Admin scope toggle — persisted in localStorage
const [adminScope, setAdminScope] = useState(
() => localStorage.getItem('admin_bu_scope') || 'my-teams'
);
const toggleAdminScope = () => { /* flip between 'my-teams' and 'all', persist */ };
// Returns the teams param string for API calls based on current scope
const getActiveTeamsParam = () => {
if (isAdmin() && adminScope === 'all') return ''; // no filter
return (user?.teams || []).join(',');
};
```
### 2. Reporting Page
- On mount, pass `?teams=${getActiveTeamsParam()}` to findings fetch
- If `user.teams` is empty and user is not Admin, show "No BU teams assigned" info panel instead of table
- Admin users get a scope toggle ("My Teams" / "All BUs") persisted in localStorage
- "My Teams" mode uses the admin's own bu_teams; "All BUs" omits the teams param entirely
- Default on login: "My Teams" so the admin's daily workflow is immediately scoped
- Counts endpoint also receives teams parameter matching current scope mode
### 3. Compliance Page
- Replace hardcoded `const TEAMS = ['STEAM', 'ACCESS-ENG']` with dynamic list from AuthContext
- Non-admin: shows only their assigned teams
- Admin in "My Teams" mode: shows only their assigned teams in the selector
- Admin in "All BUs" mode: shows all KNOWN_TEAMS in the selector
- Default `activeTeam` to first item in available teams list
- If available teams is empty and user is not Admin, show "No teams assigned" message
### 4. Exports Page
- Pass teams filter when fetching findings for export
- Respects the same scope toggle state as Reporting/Compliance
- Admin in "My Teams" exports only their BUs; "All BUs" exports everything
### 5. Admin Scope Toggle Component
- Rendered in the top nav/header area, visible only to Admin users
- Two-state toggle: "My Teams" | "All BUs"
- State stored in localStorage key `admin_bu_scope` (values: `'my-teams'` | `'all'`)
- Defaults to `'my-teams'` if no localStorage value exists
- Exposed via AuthContext so all pages can read the current scope without prop drilling
- When admin's bu_teams is empty, both modes behave identically (no filtering)
### 6. User Management UI
- Add multi-select checkbox group for BU teams in create/edit user forms
- Display assigned teams as badges in user list table
- Show warning icon for users with no teams assigned
## Environment Variable
Add to `.env.example`:
```bash
# Ivanti BU Filter — comma-separated list of BU values to sync from Ivanti.
# Broadening this pulls findings for additional BUs into the local cache.
# Users see only their assigned teams' findings (filtered at query time).
# Default: NTS-AEO-ACCESS-ENG,NTS-AEO-STEAM
IVANTI_BU_FILTER=NTS-AEO-ACCESS-ENG,NTS-AEO-STEAM,NTS-AEO-INTELDEV,NTS-AEO-ACCESS-OPS
```
## Security Considerations
- Backend compliance endpoints remain open to any valid team parameter (Admin flexibility)
- Reporting endpoint filtering is advisory — an Admin or API caller can omit teams to see all
- The `bu_teams` field is not a security boundary for compliance data; it's a UX scoping mechanism
- True data isolation would require separate databases or row-level security (out of scope for Option B)
- Audit logging captures all team assignment changes
## Migration Path
1. Run migration to add `bu_teams` column (all existing users get empty string)
2. Admin assigns teams to existing users via user management UI
3. Update `IVANTI_BU_FILTER` in `.env` to include all desired BUs
4. Trigger a manual sync to pull the broader dataset
5. Frontend automatically scopes based on user's teams after login
## Backward Compatibility
- Users with empty `bu_teams` and Admin group see everything (current behavior preserved)
- If `IVANTI_BU_FILTER` env var is not set, defaults to current hardcoded value
- Compliance backend endpoints unchanged — only frontend selector is scoped
- CVE page completely unaffected

141
.kiro/specs/multi-bu-tenancy/requirements.md Normal file
View File

@@ -0,0 +1,141 @@
# Requirements Document
## Introduction
Add per-user Business Unit (BU) team assignment so that multiple users with different BU responsibilities can share the same dashboard instance. The Reporting page (Ivanti findings) and Compliance page will display only data relevant to the logged-in user's assigned BU teams, while the CVE home page remains a shared global view. This follows "Option B" — a single Ivanti API key with a broadened sync filter, storing all BU findings in the local cache and filtering at query time based on the user's team assignments.
## Glossary
- **BU**: Business Unit — an organizational group within the Ivanti platform (e.g. NTS-AEO-STEAM, NTS-AEO-ACCESS-ENG, NTS-AEO-INTELDEV)
- **bu_teams**: A per-user field storing the comma-separated list of BU team identifiers the user is authorized to view
- **FINDINGS_FILTERS**: The hardcoded Ivanti API filter array that currently scopes synced findings to specific BU values
- **Compliance Team**: One of the allowed team identifiers used in the compliance data model (STEAM, ACCESS-ENG, ACCESS-OPS, INTELDEV)
- **Reporting Page**: The Ivanti host findings triage page that displays findings from the local cache
- **Compliance Page**: The AEO compliance posture page that displays non-compliant devices by team
- **CVE Home Page**: The shared CVE tracking page — remains unscoped and visible to all users
- **Admin**: A user in the Admin user_group who can manage users and assign BU teams
- **Team Selector**: The UI dropdown on the Compliance page that lets users switch between their assigned teams
## Requirements
### Requirement 1: Per-User BU Team Assignment
**User Story:** As an admin, I want to assign one or more BU teams to each user, so that their dashboard experience is scoped to the data relevant to their responsibilities.
#### Acceptance Criteria
1. THE users table SHALL include a `bu_teams` column storing a comma-separated list of team identifiers
2. WHEN a new user is created without explicit bu_teams, THE Dashboard SHALL default bu_teams to an empty string (no teams assigned)
3. THE Dashboard SHALL accept any combination of known team identifiers: STEAM, ACCESS-ENG, ACCESS-OPS, INTELDEV
4. THE Dashboard SHALL allow Admin users to assign or update bu_teams for any user via the user management interface
5. THE Dashboard SHALL validate that bu_teams contains only recognized team identifiers before persisting
### Requirement 2: Expose User Teams in Session
**User Story:** As a logged-in user, I want my assigned BU teams to be available in my session context, so that the frontend can scope data requests appropriately.
#### Acceptance Criteria
1. WHEN a user authenticates, THE auth middleware SHALL attach the user's parsed bu_teams array to `req.user.teams`
2. THE `GET /api/auth/me` endpoint SHALL return the user's teams array in the response
3. THE login response SHALL include the user's teams array
4. IF a user has no bu_teams assigned (empty string), THEN the teams array SHALL be empty `[]`
### Requirement 3: Broaden Ivanti Sync Filter
**User Story:** As a system operator, I want the Ivanti sync to pull findings for all configured BU values, so that users from any BU can see their relevant findings without needing separate API keys.
#### Acceptance Criteria
1. THE FINDINGS_FILTERS BU value SHALL be configurable via an environment variable `IVANTI_BU_FILTER` rather than hardcoded
2. IF `IVANTI_BU_FILTER` is not set, THE Dashboard SHALL default to the current value `'NTS-AEO-ACCESS-ENG,NTS-AEO-STEAM'` for backward compatibility
3. THE CLOSED_COUNT_FILTERS SHALL use the same configurable BU filter value
4. THE Dashboard SHALL document the `IVANTI_BU_FILTER` variable in `.env.example` with usage instructions
### Requirement 4: Filter Reporting Page by User's Teams
**User Story:** As a user assigned to specific BU teams, I want the Reporting page to show only findings belonging to my BUs, so that I see relevant data without noise from other teams.
#### Acceptance Criteria
1. THE `GET /api/ivanti/findings` endpoint SHALL accept an optional `teams` query parameter
2. WHEN `teams` is provided, THE endpoint SHALL filter the cached findings to only those whose `buOwnership` matches one of the specified team values
3. WHEN `teams` is not provided, THE endpoint SHALL return all cached findings (backward-compatible for Admin or unscoped views)
4. THE frontend Reporting page SHALL pass the logged-in user's teams when fetching findings
5. IF a user has no teams assigned (empty array) and the user is not Admin, THE Reporting page SHALL display a message indicating no BU teams are configured and direct them to contact an admin
6. Admin users SHALL have a scope toggle with three modes: "My Teams" (filters to admin's own bu_teams), "All BUs" (shows everything unfiltered)
7. THE Admin scope toggle SHALL default to "My Teams" so the admin's daily workflow view is scoped to their assigned BUs
8. WHEN an Admin switches to "All BUs" mode, THE Dashboard SHALL fetch findings without a teams filter
9. THE scope toggle selection SHALL persist across page navigations within the same session (stored in component state or localStorage)
### Requirement 5: Filter Compliance Page by User's Teams
**User Story:** As a user assigned to specific BU teams, I want the Compliance page team selector to show only my assigned teams, so that I cannot accidentally view or interact with another team's compliance data.
#### Acceptance Criteria
1. THE frontend Compliance page team selector SHALL display only the teams present in the user's teams array
2. IF a user has only one team assigned, THE team selector SHALL default to that team and may be hidden or disabled
3. IF a user has no teams assigned and the user is not Admin, THE Compliance page SHALL display a message indicating no teams are configured
4. Admin users SHALL see all available teams in the selector regardless of their personal bu_teams assignment
5. Admin users SHALL have the same scope toggle as the Reporting page: "My Teams" shows only their assigned teams in the selector, "All BUs" shows all available teams
6. THE Admin scope toggle state SHALL be shared across Reporting and Compliance pages (single toggle, consistent behavior)
7. THE backend compliance endpoints SHALL continue to accept any valid team parameter (authorization is frontend-scoped, not backend-enforced, to keep the API flexible for Admin use)
### Requirement 6: Admin User Management for BU Teams
**User Story:** As an admin, I want a UI to assign and modify BU teams for users, so that I can onboard new team members and adjust assignments as responsibilities change.
#### Acceptance Criteria
1. THE user management create form SHALL include a multi-select or checkbox group for BU team assignment
2. THE user management edit form SHALL include a multi-select or checkbox group for BU team assignment
3. WHEN an admin updates a user's bu_teams, THE Dashboard SHALL log the change in the audit trail with previous and new values
4. THE user list table SHALL display each user's assigned BU teams
5. THE Dashboard SHALL provide a visual indicator when a user has no teams assigned (warning state)
### Requirement 7: CVE Home Page Remains Shared
**User Story:** As any user, I want the CVE tracking page to remain a shared global view, so that all teams can collaborate on vulnerability tracking regardless of BU assignment.
#### Acceptance Criteria
1. THE CVE home page SHALL NOT filter data based on the user's bu_teams
2. ALL authenticated users SHALL see the same CVE data regardless of team assignment
3. THE CVE creation, editing, and status workflows SHALL remain unaffected by bu_teams
### Requirement 8: Ivanti Findings Count Scoping
**User Story:** As a user, I want the findings count badges and status cards on the Reporting page to reflect only my BU's findings, so that metrics are meaningful to my scope.
#### Acceptance Criteria
1. THE `GET /api/ivanti/findings/counts` endpoint SHALL accept an optional `teams` parameter and return counts filtered to those BUs
2. THE open/closed count cards on the Reporting page SHALL reflect the user's scoped findings
3. WHEN viewing all BUs (Admin toggle), THE counts SHALL reflect the full unfiltered totals
### Requirement 9: Export Scoping
**User Story:** As a user exporting findings data, I want exports to respect my BU scope, so that I don't inadvertently share another team's data.
#### Acceptance Criteria
1. THE Exports page Ivanti findings exports SHALL filter by the user's teams by default
2. Admin users in "My Teams" mode SHALL export only their assigned BUs; in "All BUs" mode SHALL export everything
3. THE exported filename or metadata SHALL indicate which BU scope was applied
### Requirement 10: Admin Scope Toggle
**User Story:** As an admin with assigned BU teams, I want a global scope toggle that lets me switch between "My Teams" (my daily workflow) and "All BUs" (full visibility), so that I can work efficiently in my own BU context while retaining the ability to see the full picture when needed.
#### Acceptance Criteria
1. THE Dashboard SHALL display a scope toggle control visible only to Admin users
2. THE scope toggle SHALL offer two modes: "My Teams" and "All BUs"
3. WHEN "My Teams" is active, THE Dashboard SHALL filter Reporting, Compliance, and Exports to the admin's own bu_teams assignment (identical behavior to a non-admin user)
4. WHEN "All BUs" is active, THE Dashboard SHALL show all data across all BUs without filtering
5. THE scope toggle SHALL default to "My Teams" on login so the admin's daily workflow is immediately relevant
6. THE scope toggle state SHALL persist in localStorage so it survives page refreshes within the same browser
7. THE scope toggle SHALL be displayed in a consistent location (e.g. top nav bar or header area) accessible from any page
8. WHEN the admin's bu_teams is empty, "My Teams" mode SHALL behave the same as "All BUs" (no filtering applied)

201
.kiro/specs/multi-bu-tenancy/tasks.md Normal file
View File

@@ -0,0 +1,201 @@
# Implementation Plan: Multi-BU Tenancy (Option B)
## Overview
Add per-user BU team assignment with query-time filtering on the Reporting and Compliance pages. Uses a single broadened Ivanti sync and filters cached findings based on the logged-in user's assigned teams. The CVE home page remains shared/global.
## Tasks
- [x] 1. Database migration and shared teams constant
- [x] 1.1 Create `backend/helpers/teams.js` shared constant
- Export `KNOWN_TEAMS = ['STEAM', 'ACCESS-ENG', 'ACCESS-OPS', 'INTELDEV']`
- Export `validateTeams(teamsString)` helper that parses and validates a comma-separated string
- Returns `{ valid: boolean, teams: string[], invalid: string[] }`
- _Requirements: 1.3, 1.5_
- [x] 1.2 Create `backend/migrations/add_user_bu_teams.js`
- Add `bu_teams` column (TEXT, NOT NULL, DEFAULT '') to users table
- Idempotent — check if column exists before adding
- Follow existing migration pattern (open db, serialize, log progress, close)
- _Requirements: 1.1, 1.2_
- [x] 1.3 Update `backend/setup.js` to include bu_teams in fresh schema
- Add `bu_teams TEXT NOT NULL DEFAULT ''` to the users CREATE TABLE statement
- _Requirements: 1.1, 1.2_
- [x] 2. Backend auth changes — expose teams in session
- [x] 2.1 Update `backend/middleware/auth.js` requireAuth
- Ensure the session JOIN query SELECTs `bu_teams` from users table
- Parse `bu_teams` into array and attach as `req.user.teams`
- Empty string becomes empty array `[]`
- _Requirements: 2.1, 2.4_
- [x] 2.2 Update `backend/routes/auth.js` login response
- Include `teams` array in the login success response object
- _Requirements: 2.3_
- [x] 2.3 Update `backend/routes/auth.js` GET /me endpoint
- Include `teams` array in the /me response
- _Requirements: 2.2_
- [x] 3. User management — CRUD for bu_teams
- [x] 3.1 Update `backend/routes/users.js` POST / (create user)
- Accept optional `bu_teams` field in request body
- Validate using `validateTeams()` helper — return 400 if invalid teams present
- Store validated comma-separated string in DB
- _Requirements: 1.4, 1.5, 6.1_
- [x] 3.2 Update `backend/routes/users.js` PATCH /:id (update user)
- Accept optional `bu_teams` field in request body
- Validate using `validateTeams()` helper — return 400 if invalid teams present
- Log previous and new bu_teams values in audit trail
- _Requirements: 1.4, 1.5, 6.2, 6.3_
- [x] 3.3 Update `backend/routes/users.js` GET endpoints
- Include `bu_teams` (as raw string) and `teams` (as parsed array) in user response objects
- _Requirements: 6.4_
- [x] 4. Checkpoint: Verify migration and auth
- Run migration, create a test user with bu_teams, login, verify /me returns teams array.
- [x] 5. Broaden Ivanti sync filter
- [x] 5.1 Make FINDINGS_FILTERS BU value configurable in `backend/routes/ivantiFindings.js`
- Read `process.env.IVANTI_BU_FILTER` — default to `'NTS-AEO-ACCESS-ENG,NTS-AEO-STEAM'`
- Replace hardcoded value in FINDINGS_FILTERS array
- Apply same change to CLOSED_COUNT_FILTERS
- _Requirements: 3.1, 3.2, 3.3_
- [x] 5.2 Update `backend/.env.example` with IVANTI_BU_FILTER documentation
- Add commented variable with explanation of how broadening works
- _Requirements: 3.4_
- [x] 6. Add query-time team filtering to findings endpoints
- [x] 6.1 Update GET /findings endpoint in `backend/routes/ivantiFindings.js`
- Accept optional `teams` query parameter (comma-separated)
- Filter findings array by buOwnership matching (case-insensitive includes)
- Return filtered count in response
- If teams param is empty/missing, return all findings (backward-compatible)
- _Requirements: 4.1, 4.2, 4.3_
- [x] 6.2 Update GET /findings/counts endpoint
- Accept optional `teams` query parameter
- Filter open count by matching buOwnership against provided teams
- Closed count: if teams provided, note in response that closed count is approximate (full cache doesn't store per-finding closed data)
- _Requirements: 8.1, 8.2_
- [x] 6.3 Update GET /findings/counts/history endpoint (if team-scoped history is needed)
- Consider whether historical counts should be team-scoped or remain global
- For MVP: keep global (history is aggregate), document limitation
- _Requirements: 8.3_
- [x] 7. Checkpoint: Verify backend filtering
- Test GET /findings with and without teams param. Verify filtering works correctly.
- [x] 8. Frontend AuthContext — expose teams and admin scope toggle
- [x] 8.1 Update `frontend/src/contexts/AuthContext.js`
- Store `user.teams` from /me response
- Add `hasTeams()` helper — returns true if teams array is non-empty
- Add `isTeamMember(team)` helper — returns true if team is in user's teams or user is Admin
- Add `adminScope` state backed by localStorage key `admin_bu_scope` (default: `'my-teams'`)
- Add `toggleAdminScope()` function to flip between `'my-teams'` and `'all'`
- Add `getActiveTeamsParam()` helper — returns comma-joined teams for API calls; returns empty string when Admin is in "All BUs" mode
- Add `getAvailableTeams(knownTeams)` helper — returns user's teams, or all knownTeams if Admin in "All BUs" mode
- _Requirements: 2.2, 4.4, 4.6, 4.7, 4.9, 10.1, 10.2, 10.5, 10.6_
- [x] 9. Frontend Admin Scope Toggle component
- [x] 9.1 Create `frontend/src/components/AdminScopeToggle.js`
- Render a two-state toggle: "My Teams" | "All BUs"
- Only visible when `isAdmin()` is true
- Calls `toggleAdminScope()` from AuthContext on click
- Styled to fit in the top nav/header area
- _Requirements: 10.1, 10.2, 10.7_
- [x] 9.2 Mount AdminScopeToggle in the app header/nav
- Place in NavDrawer or top bar so it's accessible from any page
- _Requirements: 10.7_
- [x] 10. Frontend Reporting Page — scope by teams
- [x] 10.1 Update findings fetch in `ReportingPage.js`
- Use `getActiveTeamsParam()` from AuthContext to build the teams query param
- If user has no teams and is not Admin, show "No BU teams assigned" info panel instead of table
- _Requirements: 4.4, 4.5_
- [x] 10.2 Update counts fetch in `ReportingPage.js`
- Append teams param to counts endpoint using same `getActiveTeamsParam()`
- _Requirements: 8.2_
- [x] 10.3 Re-fetch findings when admin scope toggle changes
- Listen to `adminScope` value from AuthContext; trigger re-fetch on change
- _Requirements: 4.8_
- [x] 11. Frontend Compliance Page — scope team selector
- [x] 11.1 Update `CompliancePage.js` TEAMS constant
- Replace hardcoded `['STEAM', 'ACCESS-ENG']` with `getAvailableTeams(KNOWN_TEAMS)` from AuthContext
- Non-admin sees only their assigned teams
- Admin in "My Teams" mode sees only their assigned teams
- Admin in "All BUs" mode sees all KNOWN_TEAMS
- _Requirements: 5.1, 5.4, 5.5_
- [x] 11.2 Handle single-team and no-team states
- If user has one team: default to it, optionally hide selector
- If user has no teams and is not Admin: show "No teams assigned" message
- _Requirements: 5.2, 5.3_
- [ ] 11.3 Update `ComplianceChartsPanel.js` team colors
- Ensure chart only renders data for user's available teams
- _Requirements: 5.1_
- [x] 11.4 Re-render team selector when admin scope toggle changes
- Listen to `adminScope` from AuthContext; update available teams list on change
- _Requirements: 5.5, 5.6_
- [x] 12. Frontend Exports Page — scope by teams
- [x] 12.1 Update `ExportsPage.js` findings fetch
- Use `getActiveTeamsParam()` to pass teams filter
- _Requirements: 9.1_
- [x] 12.2 Add BU scope indicator to export filenames
- Append team names to exported file names (e.g. `findings-STEAM-2026-05-05.xlsx`)
- When in "All BUs" mode, use `findings-ALL-2026-05-05.xlsx`
- _Requirements: 9.3_
- [ ] 13. Frontend User Management — BU team assignment UI
- [ ] 13.1 Add team multi-select to create user form in `UserManagement.js`
- Checkbox group with all KNOWN_TEAMS options
- Sends comma-separated string as `bu_teams` field
- _Requirements: 6.1_
- [ ] 13.2 Add team multi-select to edit user form
- Pre-populate with user's current teams
- _Requirements: 6.2_
- [ ] 13.3 Display teams in user list table
- Show team badges for each user
- Show warning indicator for users with no teams assigned
- _Requirements: 6.4, 6.5_
- [ ] 14. Verify CVE page is unaffected
- [ ] 14.1 Confirm no team filtering on CVE routes or frontend
- Verify CVE page does not reference user.teams for data fetching
- _Requirements: 7.1, 7.2, 7.3_
- [ ] 15. Final checkpoint: End-to-end verification
- Create two test users with different bu_teams assignments. Log in as each and verify:
- Reporting page shows only their BU's findings
- Compliance page shows only their teams in selector
- CVE page shows same data for both
- Exports are scoped correctly
- Admin scope toggle switches between "My Teams" and "All BUs" correctly
- Admin in "My Teams" mode sees same scoped view as a regular user with same teams
- Toggle state persists across page refresh
## Notes
- The `bu_teams` field is a UX scoping mechanism, not a hard security boundary. Admin and direct API callers can always access all data.
- The buOwnership matching uses case-insensitive `includes()` to handle Ivanti's full BU names (e.g. "NTS-AEO-STEAM" matches user team "STEAM").
- Historical counts (IvantiCountsChart) remain global for MVP — per-BU historical tracking would require schema changes to the counts history table.
- The compliance backend endpoints are unchanged — only the frontend team selector is scoped.
- After running the migration, an admin must assign bu_teams to existing users. Until then, non-Admin users with empty teams will see the "no teams assigned" message on Reporting/Compliance.
- The Admin scope toggle defaults to "My Teams" so that your daily workflow (STEAM + ACCESS-ENG) is the default view. Switch to "All BUs" when you need the full picture across all business units.
- The scope toggle state is shared across Reporting, Compliance, and Exports — one toggle controls all three pages consistently.
- Admin users with empty bu_teams: "My Teams" mode behaves identically to "All BUs" (no filtering in either case).

1
.kiro/specs/postgres-migration/.config.kiro Normal file
View File

@@ -0,0 +1 @@
{"specId": "b93a94ad-abfd-4543-91bf-eb5a6cdd7896", "workflowType": "requirements-first", "specType": "feature"}

763
.kiro/specs/postgres-migration/design.md Normal file
View File

@@ -0,0 +1,763 @@
# Design Document: PostgreSQL Migration
## Overview
Migrate the CVE Dashboard backend from SQLite (`cve_database.db`, 13MB) to PostgreSQL 16 running in a dedicated Docker container (`steam-postgres`) on port 5433. The primary architectural change is decomposing the monolithic `ivanti_findings_cache.findings_json` blob (2.6MB TEXT column) into individual rows in an `ivanti_findings` table. This eliminates JSON parsing on every request, enables indexed per-BU filtering, provides per-BU closed finding counts, and removes SQLite's single-writer lock that blocks reads during sync.
The Postgres instance is fully isolated from the existing Postgres on port 5432 (belonging to another project). The frontend requires zero changes — the API contract remains identical.
## Architecture
```mermaid
graph TB
subgraph "Frontend (port 3000)"
FE[React SPA]
end
subgraph "Backend (port 3001 prod / 3003 dev)"
SERVER[Express Server]
POOL[pg Pool - max 10 connections]
ROUTES[Route Handlers - async/await]
SYNC[Ivanti Sync Logic]
end
subgraph "Docker: steam-postgres (port 5433)"
PG[(PostgreSQL 16 Alpine)]
DB[cve_dashboard database]
VOL[steam-pgdata volume]
end
subgraph "Existing Infrastructure (DO NOT TOUCH)"
OTHER_PG[(Other Postgres - port 5432)]
SQLITE[(SQLite backup - cve_database.db)]
end
FE -->|Same API contract| SERVER
SERVER --> ROUTES
ROUTES --> POOL
SYNC --> POOL
POOL -->|DATABASE_URL| PG
PG --> DB
DB --> VOL
```
### Key Architecture Decisions
| Decision | Rationale |
|----------|-----------|
| Dedicated Docker container on port 5433 | Isolation from existing Postgres on 5432; independent lifecycle |
| `pg` package with connection pool (max 10) | Concurrent reads during writes; no single-writer lock |
| Individual finding rows instead of JSON blob | Indexed queries, per-BU filtering in SQL, no JSON.parse |
| Closed findings stored with `bu_ownership` | Enables per-BU closed counts (currently only global count) |
| Batch upsert via `INSERT ... ON CONFLICT` | Idempotent sync; no data loss on re-runs |
| Blue-green cutover on same port | <30s downtime; instant rollback by reverting .env |
## Components and Interfaces
### 1. Connection Pool Module (`backend/db.js`)
New module that creates and exports a `pg` Pool instance. All route files import this instead of receiving a `db` (sqlite3) parameter.
```js
// backend/db.js
const { Pool } = require('pg');
const pool = new Pool({
connectionString: process.env.DATABASE_URL,
// postgresql://steam:<password>@localhost:5433/cve_dashboard
max: 10,
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 5000,
});
// Log pool errors (connection drops, etc.)
pool.on('error', (err) => {
console.error('[DB Pool] Unexpected error on idle client:', err.message);
});
// Warn when approaching pool exhaustion
let activeCount = 0;
pool.on('acquire', () => {
activeCount++;
if (activeCount >= 8) {
console.warn(`[DB Pool] WARNING: ${activeCount}/10 connections active — approaching exhaustion`);
}
});
pool.on('release', () => { activeCount--; });
module.exports = pool;
```
### 2. Route Migration Pattern
Every route file changes from callback-based SQLite to async/await Postgres:
**Before (SQLite):**
```js
function createUsersRouter(db, requireAuth, requireGroup, logAudit) {
router.get('/', async (req, res) => {
try {
const users = await new Promise((resolve, reject) => {
db.all(
'SELECT id, username, email, user_group AS "group" FROM users ORDER BY created_at DESC',
(err, rows) => {
if (err) reject(err);
else resolve(rows);
}
);
});
res.json(users);
} catch (err) {
res.status(500).json({ error: 'Failed to fetch users' });
}
});
}
```
**After (Postgres):**
```js
const pool = require('../db');
function createUsersRouter(requireAuth, requireGroup, logAudit) {
router.get('/', async (req, res) => {
try {
const { rows: users } = await pool.query(
'SELECT id, username, email, user_group AS "group" FROM users ORDER BY created_at DESC'
);
res.json(users);
} catch (err) {
console.error('Get users error:', err);
res.status(500).json({ error: 'Failed to fetch users' });
}
});
}
```
### 3. Query Pattern Translation
| SQLite Pattern | Postgres Equivalent |
|----------------|-------------------|
| `db.get(sql, [params], callback)` | `const { rows } = await pool.query(sql, [params]); const row = rows[0];` |
| `db.all(sql, [params], callback)` | `const { rows } = await pool.query(sql, [params]);` |
| `db.run(sql, [params], callback)` | `await pool.query(sql, [params]);` or with `RETURNING` |
| `?` placeholders | `$1, $2, $3...` numbered params |
| `INSERT OR IGNORE` | `INSERT ... ON CONFLICT DO NOTHING` |
| `datetime('now')` | `NOW()` |
| `LIKE` (case-sensitive) | `ILIKE` (case-insensitive) |
### 4. Ivanti Sync Component (Rewritten)
The sync logic changes from "serialize all findings to JSON blob" to "upsert individual rows":
```js
// backend/routes/ivantiFindings.js — sync logic (simplified)
async function syncFindings(pool) {
const allFindings = await fetchAllFromIvanti(); // paginated API calls
// Batch upsert in chunks of 100
for (let i = 0; i < allFindings.length; i += 100) {
const batch = allFindings.slice(i, i + 100);
const values = [];
const placeholders = batch.map((f, idx) => {
const offset = idx * 14;
values.push(f.id, f.hostId, f.title, f.severity, f.vrrGroup,
f.hostName, f.ipAddress, f.dns, f.status, f.slaStatus,
f.dueDate, f.lastFoundOn, f.buOwnership, f.cves || []);
return `($${offset+1},$${offset+2},$${offset+3},$${offset+4},$${offset+5},
$${offset+6},$${offset+7},$${offset+8},$${offset+9},$${offset+10},
$${offset+11},$${offset+12},$${offset+13},$${offset+14},'open')`;
});
await pool.query(`
INSERT INTO ivanti_findings (id, host_id, title, severity, vrr_group,
host_name, ip_address, dns, status, sla_status,
due_date, last_found_on, bu_ownership, cves, state)
VALUES ${placeholders.join(',')}
ON CONFLICT (id) DO UPDATE SET
title = EXCLUDED.title,
severity = EXCLUDED.severity,
host_name = EXCLUDED.host_name,
ip_address = EXCLUDED.ip_address,
dns = EXCLUDED.dns,
status = EXCLUDED.status,
sla_status = EXCLUDED.sla_status,
due_date = EXCLUDED.due_date,
last_found_on = EXCLUDED.last_found_on,
bu_ownership = EXCLUDED.bu_ownership,
cves = EXCLUDED.cves,
state = EXCLUDED.state,
synced_at = NOW()
`, values);
}
// Update sync metadata
await pool.query(`
UPDATE ivanti_sync_state SET
total = (SELECT COUNT(*) FROM ivanti_findings WHERE state = 'open'),
synced_at = NOW(),
sync_status = 'success',
error_message = NULL
WHERE id = 1
`);
}
```
### 5. Auth Middleware Migration
```js
// backend/middleware/auth.js — After
const pool = require('../db');
function requireAuth() {
return async (req, res, next) => {
const sessionId = req.cookies?.session_id;
if (!sessionId) return res.status(401).json({ error: 'Authentication required' });
try {
const { rows } = await pool.query(
`SELECT s.*, u.id as user_id, u.username, u.email, u.role,
u.user_group, u.bu_teams, u.is_active
FROM sessions s
JOIN users u ON s.user_id = u.id
WHERE s.session_id = $1 AND s.expires_at > NOW()`,
[sessionId]
);
const session = rows[0];
if (!session) return res.status(401).json({ error: 'Session expired or invalid' });
if (!session.is_active) return res.status(401).json({ error: 'Account is disabled' });
req.user = {
id: session.user_id,
username: session.username,
email: session.email,
role: session.role,
group: session.user_group,
teams: session.bu_teams ? session.bu_teams.split(',').filter(Boolean) : []
};
next();
} catch (err) {
console.error('Auth middleware error:', err);
return res.status(500).json({ error: 'Authentication error' });
}
};
}
```
## Data Models
### Complete DDL for `ivanti_findings` Table
```sql
CREATE TABLE IF NOT EXISTS ivanti_findings (
id TEXT PRIMARY KEY, -- Ivanti finding ID (e.g. "HF-12345")
host_id INTEGER,
title TEXT NOT NULL DEFAULT '',
severity NUMERIC(4,2) NOT NULL DEFAULT 0,
vrr_group TEXT NOT NULL DEFAULT '',
host_name TEXT NOT NULL DEFAULT '',
ip_address TEXT NOT NULL DEFAULT '',
dns TEXT NOT NULL DEFAULT '',
status TEXT NOT NULL DEFAULT '',
sla_status TEXT NOT NULL DEFAULT '',
due_date DATE,
last_found_on DATE,
bu_ownership TEXT NOT NULL DEFAULT '',
cves TEXT[] DEFAULT '{}', -- Postgres array type
workflow_id TEXT,
workflow_state TEXT,
workflow_type TEXT,
state TEXT NOT NULL DEFAULT 'open' CHECK (state IN ('open', 'closed')),
note TEXT NOT NULL DEFAULT '', -- Merged from ivanti_finding_notes
override_host_name TEXT, -- Merged from ivanti_finding_overrides
override_dns TEXT, -- Merged from ivanti_finding_overrides
synced_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Performance indexes
CREATE INDEX IF NOT EXISTS idx_findings_state ON ivanti_findings(state);
CREATE INDEX IF NOT EXISTS idx_findings_bu ON ivanti_findings(bu_ownership);
CREATE INDEX IF NOT EXISTS idx_findings_severity ON ivanti_findings(severity);
CREATE INDEX IF NOT EXISTS idx_findings_state_bu ON ivanti_findings(state, bu_ownership);
```
### Core Tables (Postgres DDL)
```sql
-- Users
CREATE TABLE IF NOT EXISTS users (
id SERIAL PRIMARY KEY,
username VARCHAR(50) UNIQUE NOT NULL,
email VARCHAR(255) UNIQUE NOT NULL,
password_hash VARCHAR(255) NOT NULL,
role VARCHAR(20) NOT NULL DEFAULT 'viewer' CHECK (role IN ('admin', 'editor', 'viewer')),
is_active BOOLEAN DEFAULT TRUE,
created_at TIMESTAMPTZ DEFAULT NOW(),
last_login TIMESTAMPTZ,
user_group VARCHAR(20) NOT NULL DEFAULT 'Read_Only',
bu_teams TEXT NOT NULL DEFAULT ''
);
-- Sessions
CREATE TABLE IF NOT EXISTS sessions (
id SERIAL PRIMARY KEY,
session_id VARCHAR(255) UNIQUE NOT NULL,
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
expires_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_sessions_session_id ON sessions(session_id);
CREATE INDEX IF NOT EXISTS idx_sessions_user_id ON sessions(user_id);
CREATE INDEX IF NOT EXISTS idx_sessions_expires ON sessions(expires_at);
-- Audit Logs
CREATE TABLE IF NOT EXISTS audit_logs (
id SERIAL PRIMARY KEY,
user_id INTEGER,
username VARCHAR(50) NOT NULL,
action VARCHAR(50) NOT NULL,
entity_type VARCHAR(50) NOT NULL,
entity_id VARCHAR(100),
details TEXT,
ip_address VARCHAR(45),
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_audit_created_at ON audit_logs(created_at);
-- CVEs
CREATE TABLE IF NOT EXISTS cves (
id SERIAL PRIMARY KEY,
cve_id VARCHAR(20) NOT NULL,
vendor VARCHAR(100) NOT NULL,
severity VARCHAR(20) NOT NULL,
description TEXT,
published_date DATE,
status VARCHAR(50) DEFAULT 'Open',
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
created_by INTEGER,
UNIQUE(cve_id, vendor)
);
-- Jira Tickets
CREATE TABLE IF NOT EXISTS jira_tickets (
id SERIAL PRIMARY KEY,
cve_id TEXT NOT NULL,
vendor TEXT NOT NULL,
ticket_key TEXT NOT NULL,
url TEXT,
summary TEXT,
status TEXT DEFAULT 'Open' CHECK(status IN ('Open', 'In Progress', 'Closed')),
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
-- Ivanti Sync State (replaces ivanti_findings_cache metadata)
CREATE TABLE IF NOT EXISTS ivanti_sync_state (
id INTEGER PRIMARY KEY DEFAULT 1 CHECK (id = 1),
total INTEGER DEFAULT 0,
workflows_json TEXT DEFAULT '[]',
synced_at TIMESTAMPTZ,
sync_status TEXT DEFAULT 'never',
error_message TEXT
);
-- Ivanti Counts Cache (for FP workflow counts)
CREATE TABLE IF NOT EXISTS ivanti_counts_cache (
id INTEGER PRIMARY KEY DEFAULT 1 CHECK (id = 1),
open_count INTEGER DEFAULT 0,
closed_count INTEGER DEFAULT 0,
synced_at TIMESTAMPTZ,
fp_workflow_counts_json TEXT DEFAULT '{}',
fp_id_counts_json TEXT DEFAULT '{}'
);
-- Ivanti Counts History
CREATE TABLE IF NOT EXISTS ivanti_counts_history (
id SERIAL PRIMARY KEY,
open_count INTEGER NOT NULL,
closed_count INTEGER NOT NULL,
recorded_at TIMESTAMPTZ DEFAULT NOW()
);
-- Ivanti Finding Archives
CREATE TABLE IF NOT EXISTS ivanti_finding_archives (
id SERIAL PRIMARY KEY,
finding_id TEXT NOT NULL UNIQUE,
finding_title TEXT NOT NULL DEFAULT '',
host_name TEXT NOT NULL DEFAULT '',
ip_address TEXT NOT NULL DEFAULT '',
current_state TEXT NOT NULL CHECK(current_state IN ('ARCHIVED','RETURNED','CLOSED','CLOSED_GONE')),
last_severity NUMERIC(4,2) NOT NULL DEFAULT 0,
first_archived_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
last_transition_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Ivanti Archive Transitions
CREATE TABLE IF NOT EXISTS ivanti_archive_transitions (
id SERIAL PRIMARY KEY,
archive_id INTEGER NOT NULL REFERENCES ivanti_finding_archives(id),
from_state TEXT NOT NULL,
to_state TEXT NOT NULL,
severity_at_transition NUMERIC(4,2) NOT NULL DEFAULT 0,
reason TEXT NOT NULL DEFAULT '',
transitioned_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Ivanti Sync Anomaly Log
CREATE TABLE IF NOT EXISTS ivanti_sync_anomaly_log (
id SERIAL PRIMARY KEY,
sync_timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
open_count_delta INTEGER NOT NULL DEFAULT 0,
closed_count_delta INTEGER NOT NULL DEFAULT 0,
newly_archived_count INTEGER NOT NULL DEFAULT 0,
returned_count INTEGER NOT NULL DEFAULT 0,
classification_json TEXT NOT NULL DEFAULT '{}',
return_classification_json TEXT NOT NULL DEFAULT '{}',
is_significant BOOLEAN NOT NULL DEFAULT FALSE,
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Ivanti Finding BU History
CREATE TABLE IF NOT EXISTS ivanti_finding_bu_history (
id SERIAL PRIMARY KEY,
finding_id TEXT NOT NULL,
finding_title TEXT NOT NULL DEFAULT '',
host_name TEXT NOT NULL DEFAULT '',
previous_bu TEXT NOT NULL,
new_bu TEXT NOT NULL,
detected_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Ivanti FP Submissions
CREATE TABLE IF NOT EXISTS ivanti_fp_submissions (
id SERIAL PRIMARY KEY,
user_id INTEGER NOT NULL,
username TEXT NOT NULL,
ivanti_workflow_batch_id INTEGER,
ivanti_generated_id TEXT,
ivanti_workflow_batch_uuid TEXT,
workflow_name TEXT NOT NULL,
reason TEXT NOT NULL,
description TEXT,
expiration_date TEXT NOT NULL,
scope_override TEXT NOT NULL DEFAULT 'Authorized',
finding_ids_json TEXT NOT NULL,
queue_item_ids_json TEXT NOT NULL,
attachment_count INTEGER DEFAULT 0,
attachment_results_json TEXT,
status TEXT NOT NULL DEFAULT 'success' CHECK(status IN ('success', 'partial', 'failed')),
lifecycle_status TEXT NOT NULL DEFAULT 'submitted' CHECK(lifecycle_status IN ('submitted', 'approved', 'rejected', 'rework', 'resubmitted')),
error_message TEXT,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NULL
);
-- Ivanti FP Submission History
CREATE TABLE IF NOT EXISTS ivanti_fp_submission_history (
id SERIAL PRIMARY KEY,
submission_id INTEGER NOT NULL REFERENCES ivanti_fp_submissions(id) ON DELETE CASCADE,
user_id INTEGER NOT NULL,
username TEXT NOT NULL,
change_type TEXT NOT NULL CHECK(change_type IN ('created', 'fields_updated', 'findings_added', 'attachments_added', 'status_changed')),
change_details_json TEXT,
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Ivanti Todo Queue
CREATE TABLE IF NOT EXISTS ivanti_todo_queue (
id SERIAL PRIMARY KEY,
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
finding_id TEXT NOT NULL,
finding_title TEXT,
cves_json TEXT,
ip_address TEXT,
hostname TEXT,
vendor TEXT NOT NULL,
workflow_type TEXT NOT NULL CHECK(workflow_type IN ('FP', 'Archer', 'CARD', 'GRANITE')),
status TEXT NOT NULL DEFAULT 'pending' CHECK(status IN ('pending', 'complete')),
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_todo_queue_user ON ivanti_todo_queue(user_id, status);
-- Atlas Action Plans Cache
CREATE TABLE IF NOT EXISTS atlas_action_plans_cache (
id SERIAL PRIMARY KEY,
host_id INTEGER NOT NULL UNIQUE,
has_action_plan BOOLEAN NOT NULL DEFAULT FALSE,
plan_count INTEGER NOT NULL DEFAULT 0,
plans_json TEXT NOT NULL DEFAULT '[]',
synced_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Compliance Tables
CREATE TABLE IF NOT EXISTS compliance_uploads (
id SERIAL PRIMARY KEY,
filename TEXT NOT NULL,
report_date TEXT,
uploaded_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
uploaded_at TIMESTAMPTZ DEFAULT NOW(),
new_count INTEGER DEFAULT 0,
resolved_count INTEGER DEFAULT 0,
recurring_count INTEGER DEFAULT 0,
summary_json TEXT
);
CREATE TABLE IF NOT EXISTS compliance_items (
id SERIAL PRIMARY KEY,
upload_id INTEGER NOT NULL REFERENCES compliance_uploads(id) ON DELETE CASCADE,
hostname TEXT NOT NULL,
ip_address TEXT,
device_type TEXT,
team TEXT,
metric_id TEXT NOT NULL,
metric_desc TEXT,
category TEXT,
extra_json TEXT,
status TEXT NOT NULL DEFAULT 'active' CHECK(status IN ('active', 'resolved')),
first_seen_upload_id INTEGER REFERENCES compliance_uploads(id) ON DELETE SET NULL,
resolved_upload_id INTEGER REFERENCES compliance_uploads(id) ON DELETE SET NULL,
seen_count INTEGER DEFAULT 1,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE IF NOT EXISTS compliance_notes (
id SERIAL PRIMARY KEY,
hostname TEXT NOT NULL,
metric_id TEXT NOT NULL,
note TEXT NOT NULL,
group_id TEXT,
created_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Knowledge Base
CREATE TABLE IF NOT EXISTS knowledge_base (
id SERIAL PRIMARY KEY,
title VARCHAR(255) NOT NULL,
slug VARCHAR(255) UNIQUE NOT NULL,
description TEXT,
category VARCHAR(100),
file_path VARCHAR(500),
file_name VARCHAR(255),
file_type VARCHAR(50),
file_size INTEGER,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
created_by INTEGER REFERENCES users(id)
);
-- Archer Tickets
CREATE TABLE IF NOT EXISTS archer_tickets (
id SERIAL PRIMARY KEY,
exc_number TEXT NOT NULL UNIQUE,
archer_url TEXT,
status TEXT DEFAULT 'Draft' CHECK(status IN ('Draft', 'Open', 'Under Review', 'Accepted')),
cve_id TEXT NOT NULL,
vendor TEXT NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
-- Documents
CREATE TABLE IF NOT EXISTS documents (
id SERIAL PRIMARY KEY,
cve_id VARCHAR(20) NOT NULL,
vendor VARCHAR(100) NOT NULL,
name VARCHAR(255) NOT NULL,
type VARCHAR(50) NOT NULL,
file_path VARCHAR(500) NOT NULL,
file_size VARCHAR(20),
mime_type VARCHAR(100),
uploaded_at TIMESTAMPTZ DEFAULT NOW(),
notes TEXT
);
-- Required Documents (seed data)
CREATE TABLE IF NOT EXISTS required_documents (
id SERIAL PRIMARY KEY,
vendor VARCHAR(100) NOT NULL,
document_type VARCHAR(50) NOT NULL,
is_mandatory BOOLEAN DEFAULT TRUE,
description TEXT
);
```
### Per-BU Count Queries
```sql
-- Open count for specific BUs (used by counts endpoint with teams filter)
SELECT COUNT(*) FROM ivanti_findings
WHERE state = 'open' AND bu_ownership ILIKE ANY(ARRAY['%STEAM%', '%ACCESS-ENG%']);
-- Closed count for specific BUs
SELECT COUNT(*) FROM ivanti_findings
WHERE state = 'closed' AND bu_ownership ILIKE ANY(ARRAY['%STEAM%', '%ACCESS-ENG%']);
-- Aggregated counts grouped by BU and state (single query)
SELECT bu_ownership, state, COUNT(*) as count
FROM ivanti_findings
GROUP BY bu_ownership, state;
-- Global totals (no filter — backward compatible)
SELECT state, COUNT(*) as count
FROM ivanti_findings
GROUP BY state;
```
### Data Migration Script Design (`backend/scripts/migrate-to-postgres.js`)
```mermaid
flowchart TD
A[Open SQLite read-only] --> B[Connect to Postgres pool]
B --> C[Create all tables IF NOT EXISTS]
C --> D[Copy simple tables]
D --> E[Parse findings_json blob]
E --> F[Insert individual finding rows state=open]
F --> G[Merge ivanti_finding_notes → findings.note]
G --> H[Merge ivanti_finding_overrides → findings.override_*]
H --> I[Verify row counts]
I --> J[Print summary report]
```
The migration script:
1. Opens SQLite with `OPEN_READONLY` flag
2. Connects to Postgres via `DATABASE_URL`
3. Creates schema idempotently (`IF NOT EXISTS`)
4. Copies each table using batch inserts with `ON CONFLICT` for idempotency
5. Special handling for findings: parses `findings_json`, creates one row per finding
6. Merges notes and overrides into the corresponding finding rows
7. Verifies source vs destination row counts
8. Never modifies the SQLite file
## Correctness Properties
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
### Property 1: Upsert Idempotence
*For any* finding synced N times (N ≥ 1) with the same finding ID, the `ivanti_findings` table SHALL contain exactly one row for that finding ID, with the data from the most recent sync.
**Validates: Requirements 3.5, 6.6**
### Property 2: Finding Storage Preserves State and BU Ownership
*For any* finding (open or closed) stored in `ivanti_findings`, querying it back by ID SHALL return the same `state` and `bu_ownership` values it was stored with.
**Validates: Requirements 3.4, 4.1**
### Property 3: Count Query Accuracy
*For any* set of findings in `ivanti_findings` and any BU filter (including empty/no filter), the count query result SHALL equal the actual number of rows matching that filter and state combination.
**Validates: Requirements 4.2, 4.3, 4.5**
### Property 4: Migration Data Preservation (Findings)
*For any* finding in the source `findings_json` blob with associated notes (from `ivanti_finding_notes`) and overrides (from `ivanti_finding_overrides`), the migrated `ivanti_findings` row SHALL contain the finding data with `state = 'open'`, the correct `note` value, and the correct `override_host_name`/`override_dns` values.
**Validates: Requirements 7.4, 7.5, 7.6**
### Property 5: Migration Table Copy Preservation
*For any* table copied from SQLite to Postgres, the row count in Postgres SHALL equal the row count in SQLite, and each row's data SHALL be equivalent (accounting for type conversions: 0/1 → boolean, DATETIME → TIMESTAMPTZ).
**Validates: Requirements 7.7, 7.8**
### Property 6: Migration Idempotence
*For any* initial state of the SQLite and Postgres databases, running the migration script N times (N ≥ 1) SHALL produce the same final state in Postgres as running it exactly once (no duplicate rows, no errors).
**Validates: Requirements 7.3, 7.9**
### Property 7: Migration Source Safety
*For any* execution of the migration script, the SQLite database file SHALL remain byte-for-byte identical before and after (checksum unchanged).
**Validates: Requirements 7.10**
### Property 8: Schema Creation Idempotence
*For any* number of times the schema creation DDL is executed against the same database, the resulting schema SHALL be identical (no errors, same tables, same indexes, same constraints).
**Validates: Requirements 2.5**
### Property 9: API Response Shape Preservation
*For any* valid API request to any endpoint, the response JSON structure (top-level keys and value types) after migration SHALL be identical to the pre-migration response structure.
**Validates: Requirements 10.1**
## Error Handling
| Error Scenario | Handling Strategy |
|----------------|-------------------|
| Pool connection failure | `pool.on('error')` logs error; automatic reconnection on next query attempt |
| Pool exhaustion (all 10 busy) | Queries queue internally; warning logged at 8 active connections |
| Query timeout | `connectionTimeoutMillis: 5000` — rejects after 5s with error |
| Sync failure mid-batch | Transaction rollback; `sync_status = 'error'` with message; previous data preserved |
| Migration script failure | Idempotent design — safe to re-run; prints error and exits with code 1 |
| Docker container crash | `--restart unless-stopped` auto-recovers; pool reconnects on next query |
| Invalid finding data | `NOT NULL DEFAULT ''` columns prevent null constraint violations; CHECK constraints reject invalid state values |
| Rollback needed | Stop Postgres backend → revert `.env` → restart SQLite backend; SQLite file always preserved |
### Error Response Format (Unchanged)
All error responses maintain the existing format:
```json
{ "error": "Human-readable error message" }
```
With appropriate HTTP status codes: 400 (validation), 401 (auth), 403 (permission), 404 (not found), 500 (server error).
## Testing Strategy
### Unit Tests (Example-Based)
- Verify each route returns correct response shape with known test data
- Verify auth middleware rejects expired sessions
- Verify parameter placeholder conversion (`?` → `$1`) in all queries
- Verify schema DDL executes without errors
- Verify migration script handles empty tables gracefully
### Property-Based Tests
Property-based testing is appropriate for this feature because the core operations (upsert, count queries, data migration) have clear input/output behavior with universal properties that hold across a wide input space.
**Library**: [fast-check](https://github.com/dubzzz/fast-check) (JavaScript PBT library)
**Configuration**: Minimum 100 iterations per property test.
**Tag format**: `Feature: postgres-migration, Property {number}: {property_text}`
Each correctness property (1-9) maps to a single property-based test:
- Property 1: Generate random findings, upsert each N times, verify exactly one row per ID
- Property 2: Generate findings with random state/bu_ownership, store and retrieve, verify equality
- Property 3: Generate random finding sets, insert, run count queries with random filters, verify accuracy
- Property 4: Generate random findings JSON with notes/overrides, run migration logic, verify merged output
- Property 5: Generate random table rows, copy via migration, verify count and data equivalence
- Property 6: Run migration logic N times on same input, verify final state equals single-run state
- Property 7: Checksum SQLite before/after migration, verify unchanged
- Property 8: Run schema DDL N times, verify no errors and same schema
- Property 9: Compare response shapes between SQLite and Postgres backends for same requests
### Integration Tests
- Docker container health check (port 5433 accessible)
- Full sync cycle: trigger sync → verify rows created → verify counts endpoint
- Concurrent read during write: start sync, simultaneously query findings, verify no blocking
- Performance: GET /findings < 500ms, GET /counts < 100ms with 6000+ rows
- Cutover simulation: stop/start backend, verify API responds correctly
### Development Isolation
- Test backend runs on port 3003 with `DATABASE_URL` pointing to Postgres
- Production backend continues on port 3001 with SQLite (no `DATABASE_URL` set)
- Switching is controlled by presence of `DATABASE_URL` environment variable
- All work on `feature/multi-tenancy` branch

158
.kiro/specs/postgres-migration/requirements.md Normal file
View File

@@ -0,0 +1,158 @@
# Requirements Document
## Introduction
Migrate the CVE Dashboard (STEAM Security Dashboard) backend from SQLite to PostgreSQL 16. The current SQLite architecture stores all Ivanti findings as a single 2.6MB JSON blob (`ivanti_findings_cache.findings_json`) that must be parsed on every API request, causing 5-10 second load times. Additionally, SQLite's single-writer lock blocks reads during sync writes, and per-BU closed finding counts are unavailable (only a global count exists). PostgreSQL enables individual finding rows with indexed columns, per-BU open and closed counts, connection pooling for concurrent access, and proper type support. The Postgres instance runs in a dedicated Docker container on port 5433, isolated from the existing Postgres on port 5432 which belongs to another project.
## Glossary
- **SQLite**: The current embedded database engine storing all data in `backend/cve_database.db` (13MB total)
- **PostgreSQL_16**: The target relational database running in a Docker container (`steam-postgres`) on port 5433
- **findings_json**: The current TEXT column in `ivanti_findings_cache` storing all Ivanti findings as a serialized JSON array (2.6MB+)
- **ivanti_findings**: The new Postgres table storing each finding as an individual row with indexed columns
- **Pool**: A `pg` (node-postgres) connection pool managing up to 10 concurrent database connections
- **DATABASE_URL**: Environment variable containing the Postgres connection string (`postgresql://steam:<password>@localhost:5433/cve_dashboard`)
- **Migration_Script**: A one-time Node.js script (`backend/scripts/migrate-to-postgres.js`) that reads from SQLite and writes to Postgres
- **Cutover**: The moment production switches from SQLite backend on port 3001 to Postgres backend on port 3001
- **steam-postgres**: The Docker container name for the CVE Dashboard's dedicated PostgreSQL instance
- **steam-pgdata**: The Docker volume providing persistent storage for the Postgres data directory
- **bu_ownership**: The Ivanti finding field containing the BU assignment (e.g. "NTS-AEO-STEAM")
- **Upsert**: An INSERT that updates the existing row on primary key conflict (`INSERT ... ON CONFLICT DO UPDATE`)
## Requirements
### Requirement 1: Dedicated PostgreSQL Docker Container
**User Story:** As a system operator, I want a dedicated Postgres 16 container for the CVE Dashboard on port 5433, so that it is fully isolated from the existing Postgres instance on port 5432 belonging to another project.
#### Acceptance Criteria
1. THE Infrastructure SHALL run PostgreSQL 16 (Alpine) in a Docker container named `steam-postgres`
2. THE container SHALL map host port 5433 to container port 5432
3. THE container SHALL use a Docker volume named `steam-pgdata` for persistent data storage
4. THE container SHALL be configured with `--restart unless-stopped` for automatic recovery after host reboots
5. THE container SHALL create a database named `cve_dashboard` with a user named `steam`
6. THE Infrastructure SHALL NOT modify or affect the existing Postgres instance on port 5432
### Requirement 2: Schema Parity with Proper Types
**User Story:** As a developer, I want all existing SQLite tables recreated in Postgres with proper types and constraints, so that no functionality is lost during migration.
#### Acceptance Criteria
1. THE Postgres schema SHALL include all tables from the current SQLite schema: users, sessions, audit_logs, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, compliance_uploads, compliance_items, compliance_notes, ivanti_sync_state, ivanti_finding_notes, ivanti_counts_cache, ivanti_counts_history, ivanti_finding_overrides, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue, atlas_action_plans_cache
2. THE Postgres schema SHALL use appropriate types: SERIAL for auto-increment, TIMESTAMPTZ for timestamps, BOOLEAN for booleans, NUMERIC for decimals, TEXT[] for arrays, DATE for date-only fields
3. THE Postgres schema SHALL preserve all existing CHECK constraints and foreign key relationships
4. THE Postgres schema SHALL include the `bu_teams` column on the users table (required by multi-BU tenancy feature)
5. THE schema creation SHALL be idempotent using `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS`
### Requirement 3: Findings Table Redesign
**User Story:** As a user, I want findings stored as individual rows with indexed columns, so that filtering by BU ownership and state is instant instead of requiring full JSON blob parsing.
#### Acceptance Criteria
1. THE system SHALL replace the `ivanti_findings_cache.findings_json` blob with an `ivanti_findings` table containing one row per finding
2. THE `ivanti_findings` table SHALL include columns: id (TEXT PRIMARY KEY, Ivanti finding ID), host_id (INTEGER), title (TEXT), severity (NUMERIC), vrr_group (TEXT), host_name (TEXT), ip_address (TEXT), dns (TEXT), status (TEXT), sla_status (TEXT), due_date (DATE), last_found_on (DATE), bu_ownership (TEXT), cves (TEXT[] array), workflow_id (TEXT), workflow_state (TEXT), workflow_type (TEXT), state (TEXT with CHECK constraint for 'open' or 'closed'), note (TEXT), override_host_name (TEXT), override_dns (TEXT), synced_at (TIMESTAMPTZ), created_at (TIMESTAMPTZ)
3. THE system SHALL create indexes on: state, bu_ownership, severity, and a composite index on (state, bu_ownership)
4. THE system SHALL store both open AND closed findings as individual rows with their respective state values
5. WHEN findings are synced from Ivanti, THE system SHALL upsert rows using `INSERT ... ON CONFLICT (id) DO UPDATE` rather than replacing a JSON blob
### Requirement 4: Per-BU Closed Finding Counts
**User Story:** As a user viewing the Reporting page with a BU scope filter, I want accurate open and closed counts for my selected BUs, so that the dashboard shows meaningful per-team metrics instead of only a global count.
#### Acceptance Criteria
1. THE system SHALL store closed findings with their `bu_ownership` field preserved as individual rows with `state = 'closed'`
2. THE counts endpoint SHALL derive per-BU counts using `SELECT COUNT(*) ... WHERE state = ? AND bu_ownership ILIKE ?` queries
3. WHEN a teams filter is provided, THE counts endpoint SHALL return open and closed counts scoped to those BUs
4. WHEN no filter is applied, THE counts endpoint SHALL return global totals (backward compatible with current behavior)
5. THE system SHALL support aggregated counts grouped by bu_ownership and state in a single query
### Requirement 5: Connection Pooling
**User Story:** As a system operator, I want the backend to use connection pooling via the pg package, so that multiple concurrent requests are handled efficiently without blocking and reads are never blocked by writes.
#### Acceptance Criteria
1. THE backend SHALL use the `pg` npm package with a Pool instance (maximum pool size: 10 connections)
2. THE Pool SHALL read the connection string from the `DATABASE_URL` environment variable
3. IF the Pool detects connection errors, THEN THE Pool SHALL attempt automatic reconnection
4. THE Pool SHALL log a warning when active connections reach 8 (approaching exhaustion)
5. ALL database operations SHALL use async/await with the Pool (replacing all callback-based SQLite patterns)
### Requirement 6: Backend Route Migration
**User Story:** As a developer, I want all SQLite-specific code replaced with Postgres equivalents using async/await, so that the codebase uses a single database driver consistently.
#### Acceptance Criteria
1. THE backend SHALL replace all `db.get(sql, params, callback)` calls with `pool.query(sql, params)` returning `rows[0]`
2. THE backend SHALL replace all `db.all(sql, params, callback)` calls with `pool.query(sql, params)` returning `rows`
3. THE backend SHALL replace all `db.run(sql, params, callback)` calls with `pool.query(sql, params)` using `RETURNING` clauses where the inserted/updated row is needed
4. THE backend SHALL replace `?` parameter placeholders with `$1, $2, $3...` Postgres numbered parameter syntax
5. THE backend SHALL remove all callback-based database patterns in favor of async/await with try/catch error handling
6. THE Ivanti sync logic SHALL write individual finding rows via upsert instead of serializing to a JSON blob
### Requirement 7: Data Migration Script
**User Story:** As a system operator, I want a one-time migration script that copies all data from SQLite to Postgres, so that no data is lost during the transition.
#### Acceptance Criteria
1. THE Migration_Script SHALL open the SQLite database in read-only mode
2. THE Migration_Script SHALL connect to Postgres using the DATABASE_URL connection string
3. THE Migration_Script SHALL create all tables idempotently before inserting data
4. THE Migration_Script SHALL parse the `findings_json` blob and insert individual finding rows into the `ivanti_findings` table with `state = 'open'`
5. THE Migration_Script SHALL merge `ivanti_finding_notes` into the corresponding `ivanti_findings.note` column
6. THE Migration_Script SHALL merge `ivanti_finding_overrides` into the corresponding `ivanti_findings.override_*` columns
7. THE Migration_Script SHALL copy all other tables preserving their data: users, sessions, cves, documents, jira_tickets, archer_tickets, knowledge_base, audit_logs, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, atlas_action_plans_cache, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue
8. THE Migration_Script SHALL verify row counts after migration and print a summary with any discrepancies
9. THE Migration_Script SHALL be idempotent (safe to run multiple times using ON CONFLICT/upsert logic)
10. THE Migration_Script SHALL never modify the SQLite database file
### Requirement 8: Zero-Impact Cutover
**User Story:** As a system operator, I want the cutover from SQLite to Postgres to take under 30 seconds with a clear rollback path, so that downtime is minimal and reversible.
#### Acceptance Criteria
1. THE cutover procedure SHALL consist of: stop old backend, start new backend on the same port (3001)
2. THE frontend SHALL NOT require any changes for the cutover (same API contract, same port, same URL)
3. IF issues are detected after cutover, THEN THE system SHALL support rollback by reverting the .env configuration and restarting the SQLite-based backend
4. THE SQLite database file SHALL be preserved indefinitely as a backup after cutover
5. THE cutover downtime SHALL not exceed 30 seconds
### Requirement 9: Performance Improvement
**User Story:** As a user, I want the dashboard to load findings in under 1 second regardless of BU scope, so that switching between BU views feels instant compared to the current 5-10 second load times.
#### Acceptance Criteria
1. THE `GET /api/ivanti/findings` endpoint SHALL respond in under 500ms for any BU filter combination
2. THE `GET /api/ivanti/findings/counts` endpoint SHALL respond in under 100ms for any BU filter
3. THE Ivanti sync process SHALL complete without blocking concurrent read requests (Postgres MVCC)
4. THE system SHALL eliminate JSON blob parsing from the findings read path entirely
### Requirement 10: API Backward Compatibility
**User Story:** As a developer, I want the API contract to remain unchanged after migration, so that the frontend works identically without code changes.
#### Acceptance Criteria
1. ALL existing API endpoints SHALL return the same response shape after migration
2. THE `GET /api/ivanti/findings` response SHALL include: findings array, total count, synced_at timestamp, sync_status, and error_message fields
3. THE authentication and session system SHALL work identically (cookie-based sessions, same expiration behavior)
4. THE frontend SHALL require zero code changes specifically for the database migration (multi-BU filtering changes are a separate feature)
### Requirement 11: Development and Testing Isolation
**User Story:** As a developer, I want to test the Postgres backend on port 3003 while production continues on port 3001 with SQLite, so that development does not disrupt the live system.
#### Acceptance Criteria
1. WHILE the migration is in development, THE test backend SHALL run on port 3003
2. WHILE the migration is in development, THE production backend SHALL continue running on port 3001 with SQLite
3. THE system SHALL support switching between SQLite and Postgres via environment variable configuration (DATABASE_URL presence or DB_TYPE flag)
4. ALL development work SHALL occur on the existing `feature/multi-tenancy` branch

294
.kiro/specs/postgres-migration/tasks.md Normal file
View File

@@ -0,0 +1,294 @@
# Implementation Plan: PostgreSQL Migration
## Overview
Migrate the CVE Dashboard backend from SQLite to PostgreSQL 16. Replace the monolithic `findings_json` blob (2.6MB) with individual indexed rows in `ivanti_findings`, enable per-BU closed counts, and eliminate the single-writer lock. All work on the `feature/multi-tenancy` branch. Docker container `steam-postgres` on port 5433, test backend on port 3003, production on port 3001.
## Tasks
- [x] 1. Infrastructure setup and connection pool
- [x] 1.1 Install `pg` dependency and configure environment
- Run `npm install pg` in `backend/`
- Add `DATABASE_URL=postgresql://steam:<password>@localhost:5433/cve_dashboard` to `backend/.env`
- Add `DATABASE_URL` placeholder to `backend/.env.example`
- _Requirements: 1.5, 5.1, 5.2_
- [x] 1.2 Create `backend/db.js` connection pool module
- Import `pg` and create a `Pool` instance reading from `DATABASE_URL`
- Set `max: 10`, `idleTimeoutMillis: 30000`, `connectionTimeoutMillis: 5000`
- Add `pool.on('error')` handler logging unexpected errors
- Track active connections; log warning when count reaches 8
- Export the pool instance for use by all route files
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5_
- [x] 1.3 Create Docker run command documentation
- Document the `docker run` command for `steam-postgres` container (port 5433:5432, volume `steam-pgdata`, `--restart unless-stopped`, Postgres 16 Alpine)
- Verify container creates `cve_dashboard` database with `steam` user
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6_
- [x] 2. Schema creation
- [x] 2.1 Create `backend/db-schema.sql` with complete Postgres DDL
- Define all tables with proper Postgres types: SERIAL, TIMESTAMPTZ, BOOLEAN, NUMERIC, TEXT[], DATE
- Include `ivanti_findings` table with TEXT PRIMARY KEY (`id`), all columns per design, CHECK constraint on `state`
- Include `ivanti_sync_state` table (single-row pattern, replaces `ivanti_findings_cache` metadata)
- Include all other tables: users (with `bu_teams`), sessions, audit_logs, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_cache, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue, atlas_action_plans_cache
- Include all indexes: findings (state, bu_ownership, severity, state+bu_ownership composite), sessions (session_id, user_id, expires_at), audit_logs (created_at), todo_queue (user_id+status)
- Include all foreign key relationships and CHECK constraints
- Use `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS` for idempotence
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 3.1, 3.2, 3.3_
- [ ]* 2.2 Write property test for schema creation idempotence
- **Property 8: Schema Creation Idempotence**
- Run schema DDL N times against the same database, verify no errors and same resulting schema each time
- **Validates: Requirements 2.5**
- [x] 2.3 Create schema initialization in `backend/setup.js`
- Read `db-schema.sql` and execute via pool.query
- Make callable on server startup or as standalone script
- Seed `ivanti_sync_state` row (id=1) and `ivanti_counts_cache` row (id=1) if not exists
- _Requirements: 2.1, 2.5_
- [x] 3. Checkpoint — Verify infrastructure
- Ensure Docker container is running on port 5433, pool connects successfully, schema creates without errors. Ask the user if questions arise.
- [-] 4. Migrate auth and session system
- [x] 4.1 Update `backend/middleware/auth.js`
- Replace `db.get()` callback with `pool.query()` async/await
- Change `?` placeholders to `$1, $2, ...` numbered params
- Join sessions and users tables, check `expires_at > NOW()`
- Return 401 for missing/expired sessions, 500 for query errors
- _Requirements: 6.1, 6.4, 6.5, 10.3_
- [x] 4.2 Update `backend/routes/auth.js`
- Replace all `db.get/db.all/db.run` with `pool.query`
- Login: query user by username, create session with `RETURNING`
- Logout: delete session by session_id
- Password change: update password_hash
- Profile/me endpoint: query user by session
- Use `$1, $2...` placeholders throughout
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 10.3_
- [x] 4.3 Update `backend/routes/users.js`
- Replace all sqlite3 calls with pool.query
- CRUD operations: list users, create user, update user, delete user
- Use `RETURNING` clause for inserts/updates where row data is needed
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
- [x] 4.4 Update `backend/server.js` database initialization
- Remove sqlite3 database opening and `db` object creation
- Import pool from `backend/db.js`
- Remove passing `db` parameter to route factory functions
- Update inline CVE/document/vendor routes to use pool.query
- _Requirements: 6.1, 6.4, 6.5_
- [ ] 5. Checkpoint — Verify auth and core routes
- Ensure login, logout, session validation, user CRUD, and CVE routes work on port 3003. Ask the user if questions arise.
- [-] 6. Migrate remaining route files
- [x] 6.1 Update `backend/routes/jiraTickets.js`
- Replace sqlite3 calls with pool.query, update placeholders
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.2 Update `backend/routes/archerTickets.js`
- Replace sqlite3 calls with pool.query, update placeholders
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.3 Update `backend/routes/knowledgeBase.js`
- Replace sqlite3 calls with pool.query, update placeholders
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.4 Update `backend/routes/compliance.js`
- Replace sqlite3 calls with pool.query, update placeholders
- Handle compliance_items, compliance_uploads, compliance_notes queries
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
- [x] 6.5 Update `backend/routes/auditLog.js` and `backend/helpers/auditLog.js`
- Replace sqlite3 db.run/db.all with pool.query
- Update the audit logging helper to use async pool.query
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.6 Update `backend/routes/ivantiWorkflows.js`
- Replace sqlite3 calls with pool.query, update placeholders
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.7 Update `backend/routes/ivantiFpWorkflow.js`
- Replace sqlite3 calls with pool.query, update placeholders
- Handle fp_submissions and fp_submission_history tables
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.8 Update `backend/routes/ivantiTodoQueue.js`
- Replace sqlite3 calls with pool.query, update placeholders
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.9 Update `backend/routes/ivantiArchive.js`
- Replace sqlite3 calls with pool.query, update placeholders
- Handle archive and transition table queries
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.10 Update `backend/routes/atlas.js`
- Replace sqlite3 calls with pool.query, update placeholders
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.11 Update `backend/routes/cardApi.js` and `backend/routes/feedback.js`
- Replace any sqlite3 calls with pool.query if present
- _Requirements: 6.1, 6.4, 6.5_
- [ ] 7. Checkpoint — Verify all route migrations
- Ensure all non-findings routes work correctly on port 3003. Ask the user if questions arise.
- [-] 8. Rewrite Ivanti findings sync and read logic
- [x] 8.1 Rewrite sync logic in `backend/routes/ivantiFindings.js`
- Replace JSON blob serialization with batch upsert of individual rows
- Use `INSERT INTO ivanti_findings (...) VALUES ... ON CONFLICT (id) DO UPDATE SET ...` in batches of 100
- Sync both open and closed findings as individual rows with correct `state` value
- Update `ivanti_sync_state` with total count, synced_at, sync_status after sync
- Preserve existing note and override values during upsert (do not overwrite user-set fields)
- _Requirements: 3.4, 3.5, 6.6, 9.3_
- [ ]* 8.2 Write property test for upsert idempotence
- **Property 1: Upsert Idempotence**
- Generate random findings, upsert each N times with same ID, verify exactly one row per ID with most recent data
- **Validates: Requirements 3.5, 6.6**
- [x] 8.3 Rewrite read endpoints in `backend/routes/ivantiFindings.js`
- GET /findings: `SELECT * FROM ivanti_findings WHERE state = 'open'` with optional BU filter via `bu_ownership ILIKE`
- Return response shape: `{ findings, total, synced_at, sync_status, error_message }`
- GET /findings/counts: derive open/closed counts from `ivanti_findings` with optional BU filter
- Support `teams` query parameter for per-BU scoped counts using `ILIKE ANY(ARRAY[...])`
- GET /findings/counts/history: unchanged (reads from `ivanti_counts_history`)
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 9.1, 9.2, 10.1, 10.2_
- [ ]* 8.4 Write property test for finding state and BU preservation
- **Property 2: Finding Storage Preserves State and BU Ownership**
- Generate findings with random state/bu_ownership, store and retrieve by ID, verify equality
- **Validates: Requirements 3.4, 4.1**
- [ ]* 8.5 Write property test for count query accuracy
- **Property 3: Count Query Accuracy**
- Generate random finding sets, insert into DB, run count queries with random BU filters, verify counts match actual row counts
- **Validates: Requirements 4.2, 4.3, 4.5**
- [x] 8.6 Update note and override endpoints
- PUT /findings/:id/note: `UPDATE ivanti_findings SET note = $1 WHERE id = $2`
- PUT /findings/:id/override: `UPDATE ivanti_findings SET override_host_name = $1, override_dns = $2 WHERE id = $3`
- _Requirements: 6.1, 6.4_
- [ ] 9. Checkpoint — Verify findings redesign
- Ensure sync creates individual rows, read endpoints return correct shape, counts work with and without BU filter. Ask the user if questions arise.
- [-] 10. Data migration script
- [x] 10.1 Create `backend/scripts/migrate-to-postgres.js`
- Open SQLite database in read-only mode (`OPEN_READONLY`)
- Connect to Postgres via `DATABASE_URL`
- Run schema creation (idempotent) before inserting data
- Copy all simple tables: users, sessions, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, audit_logs, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, atlas_action_plans_cache, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue
- Handle type conversions: SQLite 0/1 → Postgres boolean, DATETIME strings → TIMESTAMPTZ
- Use batch inserts with `ON CONFLICT` for idempotency (safe to re-run)
- _Requirements: 7.1, 7.2, 7.3, 7.7, 7.9, 7.10_
- [x] 10.2 Implement findings migration with note/override merging
- Parse `ivanti_findings_cache.findings_json` blob into individual objects
- Insert each finding as a row in `ivanti_findings` with `state = 'open'`
- Query `ivanti_finding_notes` and merge each note into the corresponding `ivanti_findings.note` column
- Query `ivanti_finding_overrides` and merge into `ivanti_findings.override_host_name` / `override_dns`
- _Requirements: 7.4, 7.5, 7.6_
- [x] 10.3 Add verification and summary reporting
- After migration, query row counts for every table in both SQLite and Postgres
- Print comparison table showing source count vs destination count
- Flag any discrepancies with warnings
- Exit with code 0 on success, code 1 on any errors
- _Requirements: 7.8, 7.9_
- [ ]* 10.4 Write property test for migration data preservation (findings)
- **Property 4: Migration Data Preservation (Findings)**
- Generate random findings JSON with associated notes and overrides, run migration logic, verify merged output matches expected values
- **Validates: Requirements 7.4, 7.5, 7.6**
- [ ]* 10.5 Write property test for migration table copy preservation
- **Property 5: Migration Table Copy Preservation**
- Generate random table rows, copy via migration logic, verify row counts and data equivalence (accounting for type conversions)
- **Validates: Requirements 7.7, 7.8**
- [ ]* 10.6 Write property test for migration idempotence
- **Property 6: Migration Idempotence**
- Run migration logic N times on same input data, verify final Postgres state equals single-run state (no duplicates)
- **Validates: Requirements 7.3, 7.9**
- [ ]* 10.7 Write property test for migration source safety
- **Property 7: Migration Source Safety**
- Checksum SQLite file before and after migration script execution, verify bytes unchanged
- **Validates: Requirements 7.10**
- [ ] 11. Checkpoint — Verify data migration
- Run migration script against test Postgres instance, verify all row counts match, verify findings have merged notes/overrides. Ask the user if questions arise.
- [ ] 12. Frontend updates for per-BU closed counts
- [ ] 12.1 Update ReportingPage donut chart to show per-BU closed counts
- Remove "N/A" fallback for closed count when BU filter is active
- Display actual closed count from the updated `/api/ivanti/findings/counts` endpoint
- Pass `teams` parameter to counts endpoint for server-side BU filtering
- _Requirements: 4.3, 4.4_
- [ ]* 12.2 Write property test for API response shape preservation
- **Property 9: API Response Shape Preservation**
- For various valid API requests, verify response JSON structure (top-level keys and value types) matches expected contract
- **Validates: Requirements 10.1**
- [ ] 13. Testing and verification
- [ ] 13.1 Run backend on port 3003 and verify all endpoints
- Set `PORT=3003` in test environment with `DATABASE_URL` pointing to Postgres
- Verify auth endpoints (login, logout, me, password change)
- Verify findings endpoints (list, counts, sync, notes, overrides)
- Verify all other routes (CVEs, Jira, Archer, compliance, knowledge base, audit log)
- Compare response shapes against production SQLite backend on port 3001
- _Requirements: 9.1, 9.2, 10.1, 10.2, 10.3, 10.4, 11.1, 11.2, 11.3_
- [ ] 13.2 Performance verification
- Confirm GET /api/ivanti/findings responds in <500ms with full dataset
- Confirm GET /api/ivanti/findings/counts responds in <100ms
- Confirm sync does not block concurrent read requests
- _Requirements: 9.1, 9.2, 9.3, 9.4_
- [ ] 13.3 Run existing test suite against Postgres backend
- Verify all existing property tests and unit tests pass
- _Requirements: 10.1, 10.3_
- [ ] 14. Cutover to production
- [ ] 14.1 Execute cutover procedure
- Run final Ivanti sync on SQLite production backend
- Run migration script to copy latest data to Postgres
- Stop production backend on port 3001
- Update production `.env` with `DATABASE_URL`
- Start new backend on port 3001 with Postgres
- Verify frontend loads and API responds correctly
- _Requirements: 8.1, 8.2, 8.3_
- [ ] 14.2 Preserve SQLite backup and document rollback
- Keep `backend/cve_database.db` as permanent backup (do not delete)
- Document rollback procedure: stop backend → remove DATABASE_URL from .env → restart
- _Requirements: 8.3, 8.4_
- [ ] 15. Cleanup
- [ ] 15.1 Remove SQLite dependency and legacy code
- Run `npm uninstall sqlite3` from `backend/`
- Remove any remaining sqlite3 imports or `db` parameter passing
- Remove the old `ivanti_findings_cache` and `ivanti_finding_notes` / `ivanti_finding_overrides` table references
- Update `backend/.env.example` and README with Postgres prerequisites
- _Requirements: 6.5, 11.4_
- [ ] 16. Final checkpoint — Ensure all tests pass
- Ensure all tests pass, production is stable on Postgres, and SQLite backup is preserved. Ask the user if questions arise.
## Notes
- Tasks marked with `*` are optional and can be skipped for faster MVP
- Each task references specific requirements for traceability
- Checkpoints ensure incremental validation between major phases
- Property tests validate universal correctness properties from the design document
- All work on `feature/multi-tenancy` branch
- Docker container `steam-postgres` on port 5433 (NOT 5432 — that belongs to another project)
- Test backend on port 3003 during development; production stays on port 3001 with SQLite until cutover
- The `pg` package connection pool handles concurrent reads during sync writes (MVCC)
- Batch upserts in chunks of 100 for sync performance
- SQLite file is never modified — opened read-only during migration

1
.kiro/specs/ticket-linking/.config.kiro Normal file
View File

@@ -0,0 +1 @@
{"specId": "c9369370-989b-476a-8305-b62401390b71", "workflowType": "requirements-first", "specType": "feature"}

311
.kiro/specs/ticket-linking/design.md Normal file
View File

@@ -0,0 +1,311 @@
# Design Document: Ticket Linking
## Overview
The ticket-linking feature introduces a generic many-to-many association system between the three entity types in the CVE Dashboard: Archer tickets (EXC-XXXX), Jira tickets (PROJECT-XXXX), and CVEs (CVE-YYYY-NNNNN). A new `ticket_links` PostgreSQL table stores directional link records, but the system treats them as bidirectional — querying from either side returns the link. The existing `cve_id`/`vendor` foreign key columns on `archer_tickets` and `jira_tickets` remain unchanged; this feature is purely additive.
The implementation spans three layers:
1. **Database** — A `ticket_links` table with CHECK constraints, unique index, and bidirectional query indexes.
2. **Backend API** — A new Express router (`routes/links.js`) exposing POST/GET/DELETE endpoints behind `requireAuth()`.
3. **Frontend** — A reusable `LinkedItems` React component rendered on Archer and Jira detail views.
## Architecture
```mermaid
flowchart TD
subgraph Frontend
A[ArcherTicketDetail] --> C[LinkedItems Component]
B[JiraPage Detail View] --> C
C -->|POST /api/links| D[Links Router]
C -->|GET /api/links?type=&id=| D
C -->|DELETE /api/links/:id| D
end
subgraph Backend
D --> E[Validation Layer]
E --> F[PostgreSQL Pool]
end
subgraph Database
F --> G[ticket_links table]
F --> H[audit_logs table]
end
```
### Design Decisions
| Decision | Rationale |
|----------|-----------|
| TEXT columns for source/target types and IDs | Avoids polymorphic FK complexity; entity types have well-defined ID patterns validated at the application layer |
| Bidirectional queries via OR conditions | Single query returns links regardless of which side was the "source" at creation time |
| Duplicate prevention checks both (A→B) and (B→A) | Treats links as undirected edges — prevents logical duplicates stored in different directions |
| Separate `ticket_links` table (not modifying existing FKs) | Purely additive; zero risk to existing Archer/Jira/CVE workflows |
| Relationship types as TEXT with CHECK constraint | Extensible without migrations; initial set: `related`, `spawned_by`, `blocks` |
| Auto-detect entity type from key format | Reduces user friction — no need to manually select a type dropdown |
## Components and Interfaces
### Backend: `routes/links.js`
```javascript
// Express router factory — follows existing project pattern
function createLinksRouter() {
const router = express.Router();
// POST /api/links — Create a new link
router.post('/', requireAuth(), async (req, res) => { /* ... */ });
// GET /api/links?type=archer&id=EXC-6056 — Query links for an entity
router.get('/', requireAuth(), async (req, res) => { /* ... */ });
// DELETE /api/links/:id — Remove a link by ID
router.delete('/:id', requireAuth(), async (req, res) => { /* ... */ });
return router;
}
```
**Validation helpers:**
```javascript
const ENTITY_PATTERNS = {
archer: /^EXC-\d{4,}$/,
jira: /^[A-Z][A-Z0-9_]+-\d+$/,
cve: /^CVE-\d{4}-\d{4,}$/
};
const VALID_ENTITY_TYPES = ['archer', 'jira', 'cve'];
const VALID_RELATIONSHIPS = ['related', 'spawned_by', 'blocks'];
function detectEntityType(key) {
if (ENTITY_PATTERNS.cve.test(key)) return 'cve';
if (ENTITY_PATTERNS.archer.test(key)) return 'archer';
if (ENTITY_PATTERNS.jira.test(key)) return 'jira';
return null;
}
```
### Frontend: `LinkedItems` Component
```
Props:
- entityType: 'archer' | 'jira' | 'cve'
- entityId: string (e.g., 'EXC-6056')
State:
- links: Array<LinkRecord>
- loading: boolean
- showAddForm: boolean
- error: string | null
```
The component fetches links on mount via `GET /api/links?type={entityType}&id={entityId}`, renders a list of linked items with type badges and relationship labels, and provides "Add Link" / "Remove" controls.
### API Contract
**POST /api/links**
```json
// Request
{
"source_type": "archer",
"source_id": "EXC-6056",
"target_type": "jira",
"target_id": "STEAM-1234",
"relationship": "related"
}
// Response 201
{
"id": 42,
"source_type": "archer",
"source_id": "EXC-6056",
"target_type": "jira",
"target_id": "STEAM-1234",
"relationship": "related",
"created_by": 7,
"created_at": "2025-01-15T10:30:00Z"
}
```
**GET /api/links?type=archer&id=EXC-6056**
```json
// Response 200
[
{
"id": 42,
"linked_type": "jira",
"linked_id": "STEAM-1234",
"relationship": "related",
"created_by_username": "jsmith",
"created_at": "2025-01-15T10:30:00Z"
}
]
```
**DELETE /api/links/:id**
```json
// Response 200
{ "message": "Link deleted successfully" }
```
## Data Models
### `ticket_links` Table
```sql
CREATE TABLE IF NOT EXISTS ticket_links (
id SERIAL PRIMARY KEY,
source_type TEXT NOT NULL CHECK (source_type IN ('archer', 'jira', 'cve')),
source_id TEXT NOT NULL,
target_type TEXT NOT NULL CHECK (target_type IN ('archer', 'jira', 'cve')),
target_id TEXT NOT NULL,
relationship TEXT NOT NULL DEFAULT 'related'
CHECK (relationship IN ('related', 'spawned_by', 'blocks')),
created_by INTEGER REFERENCES users(id),
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE (source_type, source_id, target_type, target_id)
);
CREATE INDEX IF NOT EXISTS idx_ticket_links_source
ON ticket_links(source_type, source_id);
CREATE INDEX IF NOT EXISTS idx_ticket_links_target
ON ticket_links(target_type, target_id);
```
### Bidirectional Query Pattern
```sql
-- Find all links involving entity (type=?, id=?)
SELECT * FROM ticket_links
WHERE (source_type = $1 AND source_id = $2)
OR (target_type = $1 AND target_id = $2);
```
### Duplicate Prevention Query
Before inserting a new link (A→B), check that neither (A→B) nor (B→A) already exists:
```sql
SELECT id FROM ticket_links
WHERE (source_type = $1 AND source_id = $2 AND target_type = $3 AND target_id = $4)
OR (source_type = $3 AND source_id = $4 AND target_type = $1 AND target_id = $2);
```
### Migration File
A new migration `add_ticket_links_table.js` will be added to `backend/migrations/` following the existing pattern (idempotent DDL via `CREATE TABLE IF NOT EXISTS`).
## Correctness Properties
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
### Property 1: Entity Type Detection from Key Format
*For any* valid entity key matching one of the known patterns (EXC-XXXX for Archer, PROJECT-XXXX for Jira, CVE-YYYY-NNNNN for CVE), the `detectEntityType` function SHALL return the correct entity type; and *for any* string not matching any known pattern, it SHALL return null.
**Validates: Requirements 6.4, 1.3, 2.4**
### Property 2: Entity ID Format Validation
*For any* (entity_type, entity_id) pair where the entity_id does not match the expected regex pattern for that type, the Link_Service validation SHALL reject the input; and *for any* pair where the entity_id does match, validation SHALL accept it.
**Validates: Requirements 2.5**
### Property 3: Self-Link Rejection
*For any* valid entity (type, id), attempting to create a link where source and target are identical SHALL be rejected with a validation error, and the ticket_links table SHALL remain unchanged.
**Validates: Requirements 2.2**
### Property 4: Bidirectional Duplicate Prevention
*For any* two distinct entities A and B, after successfully creating a link from A to B, attempting to create either A→B again or B→A SHALL be rejected with a conflict error.
**Validates: Requirements 2.3, 8.3**
### Property 5: Bidirectional Query Completeness
*For any* entity E that appears in the ticket_links table (as either source or target), querying links for E SHALL return every link record where E appears on either side, and SHALL NOT return any link record where E does not appear.
**Validates: Requirements 3.1, 8.1, 8.2**
### Property 6: Link Creation Round-Trip
*For any* valid (source_type, source_id, target_type, target_id, relationship) tuple where source ≠ target and no duplicate exists, creating the link and then querying for it SHALL return a record with matching source_type, source_id, target_type, target_id, and relationship values.
**Validates: Requirements 2.1, 3.2**
### Property 7: Delete Removes Link from Queries
*For any* existing link record, after deleting it by ID, querying links for either the source or target entity SHALL no longer include that link in the results.
**Validates: Requirements 4.1**
## Error Handling
### Backend Error Responses
| Scenario | HTTP Status | Response Body |
|----------|-------------|---------------|
| Missing/invalid auth | 401 | `{ "error": "Authentication required" }` |
| Invalid entity type | 400 | `{ "error": "Invalid entity type. Must be one of: archer, jira, cve" }` |
| Invalid entity ID format | 400 | `{ "error": "Invalid ID format for type '{type}'. Expected: {pattern}" }` |
| Self-link attempt | 400 | `{ "error": "Cannot link an entity to itself" }` |
| Duplicate link (either direction) | 409 | `{ "error": "A link between these entities already exists", "existing_id": N }` |
| Link not found on delete | 404 | `{ "error": "Link not found" }` |
| Database error | 500 | `{ "error": "Internal server error" }` |
| Missing required fields | 400 | `{ "error": "source_type, source_id, target_type, target_id are required" }` |
### Frontend Error Handling
- API errors are caught and displayed inline in the LinkedItems component (toast or inline message).
- Network failures show a generic "Unable to reach server" message with a retry option.
- The form is NOT cleared on error so the user can correct and resubmit.
- Optimistic UI is NOT used — the list only refreshes after confirmed server success.
### Edge Cases
- **Concurrent duplicate creation**: The UNIQUE constraint on the table acts as a final safety net if two requests pass the application-layer check simultaneously. The second insert will fail with a Postgres 23505 error, which the backend maps to a 409 response.
- **Deleted entities**: Links are not cascade-deleted when an entity is removed from its source table. Orphaned links are acceptable — the UI will show the ID but may indicate "entity not found" if navigation fails.
- **Very long entity IDs**: Entity IDs are TEXT columns with no length limit at the DB level, but the regex patterns enforce reasonable lengths (Archer: 4+ digits, Jira: standard key format, CVE: standard format).
## Testing Strategy
### Unit Tests (Example-Based)
- Validation function tests: specific valid/invalid inputs for each entity type
- Default relationship value when omitted
- Auth rejection (401) for unauthenticated requests
- Empty state rendering in LinkedItems component
- Form display/hide toggle on "Add Link" button click
- Error message display on API failure
- Confirmation prompt on remove action
### Property-Based Tests
**Library**: [fast-check](https://github.com/dubzzz/fast-check) (already compatible with the project's Jest test runner)
**Configuration**: Minimum 100 iterations per property test.
Each property test will be tagged with a comment referencing the design property:
```
// Feature: ticket-linking, Property N: {property_text}
```
Properties to implement:
1. **Entity type detection** — Generate random strings matching/not-matching patterns, verify `detectEntityType` correctness
2. **Entity ID format validation** — Generate (type, id) pairs with valid/invalid formats, verify acceptance/rejection
3. **Self-link rejection** — Generate random valid entities, verify self-link is always rejected
4. **Bidirectional duplicate prevention** — Generate entity pairs, create link, verify both directions are blocked
5. **Bidirectional query completeness** — Generate a set of links, verify query returns exactly the correct subset
6. **Link creation round-trip** — Generate valid link inputs, create and query, verify data integrity
7. **Delete removes from queries** — Generate links, delete one, verify it disappears from both sides
### Integration Tests
- Full POST → GET → DELETE lifecycle against a test database
- Audit log entries created on link creation and deletion
- UNIQUE constraint fires on concurrent duplicate inserts
- Migration is idempotent (can run multiple times without error)
- Existing archer_tickets and jira_tickets schemas unchanged after migration

123
.kiro/specs/ticket-linking/requirements.md Normal file
View File

@@ -0,0 +1,123 @@
# Requirements Document
## Introduction
The CVE Dashboard currently tracks three entity types — Archer tickets (risk acceptance exceptions), Jira tickets (work items), and CVEs (vulnerabilities) — but relationships between them are limited to a single primary CVE foreign key on each ticket. This feature introduces a generic `ticket_links` table that enables bidirectional, many-to-many associations between any combination of these entities. Users will be able to view, create, and remove links from ticket detail views in the UI.
## Glossary
- **Link_Service**: The backend service responsible for creating, querying, and deleting ticket links.
- **Link_UI**: The frontend component that displays linked items and provides controls for adding/removing links.
- **Ticket_Links_Table**: The PostgreSQL table storing associations between entities.
- **Entity**: One of the three linkable types in the system — an Archer ticket, a Jira ticket, or a CVE.
- **Source**: The entity on the left side of a link record.
- **Target**: The entity on the right side of a link record.
- **Entity_Type**: A classification string identifying the kind of entity: `archer`, `jira`, or `cve`.
- **Entity_ID**: The unique identifier for an entity within its type (e.g., `EXC-6056`, `STEAM-1234`, `CVE-2025-0905`).
- **Relationship**: A label describing the nature of a link (e.g., `related`, `spawned_by`, `blocks`).
## Requirements
### Requirement 1: Create the ticket_links Table
**User Story:** As a database administrator, I want a dedicated table for storing entity associations, so that any entity can be linked to any other entity without schema changes.
#### Acceptance Criteria
1. THE Ticket_Links_Table SHALL store a source entity (source_type, source_id) and a target entity (target_type, target_id) in each row.
2. THE Ticket_Links_Table SHALL enforce a UNIQUE constraint on the combination of (source_type, source_id, target_type, target_id) to prevent duplicate links.
3. THE Ticket_Links_Table SHALL restrict source_type and target_type values to `archer`, `jira`, or `cve`.
4. THE Ticket_Links_Table SHALL store a relationship label defaulting to `related`.
5. THE Ticket_Links_Table SHALL record the user who created the link via a created_by foreign key to the users table.
6. THE Ticket_Links_Table SHALL record the creation timestamp defaulting to the current time.
7. THE Ticket_Links_Table SHALL include indexes on (source_type, source_id) and (target_type, target_id) to support efficient bidirectional queries.
### Requirement 2: Create a Link
**User Story:** As a security analyst, I want to create a link between two entities, so that I can document relationships between Archer exceptions, Jira work items, and CVEs.
#### Acceptance Criteria
1. WHEN a valid source entity, target entity, and relationship are provided, THE Link_Service SHALL insert a new row into the Ticket_Links_Table and return the created link record.
2. WHEN the source entity and target entity are identical (same type and same ID), THE Link_Service SHALL reject the request with a validation error.
3. WHEN a link between the same source and target already exists, THE Link_Service SHALL reject the request with a conflict error indicating the duplicate.
4. WHEN the source_type or target_type is not one of `archer`, `jira`, or `cve`, THE Link_Service SHALL reject the request with a validation error.
5. WHEN the entity ID format does not match the expected pattern for its type, THE Link_Service SHALL reject the request with a validation error.
6. THE Link_Service SHALL require the user to be authenticated before creating a link.
7. THE Link_Service SHALL log an audit entry when a link is successfully created.
### Requirement 3: Query Links for an Entity
**User Story:** As a security analyst, I want to see all items linked to a given entity, so that I can understand the full context of a ticket or vulnerability.
#### Acceptance Criteria
1. WHEN an entity type and entity ID are provided, THE Link_Service SHALL return all links where the entity appears as either source or target (bidirectional query).
2. THE Link_Service SHALL return each linked entity's type, ID, relationship label, creator, and creation timestamp.
3. WHEN no links exist for the given entity, THE Link_Service SHALL return an empty list.
4. THE Link_Service SHALL require the user to be authenticated before querying links.
### Requirement 4: Delete a Link
**User Story:** As a security analyst, I want to remove a link between two entities, so that I can correct mistakes or remove outdated associations.
#### Acceptance Criteria
1. WHEN a valid link ID is provided, THE Link_Service SHALL delete the corresponding row from the Ticket_Links_Table.
2. WHEN the provided link ID does not exist, THE Link_Service SHALL return a not-found error.
3. THE Link_Service SHALL require the user to be authenticated before deleting a link.
4. THE Link_Service SHALL log an audit entry when a link is successfully deleted.
### Requirement 5: Display Linked Items in the UI
**User Story:** As a security analyst, I want to see a "Linked Items" section on ticket detail views, so that I can quickly navigate between related entities.
#### Acceptance Criteria
1. WHILE viewing an Archer ticket detail page, THE Link_UI SHALL display a "Linked Items" section listing all entities linked to that Archer ticket.
2. WHILE viewing a Jira ticket detail page, THE Link_UI SHALL display a "Linked Items" section listing all entities linked to that Jira ticket.
3. WHEN no links exist for the displayed entity, THE Link_UI SHALL show an empty state message indicating no linked items.
4. THE Link_UI SHALL display each linked item with its entity type, entity ID, and relationship label.
5. THE Link_UI SHALL render each linked item's entity ID as a navigable link to that entity's detail view.
### Requirement 6: Add a Link from the UI
**User Story:** As a security analyst, I want an "Add Link" button on ticket detail views, so that I can create associations without leaving the page.
#### Acceptance Criteria
1. WHEN the user clicks the "Add Link" button, THE Link_UI SHALL display a form allowing the user to enter a target entity key and select a relationship type.
2. WHEN the user submits the form with a valid entity key, THE Link_UI SHALL call the Link_Service to create the link and refresh the Linked Items section.
3. WHEN the Link_Service returns a validation or conflict error, THE Link_UI SHALL display the error message to the user without clearing the form.
4. THE Link_UI SHALL auto-detect the entity type from the entered key format (EXC-XXXX for Archer, PROJECT-XXXX for Jira, CVE-YYYY-NNNNN for CVE).
### Requirement 7: Remove a Link from the UI
**User Story:** As a security analyst, I want to remove a link directly from the Linked Items section, so that I can manage associations without navigating away.
#### Acceptance Criteria
1. THE Link_UI SHALL display a remove control on each linked item in the Linked Items section.
2. WHEN the user activates the remove control, THE Link_UI SHALL prompt for confirmation before proceeding.
3. WHEN the user confirms removal, THE Link_UI SHALL call the Link_Service to delete the link and refresh the Linked Items section.
4. WHEN the Link_Service returns an error during removal, THE Link_UI SHALL display the error message to the user.
### Requirement 8: Bidirectional Link Symmetry
**User Story:** As a security analyst, I want links to be visible from both sides of the association, so that navigating from either entity shows the connection.
#### Acceptance Criteria
1. WHEN a link is created from Entity A to Entity B, THE Link_Service SHALL return that link when queried from Entity A.
2. WHEN a link is created from Entity A to Entity B, THE Link_Service SHALL return that link when queried from Entity B.
3. THE Link_Service SHALL treat (source=A, target=B) and (source=B, target=A) as the same logical link — creating one SHALL prevent creating the other.
### Requirement 9: Links are Optional
**User Story:** As a security analyst, I want the linking feature to be purely additive, so that existing tickets without links continue to function normally.
#### Acceptance Criteria
1. THE Link_Service SHALL not require any entity to have links — entities with zero links remain fully functional.
2. THE Link_Service SHALL not modify the existing cve_id and vendor foreign key columns on archer_tickets or jira_tickets.
3. WHEN an entity has no links, THE Link_UI SHALL display the empty state without errors or warnings.

169
.kiro/specs/ticket-linking/tasks.md Normal file
View File

@@ -0,0 +1,169 @@
# Implementation Plan: Ticket Linking
## Overview
Add a generic bidirectional many-to-many linking system between Archer tickets, Jira tickets, and CVEs. Implementation spans three layers: a `ticket_links` PostgreSQL table with CHECK constraints and bidirectional indexes, a new Express router at `backend/routes/links.js` with POST/GET/DELETE endpoints behind `requireAuth()`, and a reusable `LinkedItems` React component mounted on Archer and Jira detail views. Entity type is auto-detected from key format. Audit logging on create/delete.
## Tasks
- [ ] 1. Database schema and migration
- [ ] 1.1 Create migration file `backend/migrations/add_ticket_links_table.js`
- Create `ticket_links` table with columns: id (SERIAL PK), source_type (TEXT NOT NULL), source_id (TEXT NOT NULL), target_type (TEXT NOT NULL), target_id (TEXT NOT NULL), relationship (TEXT NOT NULL DEFAULT 'related'), created_by (INTEGER REFERENCES users(id)), created_at (TIMESTAMPTZ DEFAULT NOW())
- Add CHECK constraints: source_type IN ('archer', 'jira', 'cve'), target_type IN ('archer', 'jira', 'cve'), relationship IN ('related', 'spawned_by', 'blocks')
- Add UNIQUE constraint on (source_type, source_id, target_type, target_id)
- Add indexes: idx_ticket_links_source ON (source_type, source_id), idx_ticket_links_target ON (target_type, target_id)
- Use `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS` for idempotence
- Use the PostgreSQL pool from `backend/db.js` (not SQLite)
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7_
- [ ] 1.2 Add `ticket_links` DDL to `backend/db-schema.sql`
- Append the CREATE TABLE and CREATE INDEX statements to the existing schema file for documentation and fresh-install setup
- _Requirements: 1.1, 1.2, 1.3, 1.7_
- [ ] 1.3 Run migration against development database
- Execute the migration script to create the table on the running Postgres instance
- Verify table exists with correct constraints via a quick SELECT
- _Requirements: 1.1_
- [ ] 2. Checkpoint — Verify database schema
- Ensure the `ticket_links` table exists with correct columns, constraints, and indexes. Ask the user if questions arise.
- [ ] 3. Backend API route
- [ ] 3.1 Create `backend/routes/links.js` with validation helpers
- Define ENTITY_PATTERNS regex map: archer `/^EXC-\d{4,}$/`, jira `/^[A-Z][A-Z0-9_]+-\d+$/`, cve `/^CVE-\d{4}-\d{4,}$/`
- Implement `detectEntityType(key)` function that returns 'cve', 'archer', 'jira', or null based on pattern matching
- Implement `validateEntityId(type, id)` that checks the ID matches the pattern for the given type
- Define VALID_RELATIONSHIPS array: ['related', 'spawned_by', 'blocks']
- _Requirements: 2.4, 2.5, 6.4_
- [ ]* 3.2 Write property test for entity type detection (Property 1)
- **Property 1: Entity Type Detection from Key Format**
- Generate random strings matching EXC-XXXX, PROJECT-XXXX, CVE-YYYY-NNNNN patterns and verify `detectEntityType` returns correct type; generate non-matching strings and verify null return
- **Validates: Requirements 6.4, 1.3, 2.4**
- [ ]* 3.3 Write property test for entity ID format validation (Property 2)
- **Property 2: Entity ID Format Validation**
- Generate (entity_type, entity_id) pairs with valid and invalid formats, verify validation accepts/rejects correctly
- **Validates: Requirements 2.5**
- [ ] 3.4 Implement POST /api/links endpoint
- Require authentication via `requireAuth()`
- Validate required fields: source_type, source_id, target_type, target_id
- Validate entity types are in ['archer', 'jira', 'cve']
- Validate entity ID formats match expected patterns for their types
- Reject self-links (same type AND same ID) with 400
- Check for duplicate links in both directions (A→B and B→A) with SELECT query, return 409 if exists
- Insert new row into ticket_links with created_by from req.user.id
- Log audit entry via `logAudit()` with action 'link_create'
- Return 201 with created link record
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 8.3_
- [ ]* 3.5 Write property test for self-link rejection (Property 3)
- **Property 3: Self-Link Rejection**
- Generate valid entities, attempt self-link creation, verify rejection with 400 status
- **Validates: Requirements 2.2**
- [ ]* 3.6 Write property test for bidirectional duplicate prevention (Property 4)
- **Property 4: Bidirectional Duplicate Prevention**
- Generate entity pairs, create link A→B, attempt A→B again and B→A, verify both return 409
- **Validates: Requirements 2.3, 8.3**
- [ ] 3.7 Implement GET /api/links endpoint
- Require authentication via `requireAuth()`
- Accept query params: `type` (entity type) and `id` (entity ID)
- Validate type and id are provided
- Execute bidirectional query: SELECT where (source_type=$1 AND source_id=$2) OR (target_type=$1 AND target_id=$2)
- Transform results to return `linked_type`, `linked_id` (the "other" side from the queried entity), relationship, created_by_username (JOIN with users table), created_at
- Return empty array when no links exist
- _Requirements: 3.1, 3.2, 3.3, 3.4, 8.1, 8.2_
- [ ]* 3.8 Write property test for bidirectional query completeness (Property 5)
- **Property 5: Bidirectional Query Completeness**
- Generate a set of links, query for a specific entity, verify result contains exactly the links where that entity appears on either side
- **Validates: Requirements 3.1, 8.1, 8.2**
- [ ]* 3.9 Write property test for link creation round-trip (Property 6)
- **Property 6: Link Creation Round-Trip**
- Generate valid link inputs (source ≠ target, no duplicate), create link, query for it, verify returned record matches input values
- **Validates: Requirements 2.1, 3.2**
- [ ] 3.10 Implement DELETE /api/links/:id endpoint
- Require authentication via `requireAuth()`
- Query the link by ID to verify it exists, return 404 if not found
- Delete the row from ticket_links
- Log audit entry via `logAudit()` with action 'link_delete', including source/target details
- Return 200 with success message
- _Requirements: 4.1, 4.2, 4.3, 4.4_
- [ ]* 3.11 Write property test for delete removes link from queries (Property 7)
- **Property 7: Delete Removes Link from Queries**
- Generate links, delete one by ID, verify querying from either side no longer includes the deleted link
- **Validates: Requirements 4.1**
- [ ] 3.12 Mount links router in `backend/server.js`
- Import `createLinksRouter` from `./routes/links`
- Add `app.use('/api/links', createLinksRouter())` alongside existing route mounts
- _Requirements: 2.6, 3.4, 4.3_
- [ ] 4. Checkpoint — Verify backend API
- Ensure POST/GET/DELETE endpoints work correctly, validation rejects bad input, duplicates return 409, bidirectional queries return links from both sides, and audit logs are created. Ask the user if questions arise.
- [ ] 5. Frontend LinkedItems component
- [ ] 5.1 Create `frontend/src/components/LinkedItems.js`
- Accept props: `entityType` ('archer' | 'jira' | 'cve') and `entityId` (string)
- Manage state: links array, loading boolean, showAddForm boolean, error string
- On mount, fetch links from `GET /api/links?type={entityType}&id={entityId}`
- Render "Linked Items" section header
- Display empty state message when no links exist
- Render each linked item with: entity type badge, entity ID as navigable link, relationship label
- Display remove button (×) on each linked item
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 7.1_
- [ ] 5.2 Implement "Add Link" form in LinkedItems component
- "Add Link" button toggles form visibility
- Form contains: text input for target entity key, dropdown for relationship type (related, spawned_by, blocks)
- Auto-detect entity type from entered key using same regex patterns as backend
- On submit, call POST /api/links with source (current entity) and target (entered key)
- On success, refresh the links list
- On error, display error message inline without clearing the form
- _Requirements: 6.1, 6.2, 6.3, 6.4_
- [ ] 5.3 Implement link removal in LinkedItems component
- Remove button triggers confirmation prompt
- On confirm, call DELETE /api/links/:id
- On success, refresh the links list
- On error, display error message to user
- _Requirements: 7.1, 7.2, 7.3, 7.4_
- [ ] 6. Integrate into existing views
- [ ] 6.1 Mount LinkedItems on Archer ticket detail view
- Import LinkedItems component
- Render with `entityType="archer"` and `entityId={excNumber}` (the EXC-XXXX identifier)
- Place in the detail panel below existing ticket information
- _Requirements: 5.1, 8.1, 8.2, 9.1_
- [ ] 6.2 Mount LinkedItems on Jira ticket detail/page view
- Import LinkedItems component
- Render with `entityType="jira"` and `entityId={ticketKey}` (the PROJECT-XXXX identifier)
- Place in the detail view below existing ticket information
- _Requirements: 5.2, 8.1, 8.2, 9.1_
- [ ] 7. Checkpoint — Verify end-to-end flow
- Ensure LinkedItems renders on both Archer and Jira detail views, links can be created/viewed/deleted from the UI, bidirectional visibility works (link created from Archer shows on Jira side), and entities with no links show empty state without errors. Ask the user if questions arise.
- [ ] 8. Final checkpoint — Ensure all tests pass
- Ensure all tests pass, audit log entries are created on link create/delete, and the feature is fully functional. Ask the user if questions arise.
## Notes
- Tasks marked with `*` are optional and can be skipped for faster MVP
- Each task references specific requirements for traceability
- Checkpoints ensure incremental validation between major phases
- Property tests validate universal correctness properties from the design document
- The design uses JavaScript — all code examples and implementations use Node.js/Express/React
- Entity IDs are text-based (EXC-6056, STEAM-1234, CVE-2025-0905) — not integer PKs
- The existing `cve_id` column on archer/jira tickets remains unchanged (backward compatible)
- Links are purely additive — they supplement the primary CVE relationship, not replace it
- Auto-detection of entity type from ID format reduces user friction
- Duplicate prevention checks both directions (A→B and B→A) since links are logically undirected
- The UNIQUE constraint on the table acts as a final safety net for concurrent duplicate inserts

1
.kiro/specs/vcl-compliance-reporting/.config.kiro Normal file
View File

@@ -0,0 +1 @@
{"specId": "a7c3e1d4-9f82-4b6a-a3d1-7e5f2c8b9a04", "workflowType": "requirements-first", "specType": "feature"}

505
.kiro/specs/vcl-compliance-reporting/design.md Normal file
View File

@@ -0,0 +1,505 @@
# Design Document: VCL Compliance Reporting
## Overview
This feature adds an executive-level VCL (Vulnerability Compliance Level) reporting page to the existing Compliance module, extends device records with remediation tracking fields (resolution date, remediation plan), and introduces a bulk upload mechanism for updating device metadata in batch. The VCL Report Page mirrors the layout of the leadership's existing spreadsheet deck — summary statistics bar, trend chart with forecast, non-compliant asset donut chart, heavy hitters table, and vertical breakdown table with burndown projections.
The implementation builds on the existing `compliance.js` route module, `compliance_items` table, and `CompliancePage.js` frontend component. New backend endpoints compute VCL statistics from existing data plus the new `resolution_date` and `remediation_plan` columns. The frontend adds a new `VCLReportPage.js` component accessible from the Compliance module navigation.
## Architecture
```mermaid
sequenceDiagram
participant U as User
participant FE as React Frontend
participant BE as Express Backend
participant DB as PostgreSQL
Note over FE,DB: Device Metadata Update (single device)
U->>FE: Edit resolution_date / remediation_plan in DetailPanel
FE->>BE: PATCH /api/compliance/items/:hostname/metadata
BE->>DB: UPDATE compliance_items SET resolution_date, remediation_plan WHERE hostname = $1
BE-->>FE: 200 OK { updated: count }
Note over FE,DB: VCL Report Page Load
FE->>BE: GET /api/compliance/vcl/stats
BE->>DB: Aggregate compliance_items (counts, percentages, categorization)
DB-->>BE: Raw counts
BE->>BE: Compute stats, categorization, heavy hitters, vertical breakdown
BE-->>FE: JSON { stats, donut, heavyHitters, verticalBreakdown }
FE->>BE: GET /api/compliance/vcl/trend
BE->>DB: Monthly aggregation from compliance_uploads + compliance_items history
DB-->>BE: Monthly data points
BE->>BE: Compute actuals + forecast
BE-->>FE: JSON { months: [...] }
Note over U,DB: Bulk Upload Flow
U->>FE: Select xlsx file in bulk upload control
FE->>FE: Parse xlsx with 'xlsx' library (client-side)
FE->>FE: Map columns, validate fields, match hostnames
FE->>BE: POST /api/compliance/vcl/bulk-preview { rows: [...] }
BE->>DB: Match hostnames against compliance_items
BE-->>FE: JSON { matched, unmatched, changes, invalid }
FE->>FE: Display Diff_Preview
U->>FE: Confirm changes
FE->>BE: POST /api/compliance/vcl/bulk-commit { changes: [...] }
BE->>DB: BEGIN; UPDATE compliance_items ...; COMMIT;
BE-->>FE: 200 OK { committed: count }
```
### Data Flow Summary
1. **Device metadata** — stored directly on `compliance_items` rows. Updated via PATCH endpoint (single) or bulk commit (batch).
2. **VCL statistics** — computed on-demand from current `compliance_items` state. No separate materialized table needed since the dataset is small (~1000 devices).
3. **Trend data** — derived from `compliance_uploads` history (existing) plus monthly snapshots of compliance percentages stored in a new `compliance_snapshots` table.
4. **Burndown projections** — computed from `resolution_date` values on active non-compliant items, bucketed by month.
## Components and Interfaces
### Backend
#### New Endpoints (added to `backend/routes/compliance.js`)
**`PATCH /api/compliance/items/:hostname/metadata`**
Updates resolution_date and/or remediation_plan for all active items matching a hostname.
- Auth: `requireAuth()`, `requireGroup('Admin', 'Standard_User')`
- Body: `{ resolution_date?: string|null, remediation_plan?: string|null }`
- Validation: resolution_date must be a valid ISO date or null; remediation_plan must be <= 2000 chars
- Response: `{ updated: number }`
**`GET /api/compliance/vcl/stats`**
Returns computed VCL executive summary statistics.
- Auth: `requireAuth()`
- Response:
```json
{
"stats": {
"total_devices": 1200,
"in_scope": 1100,
"compliant": 950,
"non_compliant": 150,
"remediations_required": 150,
"compliance_pct": 86,
"target_pct": 95
},
"donut": {
"blocked": { "count": 45, "pct": 30 },
"in_progress": { "count": 105, "pct": 70 }
},
"heavy_hitters": [
{ "vertical": "Network Ops", "team": "STEAM", "non_compliant": 42, "compliance_date": "2026-06-30", "notes": "..." }
],
"vertical_breakdown": [
{
"vertical": "Network Ops",
"compliance_pct": 82,
"team": "STEAM",
"non_compliant": 42,
"actual_burndown": { "2026-01": 5, "2026-02": 8 },
"forecast_burndown": { "2026-03": 10, "2026-04": 12 },
"blockers": 8,
"risk_acceptances": 3,
"notes": ""
}
]
}
```
**`GET /api/compliance/vcl/trend`**
Returns monthly compliance trend data for the overview chart.
- Auth: `requireAuth()`
- Query params: none
- Response:
```json
{
"months": [
{
"month": "2026-01",
"compliant_count": 900,
"compliance_pct": 82,
"forecast_pct": null,
"target_pct": 95
}
]
}
```
Forecast is computed using linear regression on the last 3+ months of actual data, projected forward.
**`POST /api/compliance/vcl/bulk-preview`**
Accepts parsed bulk upload rows and returns a diff preview.
- Auth: `requireAuth()`, `requireGroup('Admin', 'Standard_User')`
- Body: `{ rows: [{ hostname, resolution_date?, remediation_plan?, notes? }] }`
- Response:
```json
{
"matched": 850,
"unmatched": 12,
"changes": 200,
"invalid": 5,
"details": [
{
"hostname": "srv-001",
"status": "changed",
"fields": {
"resolution_date": { "old": null, "new": "2026-06-15" },
"remediation_plan": { "old": "", "new": "Patch in next window" }
}
}
],
"unmatched_rows": ["unknown-host-1"],
"invalid_rows": [{ "hostname": "srv-bad", "errors": ["resolution_date: invalid date format"] }]
}
```
**`POST /api/compliance/vcl/bulk-commit`**
Commits validated bulk changes in a single transaction.
- Auth: `requireAuth()`, `requireGroup('Admin', 'Standard_User')`
- Body: `{ changes: [{ hostname, resolution_date?, remediation_plan?, notes? }] }`
- Response: `{ committed: number }`
- Audit: logs `compliance_bulk_update` action
#### Pure Helper Functions (exported for testing)
```javascript
// Truncates text to maxLen chars with ellipsis
function truncateText(text, maxLen = 80) { ... }
// Validates remediation_plan length
function validateRemediationPlan(text) { ... }
// Validates a date string (ISO format)
function isValidDateString(str) { ... }
// Computes VCL summary stats from device rows
function computeVCLStats(items, targetPct) { ... }
// Categorizes non-compliant devices into blocked/in-progress
function categorizeNonCompliant(items) { ... }
// Ranks verticals by non-compliant count descending
function rankHeavyHitters(verticalData) { ... }
// Computes forecasted burndown from resolution_date values
function computeForecastBurndown(items) { ... }
// Matches uploaded rows to existing devices by hostname
function matchByHostname(uploadedRows, existingHostnames) { ... }
// Computes diff between uploaded values and current DB values
function computeBulkDiff(matchedRows, currentData) { ... }
// Maps column headers to known field names
function mapColumnHeaders(headers) { ... }
// Formats a decimal as a whole-number percentage string
function formatPct(decimal) { ... }
```
### Frontend
#### New Component: `VCLReportPage.js`
Located at `frontend/src/components/pages/VCLReportPage.js`. Accessible via a tab/button on the existing CompliancePage or as a separate nav entry.
**Sub-components:**
| Component | Purpose |
|-----------|---------|
| `VCLStatsBar` | Horizontal bar with 7 stat cards (Total, In-Scope, Compliant, Non-Compliant, Remediations, Current %, Target %) |
| `ComplianceOverviewChart` | Recharts ComposedChart — bars for compliant count, solid line for actual %, dashed line for forecast %, ReferenceLine for target |
| `NonCompliantDonutChart` | Recharts PieChart (donut) — Blocked vs In-Progress segments |
| `HeavyHittersTable` | Sorted table of top verticals by non-compliant count |
| `VerticalBreakdownTable` | Full breakdown table with burndown columns |
| `BulkUploadModal` | Modal with file picker, column mapping preview, diff display, confirm/cancel |
#### Modified Component: `ComplianceDetailPanel.js`
Add two new fields to the device detail panel:
- **Resolution Date** — `<input type="date">` with save on blur/enter
- **Remediation Plan** — `<textarea>` with character counter (max 2000) and save button
#### Modified Component: `CompliancePage.js`
- Add "VCL Report" tab/button in the page header that navigates to VCLReportPage
- Add `resolution_date` and `remediation_plan` columns to the device table
### Chart Specifications
#### Compliance Overview Chart (Recharts ComposedChart)
```javascript
<ComposedChart data={months}>
<CartesianGrid stroke="rgba(255,255,255,0.05)" strokeDasharray="3 3" />
<XAxis dataKey="month" tick={AXIS_STYLE} />
<YAxis yAxisId="count" tick={AXIS_STYLE} />
<YAxis yAxisId="pct" orientation="right" domain={[0, 100]} unit="%" tick={AXIS_STYLE} />
<Bar yAxisId="count" dataKey="compliant_count" fill="#10B981" fillOpacity={0.7} />
<Line yAxisId="pct" dataKey="compliance_pct" stroke={TEAL} strokeWidth={2} dot={{ r: 3 }} />
<Line yAxisId="pct" dataKey="forecast_pct" stroke={TEAL} strokeWidth={2} strokeDasharray="5 3" dot={false} />
<ReferenceLine yAxisId="pct" y={targetPct} stroke="#F59E0B" strokeDasharray="4 4" label="Target" />
</ComposedChart>
```
#### Non-Compliant Assets Donut (Recharts PieChart)
```javascript
<PieChart>
<Pie data={donutData} innerRadius={60} outerRadius={90} dataKey="count" nameKey="name">
<Cell fill="#EF4444" /> {/* Blocked */}
<Cell fill="#F59E0B" /> {/* In-Progress */}
</Pie>
<Legend />
</PieChart>
```
## Data Models
### Schema Changes to `compliance_items`
Two new columns:
```sql
ALTER TABLE compliance_items ADD COLUMN IF NOT EXISTS resolution_date DATE DEFAULT NULL;
ALTER TABLE compliance_items ADD COLUMN IF NOT EXISTS remediation_plan TEXT DEFAULT NULL;
```
- `resolution_date` — target date for remediation completion. NULL means no date set.
- `remediation_plan` — free-text description of the fix approach. NULL or empty means no plan documented. Max 2000 characters enforced at application layer.
### New Table: `compliance_snapshots`
Stores monthly compliance percentage snapshots for trend charting. One row per vertical per month.
```sql
CREATE TABLE IF NOT EXISTS compliance_snapshots (
id SERIAL PRIMARY KEY,
snapshot_month TEXT NOT NULL, -- 'YYYY-MM' format
vertical TEXT NOT NULL,
total_devices INTEGER NOT NULL DEFAULT 0,
compliant INTEGER NOT NULL DEFAULT 0,
non_compliant INTEGER NOT NULL DEFAULT 0,
compliance_pct NUMERIC(5,2) DEFAULT 0,
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(snapshot_month, vertical)
);
CREATE INDEX IF NOT EXISTS idx_compliance_snapshots_month
ON compliance_snapshots(snapshot_month);
```
Snapshots are created automatically when a new compliance upload is committed — the commit logic inserts/updates the snapshot for the current month.
### Migration Script: `backend/migrations/add_vcl_reporting_columns.js`
```javascript
const pool = require('../db');
async function run() {
console.log('Starting VCL reporting migration...');
try {
await pool.query(`ALTER TABLE compliance_items ADD COLUMN IF NOT EXISTS resolution_date DATE DEFAULT NULL`);
console.log('✓ resolution_date column added');
await pool.query(`ALTER TABLE compliance_items ADD COLUMN IF NOT EXISTS remediation_plan TEXT DEFAULT NULL`);
console.log('✓ remediation_plan column added');
await pool.query(`
CREATE TABLE IF NOT EXISTS compliance_snapshots (
id SERIAL PRIMARY KEY,
snapshot_month TEXT NOT NULL,
vertical TEXT NOT NULL,
total_devices INTEGER NOT NULL DEFAULT 0,
compliant INTEGER NOT NULL DEFAULT 0,
non_compliant INTEGER NOT NULL DEFAULT 0,
compliance_pct NUMERIC(5,2) DEFAULT 0,
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(snapshot_month, vertical)
)
`);
console.log('✓ compliance_snapshots table created');
await pool.query(`CREATE INDEX IF NOT EXISTS idx_compliance_snapshots_month ON compliance_snapshots(snapshot_month)`);
console.log('✓ compliance_snapshots index created');
} catch (err) {
console.error('Migration error:', err.message);
process.exit(1);
}
console.log('Migration complete.');
process.exit(0);
}
run();
```
## Correctness Properties
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
### Property 1: Device Metadata Persistence Round-Trip
*For any* valid resolution_date (ISO date string or null) and any valid remediation_plan (string of 0–2000 characters or null), saving the metadata via the update endpoint and then fetching the device should return the same resolution_date and remediation_plan values.
**Validates: Requirements 1.3, 2.3**
### Property 2: Text Truncation
*For any* string, `truncateText(text, 80)` should return the original string if its length is <= 80, or the first 80 characters followed by "…" if its length exceeds 80. The output length should never exceed 81 characters (80 + ellipsis).
**Validates: Requirements 2.4**
### Property 3: Remediation Plan Length Validation
*For any* string, `validateRemediationPlan(text)` should return valid if and only if the string length is <= 2000 characters. Strings exceeding 2000 characters should be flagged as invalid.
**Validates: Requirements 2.5, 9.4**
### Property 4: Summary Statistics Computation Invariants
*For any* set of compliance items with total, compliant, and non-compliant counts where total >= compliant >= 0 and non_compliant = total - compliant, `computeVCLStats(items, target)` should produce: non_compliant + compliant = total, compliance_pct = Math.round((compliant / total) * 100) when total > 0, and compliance_pct = 0 when total = 0.
**Validates: Requirements 3.2, 7.3**
### Property 5: Percentage Formatting
*For any* decimal number between 0 and 1 (inclusive), `formatPct(decimal)` should return `Math.round(decimal * 100) + '%'`. The output should always match the regex pattern `/^\d{1,3}%$/`.
**Validates: Requirements 3.3**
### Property 6: Non-Compliant Device Categorization Partition
*For any* array of non-compliant device objects, `categorizeNonCompliant(items)` should produce two groups (blocked, in_progress) where: every input item appears in exactly one group, blocked.count + in_progress.count = items.length, and each group's percentage equals Math.round((group.count / items.length) * 100) when items.length > 0.
**Validates: Requirements 5.2, 5.3**
### Property 7: Heavy Hitters Descending Sort
*For any* array of vertical objects with non_compliant counts, `rankHeavyHitters(verticals)` should return the array sorted in strictly non-increasing order by non_compliant count. For all consecutive pairs (a, b) in the output, a.non_compliant >= b.non_compliant.
**Validates: Requirements 6.1, 6.3**
### Property 8: Forecasted Burndown Projection
*For any* set of non-compliant devices with resolution_date values (some null, some valid future dates), `computeForecastBurndown(items)` should produce monthly buckets where: the sum of all monthly forecast counts equals the number of items with non-null resolution_dates, and each item with a resolution_date appears in exactly the bucket corresponding to its resolution month.
**Validates: Requirements 7.5**
### Property 9: Hostname Matching with Unmatched Flagging
*For any* array of uploaded rows (each with a hostname) and a set of existing hostnames, `matchByHostname(rows, existing)` should produce: matched rows (hostname exists in the set) + unmatched rows (hostname not in set) = total input rows. Every matched row's hostname must be in the existing set, and every unmatched row's hostname must not be in the existing set.
**Validates: Requirements 8.2, 8.7**
### Property 10: Bulk Diff Change Detection
*For any* array of matched row pairs (uploaded value, current DB value) for fields resolution_date and remediation_plan, `computeBulkDiff(matched, current)` should flag a row as "changed" if and only if at least one field value differs between uploaded and current. Rows where all fields are identical should be flagged as "unchanged".
**Validates: Requirements 8.3, 8.4**
### Property 11: Column Header Mapping
*For any* array of column header strings, `mapColumnHeaders(headers)` should: return a mapping that includes "hostname" if any header case-insensitively matches "Hostname", include "resolution_date" if any header matches "Resolution Date", include "remediation_plan" if any header matches "Remediation Plan", and include "notes" if any header matches "Notes". Headers not matching any known field should be ignored.
**Validates: Requirements 9.2**
### Property 12: Date String Validation
*For any* string, `isValidDateString(str)` should return true if and only if the string can be parsed into a valid Date object representing a real calendar date (e.g., "2026-02-30" is invalid). Null and empty string should return false.
**Validates: Requirements 9.3**
### Property 13: Row Count Arithmetic Invariant
*For any* bulk upload preview result with matched, unmatched, and invalid counts, the sum matched + unmatched must equal the total number of input rows. Additionally, within matched rows, changed + unchanged must equal matched count.
**Validates: Requirements 9.6**
## Error Handling
### Device Metadata Update Errors
| Condition | HTTP Status | Response | Behavior |
|-----------|-------------|----------|----------|
| Hostname not found | 404 | `{ "error": "Device not found" }` | No state change |
| Invalid date format | 400 | `{ "error": "Invalid resolution_date format" }` | No state change |
| Remediation plan > 2000 chars | 400 | `{ "error": "Remediation plan exceeds 2000 characters" }` | No state change |
| Database error | 500 | `{ "error": "Failed to update device metadata" }` | No state change |
### VCL Stats Endpoint Errors
| Condition | HTTP Status | Response | Behavior |
|-----------|-------------|----------|----------|
| No compliance data | 200 | `{ "stats": { all zeros }, ... }` | Return empty/zero stats gracefully |
| Database error | 500 | `{ "error": "Database error" }` | Log error |
### Bulk Upload Errors
| Condition | HTTP Status | Response | Behavior |
|-----------|-------------|----------|----------|
| No rows in file | 400 | `{ "error": "File contains no data rows" }` | No state change |
| No Hostname column | 400 | `{ "error": "File must contain a Hostname column" }` | No state change |
| No updatable columns | 400 | `{ "error": "No updatable fields found (need Resolution Date, Remediation Plan, or Notes)" }` | No state change |
| File exceeds 2000 rows | 400 | `{ "error": "File exceeds maximum of 2000 rows" }` | No state change |
| Transaction failure on commit | 500 | `{ "error": "Failed to commit changes" }` | Full rollback, no partial updates |
### Frontend Error Handling
- API failures display inline error messages (red text, monospace, consistent with existing patterns)
- Bulk upload validation errors are shown per-row in the diff preview with red highlighting
- Network errors show a retry prompt
- File parsing errors (corrupt xlsx) show a user-friendly message suggesting re-export from the source
## Testing Strategy
### Property-Based Testing
Use `fast-check` as the property-based testing library (already used in this project). Each correctness property maps to a single property-based test with a minimum of 100 iterations.
Property tests focus on the pure helper functions exported from the compliance route module:
- `truncateText` — Property 2
- `validateRemediationPlan` — Property 3
- `computeVCLStats` — Property 4
- `formatPct` — Property 5
- `categorizeNonCompliant` — Property 6
- `rankHeavyHitters` — Property 7
- `computeForecastBurndown` — Property 8
- `matchByHostname` — Property 9
- `computeBulkDiff` — Property 10
- `mapColumnHeaders` — Property 11
- `isValidDateString` — Property 12
Tag format: **Feature: vcl-compliance-reporting, Property {number}: {title}**
Test file: `backend/__tests__/vcl-compliance-reporting.property.test.js`
### Unit Testing
Unit tests cover specific examples, edge cases, and integration points:
- **PATCH metadata endpoint** — happy path, invalid date, plan too long, hostname not found
- **VCL stats with no data** — verify zero/empty response
- **Bulk preview with all unmatched** — verify correct counts
- **Bulk preview with mixed valid/invalid** — verify row classification
- **Bulk commit transactional** — verify all-or-nothing behavior
- **Donut chart with single category** — verify full donut rendering
- **Trend chart with < 2 months** — verify no forecast line
- **Vertical with zero non-compliant** — verify zero display
Test file: `backend/__tests__/vcl-compliance-reporting.test.js`
### Integration Testing
- Full bulk upload flow: parse → preview → commit → verify DB state
- Device metadata update → verify VCL stats reflect the change
- Snapshot creation on upload commit → verify trend data includes new month

135
.kiro/specs/vcl-compliance-reporting/requirements.md Normal file
View File

@@ -0,0 +1,135 @@
# Requirements Document
## Introduction
This feature adds executive-level VCL (Vulnerability Compliance Level) reporting capabilities to the STEAM Security Dashboard's Compliance module. Leadership requires a dedicated reporting page that mirrors the layout of their existing VCL spreadsheet deck used for executive presentations. The feature also extends device records with remediation tracking fields and introduces bulk upload support for updating approximately 1,000 device records at once with resolution dates, remediation plans, and notes.
## Glossary
- **Dashboard**: The STEAM Security Dashboard application
- **Compliance_Module**: The existing AEO Compliance section of the Dashboard, which handles weekly xlsx upload, diff preview, per-team metric health cards, and device-level violation tracking
- **VCL_Report_Page**: A new page within the Compliance_Module that displays executive-level compliance summary statistics, trend charts, and vertical breakdown tables
- **Device_Record**: A compliance_items row representing a non-compliant asset tracked in the system, identified by hostname and metric_id
- **Resolution_Date**: A date field on a Device_Record indicating the target date by which remediation is expected to be complete
- **Remediation_Plan**: A free-text field on a Device_Record describing the planned approach to bring the device into compliance
- **Vertical**: A logical grouping of devices by business function or organizational unit (e.g., a team or department responsible for a set of assets)
- **Compliance_Percentage**: The ratio of compliant devices to total in-scope devices, expressed as a percentage
- **Target_Compliance**: The organization-defined compliance percentage threshold that all verticals are expected to meet
- **Burndown**: A month-over-month projection of how many non-compliant devices will be remediated, shown as actual (historical) and forecasted (future) values
- **Heavy_Hitter**: A vertical with a disproportionately high count of non-compliant devices relative to other verticals
- **Bulk_Upload**: The process of uploading an xlsx file to update multiple Device_Records simultaneously with remediation metadata
- **Diff_Preview**: A summary view showing proposed changes before they are committed to the database, allowing the user to review and confirm or cancel
## Requirements
### Requirement 1: Resolution Date Field on Device Records
**User Story:** As a compliance analyst, I want to set a target resolution date on each non-compliant device, so that leadership can track remediation timelines.
#### Acceptance Criteria
1. THE Dashboard SHALL store a Resolution_Date field on each Device_Record in the compliance_items table
2. WHEN a user opens the ComplianceDetailPanel for a device, THE Dashboard SHALL display the Resolution_Date field as an editable date input
3. WHEN the user sets or changes the Resolution_Date value and saves, THE Dashboard SHALL persist the updated value to the database
4. THE Dashboard SHALL display the Resolution_Date as a column in the compliance device table view
5. IF the user clears the Resolution_Date field, THEN THE Dashboard SHALL store a null value and display the column cell as empty
### Requirement 2: Remediation Plan Field on Device Records
**User Story:** As a compliance analyst, I want to document a remediation plan for each non-compliant device, so that teams have a clear record of the intended fix.
#### Acceptance Criteria
1. THE Dashboard SHALL store a Remediation_Plan field on each Device_Record in the compliance_items table
2. WHEN a user opens the ComplianceDetailPanel for a device, THE Dashboard SHALL display the Remediation_Plan field as an editable text area
3. WHEN the user enters or modifies the Remediation_Plan text and saves, THE Dashboard SHALL persist the updated value to the database
4. THE Dashboard SHALL display the Remediation_Plan as a column in the compliance device table view, truncated to 80 characters with an ellipsis when the text exceeds that length
5. THE Dashboard SHALL limit the Remediation_Plan field to 2000 characters
### Requirement 3: VCL Executive Summary Statistics Bar
**User Story:** As an executive, I want to see high-level compliance statistics at a glance, so that I can quickly assess the organization's security posture.
#### Acceptance Criteria
1. THE VCL_Report_Page SHALL display a summary statistics bar containing: Total Devices, In-Scope count, Compliant count, Non-Compliant count, Remediations Required count, Current Compliance_Percentage, and Target_Compliance percentage
2. WHEN the VCL_Report_Page loads, THE Dashboard SHALL compute the summary statistics from the current compliance data in the database
3. THE Dashboard SHALL format Compliance_Percentage and Target_Compliance as whole-number percentages with a percent sign
4. WHEN the underlying compliance data changes due to a new upload, THE Dashboard SHALL reflect updated statistics upon page refresh
### Requirement 4: Compliance Overview Trend Chart
**User Story:** As an executive, I want to see a monthly compliance trend chart, so that I can understand whether compliance is improving over time.
#### Acceptance Criteria
1. THE VCL_Report_Page SHALL display a "Compliance Overview" chart showing monthly compliance data
2. THE Chart SHALL render compliant asset counts as vertical bars for each month
3. THE Chart SHALL render the actual Compliance_Percentage as a solid line overlaid on the bar chart
4. THE Chart SHALL render the forecasted Compliance_Percentage as a dashed line extending beyond the current month
5. THE Chart SHALL render the Target_Compliance threshold as a horizontal reference line
6. WHEN fewer than two months of historical data exist, THE Dashboard SHALL display the chart with available data points without forecasting
### Requirement 5: Non-Compliant Assets Status Chart
**User Story:** As an executive, I want to see the breakdown of non-compliant asset statuses, so that I can understand how many are blocked versus in progress.
#### Acceptance Criteria
1. THE VCL_Report_Page SHALL display a "Status of Non-Compliant Assets" donut chart
2. THE Chart SHALL segment non-compliant devices into "Blocked" and "In-Progress" categories
3. THE Chart SHALL display the count and percentage for each segment
4. WHEN all non-compliant devices belong to a single category, THE Chart SHALL render a full donut in that category's color with the count displayed
### Requirement 6: Heavy Hitters Table
**User Story:** As an executive, I want to see which verticals have the most non-compliant devices, so that I can focus leadership attention on the highest-impact areas.
#### Acceptance Criteria
1. THE VCL_Report_Page SHALL display a "Heavy Hitters" table showing verticals ranked by non-compliant device count in descending order
2. THE Table SHALL include columns: Vertical name with Responsible Team, Non-Compliant count, Compliance Date (target), and Notes
3. THE Dashboard SHALL derive the Heavy Hitters list from verticals with the highest non-compliant device counts
4. WHEN a vertical has no target Compliance Date set, THE Table SHALL display the cell as empty
### Requirement 7: Vertical Breakdown Table
**User Story:** As an executive, I want a detailed breakdown of compliance by vertical with burndown projections, so that I can track each team's remediation progress against their timeline.
#### Acceptance Criteria
1. THE VCL_Report_Page SHALL display a detailed vertical breakdown table
2. THE Table SHALL include columns: Vertical, Current Vertical Compliance_Percentage, Responsible Team, Non-Compliant Devices count, Actual Burndown (by month), Forecasted Burndown (by month from current month through Q4 2026), Count Blockers, Count RAs, and Notes
3. THE Dashboard SHALL compute Current Vertical Compliance_Percentage as the ratio of compliant to total in-scope devices within each vertical
4. THE Dashboard SHALL populate Actual Burndown columns with historical monthly remediation counts from upload history
5. THE Dashboard SHALL populate Forecasted Burndown columns with projected monthly remediation counts based on Resolution_Date values of non-compliant devices in each vertical
6. WHEN a vertical has no non-compliant devices, THE Table SHALL display zero values for Non-Compliant Devices, Blockers, and RAs columns
### Requirement 8: Bulk Upload for Device Remediation Metadata
**User Story:** As a compliance analyst, I want to upload an xlsx file to update resolution dates, remediation plans, and notes for many devices at once, so that I do not have to edit each device individually.
#### Acceptance Criteria
1. THE Dashboard SHALL provide a bulk upload control on the VCL_Report_Page that accepts an xlsx file
2. WHEN the user uploads an xlsx file, THE Dashboard SHALL parse the file and match rows to existing Device_Records by hostname
3. WHEN the file is parsed successfully, THE Dashboard SHALL display a Diff_Preview showing: total rows matched, rows with changes, and a summary of fields to be updated (Resolution_Date, Remediation_Plan, Notes)
4. THE Diff_Preview SHALL highlight rows where the uploaded value differs from the current database value
5. WHEN the user confirms the Diff_Preview, THE Dashboard SHALL commit all changes to the database in a single transaction
6. IF the user cancels the Diff_Preview, THEN THE Dashboard SHALL discard the parsed data without modifying any records
7. IF a row in the uploaded file references a hostname not found in the database, THEN THE Dashboard SHALL flag that row as unmatched in the Diff_Preview and exclude it from the commit
8. THE Dashboard SHALL support xlsx files containing up to 2000 rows without timeout or failure
### Requirement 9: Bulk Upload Field Mapping and Validation
**User Story:** As a compliance analyst, I want the bulk upload to validate my data before committing, so that I do not accidentally corrupt device records with malformed data.
#### Acceptance Criteria
1. THE Dashboard SHALL require the uploaded xlsx file to contain a "Hostname" column for row matching
2. THE Dashboard SHALL recognize and map the following optional columns: "Resolution Date", "Remediation Plan", "Notes"
3. WHEN a Resolution Date value cannot be parsed as a valid date, THE Dashboard SHALL flag that cell as invalid in the Diff_Preview and exclude the row from the commit
4. WHEN a Remediation Plan value exceeds 2000 characters, THE Dashboard SHALL flag that cell as invalid in the Diff_Preview and exclude the row from the commit
5. IF the uploaded file contains no recognizable columns beyond Hostname, THEN THE Dashboard SHALL display an error message indicating no updatable fields were found
6. THE Dashboard SHALL display the count of valid rows, invalid rows, and unmatched rows in the Diff_Preview summary

158
.kiro/specs/vcl-compliance-reporting/tasks.md Normal file
View File

@@ -0,0 +1,158 @@
# Tasks
## Task 1: Database Migration
- [x] 1.1 Create migration file `backend/migrations/add_vcl_reporting_columns.js` that adds `resolution_date DATE DEFAULT NULL` and `remediation_plan TEXT DEFAULT NULL` columns to the `compliance_items` table using `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`
- [x] 1.2 In the same migration, create the `compliance_snapshots` table with columns: id (SERIAL PK), snapshot_month (TEXT NOT NULL), vertical (TEXT NOT NULL), total_devices (INTEGER), compliant (INTEGER), non_compliant (INTEGER), compliance_pct (NUMERIC(5,2)), created_at (TIMESTAMPTZ), with a UNIQUE constraint on (snapshot_month, vertical)
- [x] 1.3 Add index `idx_compliance_snapshots_month` on `compliance_snapshots(snapshot_month)`
- [x] 1.4 Run the migration and verify columns and table exist in the database
## Task 2: Backend — Pure Helper Functions
- [x] 2.1 Implement and export `truncateText(text, maxLen)` — returns original if length <= maxLen, otherwise first maxLen chars + "…"
- [x] 2.2 Implement and export `validateRemediationPlan(text)` — returns `{ valid: true }` if length <= 2000, otherwise `{ valid: false, error: "..." }`
- [x] 2.3 Implement and export `isValidDateString(str)` — returns true only for strings parseable as real calendar dates (rejects "2026-02-30", null, empty)
- [x] 2.4 Implement and export `formatPct(decimal)` — returns `Math.round(decimal * 100) + '%'`
- [x] 2.5 Implement and export `computeVCLStats(items, targetPct)` — computes total, in_scope, compliant, non_compliant, remediations_required, compliance_pct from an array of device objects
- [x] 2.6 Implement and export `categorizeNonCompliant(items)` — partitions items into "blocked" (no resolution_date set) and "in_progress" (resolution_date set) with counts and percentages
- [x] 2.7 Implement and export `rankHeavyHitters(verticalData)` — sorts verticals by non_compliant count descending
- [x] 2.8 Implement and export `computeForecastBurndown(items)` — buckets items by resolution_date month, returns monthly forecast counts
- [x] 2.9 Implement and export `matchByHostname(uploadedRows, existingHostnames)` — returns { matched, unmatched } arrays
- [x] 2.10 Implement and export `computeBulkDiff(matchedRows, currentData)` — compares uploaded vs current values, returns change details
- [x] 2.11 Implement and export `mapColumnHeaders(headers)` — maps column header strings to known field names (case-insensitive matching)
## Task 3: Backend — Device Metadata Endpoint
- [x] 3.1 Add `PATCH /items/:hostname/metadata` endpoint to `backend/routes/compliance.js` with `requireAuth()` and `requireGroup('Admin', 'Standard_User')` middleware
- [x] 3.2 Implement request body validation: resolution_date must be a valid ISO date or null, remediation_plan must be <= 2000 chars or null
- [x] 3.3 Update all active `compliance_items` rows matching the hostname with the provided fields
- [x] 3.4 Return 404 if no rows match the hostname, 200 with `{ updated: count }` on success
- [x] 3.5 Log audit entry with action `compliance_metadata_update`
## Task 4: Backend — VCL Stats Endpoint
- [x] 4.1 Add `GET /vcl/stats` endpoint to `backend/routes/compliance.js` with `requireAuth()` middleware
- [x] 4.2 Query compliance_items to compute summary statistics (total devices, compliant, non-compliant counts) using `computeVCLStats`
- [x] 4.3 Compute donut chart data using `categorizeNonCompliant` on active non-compliant items
- [x] 4.4 Compute heavy hitters using `rankHeavyHitters` grouped by team/vertical
- [x] 4.5 Compute vertical breakdown with actual burndown (from upload history) and forecast burndown (from resolution_dates) using `computeForecastBurndown`
- [x] 4.6 Return the combined JSON response with stats, donut, heavy_hitters, and vertical_breakdown
## Task 5: Backend — VCL Trend Endpoint
- [x] 5.1 Add `GET /vcl/trend` endpoint to `backend/routes/compliance.js` with `requireAuth()` middleware
- [x] 5.2 Query `compliance_snapshots` table for monthly compliance data points
- [x] 5.3 Compute forecast line using linear regression on the last 3+ months of actual data
- [x] 5.4 Return monthly data array with compliant_count, compliance_pct, forecast_pct, and target_pct
## Task 6: Backend — Bulk Upload Endpoints
- [x] 6.1 Add `POST /vcl/bulk-preview` endpoint with `requireAuth()` and `requireGroup('Admin', 'Standard_User')` middleware
- [x] 6.2 Validate request body: require rows array, enforce 2000 row limit, require hostname field on each row
- [x] 6.3 Use `mapColumnHeaders` to identify updatable fields; return 400 if no updatable fields found
- [x] 6.4 Use `matchByHostname` to separate matched/unmatched rows
- [x] 6.5 Validate each row's fields using `isValidDateString` and `validateRemediationPlan`; flag invalid rows
- [x] 6.6 Use `computeBulkDiff` to identify changed rows and return the full diff preview response
- [x] 6.7 Add `POST /vcl/bulk-commit` endpoint with `requireAuth()` and `requireGroup('Admin', 'Standard_User')` middleware
- [x] 6.8 Accept validated changes array, execute all updates in a single PostgreSQL transaction
- [x] 6.9 Log audit entry with action `compliance_bulk_update` including count of rows updated
## Task 7: Backend — Snapshot Creation on Upload
- [x] 7.1 Modify the existing `persistUpload` function in `compliance.js` to insert/update a `compliance_snapshots` row for the current month after each upload commit
- [x] 7.2 Compute per-vertical compliance percentages at snapshot time from the current state of compliance_items
## Task 8: Frontend — Device Detail Panel Fields
- [x] 8.1 Add a "Resolution Date" section to `ComplianceDetailPanel.js` with an `<input type="date">` field
- [x] 8.2 Add a "Remediation Plan" section with a `<textarea>` field and character counter showing current/max (2000)
- [x] 8.3 Implement save logic that calls `PATCH /api/compliance/items/:hostname/metadata` on blur or explicit save button click
- [x] 8.4 Display loading state during save and error message on failure
- [x] 8.5 Fetch and display existing resolution_date and remediation_plan values when the panel opens (extend the existing GET /items/:hostname response)
## Task 9: Frontend — Device Table Columns
- [x] 9.1 Add "Resolution Date" column to the device table in `CompliancePage.js`, displaying the date or empty cell
- [x] 9.2 Add "Remediation Plan" column to the device table, truncated to 80 characters with ellipsis using `truncateText` logic
## Task 10: Frontend — VCL Report Page Shell
- [x] 10.1 Create `frontend/src/components/pages/VCLReportPage.js` with the page layout structure (header, stats bar area, charts area, tables area)
- [x] 10.2 Add navigation to VCLReportPage from CompliancePage (tab or button in the page header)
- [x] 10.3 Implement data fetching on mount: call `/api/compliance/vcl/stats` and `/api/compliance/vcl/trend`
- [x] 10.4 Add loading and error states consistent with existing page patterns
## Task 11: Frontend — VCL Stats Bar
- [x] 11.1 Implement `VCLStatsBar` component displaying 7 stat cards in a horizontal row: Total Devices, In-Scope, Compliant, Non-Compliant, Remediations Required, Current %, Target %
- [x] 11.2 Style stat cards with the dark tactical theme (monospace font, teal accents, gradient backgrounds)
- [x] 11.3 Format percentage values as whole numbers with percent sign
## Task 12: Frontend — Compliance Overview Trend Chart
- [x] 12.1 Implement `ComplianceOverviewChart` using Recharts ComposedChart with dual Y-axes (count left, percentage right)
- [x] 12.2 Render compliant asset counts as green bars on the count axis
- [x] 12.3 Render actual compliance percentage as a solid teal line on the percentage axis
- [x] 12.4 Render forecasted compliance percentage as a dashed teal line
- [x] 12.5 Render target compliance as a horizontal amber ReferenceLine
- [x] 12.6 Handle edge case: when < 2 months of data, show available points without forecast line
## Task 13: Frontend — Non-Compliant Assets Donut Chart
- [x] 13.1 Implement `NonCompliantDonutChart` using Recharts PieChart with innerRadius/outerRadius for donut shape
- [x] 13.2 Render "Blocked" segment in red (#EF4444) and "In-Progress" segment in amber (#F59E0B)
- [x] 13.3 Display count and percentage labels for each segment
- [x] 13.4 Handle edge case: single category renders full donut in that color
## Task 14: Frontend — Heavy Hitters Table
- [x] 14.1 Implement `HeavyHittersTable` component with columns: Vertical (with team), Non-Compliant count, Compliance Date, Notes
- [x] 14.2 Display rows sorted by non-compliant count descending (pre-sorted from API)
- [x] 14.3 Display empty cell when compliance date is null
- [x] 14.4 Style table with dark theme (border colors, monospace text, teal header accents)
## Task 15: Frontend — Vertical Breakdown Table
- [x] 15.1 Implement `VerticalBreakdownTable` component with all specified columns: Vertical, Compliance %, Team, Non-Compliant, Actual Burndown months, Forecast Burndown months, Blockers, RAs, Notes
- [x] 15.2 Render actual burndown columns with historical monthly counts
- [x] 15.3 Render forecast burndown columns from current month through Q4 2026
- [x] 15.4 Display zero values for verticals with no non-compliant devices
## Task 16: Frontend — Bulk Upload Modal
- [x] 16.1 Create `BulkUploadModal` component with file picker accepting .xlsx files
- [x] 16.2 Parse uploaded xlsx file client-side using the `xlsx` library to extract rows and headers
- [x] 16.3 Use column mapping logic to identify Hostname, Resolution Date, Remediation Plan, Notes columns
- [x] 16.4 Validate fields client-side (date format, plan length) before sending to backend
- [x] 16.5 Call `POST /api/compliance/vcl/bulk-preview` with parsed rows and display the diff preview
- [x] 16.6 Render diff preview showing: matched count, unmatched count, changes count, invalid count
- [x] 16.7 Highlight changed rows with field-level old/new value comparison
- [x] 16.8 Display unmatched hostnames and invalid rows with error details
- [x] 16.9 Implement confirm button that calls `POST /api/compliance/vcl/bulk-commit` and closes modal on success
- [x] 16.10 Implement cancel button that discards data and closes modal without changes
- [x] 16.11 Show error message if file has no Hostname column or no updatable fields
## Task 17: Property-Based Tests
- [x] 17.1 Create `backend/__tests__/vcl-compliance-reporting.property.test.js` with fast-check
- [x] 17.2 Implement Property 2 test: truncateText returns original for short strings, truncated + ellipsis for long strings (min 100 iterations)
- [x] 17.3 Implement Property 3 test: validateRemediationPlan accepts strings <= 2000 chars, rejects longer (min 100 iterations)
- [x] 17.4 Implement Property 4 test: computeVCLStats produces correct arithmetic relationships (non_compliant + compliant = total, correct percentage) (min 100 iterations)
- [x] 17.5 Implement Property 5 test: formatPct produces correct percentage string matching /^\d{1,3}%$/ (min 100 iterations)
- [x] 17.6 Implement Property 6 test: categorizeNonCompliant partitions all items into exactly two groups summing to total (min 100 iterations)
- [x] 17.7 Implement Property 7 test: rankHeavyHitters output is sorted in non-increasing order by non_compliant (min 100 iterations)
- [x] 17.8 Implement Property 8 test: computeForecastBurndown monthly bucket sum equals count of items with non-null resolution_dates (min 100 iterations)
- [x] 17.9 Implement Property 9 test: matchByHostname matched + unmatched = total input, matched hostnames all exist in set (min 100 iterations)
- [x] 17.10 Implement Property 10 test: computeBulkDiff flags row as changed iff at least one field differs (min 100 iterations)
- [x] 17.11 Implement Property 11 test: mapColumnHeaders correctly identifies known columns case-insensitively (min 100 iterations)
- [x] 17.12 Implement Property 12 test: isValidDateString rejects invalid calendar dates and non-date strings (min 100 iterations)
- [x] 17.13 Implement Property 13 test: bulk preview row counts (matched + unmatched = total) invariant holds (min 100 iterations)
## Task 18: Unit and Integration Tests
- [x] 18.1 Write unit tests for PATCH /items/:hostname/metadata endpoint (happy path, invalid date, plan too long, not found)
- [x] 18.2 Write unit tests for GET /vcl/stats with no data (verify zero/empty response)
- [x] 18.3 Write unit tests for bulk preview with all unmatched hostnames
- [x] 18.4 Write unit tests for bulk preview with mixed valid/invalid rows
- [x] 18.5 Write integration test for full bulk upload flow: preview → commit → verify DB state
- [x] 18.6 Write unit test for trend endpoint with < 2 months of data (no forecast)