Update .kiro: remove SQLite hooks, add PostgreSQL migration hook, add workflow steering, sync specs

This commit is contained in:
Jordan Ramos
2026-05-12 14:45:58 -06:00
parent 3ee8487286
commit 1bb8ec1658
35 changed files with 4645 additions and 48 deletions

View File

@@ -0,0 +1 @@
{"specId": "b93a94ad-abfd-4543-91bf-eb5a6cdd7896", "workflowType": "requirements-first", "specType": "feature"}

View File

@@ -0,0 +1,763 @@
# Design Document: PostgreSQL Migration
## Overview
Migrate the CVE Dashboard backend from SQLite (`cve_database.db`, 13MB) to PostgreSQL 16 running in a dedicated Docker container (`steam-postgres`) on port 5433. The primary architectural change is decomposing the monolithic `ivanti_findings_cache.findings_json` blob (2.6MB TEXT column) into individual rows in an `ivanti_findings` table. This eliminates JSON parsing on every request, enables indexed per-BU filtering, provides per-BU closed finding counts, and removes SQLite's single-writer lock that blocks reads during sync.
The Postgres instance is fully isolated from the existing Postgres on port 5432 (belonging to another project). The frontend requires zero changes — the API contract remains identical.
## Architecture
```mermaid
graph TB
subgraph "Frontend (port 3000)"
FE[React SPA]
end
subgraph "Backend (port 3001 prod / 3003 dev)"
SERVER[Express Server]
POOL[pg Pool - max 10 connections]
ROUTES[Route Handlers - async/await]
SYNC[Ivanti Sync Logic]
end
subgraph "Docker: steam-postgres (port 5433)"
PG[(PostgreSQL 16 Alpine)]
DB[cve_dashboard database]
VOL[steam-pgdata volume]
end
subgraph "Existing Infrastructure (DO NOT TOUCH)"
OTHER_PG[(Other Postgres - port 5432)]
SQLITE[(SQLite backup - cve_database.db)]
end
FE -->|Same API contract| SERVER
SERVER --> ROUTES
ROUTES --> POOL
SYNC --> POOL
POOL -->|DATABASE_URL| PG
PG --> DB
DB --> VOL
```
### Key Architecture Decisions
| Decision | Rationale |
|----------|-----------|
| Dedicated Docker container on port 5433 | Isolation from existing Postgres on 5432; independent lifecycle |
| `pg` package with connection pool (max 10) | Concurrent reads during writes; no single-writer lock |
| Individual finding rows instead of JSON blob | Indexed queries, per-BU filtering in SQL, no JSON.parse |
| Closed findings stored with `bu_ownership` | Enables per-BU closed counts (currently only global count) |
| Batch upsert via `INSERT ... ON CONFLICT` | Idempotent sync; no data loss on re-runs |
| Blue-green cutover on same port | <30s downtime; instant rollback by reverting .env |
## Components and Interfaces
### 1. Connection Pool Module (`backend/db.js`)
New module that creates and exports a `pg` Pool instance. All route files import this instead of receiving a `db` (sqlite3) parameter.
```js
// backend/db.js
const { Pool } = require('pg');
const pool = new Pool({
connectionString: process.env.DATABASE_URL,
// postgresql://steam:<password>@localhost:5433/cve_dashboard
max: 10,
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 5000,
});
// Log pool errors (connection drops, etc.)
pool.on('error', (err) => {
console.error('[DB Pool] Unexpected error on idle client:', err.message);
});
// Warn when approaching pool exhaustion
let activeCount = 0;
pool.on('acquire', () => {
activeCount++;
if (activeCount >= 8) {
console.warn(`[DB Pool] WARNING: ${activeCount}/10 connections active — approaching exhaustion`);
}
});
pool.on('release', () => { activeCount--; });
module.exports = pool;
```
### 2. Route Migration Pattern
Every route file changes from callback-based SQLite to async/await Postgres:
**Before (SQLite):**
```js
function createUsersRouter(db, requireAuth, requireGroup, logAudit) {
router.get('/', async (req, res) => {
try {
const users = await new Promise((resolve, reject) => {
db.all(
'SELECT id, username, email, user_group AS "group" FROM users ORDER BY created_at DESC',
(err, rows) => {
if (err) reject(err);
else resolve(rows);
}
);
});
res.json(users);
} catch (err) {
res.status(500).json({ error: 'Failed to fetch users' });
}
});
}
```
**After (Postgres):**
```js
const pool = require('../db');
function createUsersRouter(requireAuth, requireGroup, logAudit) {
router.get('/', async (req, res) => {
try {
const { rows: users } = await pool.query(
'SELECT id, username, email, user_group AS "group" FROM users ORDER BY created_at DESC'
);
res.json(users);
} catch (err) {
console.error('Get users error:', err);
res.status(500).json({ error: 'Failed to fetch users' });
}
});
}
```
### 3. Query Pattern Translation
| SQLite Pattern | Postgres Equivalent |
|----------------|-------------------|
| `db.get(sql, [params], callback)` | `const { rows } = await pool.query(sql, [params]); const row = rows[0];` |
| `db.all(sql, [params], callback)` | `const { rows } = await pool.query(sql, [params]);` |
| `db.run(sql, [params], callback)` | `await pool.query(sql, [params]);` or with `RETURNING` |
| `?` placeholders | `$1, $2, $3...` numbered params |
| `INSERT OR IGNORE` | `INSERT ... ON CONFLICT DO NOTHING` |
| `datetime('now')` | `NOW()` |
| `LIKE` (case-sensitive) | `ILIKE` (case-insensitive) |
### 4. Ivanti Sync Component (Rewritten)
The sync logic changes from "serialize all findings to JSON blob" to "upsert individual rows":
```js
// backend/routes/ivantiFindings.js — sync logic (simplified)
async function syncFindings(pool) {
const allFindings = await fetchAllFromIvanti(); // paginated API calls
// Batch upsert in chunks of 100
for (let i = 0; i < allFindings.length; i += 100) {
const batch = allFindings.slice(i, i + 100);
const values = [];
const placeholders = batch.map((f, idx) => {
const offset = idx * 14;
values.push(f.id, f.hostId, f.title, f.severity, f.vrrGroup,
f.hostName, f.ipAddress, f.dns, f.status, f.slaStatus,
f.dueDate, f.lastFoundOn, f.buOwnership, f.cves || []);
return `($${offset+1},$${offset+2},$${offset+3},$${offset+4},$${offset+5},
$${offset+6},$${offset+7},$${offset+8},$${offset+9},$${offset+10},
$${offset+11},$${offset+12},$${offset+13},$${offset+14},'open')`;
});
await pool.query(`
INSERT INTO ivanti_findings (id, host_id, title, severity, vrr_group,
host_name, ip_address, dns, status, sla_status,
due_date, last_found_on, bu_ownership, cves, state)
VALUES ${placeholders.join(',')}
ON CONFLICT (id) DO UPDATE SET
title = EXCLUDED.title,
severity = EXCLUDED.severity,
host_name = EXCLUDED.host_name,
ip_address = EXCLUDED.ip_address,
dns = EXCLUDED.dns,
status = EXCLUDED.status,
sla_status = EXCLUDED.sla_status,
due_date = EXCLUDED.due_date,
last_found_on = EXCLUDED.last_found_on,
bu_ownership = EXCLUDED.bu_ownership,
cves = EXCLUDED.cves,
state = EXCLUDED.state,
synced_at = NOW()
`, values);
}
// Update sync metadata
await pool.query(`
UPDATE ivanti_sync_state SET
total = (SELECT COUNT(*) FROM ivanti_findings WHERE state = 'open'),
synced_at = NOW(),
sync_status = 'success',
error_message = NULL
WHERE id = 1
`);
}
```
### 5. Auth Middleware Migration
```js
// backend/middleware/auth.js — After
const pool = require('../db');
function requireAuth() {
return async (req, res, next) => {
const sessionId = req.cookies?.session_id;
if (!sessionId) return res.status(401).json({ error: 'Authentication required' });
try {
const { rows } = await pool.query(
`SELECT s.*, u.id as user_id, u.username, u.email, u.role,
u.user_group, u.bu_teams, u.is_active
FROM sessions s
JOIN users u ON s.user_id = u.id
WHERE s.session_id = $1 AND s.expires_at > NOW()`,
[sessionId]
);
const session = rows[0];
if (!session) return res.status(401).json({ error: 'Session expired or invalid' });
if (!session.is_active) return res.status(401).json({ error: 'Account is disabled' });
req.user = {
id: session.user_id,
username: session.username,
email: session.email,
role: session.role,
group: session.user_group,
teams: session.bu_teams ? session.bu_teams.split(',').filter(Boolean) : []
};
next();
} catch (err) {
console.error('Auth middleware error:', err);
return res.status(500).json({ error: 'Authentication error' });
}
};
}
```
## Data Models
### Complete DDL for `ivanti_findings` Table
```sql
CREATE TABLE IF NOT EXISTS ivanti_findings (
id TEXT PRIMARY KEY, -- Ivanti finding ID (e.g. "HF-12345")
host_id INTEGER,
title TEXT NOT NULL DEFAULT '',
severity NUMERIC(4,2) NOT NULL DEFAULT 0,
vrr_group TEXT NOT NULL DEFAULT '',
host_name TEXT NOT NULL DEFAULT '',
ip_address TEXT NOT NULL DEFAULT '',
dns TEXT NOT NULL DEFAULT '',
status TEXT NOT NULL DEFAULT '',
sla_status TEXT NOT NULL DEFAULT '',
due_date DATE,
last_found_on DATE,
bu_ownership TEXT NOT NULL DEFAULT '',
cves TEXT[] DEFAULT '{}', -- Postgres array type
workflow_id TEXT,
workflow_state TEXT,
workflow_type TEXT,
state TEXT NOT NULL DEFAULT 'open' CHECK (state IN ('open', 'closed')),
note TEXT NOT NULL DEFAULT '', -- Merged from ivanti_finding_notes
override_host_name TEXT, -- Merged from ivanti_finding_overrides
override_dns TEXT, -- Merged from ivanti_finding_overrides
synced_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Performance indexes
CREATE INDEX IF NOT EXISTS idx_findings_state ON ivanti_findings(state);
CREATE INDEX IF NOT EXISTS idx_findings_bu ON ivanti_findings(bu_ownership);
CREATE INDEX IF NOT EXISTS idx_findings_severity ON ivanti_findings(severity);
CREATE INDEX IF NOT EXISTS idx_findings_state_bu ON ivanti_findings(state, bu_ownership);
```
### Core Tables (Postgres DDL)
```sql
-- Users
CREATE TABLE IF NOT EXISTS users (
id SERIAL PRIMARY KEY,
username VARCHAR(50) UNIQUE NOT NULL,
email VARCHAR(255) UNIQUE NOT NULL,
password_hash VARCHAR(255) NOT NULL,
role VARCHAR(20) NOT NULL DEFAULT 'viewer' CHECK (role IN ('admin', 'editor', 'viewer')),
is_active BOOLEAN DEFAULT TRUE,
created_at TIMESTAMPTZ DEFAULT NOW(),
last_login TIMESTAMPTZ,
user_group VARCHAR(20) NOT NULL DEFAULT 'Read_Only',
bu_teams TEXT NOT NULL DEFAULT ''
);
-- Sessions
CREATE TABLE IF NOT EXISTS sessions (
id SERIAL PRIMARY KEY,
session_id VARCHAR(255) UNIQUE NOT NULL,
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
expires_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_sessions_session_id ON sessions(session_id);
CREATE INDEX IF NOT EXISTS idx_sessions_user_id ON sessions(user_id);
CREATE INDEX IF NOT EXISTS idx_sessions_expires ON sessions(expires_at);
-- Audit Logs
CREATE TABLE IF NOT EXISTS audit_logs (
id SERIAL PRIMARY KEY,
user_id INTEGER,
username VARCHAR(50) NOT NULL,
action VARCHAR(50) NOT NULL,
entity_type VARCHAR(50) NOT NULL,
entity_id VARCHAR(100),
details TEXT,
ip_address VARCHAR(45),
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_audit_created_at ON audit_logs(created_at);
-- CVEs
CREATE TABLE IF NOT EXISTS cves (
id SERIAL PRIMARY KEY,
cve_id VARCHAR(20) NOT NULL,
vendor VARCHAR(100) NOT NULL,
severity VARCHAR(20) NOT NULL,
description TEXT,
published_date DATE,
status VARCHAR(50) DEFAULT 'Open',
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
created_by INTEGER,
UNIQUE(cve_id, vendor)
);
-- Jira Tickets
CREATE TABLE IF NOT EXISTS jira_tickets (
id SERIAL PRIMARY KEY,
cve_id TEXT NOT NULL,
vendor TEXT NOT NULL,
ticket_key TEXT NOT NULL,
url TEXT,
summary TEXT,
status TEXT DEFAULT 'Open' CHECK(status IN ('Open', 'In Progress', 'Closed')),
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
-- Ivanti Sync State (replaces ivanti_findings_cache metadata)
CREATE TABLE IF NOT EXISTS ivanti_sync_state (
id INTEGER PRIMARY KEY DEFAULT 1 CHECK (id = 1),
total INTEGER DEFAULT 0,
workflows_json TEXT DEFAULT '[]',
synced_at TIMESTAMPTZ,
sync_status TEXT DEFAULT 'never',
error_message TEXT
);
-- Ivanti Counts Cache (for FP workflow counts)
CREATE TABLE IF NOT EXISTS ivanti_counts_cache (
id INTEGER PRIMARY KEY DEFAULT 1 CHECK (id = 1),
open_count INTEGER DEFAULT 0,
closed_count INTEGER DEFAULT 0,
synced_at TIMESTAMPTZ,
fp_workflow_counts_json TEXT DEFAULT '{}',
fp_id_counts_json TEXT DEFAULT '{}'
);
-- Ivanti Counts History
CREATE TABLE IF NOT EXISTS ivanti_counts_history (
id SERIAL PRIMARY KEY,
open_count INTEGER NOT NULL,
closed_count INTEGER NOT NULL,
recorded_at TIMESTAMPTZ DEFAULT NOW()
);
-- Ivanti Finding Archives
CREATE TABLE IF NOT EXISTS ivanti_finding_archives (
id SERIAL PRIMARY KEY,
finding_id TEXT NOT NULL UNIQUE,
finding_title TEXT NOT NULL DEFAULT '',
host_name TEXT NOT NULL DEFAULT '',
ip_address TEXT NOT NULL DEFAULT '',
current_state TEXT NOT NULL CHECK(current_state IN ('ARCHIVED','RETURNED','CLOSED','CLOSED_GONE')),
last_severity NUMERIC(4,2) NOT NULL DEFAULT 0,
first_archived_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
last_transition_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Ivanti Archive Transitions
CREATE TABLE IF NOT EXISTS ivanti_archive_transitions (
id SERIAL PRIMARY KEY,
archive_id INTEGER NOT NULL REFERENCES ivanti_finding_archives(id),
from_state TEXT NOT NULL,
to_state TEXT NOT NULL,
severity_at_transition NUMERIC(4,2) NOT NULL DEFAULT 0,
reason TEXT NOT NULL DEFAULT '',
transitioned_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Ivanti Sync Anomaly Log
CREATE TABLE IF NOT EXISTS ivanti_sync_anomaly_log (
id SERIAL PRIMARY KEY,
sync_timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
open_count_delta INTEGER NOT NULL DEFAULT 0,
closed_count_delta INTEGER NOT NULL DEFAULT 0,
newly_archived_count INTEGER NOT NULL DEFAULT 0,
returned_count INTEGER NOT NULL DEFAULT 0,
classification_json TEXT NOT NULL DEFAULT '{}',
return_classification_json TEXT NOT NULL DEFAULT '{}',
is_significant BOOLEAN NOT NULL DEFAULT FALSE,
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Ivanti Finding BU History
CREATE TABLE IF NOT EXISTS ivanti_finding_bu_history (
id SERIAL PRIMARY KEY,
finding_id TEXT NOT NULL,
finding_title TEXT NOT NULL DEFAULT '',
host_name TEXT NOT NULL DEFAULT '',
previous_bu TEXT NOT NULL,
new_bu TEXT NOT NULL,
detected_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Ivanti FP Submissions
CREATE TABLE IF NOT EXISTS ivanti_fp_submissions (
id SERIAL PRIMARY KEY,
user_id INTEGER NOT NULL,
username TEXT NOT NULL,
ivanti_workflow_batch_id INTEGER,
ivanti_generated_id TEXT,
ivanti_workflow_batch_uuid TEXT,
workflow_name TEXT NOT NULL,
reason TEXT NOT NULL,
description TEXT,
expiration_date TEXT NOT NULL,
scope_override TEXT NOT NULL DEFAULT 'Authorized',
finding_ids_json TEXT NOT NULL,
queue_item_ids_json TEXT NOT NULL,
attachment_count INTEGER DEFAULT 0,
attachment_results_json TEXT,
status TEXT NOT NULL DEFAULT 'success' CHECK(status IN ('success', 'partial', 'failed')),
lifecycle_status TEXT NOT NULL DEFAULT 'submitted' CHECK(lifecycle_status IN ('submitted', 'approved', 'rejected', 'rework', 'resubmitted')),
error_message TEXT,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NULL
);
-- Ivanti FP Submission History
CREATE TABLE IF NOT EXISTS ivanti_fp_submission_history (
id SERIAL PRIMARY KEY,
submission_id INTEGER NOT NULL REFERENCES ivanti_fp_submissions(id) ON DELETE CASCADE,
user_id INTEGER NOT NULL,
username TEXT NOT NULL,
change_type TEXT NOT NULL CHECK(change_type IN ('created', 'fields_updated', 'findings_added', 'attachments_added', 'status_changed')),
change_details_json TEXT,
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Ivanti Todo Queue
CREATE TABLE IF NOT EXISTS ivanti_todo_queue (
id SERIAL PRIMARY KEY,
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
finding_id TEXT NOT NULL,
finding_title TEXT,
cves_json TEXT,
ip_address TEXT,
hostname TEXT,
vendor TEXT NOT NULL,
workflow_type TEXT NOT NULL CHECK(workflow_type IN ('FP', 'Archer', 'CARD', 'GRANITE')),
status TEXT NOT NULL DEFAULT 'pending' CHECK(status IN ('pending', 'complete')),
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_todo_queue_user ON ivanti_todo_queue(user_id, status);
-- Atlas Action Plans Cache
CREATE TABLE IF NOT EXISTS atlas_action_plans_cache (
id SERIAL PRIMARY KEY,
host_id INTEGER NOT NULL UNIQUE,
has_action_plan BOOLEAN NOT NULL DEFAULT FALSE,
plan_count INTEGER NOT NULL DEFAULT 0,
plans_json TEXT NOT NULL DEFAULT '[]',
synced_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Compliance Tables
CREATE TABLE IF NOT EXISTS compliance_uploads (
id SERIAL PRIMARY KEY,
filename TEXT NOT NULL,
report_date TEXT,
uploaded_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
uploaded_at TIMESTAMPTZ DEFAULT NOW(),
new_count INTEGER DEFAULT 0,
resolved_count INTEGER DEFAULT 0,
recurring_count INTEGER DEFAULT 0,
summary_json TEXT
);
CREATE TABLE IF NOT EXISTS compliance_items (
id SERIAL PRIMARY KEY,
upload_id INTEGER NOT NULL REFERENCES compliance_uploads(id) ON DELETE CASCADE,
hostname TEXT NOT NULL,
ip_address TEXT,
device_type TEXT,
team TEXT,
metric_id TEXT NOT NULL,
metric_desc TEXT,
category TEXT,
extra_json TEXT,
status TEXT NOT NULL DEFAULT 'active' CHECK(status IN ('active', 'resolved')),
first_seen_upload_id INTEGER REFERENCES compliance_uploads(id) ON DELETE SET NULL,
resolved_upload_id INTEGER REFERENCES compliance_uploads(id) ON DELETE SET NULL,
seen_count INTEGER DEFAULT 1,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE IF NOT EXISTS compliance_notes (
id SERIAL PRIMARY KEY,
hostname TEXT NOT NULL,
metric_id TEXT NOT NULL,
note TEXT NOT NULL,
group_id TEXT,
created_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Knowledge Base
CREATE TABLE IF NOT EXISTS knowledge_base (
id SERIAL PRIMARY KEY,
title VARCHAR(255) NOT NULL,
slug VARCHAR(255) UNIQUE NOT NULL,
description TEXT,
category VARCHAR(100),
file_path VARCHAR(500),
file_name VARCHAR(255),
file_type VARCHAR(50),
file_size INTEGER,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
created_by INTEGER REFERENCES users(id)
);
-- Archer Tickets
CREATE TABLE IF NOT EXISTS archer_tickets (
id SERIAL PRIMARY KEY,
exc_number TEXT NOT NULL UNIQUE,
archer_url TEXT,
status TEXT DEFAULT 'Draft' CHECK(status IN ('Draft', 'Open', 'Under Review', 'Accepted')),
cve_id TEXT NOT NULL,
vendor TEXT NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
-- Documents
CREATE TABLE IF NOT EXISTS documents (
id SERIAL PRIMARY KEY,
cve_id VARCHAR(20) NOT NULL,
vendor VARCHAR(100) NOT NULL,
name VARCHAR(255) NOT NULL,
type VARCHAR(50) NOT NULL,
file_path VARCHAR(500) NOT NULL,
file_size VARCHAR(20),
mime_type VARCHAR(100),
uploaded_at TIMESTAMPTZ DEFAULT NOW(),
notes TEXT
);
-- Required Documents (seed data)
CREATE TABLE IF NOT EXISTS required_documents (
id SERIAL PRIMARY KEY,
vendor VARCHAR(100) NOT NULL,
document_type VARCHAR(50) NOT NULL,
is_mandatory BOOLEAN DEFAULT TRUE,
description TEXT
);
```
### Per-BU Count Queries
```sql
-- Open count for specific BUs (used by counts endpoint with teams filter)
SELECT COUNT(*) FROM ivanti_findings
WHERE state = 'open' AND bu_ownership ILIKE ANY(ARRAY['%STEAM%', '%ACCESS-ENG%']);
-- Closed count for specific BUs
SELECT COUNT(*) FROM ivanti_findings
WHERE state = 'closed' AND bu_ownership ILIKE ANY(ARRAY['%STEAM%', '%ACCESS-ENG%']);
-- Aggregated counts grouped by BU and state (single query)
SELECT bu_ownership, state, COUNT(*) as count
FROM ivanti_findings
GROUP BY bu_ownership, state;
-- Global totals (no filter — backward compatible)
SELECT state, COUNT(*) as count
FROM ivanti_findings
GROUP BY state;
```
### Data Migration Script Design (`backend/scripts/migrate-to-postgres.js`)
```mermaid
flowchart TD
A[Open SQLite read-only] --> B[Connect to Postgres pool]
B --> C[Create all tables IF NOT EXISTS]
C --> D[Copy simple tables]
D --> E[Parse findings_json blob]
E --> F[Insert individual finding rows state=open]
F --> G[Merge ivanti_finding_notes → findings.note]
G --> H[Merge ivanti_finding_overrides → findings.override_*]
H --> I[Verify row counts]
I --> J[Print summary report]
```
The migration script:
1. Opens SQLite with `OPEN_READONLY` flag
2. Connects to Postgres via `DATABASE_URL`
3. Creates schema idempotently (`IF NOT EXISTS`)
4. Copies each table using batch inserts with `ON CONFLICT` for idempotency
5. Special handling for findings: parses `findings_json`, creates one row per finding
6. Merges notes and overrides into the corresponding finding rows
7. Verifies source vs destination row counts
8. Never modifies the SQLite file
## Correctness Properties
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
### Property 1: Upsert Idempotence
*For any* finding synced N times (N ≥ 1) with the same finding ID, the `ivanti_findings` table SHALL contain exactly one row for that finding ID, with the data from the most recent sync.
**Validates: Requirements 3.5, 6.6**
### Property 2: Finding Storage Preserves State and BU Ownership
*For any* finding (open or closed) stored in `ivanti_findings`, querying it back by ID SHALL return the same `state` and `bu_ownership` values it was stored with.
**Validates: Requirements 3.4, 4.1**
### Property 3: Count Query Accuracy
*For any* set of findings in `ivanti_findings` and any BU filter (including empty/no filter), the count query result SHALL equal the actual number of rows matching that filter and state combination.
**Validates: Requirements 4.2, 4.3, 4.5**
### Property 4: Migration Data Preservation (Findings)
*For any* finding in the source `findings_json` blob with associated notes (from `ivanti_finding_notes`) and overrides (from `ivanti_finding_overrides`), the migrated `ivanti_findings` row SHALL contain the finding data with `state = 'open'`, the correct `note` value, and the correct `override_host_name`/`override_dns` values.
**Validates: Requirements 7.4, 7.5, 7.6**
### Property 5: Migration Table Copy Preservation
*For any* table copied from SQLite to Postgres, the row count in Postgres SHALL equal the row count in SQLite, and each row's data SHALL be equivalent (accounting for type conversions: 0/1 → boolean, DATETIME → TIMESTAMPTZ).
**Validates: Requirements 7.7, 7.8**
### Property 6: Migration Idempotence
*For any* initial state of the SQLite and Postgres databases, running the migration script N times (N ≥ 1) SHALL produce the same final state in Postgres as running it exactly once (no duplicate rows, no errors).
**Validates: Requirements 7.3, 7.9**
### Property 7: Migration Source Safety
*For any* execution of the migration script, the SQLite database file SHALL remain byte-for-byte identical before and after (checksum unchanged).
**Validates: Requirements 7.10**
### Property 8: Schema Creation Idempotence
*For any* number of times the schema creation DDL is executed against the same database, the resulting schema SHALL be identical (no errors, same tables, same indexes, same constraints).
**Validates: Requirements 2.5**
### Property 9: API Response Shape Preservation
*For any* valid API request to any endpoint, the response JSON structure (top-level keys and value types) after migration SHALL be identical to the pre-migration response structure.
**Validates: Requirements 10.1**
## Error Handling
| Error Scenario | Handling Strategy |
|----------------|-------------------|
| Pool connection failure | `pool.on('error')` logs error; automatic reconnection on next query attempt |
| Pool exhaustion (all 10 busy) | Queries queue internally; warning logged at 8 active connections |
| Query timeout | `connectionTimeoutMillis: 5000` — rejects after 5s with error |
| Sync failure mid-batch | Transaction rollback; `sync_status = 'error'` with message; previous data preserved |
| Migration script failure | Idempotent design — safe to re-run; prints error and exits with code 1 |
| Docker container crash | `--restart unless-stopped` auto-recovers; pool reconnects on next query |
| Invalid finding data | `NOT NULL DEFAULT ''` columns prevent null constraint violations; CHECK constraints reject invalid state values |
| Rollback needed | Stop Postgres backend → revert `.env` → restart SQLite backend; SQLite file always preserved |
### Error Response Format (Unchanged)
All error responses maintain the existing format:
```json
{ "error": "Human-readable error message" }
```
With appropriate HTTP status codes: 400 (validation), 401 (auth), 403 (permission), 404 (not found), 500 (server error).
## Testing Strategy
### Unit Tests (Example-Based)
- Verify each route returns correct response shape with known test data
- Verify auth middleware rejects expired sessions
- Verify parameter placeholder conversion (`?``$1`) in all queries
- Verify schema DDL executes without errors
- Verify migration script handles empty tables gracefully
### Property-Based Tests
Property-based testing is appropriate for this feature because the core operations (upsert, count queries, data migration) have clear input/output behavior with universal properties that hold across a wide input space.
**Library**: [fast-check](https://github.com/dubzzz/fast-check) (JavaScript PBT library)
**Configuration**: Minimum 100 iterations per property test.
**Tag format**: `Feature: postgres-migration, Property {number}: {property_text}`
Each correctness property (1-9) maps to a single property-based test:
- Property 1: Generate random findings, upsert each N times, verify exactly one row per ID
- Property 2: Generate findings with random state/bu_ownership, store and retrieve, verify equality
- Property 3: Generate random finding sets, insert, run count queries with random filters, verify accuracy
- Property 4: Generate random findings JSON with notes/overrides, run migration logic, verify merged output
- Property 5: Generate random table rows, copy via migration, verify count and data equivalence
- Property 6: Run migration logic N times on same input, verify final state equals single-run state
- Property 7: Checksum SQLite before/after migration, verify unchanged
- Property 8: Run schema DDL N times, verify no errors and same schema
- Property 9: Compare response shapes between SQLite and Postgres backends for same requests
### Integration Tests
- Docker container health check (port 5433 accessible)
- Full sync cycle: trigger sync → verify rows created → verify counts endpoint
- Concurrent read during write: start sync, simultaneously query findings, verify no blocking
- Performance: GET /findings < 500ms, GET /counts < 100ms with 6000+ rows
- Cutover simulation: stop/start backend, verify API responds correctly
### Development Isolation
- Test backend runs on port 3003 with `DATABASE_URL` pointing to Postgres
- Production backend continues on port 3001 with SQLite (no `DATABASE_URL` set)
- Switching is controlled by presence of `DATABASE_URL` environment variable
- All work on `feature/multi-tenancy` branch

View File

@@ -0,0 +1,158 @@
# Requirements Document
## Introduction
Migrate the CVE Dashboard (STEAM Security Dashboard) backend from SQLite to PostgreSQL 16. The current SQLite architecture stores all Ivanti findings as a single 2.6MB JSON blob (`ivanti_findings_cache.findings_json`) that must be parsed on every API request, causing 5-10 second load times. Additionally, SQLite's single-writer lock blocks reads during sync writes, and per-BU closed finding counts are unavailable (only a global count exists). PostgreSQL enables individual finding rows with indexed columns, per-BU open and closed counts, connection pooling for concurrent access, and proper type support. The Postgres instance runs in a dedicated Docker container on port 5433, isolated from the existing Postgres on port 5432 which belongs to another project.
## Glossary
- **SQLite**: The current embedded database engine storing all data in `backend/cve_database.db` (13MB total)
- **PostgreSQL_16**: The target relational database running in a Docker container (`steam-postgres`) on port 5433
- **findings_json**: The current TEXT column in `ivanti_findings_cache` storing all Ivanti findings as a serialized JSON array (2.6MB+)
- **ivanti_findings**: The new Postgres table storing each finding as an individual row with indexed columns
- **Pool**: A `pg` (node-postgres) connection pool managing up to 10 concurrent database connections
- **DATABASE_URL**: Environment variable containing the Postgres connection string (`postgresql://steam:<password>@localhost:5433/cve_dashboard`)
- **Migration_Script**: A one-time Node.js script (`backend/scripts/migrate-to-postgres.js`) that reads from SQLite and writes to Postgres
- **Cutover**: The moment production switches from SQLite backend on port 3001 to Postgres backend on port 3001
- **steam-postgres**: The Docker container name for the CVE Dashboard's dedicated PostgreSQL instance
- **steam-pgdata**: The Docker volume providing persistent storage for the Postgres data directory
- **bu_ownership**: The Ivanti finding field containing the BU assignment (e.g. "NTS-AEO-STEAM")
- **Upsert**: An INSERT that updates the existing row on primary key conflict (`INSERT ... ON CONFLICT DO UPDATE`)
## Requirements
### Requirement 1: Dedicated PostgreSQL Docker Container
**User Story:** As a system operator, I want a dedicated Postgres 16 container for the CVE Dashboard on port 5433, so that it is fully isolated from the existing Postgres instance on port 5432 belonging to another project.
#### Acceptance Criteria
1. THE Infrastructure SHALL run PostgreSQL 16 (Alpine) in a Docker container named `steam-postgres`
2. THE container SHALL map host port 5433 to container port 5432
3. THE container SHALL use a Docker volume named `steam-pgdata` for persistent data storage
4. THE container SHALL be configured with `--restart unless-stopped` for automatic recovery after host reboots
5. THE container SHALL create a database named `cve_dashboard` with a user named `steam`
6. THE Infrastructure SHALL NOT modify or affect the existing Postgres instance on port 5432
### Requirement 2: Schema Parity with Proper Types
**User Story:** As a developer, I want all existing SQLite tables recreated in Postgres with proper types and constraints, so that no functionality is lost during migration.
#### Acceptance Criteria
1. THE Postgres schema SHALL include all tables from the current SQLite schema: users, sessions, audit_logs, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, compliance_uploads, compliance_items, compliance_notes, ivanti_sync_state, ivanti_finding_notes, ivanti_counts_cache, ivanti_counts_history, ivanti_finding_overrides, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue, atlas_action_plans_cache
2. THE Postgres schema SHALL use appropriate types: SERIAL for auto-increment, TIMESTAMPTZ for timestamps, BOOLEAN for booleans, NUMERIC for decimals, TEXT[] for arrays, DATE for date-only fields
3. THE Postgres schema SHALL preserve all existing CHECK constraints and foreign key relationships
4. THE Postgres schema SHALL include the `bu_teams` column on the users table (required by multi-BU tenancy feature)
5. THE schema creation SHALL be idempotent using `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS`
### Requirement 3: Findings Table Redesign
**User Story:** As a user, I want findings stored as individual rows with indexed columns, so that filtering by BU ownership and state is instant instead of requiring full JSON blob parsing.
#### Acceptance Criteria
1. THE system SHALL replace the `ivanti_findings_cache.findings_json` blob with an `ivanti_findings` table containing one row per finding
2. THE `ivanti_findings` table SHALL include columns: id (TEXT PRIMARY KEY, Ivanti finding ID), host_id (INTEGER), title (TEXT), severity (NUMERIC), vrr_group (TEXT), host_name (TEXT), ip_address (TEXT), dns (TEXT), status (TEXT), sla_status (TEXT), due_date (DATE), last_found_on (DATE), bu_ownership (TEXT), cves (TEXT[] array), workflow_id (TEXT), workflow_state (TEXT), workflow_type (TEXT), state (TEXT with CHECK constraint for 'open' or 'closed'), note (TEXT), override_host_name (TEXT), override_dns (TEXT), synced_at (TIMESTAMPTZ), created_at (TIMESTAMPTZ)
3. THE system SHALL create indexes on: state, bu_ownership, severity, and a composite index on (state, bu_ownership)
4. THE system SHALL store both open AND closed findings as individual rows with their respective state values
5. WHEN findings are synced from Ivanti, THE system SHALL upsert rows using `INSERT ... ON CONFLICT (id) DO UPDATE` rather than replacing a JSON blob
### Requirement 4: Per-BU Closed Finding Counts
**User Story:** As a user viewing the Reporting page with a BU scope filter, I want accurate open and closed counts for my selected BUs, so that the dashboard shows meaningful per-team metrics instead of only a global count.
#### Acceptance Criteria
1. THE system SHALL store closed findings with their `bu_ownership` field preserved as individual rows with `state = 'closed'`
2. THE counts endpoint SHALL derive per-BU counts using `SELECT COUNT(*) ... WHERE state = ? AND bu_ownership ILIKE ?` queries
3. WHEN a teams filter is provided, THE counts endpoint SHALL return open and closed counts scoped to those BUs
4. WHEN no filter is applied, THE counts endpoint SHALL return global totals (backward compatible with current behavior)
5. THE system SHALL support aggregated counts grouped by bu_ownership and state in a single query
### Requirement 5: Connection Pooling
**User Story:** As a system operator, I want the backend to use connection pooling via the pg package, so that multiple concurrent requests are handled efficiently without blocking and reads are never blocked by writes.
#### Acceptance Criteria
1. THE backend SHALL use the `pg` npm package with a Pool instance (maximum pool size: 10 connections)
2. THE Pool SHALL read the connection string from the `DATABASE_URL` environment variable
3. IF the Pool detects connection errors, THEN THE Pool SHALL attempt automatic reconnection
4. THE Pool SHALL log a warning when active connections reach 8 (approaching exhaustion)
5. ALL database operations SHALL use async/await with the Pool (replacing all callback-based SQLite patterns)
### Requirement 6: Backend Route Migration
**User Story:** As a developer, I want all SQLite-specific code replaced with Postgres equivalents using async/await, so that the codebase uses a single database driver consistently.
#### Acceptance Criteria
1. THE backend SHALL replace all `db.get(sql, params, callback)` calls with `pool.query(sql, params)` returning `rows[0]`
2. THE backend SHALL replace all `db.all(sql, params, callback)` calls with `pool.query(sql, params)` returning `rows`
3. THE backend SHALL replace all `db.run(sql, params, callback)` calls with `pool.query(sql, params)` using `RETURNING` clauses where the inserted/updated row is needed
4. THE backend SHALL replace `?` parameter placeholders with `$1, $2, $3...` Postgres numbered parameter syntax
5. THE backend SHALL remove all callback-based database patterns in favor of async/await with try/catch error handling
6. THE Ivanti sync logic SHALL write individual finding rows via upsert instead of serializing to a JSON blob
### Requirement 7: Data Migration Script
**User Story:** As a system operator, I want a one-time migration script that copies all data from SQLite to Postgres, so that no data is lost during the transition.
#### Acceptance Criteria
1. THE Migration_Script SHALL open the SQLite database in read-only mode
2. THE Migration_Script SHALL connect to Postgres using the DATABASE_URL connection string
3. THE Migration_Script SHALL create all tables idempotently before inserting data
4. THE Migration_Script SHALL parse the `findings_json` blob and insert individual finding rows into the `ivanti_findings` table with `state = 'open'`
5. THE Migration_Script SHALL merge `ivanti_finding_notes` into the corresponding `ivanti_findings.note` column
6. THE Migration_Script SHALL merge `ivanti_finding_overrides` into the corresponding `ivanti_findings.override_*` columns
7. THE Migration_Script SHALL copy all other tables preserving their data: users, sessions, cves, documents, jira_tickets, archer_tickets, knowledge_base, audit_logs, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, atlas_action_plans_cache, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue
8. THE Migration_Script SHALL verify row counts after migration and print a summary with any discrepancies
9. THE Migration_Script SHALL be idempotent (safe to run multiple times using ON CONFLICT/upsert logic)
10. THE Migration_Script SHALL never modify the SQLite database file
### Requirement 8: Zero-Impact Cutover
**User Story:** As a system operator, I want the cutover from SQLite to Postgres to take under 30 seconds with a clear rollback path, so that downtime is minimal and reversible.
#### Acceptance Criteria
1. THE cutover procedure SHALL consist of: stop old backend, start new backend on the same port (3001)
2. THE frontend SHALL NOT require any changes for the cutover (same API contract, same port, same URL)
3. IF issues are detected after cutover, THEN THE system SHALL support rollback by reverting the .env configuration and restarting the SQLite-based backend
4. THE SQLite database file SHALL be preserved indefinitely as a backup after cutover
5. THE cutover downtime SHALL not exceed 30 seconds
### Requirement 9: Performance Improvement
**User Story:** As a user, I want the dashboard to load findings in under 1 second regardless of BU scope, so that switching between BU views feels instant compared to the current 5-10 second load times.
#### Acceptance Criteria
1. THE `GET /api/ivanti/findings` endpoint SHALL respond in under 500ms for any BU filter combination
2. THE `GET /api/ivanti/findings/counts` endpoint SHALL respond in under 100ms for any BU filter
3. THE Ivanti sync process SHALL complete without blocking concurrent read requests (Postgres MVCC)
4. THE system SHALL eliminate JSON blob parsing from the findings read path entirely
### Requirement 10: API Backward Compatibility
**User Story:** As a developer, I want the API contract to remain unchanged after migration, so that the frontend works identically without code changes.
#### Acceptance Criteria
1. ALL existing API endpoints SHALL return the same response shape after migration
2. THE `GET /api/ivanti/findings` response SHALL include: findings array, total count, synced_at timestamp, sync_status, and error_message fields
3. THE authentication and session system SHALL work identically (cookie-based sessions, same expiration behavior)
4. THE frontend SHALL require zero code changes specifically for the database migration (multi-BU filtering changes are a separate feature)
### Requirement 11: Development and Testing Isolation
**User Story:** As a developer, I want to test the Postgres backend on port 3003 while production continues on port 3001 with SQLite, so that development does not disrupt the live system.
#### Acceptance Criteria
1. WHILE the migration is in development, THE test backend SHALL run on port 3003
2. WHILE the migration is in development, THE production backend SHALL continue running on port 3001 with SQLite
3. THE system SHALL support switching between SQLite and Postgres via environment variable configuration (DATABASE_URL presence or DB_TYPE flag)
4. ALL development work SHALL occur on the existing `feature/multi-tenancy` branch

View File

@@ -0,0 +1,294 @@
# Implementation Plan: PostgreSQL Migration
## Overview
Migrate the CVE Dashboard backend from SQLite to PostgreSQL 16. Replace the monolithic `findings_json` blob (2.6MB) with individual indexed rows in `ivanti_findings`, enable per-BU closed counts, and eliminate the single-writer lock. All work on the `feature/multi-tenancy` branch. Docker container `steam-postgres` on port 5433, test backend on port 3003, production on port 3001.
## Tasks
- [x] 1. Infrastructure setup and connection pool
- [x] 1.1 Install `pg` dependency and configure environment
- Run `npm install pg` in `backend/`
- Add `DATABASE_URL=postgresql://steam:<password>@localhost:5433/cve_dashboard` to `backend/.env`
- Add `DATABASE_URL` placeholder to `backend/.env.example`
- _Requirements: 1.5, 5.1, 5.2_
- [x] 1.2 Create `backend/db.js` connection pool module
- Import `pg` and create a `Pool` instance reading from `DATABASE_URL`
- Set `max: 10`, `idleTimeoutMillis: 30000`, `connectionTimeoutMillis: 5000`
- Add `pool.on('error')` handler logging unexpected errors
- Track active connections; log warning when count reaches 8
- Export the pool instance for use by all route files
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5_
- [x] 1.3 Create Docker run command documentation
- Document the `docker run` command for `steam-postgres` container (port 5433:5432, volume `steam-pgdata`, `--restart unless-stopped`, Postgres 16 Alpine)
- Verify container creates `cve_dashboard` database with `steam` user
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6_
- [x] 2. Schema creation
- [x] 2.1 Create `backend/db-schema.sql` with complete Postgres DDL
- Define all tables with proper Postgres types: SERIAL, TIMESTAMPTZ, BOOLEAN, NUMERIC, TEXT[], DATE
- Include `ivanti_findings` table with TEXT PRIMARY KEY (`id`), all columns per design, CHECK constraint on `state`
- Include `ivanti_sync_state` table (single-row pattern, replaces `ivanti_findings_cache` metadata)
- Include all other tables: users (with `bu_teams`), sessions, audit_logs, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_cache, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue, atlas_action_plans_cache
- Include all indexes: findings (state, bu_ownership, severity, state+bu_ownership composite), sessions (session_id, user_id, expires_at), audit_logs (created_at), todo_queue (user_id+status)
- Include all foreign key relationships and CHECK constraints
- Use `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS` for idempotence
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 3.1, 3.2, 3.3_
- [ ]* 2.2 Write property test for schema creation idempotence
- **Property 8: Schema Creation Idempotence**
- Run schema DDL N times against the same database, verify no errors and same resulting schema each time
- **Validates: Requirements 2.5**
- [x] 2.3 Create schema initialization in `backend/setup.js`
- Read `db-schema.sql` and execute via pool.query
- Make callable on server startup or as standalone script
- Seed `ivanti_sync_state` row (id=1) and `ivanti_counts_cache` row (id=1) if not exists
- _Requirements: 2.1, 2.5_
- [x] 3. Checkpoint — Verify infrastructure
- Ensure Docker container is running on port 5433, pool connects successfully, schema creates without errors. Ask the user if questions arise.
- [-] 4. Migrate auth and session system
- [x] 4.1 Update `backend/middleware/auth.js`
- Replace `db.get()` callback with `pool.query()` async/await
- Change `?` placeholders to `$1, $2, ...` numbered params
- Join sessions and users tables, check `expires_at > NOW()`
- Return 401 for missing/expired sessions, 500 for query errors
- _Requirements: 6.1, 6.4, 6.5, 10.3_
- [x] 4.2 Update `backend/routes/auth.js`
- Replace all `db.get/db.all/db.run` with `pool.query`
- Login: query user by username, create session with `RETURNING`
- Logout: delete session by session_id
- Password change: update password_hash
- Profile/me endpoint: query user by session
- Use `$1, $2...` placeholders throughout
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 10.3_
- [x] 4.3 Update `backend/routes/users.js`
- Replace all sqlite3 calls with pool.query
- CRUD operations: list users, create user, update user, delete user
- Use `RETURNING` clause for inserts/updates where row data is needed
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
- [x] 4.4 Update `backend/server.js` database initialization
- Remove sqlite3 database opening and `db` object creation
- Import pool from `backend/db.js`
- Remove passing `db` parameter to route factory functions
- Update inline CVE/document/vendor routes to use pool.query
- _Requirements: 6.1, 6.4, 6.5_
- [ ] 5. Checkpoint — Verify auth and core routes
- Ensure login, logout, session validation, user CRUD, and CVE routes work on port 3003. Ask the user if questions arise.
- [-] 6. Migrate remaining route files
- [x] 6.1 Update `backend/routes/jiraTickets.js`
- Replace sqlite3 calls with pool.query, update placeholders
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.2 Update `backend/routes/archerTickets.js`
- Replace sqlite3 calls with pool.query, update placeholders
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.3 Update `backend/routes/knowledgeBase.js`
- Replace sqlite3 calls with pool.query, update placeholders
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.4 Update `backend/routes/compliance.js`
- Replace sqlite3 calls with pool.query, update placeholders
- Handle compliance_items, compliance_uploads, compliance_notes queries
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
- [x] 6.5 Update `backend/routes/auditLog.js` and `backend/helpers/auditLog.js`
- Replace sqlite3 db.run/db.all with pool.query
- Update the audit logging helper to use async pool.query
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.6 Update `backend/routes/ivantiWorkflows.js`
- Replace sqlite3 calls with pool.query, update placeholders
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.7 Update `backend/routes/ivantiFpWorkflow.js`
- Replace sqlite3 calls with pool.query, update placeholders
- Handle fp_submissions and fp_submission_history tables
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.8 Update `backend/routes/ivantiTodoQueue.js`
- Replace sqlite3 calls with pool.query, update placeholders
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.9 Update `backend/routes/ivantiArchive.js`
- Replace sqlite3 calls with pool.query, update placeholders
- Handle archive and transition table queries
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.10 Update `backend/routes/atlas.js`
- Replace sqlite3 calls with pool.query, update placeholders
- _Requirements: 6.1, 6.4, 6.5_
- [x] 6.11 Update `backend/routes/cardApi.js` and `backend/routes/feedback.js`
- Replace any sqlite3 calls with pool.query if present
- _Requirements: 6.1, 6.4, 6.5_
- [ ] 7. Checkpoint — Verify all route migrations
- Ensure all non-findings routes work correctly on port 3003. Ask the user if questions arise.
- [-] 8. Rewrite Ivanti findings sync and read logic
- [x] 8.1 Rewrite sync logic in `backend/routes/ivantiFindings.js`
- Replace JSON blob serialization with batch upsert of individual rows
- Use `INSERT INTO ivanti_findings (...) VALUES ... ON CONFLICT (id) DO UPDATE SET ...` in batches of 100
- Sync both open and closed findings as individual rows with correct `state` value
- Update `ivanti_sync_state` with total count, synced_at, sync_status after sync
- Preserve existing note and override values during upsert (do not overwrite user-set fields)
- _Requirements: 3.4, 3.5, 6.6, 9.3_
- [ ]* 8.2 Write property test for upsert idempotence
- **Property 1: Upsert Idempotence**
- Generate random findings, upsert each N times with same ID, verify exactly one row per ID with most recent data
- **Validates: Requirements 3.5, 6.6**
- [x] 8.3 Rewrite read endpoints in `backend/routes/ivantiFindings.js`
- GET /findings: `SELECT * FROM ivanti_findings WHERE state = 'open'` with optional BU filter via `bu_ownership ILIKE`
- Return response shape: `{ findings, total, synced_at, sync_status, error_message }`
- GET /findings/counts: derive open/closed counts from `ivanti_findings` with optional BU filter
- Support `teams` query parameter for per-BU scoped counts using `ILIKE ANY(ARRAY[...])`
- GET /findings/counts/history: unchanged (reads from `ivanti_counts_history`)
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 9.1, 9.2, 10.1, 10.2_
- [ ]* 8.4 Write property test for finding state and BU preservation
- **Property 2: Finding Storage Preserves State and BU Ownership**
- Generate findings with random state/bu_ownership, store and retrieve by ID, verify equality
- **Validates: Requirements 3.4, 4.1**
- [ ]* 8.5 Write property test for count query accuracy
- **Property 3: Count Query Accuracy**
- Generate random finding sets, insert into DB, run count queries with random BU filters, verify counts match actual row counts
- **Validates: Requirements 4.2, 4.3, 4.5**
- [x] 8.6 Update note and override endpoints
- PUT /findings/:id/note: `UPDATE ivanti_findings SET note = $1 WHERE id = $2`
- PUT /findings/:id/override: `UPDATE ivanti_findings SET override_host_name = $1, override_dns = $2 WHERE id = $3`
- _Requirements: 6.1, 6.4_
- [ ] 9. Checkpoint — Verify findings redesign
- Ensure sync creates individual rows, read endpoints return correct shape, counts work with and without BU filter. Ask the user if questions arise.
- [-] 10. Data migration script
- [x] 10.1 Create `backend/scripts/migrate-to-postgres.js`
- Open SQLite database in read-only mode (`OPEN_READONLY`)
- Connect to Postgres via `DATABASE_URL`
- Run schema creation (idempotent) before inserting data
- Copy all simple tables: users, sessions, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, audit_logs, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, atlas_action_plans_cache, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue
- Handle type conversions: SQLite 0/1 → Postgres boolean, DATETIME strings → TIMESTAMPTZ
- Use batch inserts with `ON CONFLICT` for idempotency (safe to re-run)
- _Requirements: 7.1, 7.2, 7.3, 7.7, 7.9, 7.10_
- [x] 10.2 Implement findings migration with note/override merging
- Parse `ivanti_findings_cache.findings_json` blob into individual objects
- Insert each finding as a row in `ivanti_findings` with `state = 'open'`
- Query `ivanti_finding_notes` and merge each note into the corresponding `ivanti_findings.note` column
- Query `ivanti_finding_overrides` and merge into `ivanti_findings.override_host_name` / `override_dns`
- _Requirements: 7.4, 7.5, 7.6_
- [x] 10.3 Add verification and summary reporting
- After migration, query row counts for every table in both SQLite and Postgres
- Print comparison table showing source count vs destination count
- Flag any discrepancies with warnings
- Exit with code 0 on success, code 1 on any errors
- _Requirements: 7.8, 7.9_
- [ ]* 10.4 Write property test for migration data preservation (findings)
- **Property 4: Migration Data Preservation (Findings)**
- Generate random findings JSON with associated notes and overrides, run migration logic, verify merged output matches expected values
- **Validates: Requirements 7.4, 7.5, 7.6**
- [ ]* 10.5 Write property test for migration table copy preservation
- **Property 5: Migration Table Copy Preservation**
- Generate random table rows, copy via migration logic, verify row counts and data equivalence (accounting for type conversions)
- **Validates: Requirements 7.7, 7.8**
- [ ]* 10.6 Write property test for migration idempotence
- **Property 6: Migration Idempotence**
- Run migration logic N times on same input data, verify final Postgres state equals single-run state (no duplicates)
- **Validates: Requirements 7.3, 7.9**
- [ ]* 10.7 Write property test for migration source safety
- **Property 7: Migration Source Safety**
- Checksum SQLite file before and after migration script execution, verify bytes unchanged
- **Validates: Requirements 7.10**
- [ ] 11. Checkpoint — Verify data migration
- Run migration script against test Postgres instance, verify all row counts match, verify findings have merged notes/overrides. Ask the user if questions arise.
- [ ] 12. Frontend updates for per-BU closed counts
- [ ] 12.1 Update ReportingPage donut chart to show per-BU closed counts
- Remove "N/A" fallback for closed count when BU filter is active
- Display actual closed count from the updated `/api/ivanti/findings/counts` endpoint
- Pass `teams` parameter to counts endpoint for server-side BU filtering
- _Requirements: 4.3, 4.4_
- [ ]* 12.2 Write property test for API response shape preservation
- **Property 9: API Response Shape Preservation**
- For various valid API requests, verify response JSON structure (top-level keys and value types) matches expected contract
- **Validates: Requirements 10.1**
- [ ] 13. Testing and verification
- [ ] 13.1 Run backend on port 3003 and verify all endpoints
- Set `PORT=3003` in test environment with `DATABASE_URL` pointing to Postgres
- Verify auth endpoints (login, logout, me, password change)
- Verify findings endpoints (list, counts, sync, notes, overrides)
- Verify all other routes (CVEs, Jira, Archer, compliance, knowledge base, audit log)
- Compare response shapes against production SQLite backend on port 3001
- _Requirements: 9.1, 9.2, 10.1, 10.2, 10.3, 10.4, 11.1, 11.2, 11.3_
- [ ] 13.2 Performance verification
- Confirm GET /api/ivanti/findings responds in <500ms with full dataset
- Confirm GET /api/ivanti/findings/counts responds in <100ms
- Confirm sync does not block concurrent read requests
- _Requirements: 9.1, 9.2, 9.3, 9.4_
- [ ] 13.3 Run existing test suite against Postgres backend
- Verify all existing property tests and unit tests pass
- _Requirements: 10.1, 10.3_
- [ ] 14. Cutover to production
- [ ] 14.1 Execute cutover procedure
- Run final Ivanti sync on SQLite production backend
- Run migration script to copy latest data to Postgres
- Stop production backend on port 3001
- Update production `.env` with `DATABASE_URL`
- Start new backend on port 3001 with Postgres
- Verify frontend loads and API responds correctly
- _Requirements: 8.1, 8.2, 8.3_
- [ ] 14.2 Preserve SQLite backup and document rollback
- Keep `backend/cve_database.db` as permanent backup (do not delete)
- Document rollback procedure: stop backend → remove DATABASE_URL from .env → restart
- _Requirements: 8.3, 8.4_
- [ ] 15. Cleanup
- [ ] 15.1 Remove SQLite dependency and legacy code
- Run `npm uninstall sqlite3` from `backend/`
- Remove any remaining sqlite3 imports or `db` parameter passing
- Remove the old `ivanti_findings_cache` and `ivanti_finding_notes` / `ivanti_finding_overrides` table references
- Update `backend/.env.example` and README with Postgres prerequisites
- _Requirements: 6.5, 11.4_
- [ ] 16. Final checkpoint — Ensure all tests pass
- Ensure all tests pass, production is stable on Postgres, and SQLite backup is preserved. Ask the user if questions arise.
## Notes
- Tasks marked with `*` are optional and can be skipped for faster MVP
- Each task references specific requirements for traceability
- Checkpoints ensure incremental validation between major phases
- Property tests validate universal correctness properties from the design document
- All work on `feature/multi-tenancy` branch
- Docker container `steam-postgres` on port 5433 (NOT 5432 — that belongs to another project)
- Test backend on port 3003 during development; production stays on port 3001 with SQLite until cutover
- The `pg` package connection pool handles concurrent reads during sync writes (MVCC)
- Batch upserts in chunks of 100 for sync performance
- SQLite file is never modified — opened read-only during migration