Update .kiro: remove SQLite hooks, add PostgreSQL migration hook, add workflow steering, sync specs
This commit is contained in:
1
.kiro/specs/postgres-migration/.config.kiro
Normal file
1
.kiro/specs/postgres-migration/.config.kiro
Normal file
@@ -0,0 +1 @@
|
||||
{"specId": "b93a94ad-abfd-4543-91bf-eb5a6cdd7896", "workflowType": "requirements-first", "specType": "feature"}
|
||||
763
.kiro/specs/postgres-migration/design.md
Normal file
763
.kiro/specs/postgres-migration/design.md
Normal file
@@ -0,0 +1,763 @@
|
||||
# Design Document: PostgreSQL Migration
|
||||
|
||||
## Overview
|
||||
|
||||
Migrate the CVE Dashboard backend from SQLite (`cve_database.db`, 13MB) to PostgreSQL 16 running in a dedicated Docker container (`steam-postgres`) on port 5433. The primary architectural change is decomposing the monolithic `ivanti_findings_cache.findings_json` blob (2.6MB TEXT column) into individual rows in an `ivanti_findings` table. This eliminates JSON parsing on every request, enables indexed per-BU filtering, provides per-BU closed finding counts, and removes SQLite's single-writer lock that blocks reads during sync.
|
||||
|
||||
The Postgres instance is fully isolated from the existing Postgres on port 5432 (belonging to another project). The frontend requires zero changes — the API contract remains identical.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Frontend (port 3000)"
|
||||
FE[React SPA]
|
||||
end
|
||||
|
||||
subgraph "Backend (port 3001 prod / 3003 dev)"
|
||||
SERVER[Express Server]
|
||||
POOL[pg Pool - max 10 connections]
|
||||
ROUTES[Route Handlers - async/await]
|
||||
SYNC[Ivanti Sync Logic]
|
||||
end
|
||||
|
||||
subgraph "Docker: steam-postgres (port 5433)"
|
||||
PG[(PostgreSQL 16 Alpine)]
|
||||
DB[cve_dashboard database]
|
||||
VOL[steam-pgdata volume]
|
||||
end
|
||||
|
||||
subgraph "Existing Infrastructure (DO NOT TOUCH)"
|
||||
OTHER_PG[(Other Postgres - port 5432)]
|
||||
SQLITE[(SQLite backup - cve_database.db)]
|
||||
end
|
||||
|
||||
FE -->|Same API contract| SERVER
|
||||
SERVER --> ROUTES
|
||||
ROUTES --> POOL
|
||||
SYNC --> POOL
|
||||
POOL -->|DATABASE_URL| PG
|
||||
PG --> DB
|
||||
DB --> VOL
|
||||
```
|
||||
|
||||
### Key Architecture Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Dedicated Docker container on port 5433 | Isolation from existing Postgres on 5432; independent lifecycle |
|
||||
| `pg` package with connection pool (max 10) | Concurrent reads during writes; no single-writer lock |
|
||||
| Individual finding rows instead of JSON blob | Indexed queries, per-BU filtering in SQL, no JSON.parse |
|
||||
| Closed findings stored with `bu_ownership` | Enables per-BU closed counts (currently only global count) |
|
||||
| Batch upsert via `INSERT ... ON CONFLICT` | Idempotent sync; no data loss on re-runs |
|
||||
| Blue-green cutover on same port | <30s downtime; instant rollback by reverting .env |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
### 1. Connection Pool Module (`backend/db.js`)
|
||||
|
||||
New module that creates and exports a `pg` Pool instance. All route files import this instead of receiving a `db` (sqlite3) parameter.
|
||||
|
||||
```js
|
||||
// backend/db.js
|
||||
const { Pool } = require('pg');
|
||||
|
||||
const pool = new Pool({
|
||||
connectionString: process.env.DATABASE_URL,
|
||||
// postgresql://steam:<password>@localhost:5433/cve_dashboard
|
||||
max: 10,
|
||||
idleTimeoutMillis: 30000,
|
||||
connectionTimeoutMillis: 5000,
|
||||
});
|
||||
|
||||
// Log pool errors (connection drops, etc.)
|
||||
pool.on('error', (err) => {
|
||||
console.error('[DB Pool] Unexpected error on idle client:', err.message);
|
||||
});
|
||||
|
||||
// Warn when approaching pool exhaustion
|
||||
let activeCount = 0;
|
||||
pool.on('acquire', () => {
|
||||
activeCount++;
|
||||
if (activeCount >= 8) {
|
||||
console.warn(`[DB Pool] WARNING: ${activeCount}/10 connections active — approaching exhaustion`);
|
||||
}
|
||||
});
|
||||
pool.on('release', () => { activeCount--; });
|
||||
|
||||
module.exports = pool;
|
||||
```
|
||||
|
||||
### 2. Route Migration Pattern
|
||||
|
||||
Every route file changes from callback-based SQLite to async/await Postgres:
|
||||
|
||||
**Before (SQLite):**
|
||||
```js
|
||||
function createUsersRouter(db, requireAuth, requireGroup, logAudit) {
|
||||
router.get('/', async (req, res) => {
|
||||
try {
|
||||
const users = await new Promise((resolve, reject) => {
|
||||
db.all(
|
||||
'SELECT id, username, email, user_group AS "group" FROM users ORDER BY created_at DESC',
|
||||
(err, rows) => {
|
||||
if (err) reject(err);
|
||||
else resolve(rows);
|
||||
}
|
||||
);
|
||||
});
|
||||
res.json(users);
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: 'Failed to fetch users' });
|
||||
}
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
**After (Postgres):**
|
||||
```js
|
||||
const pool = require('../db');
|
||||
|
||||
function createUsersRouter(requireAuth, requireGroup, logAudit) {
|
||||
router.get('/', async (req, res) => {
|
||||
try {
|
||||
const { rows: users } = await pool.query(
|
||||
'SELECT id, username, email, user_group AS "group" FROM users ORDER BY created_at DESC'
|
||||
);
|
||||
res.json(users);
|
||||
} catch (err) {
|
||||
console.error('Get users error:', err);
|
||||
res.status(500).json({ error: 'Failed to fetch users' });
|
||||
}
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Query Pattern Translation
|
||||
|
||||
| SQLite Pattern | Postgres Equivalent |
|
||||
|----------------|-------------------|
|
||||
| `db.get(sql, [params], callback)` | `const { rows } = await pool.query(sql, [params]); const row = rows[0];` |
|
||||
| `db.all(sql, [params], callback)` | `const { rows } = await pool.query(sql, [params]);` |
|
||||
| `db.run(sql, [params], callback)` | `await pool.query(sql, [params]);` or with `RETURNING` |
|
||||
| `?` placeholders | `$1, $2, $3...` numbered params |
|
||||
| `INSERT OR IGNORE` | `INSERT ... ON CONFLICT DO NOTHING` |
|
||||
| `datetime('now')` | `NOW()` |
|
||||
| `LIKE` (case-sensitive) | `ILIKE` (case-insensitive) |
|
||||
|
||||
### 4. Ivanti Sync Component (Rewritten)
|
||||
|
||||
The sync logic changes from "serialize all findings to JSON blob" to "upsert individual rows":
|
||||
|
||||
```js
|
||||
// backend/routes/ivantiFindings.js — sync logic (simplified)
|
||||
async function syncFindings(pool) {
|
||||
const allFindings = await fetchAllFromIvanti(); // paginated API calls
|
||||
|
||||
// Batch upsert in chunks of 100
|
||||
for (let i = 0; i < allFindings.length; i += 100) {
|
||||
const batch = allFindings.slice(i, i + 100);
|
||||
const values = [];
|
||||
const placeholders = batch.map((f, idx) => {
|
||||
const offset = idx * 14;
|
||||
values.push(f.id, f.hostId, f.title, f.severity, f.vrrGroup,
|
||||
f.hostName, f.ipAddress, f.dns, f.status, f.slaStatus,
|
||||
f.dueDate, f.lastFoundOn, f.buOwnership, f.cves || []);
|
||||
return `($${offset+1},$${offset+2},$${offset+3},$${offset+4},$${offset+5},
|
||||
$${offset+6},$${offset+7},$${offset+8},$${offset+9},$${offset+10},
|
||||
$${offset+11},$${offset+12},$${offset+13},$${offset+14},'open')`;
|
||||
});
|
||||
|
||||
await pool.query(`
|
||||
INSERT INTO ivanti_findings (id, host_id, title, severity, vrr_group,
|
||||
host_name, ip_address, dns, status, sla_status,
|
||||
due_date, last_found_on, bu_ownership, cves, state)
|
||||
VALUES ${placeholders.join(',')}
|
||||
ON CONFLICT (id) DO UPDATE SET
|
||||
title = EXCLUDED.title,
|
||||
severity = EXCLUDED.severity,
|
||||
host_name = EXCLUDED.host_name,
|
||||
ip_address = EXCLUDED.ip_address,
|
||||
dns = EXCLUDED.dns,
|
||||
status = EXCLUDED.status,
|
||||
sla_status = EXCLUDED.sla_status,
|
||||
due_date = EXCLUDED.due_date,
|
||||
last_found_on = EXCLUDED.last_found_on,
|
||||
bu_ownership = EXCLUDED.bu_ownership,
|
||||
cves = EXCLUDED.cves,
|
||||
state = EXCLUDED.state,
|
||||
synced_at = NOW()
|
||||
`, values);
|
||||
}
|
||||
|
||||
// Update sync metadata
|
||||
await pool.query(`
|
||||
UPDATE ivanti_sync_state SET
|
||||
total = (SELECT COUNT(*) FROM ivanti_findings WHERE state = 'open'),
|
||||
synced_at = NOW(),
|
||||
sync_status = 'success',
|
||||
error_message = NULL
|
||||
WHERE id = 1
|
||||
`);
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Auth Middleware Migration
|
||||
|
||||
```js
|
||||
// backend/middleware/auth.js — After
|
||||
const pool = require('../db');
|
||||
|
||||
function requireAuth() {
|
||||
return async (req, res, next) => {
|
||||
const sessionId = req.cookies?.session_id;
|
||||
if (!sessionId) return res.status(401).json({ error: 'Authentication required' });
|
||||
|
||||
try {
|
||||
const { rows } = await pool.query(
|
||||
`SELECT s.*, u.id as user_id, u.username, u.email, u.role,
|
||||
u.user_group, u.bu_teams, u.is_active
|
||||
FROM sessions s
|
||||
JOIN users u ON s.user_id = u.id
|
||||
WHERE s.session_id = $1 AND s.expires_at > NOW()`,
|
||||
[sessionId]
|
||||
);
|
||||
const session = rows[0];
|
||||
if (!session) return res.status(401).json({ error: 'Session expired or invalid' });
|
||||
if (!session.is_active) return res.status(401).json({ error: 'Account is disabled' });
|
||||
|
||||
req.user = {
|
||||
id: session.user_id,
|
||||
username: session.username,
|
||||
email: session.email,
|
||||
role: session.role,
|
||||
group: session.user_group,
|
||||
teams: session.bu_teams ? session.bu_teams.split(',').filter(Boolean) : []
|
||||
};
|
||||
next();
|
||||
} catch (err) {
|
||||
console.error('Auth middleware error:', err);
|
||||
return res.status(500).json({ error: 'Authentication error' });
|
||||
}
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
## Data Models
|
||||
|
||||
### Complete DDL for `ivanti_findings` Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS ivanti_findings (
|
||||
id TEXT PRIMARY KEY, -- Ivanti finding ID (e.g. "HF-12345")
|
||||
host_id INTEGER,
|
||||
title TEXT NOT NULL DEFAULT '',
|
||||
severity NUMERIC(4,2) NOT NULL DEFAULT 0,
|
||||
vrr_group TEXT NOT NULL DEFAULT '',
|
||||
host_name TEXT NOT NULL DEFAULT '',
|
||||
ip_address TEXT NOT NULL DEFAULT '',
|
||||
dns TEXT NOT NULL DEFAULT '',
|
||||
status TEXT NOT NULL DEFAULT '',
|
||||
sla_status TEXT NOT NULL DEFAULT '',
|
||||
due_date DATE,
|
||||
last_found_on DATE,
|
||||
bu_ownership TEXT NOT NULL DEFAULT '',
|
||||
cves TEXT[] DEFAULT '{}', -- Postgres array type
|
||||
workflow_id TEXT,
|
||||
workflow_state TEXT,
|
||||
workflow_type TEXT,
|
||||
state TEXT NOT NULL DEFAULT 'open' CHECK (state IN ('open', 'closed')),
|
||||
note TEXT NOT NULL DEFAULT '', -- Merged from ivanti_finding_notes
|
||||
override_host_name TEXT, -- Merged from ivanti_finding_overrides
|
||||
override_dns TEXT, -- Merged from ivanti_finding_overrides
|
||||
synced_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Performance indexes
|
||||
CREATE INDEX IF NOT EXISTS idx_findings_state ON ivanti_findings(state);
|
||||
CREATE INDEX IF NOT EXISTS idx_findings_bu ON ivanti_findings(bu_ownership);
|
||||
CREATE INDEX IF NOT EXISTS idx_findings_severity ON ivanti_findings(severity);
|
||||
CREATE INDEX IF NOT EXISTS idx_findings_state_bu ON ivanti_findings(state, bu_ownership);
|
||||
```
|
||||
|
||||
### Core Tables (Postgres DDL)
|
||||
|
||||
```sql
|
||||
-- Users
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id SERIAL PRIMARY KEY,
|
||||
username VARCHAR(50) UNIQUE NOT NULL,
|
||||
email VARCHAR(255) UNIQUE NOT NULL,
|
||||
password_hash VARCHAR(255) NOT NULL,
|
||||
role VARCHAR(20) NOT NULL DEFAULT 'viewer' CHECK (role IN ('admin', 'editor', 'viewer')),
|
||||
is_active BOOLEAN DEFAULT TRUE,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
last_login TIMESTAMPTZ,
|
||||
user_group VARCHAR(20) NOT NULL DEFAULT 'Read_Only',
|
||||
bu_teams TEXT NOT NULL DEFAULT ''
|
||||
);
|
||||
|
||||
-- Sessions
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
id SERIAL PRIMARY KEY,
|
||||
session_id VARCHAR(255) UNIQUE NOT NULL,
|
||||
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_sessions_session_id ON sessions(session_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_sessions_user_id ON sessions(user_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_sessions_expires ON sessions(expires_at);
|
||||
|
||||
-- Audit Logs
|
||||
CREATE TABLE IF NOT EXISTS audit_logs (
|
||||
id SERIAL PRIMARY KEY,
|
||||
user_id INTEGER,
|
||||
username VARCHAR(50) NOT NULL,
|
||||
action VARCHAR(50) NOT NULL,
|
||||
entity_type VARCHAR(50) NOT NULL,
|
||||
entity_id VARCHAR(100),
|
||||
details TEXT,
|
||||
ip_address VARCHAR(45),
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_audit_created_at ON audit_logs(created_at);
|
||||
|
||||
-- CVEs
|
||||
CREATE TABLE IF NOT EXISTS cves (
|
||||
id SERIAL PRIMARY KEY,
|
||||
cve_id VARCHAR(20) NOT NULL,
|
||||
vendor VARCHAR(100) NOT NULL,
|
||||
severity VARCHAR(20) NOT NULL,
|
||||
description TEXT,
|
||||
published_date DATE,
|
||||
status VARCHAR(50) DEFAULT 'Open',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
created_by INTEGER,
|
||||
UNIQUE(cve_id, vendor)
|
||||
);
|
||||
|
||||
-- Jira Tickets
|
||||
CREATE TABLE IF NOT EXISTS jira_tickets (
|
||||
id SERIAL PRIMARY KEY,
|
||||
cve_id TEXT NOT NULL,
|
||||
vendor TEXT NOT NULL,
|
||||
ticket_key TEXT NOT NULL,
|
||||
url TEXT,
|
||||
summary TEXT,
|
||||
status TEXT DEFAULT 'Open' CHECK(status IN ('Open', 'In Progress', 'Closed')),
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Ivanti Sync State (replaces ivanti_findings_cache metadata)
|
||||
CREATE TABLE IF NOT EXISTS ivanti_sync_state (
|
||||
id INTEGER PRIMARY KEY DEFAULT 1 CHECK (id = 1),
|
||||
total INTEGER DEFAULT 0,
|
||||
workflows_json TEXT DEFAULT '[]',
|
||||
synced_at TIMESTAMPTZ,
|
||||
sync_status TEXT DEFAULT 'never',
|
||||
error_message TEXT
|
||||
);
|
||||
|
||||
-- Ivanti Counts Cache (for FP workflow counts)
|
||||
CREATE TABLE IF NOT EXISTS ivanti_counts_cache (
|
||||
id INTEGER PRIMARY KEY DEFAULT 1 CHECK (id = 1),
|
||||
open_count INTEGER DEFAULT 0,
|
||||
closed_count INTEGER DEFAULT 0,
|
||||
synced_at TIMESTAMPTZ,
|
||||
fp_workflow_counts_json TEXT DEFAULT '{}',
|
||||
fp_id_counts_json TEXT DEFAULT '{}'
|
||||
);
|
||||
|
||||
-- Ivanti Counts History
|
||||
CREATE TABLE IF NOT EXISTS ivanti_counts_history (
|
||||
id SERIAL PRIMARY KEY,
|
||||
open_count INTEGER NOT NULL,
|
||||
closed_count INTEGER NOT NULL,
|
||||
recorded_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Ivanti Finding Archives
|
||||
CREATE TABLE IF NOT EXISTS ivanti_finding_archives (
|
||||
id SERIAL PRIMARY KEY,
|
||||
finding_id TEXT NOT NULL UNIQUE,
|
||||
finding_title TEXT NOT NULL DEFAULT '',
|
||||
host_name TEXT NOT NULL DEFAULT '',
|
||||
ip_address TEXT NOT NULL DEFAULT '',
|
||||
current_state TEXT NOT NULL CHECK(current_state IN ('ARCHIVED','RETURNED','CLOSED','CLOSED_GONE')),
|
||||
last_severity NUMERIC(4,2) NOT NULL DEFAULT 0,
|
||||
first_archived_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
last_transition_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Ivanti Archive Transitions
|
||||
CREATE TABLE IF NOT EXISTS ivanti_archive_transitions (
|
||||
id SERIAL PRIMARY KEY,
|
||||
archive_id INTEGER NOT NULL REFERENCES ivanti_finding_archives(id),
|
||||
from_state TEXT NOT NULL,
|
||||
to_state TEXT NOT NULL,
|
||||
severity_at_transition NUMERIC(4,2) NOT NULL DEFAULT 0,
|
||||
reason TEXT NOT NULL DEFAULT '',
|
||||
transitioned_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Ivanti Sync Anomaly Log
|
||||
CREATE TABLE IF NOT EXISTS ivanti_sync_anomaly_log (
|
||||
id SERIAL PRIMARY KEY,
|
||||
sync_timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
open_count_delta INTEGER NOT NULL DEFAULT 0,
|
||||
closed_count_delta INTEGER NOT NULL DEFAULT 0,
|
||||
newly_archived_count INTEGER NOT NULL DEFAULT 0,
|
||||
returned_count INTEGER NOT NULL DEFAULT 0,
|
||||
classification_json TEXT NOT NULL DEFAULT '{}',
|
||||
return_classification_json TEXT NOT NULL DEFAULT '{}',
|
||||
is_significant BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Ivanti Finding BU History
|
||||
CREATE TABLE IF NOT EXISTS ivanti_finding_bu_history (
|
||||
id SERIAL PRIMARY KEY,
|
||||
finding_id TEXT NOT NULL,
|
||||
finding_title TEXT NOT NULL DEFAULT '',
|
||||
host_name TEXT NOT NULL DEFAULT '',
|
||||
previous_bu TEXT NOT NULL,
|
||||
new_bu TEXT NOT NULL,
|
||||
detected_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Ivanti FP Submissions
|
||||
CREATE TABLE IF NOT EXISTS ivanti_fp_submissions (
|
||||
id SERIAL PRIMARY KEY,
|
||||
user_id INTEGER NOT NULL,
|
||||
username TEXT NOT NULL,
|
||||
ivanti_workflow_batch_id INTEGER,
|
||||
ivanti_generated_id TEXT,
|
||||
ivanti_workflow_batch_uuid TEXT,
|
||||
workflow_name TEXT NOT NULL,
|
||||
reason TEXT NOT NULL,
|
||||
description TEXT,
|
||||
expiration_date TEXT NOT NULL,
|
||||
scope_override TEXT NOT NULL DEFAULT 'Authorized',
|
||||
finding_ids_json TEXT NOT NULL,
|
||||
queue_item_ids_json TEXT NOT NULL,
|
||||
attachment_count INTEGER DEFAULT 0,
|
||||
attachment_results_json TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'success' CHECK(status IN ('success', 'partial', 'failed')),
|
||||
lifecycle_status TEXT NOT NULL DEFAULT 'submitted' CHECK(lifecycle_status IN ('submitted', 'approved', 'rejected', 'rework', 'resubmitted')),
|
||||
error_message TEXT,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ DEFAULT NULL
|
||||
);
|
||||
|
||||
-- Ivanti FP Submission History
|
||||
CREATE TABLE IF NOT EXISTS ivanti_fp_submission_history (
|
||||
id SERIAL PRIMARY KEY,
|
||||
submission_id INTEGER NOT NULL REFERENCES ivanti_fp_submissions(id) ON DELETE CASCADE,
|
||||
user_id INTEGER NOT NULL,
|
||||
username TEXT NOT NULL,
|
||||
change_type TEXT NOT NULL CHECK(change_type IN ('created', 'fields_updated', 'findings_added', 'attachments_added', 'status_changed')),
|
||||
change_details_json TEXT,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Ivanti Todo Queue
|
||||
CREATE TABLE IF NOT EXISTS ivanti_todo_queue (
|
||||
id SERIAL PRIMARY KEY,
|
||||
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
finding_id TEXT NOT NULL,
|
||||
finding_title TEXT,
|
||||
cves_json TEXT,
|
||||
ip_address TEXT,
|
||||
hostname TEXT,
|
||||
vendor TEXT NOT NULL,
|
||||
workflow_type TEXT NOT NULL CHECK(workflow_type IN ('FP', 'Archer', 'CARD', 'GRANITE')),
|
||||
status TEXT NOT NULL DEFAULT 'pending' CHECK(status IN ('pending', 'complete')),
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_todo_queue_user ON ivanti_todo_queue(user_id, status);
|
||||
|
||||
-- Atlas Action Plans Cache
|
||||
CREATE TABLE IF NOT EXISTS atlas_action_plans_cache (
|
||||
id SERIAL PRIMARY KEY,
|
||||
host_id INTEGER NOT NULL UNIQUE,
|
||||
has_action_plan BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
plan_count INTEGER NOT NULL DEFAULT 0,
|
||||
plans_json TEXT NOT NULL DEFAULT '[]',
|
||||
synced_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Compliance Tables
|
||||
CREATE TABLE IF NOT EXISTS compliance_uploads (
|
||||
id SERIAL PRIMARY KEY,
|
||||
filename TEXT NOT NULL,
|
||||
report_date TEXT,
|
||||
uploaded_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
|
||||
uploaded_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
new_count INTEGER DEFAULT 0,
|
||||
resolved_count INTEGER DEFAULT 0,
|
||||
recurring_count INTEGER DEFAULT 0,
|
||||
summary_json TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS compliance_items (
|
||||
id SERIAL PRIMARY KEY,
|
||||
upload_id INTEGER NOT NULL REFERENCES compliance_uploads(id) ON DELETE CASCADE,
|
||||
hostname TEXT NOT NULL,
|
||||
ip_address TEXT,
|
||||
device_type TEXT,
|
||||
team TEXT,
|
||||
metric_id TEXT NOT NULL,
|
||||
metric_desc TEXT,
|
||||
category TEXT,
|
||||
extra_json TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'active' CHECK(status IN ('active', 'resolved')),
|
||||
first_seen_upload_id INTEGER REFERENCES compliance_uploads(id) ON DELETE SET NULL,
|
||||
resolved_upload_id INTEGER REFERENCES compliance_uploads(id) ON DELETE SET NULL,
|
||||
seen_count INTEGER DEFAULT 1,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS compliance_notes (
|
||||
id SERIAL PRIMARY KEY,
|
||||
hostname TEXT NOT NULL,
|
||||
metric_id TEXT NOT NULL,
|
||||
note TEXT NOT NULL,
|
||||
group_id TEXT,
|
||||
created_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Knowledge Base
|
||||
CREATE TABLE IF NOT EXISTS knowledge_base (
|
||||
id SERIAL PRIMARY KEY,
|
||||
title VARCHAR(255) NOT NULL,
|
||||
slug VARCHAR(255) UNIQUE NOT NULL,
|
||||
description TEXT,
|
||||
category VARCHAR(100),
|
||||
file_path VARCHAR(500),
|
||||
file_name VARCHAR(255),
|
||||
file_type VARCHAR(50),
|
||||
file_size INTEGER,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
created_by INTEGER REFERENCES users(id)
|
||||
);
|
||||
|
||||
-- Archer Tickets
|
||||
CREATE TABLE IF NOT EXISTS archer_tickets (
|
||||
id SERIAL PRIMARY KEY,
|
||||
exc_number TEXT NOT NULL UNIQUE,
|
||||
archer_url TEXT,
|
||||
status TEXT DEFAULT 'Draft' CHECK(status IN ('Draft', 'Open', 'Under Review', 'Accepted')),
|
||||
cve_id TEXT NOT NULL,
|
||||
vendor TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Documents
|
||||
CREATE TABLE IF NOT EXISTS documents (
|
||||
id SERIAL PRIMARY KEY,
|
||||
cve_id VARCHAR(20) NOT NULL,
|
||||
vendor VARCHAR(100) NOT NULL,
|
||||
name VARCHAR(255) NOT NULL,
|
||||
type VARCHAR(50) NOT NULL,
|
||||
file_path VARCHAR(500) NOT NULL,
|
||||
file_size VARCHAR(20),
|
||||
mime_type VARCHAR(100),
|
||||
uploaded_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
notes TEXT
|
||||
);
|
||||
|
||||
-- Required Documents (seed data)
|
||||
CREATE TABLE IF NOT EXISTS required_documents (
|
||||
id SERIAL PRIMARY KEY,
|
||||
vendor VARCHAR(100) NOT NULL,
|
||||
document_type VARCHAR(50) NOT NULL,
|
||||
is_mandatory BOOLEAN DEFAULT TRUE,
|
||||
description TEXT
|
||||
);
|
||||
```
|
||||
|
||||
### Per-BU Count Queries
|
||||
|
||||
```sql
|
||||
-- Open count for specific BUs (used by counts endpoint with teams filter)
|
||||
SELECT COUNT(*) FROM ivanti_findings
|
||||
WHERE state = 'open' AND bu_ownership ILIKE ANY(ARRAY['%STEAM%', '%ACCESS-ENG%']);
|
||||
|
||||
-- Closed count for specific BUs
|
||||
SELECT COUNT(*) FROM ivanti_findings
|
||||
WHERE state = 'closed' AND bu_ownership ILIKE ANY(ARRAY['%STEAM%', '%ACCESS-ENG%']);
|
||||
|
||||
-- Aggregated counts grouped by BU and state (single query)
|
||||
SELECT bu_ownership, state, COUNT(*) as count
|
||||
FROM ivanti_findings
|
||||
GROUP BY bu_ownership, state;
|
||||
|
||||
-- Global totals (no filter — backward compatible)
|
||||
SELECT state, COUNT(*) as count
|
||||
FROM ivanti_findings
|
||||
GROUP BY state;
|
||||
```
|
||||
|
||||
### Data Migration Script Design (`backend/scripts/migrate-to-postgres.js`)
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Open SQLite read-only] --> B[Connect to Postgres pool]
|
||||
B --> C[Create all tables IF NOT EXISTS]
|
||||
C --> D[Copy simple tables]
|
||||
D --> E[Parse findings_json blob]
|
||||
E --> F[Insert individual finding rows state=open]
|
||||
F --> G[Merge ivanti_finding_notes → findings.note]
|
||||
G --> H[Merge ivanti_finding_overrides → findings.override_*]
|
||||
H --> I[Verify row counts]
|
||||
I --> J[Print summary report]
|
||||
```
|
||||
|
||||
The migration script:
|
||||
1. Opens SQLite with `OPEN_READONLY` flag
|
||||
2. Connects to Postgres via `DATABASE_URL`
|
||||
3. Creates schema idempotently (`IF NOT EXISTS`)
|
||||
4. Copies each table using batch inserts with `ON CONFLICT` for idempotency
|
||||
5. Special handling for findings: parses `findings_json`, creates one row per finding
|
||||
6. Merges notes and overrides into the corresponding finding rows
|
||||
7. Verifies source vs destination row counts
|
||||
8. Never modifies the SQLite file
|
||||
|
||||
## Correctness Properties
|
||||
|
||||
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
|
||||
|
||||
### Property 1: Upsert Idempotence
|
||||
|
||||
*For any* finding synced N times (N ≥ 1) with the same finding ID, the `ivanti_findings` table SHALL contain exactly one row for that finding ID, with the data from the most recent sync.
|
||||
|
||||
**Validates: Requirements 3.5, 6.6**
|
||||
|
||||
### Property 2: Finding Storage Preserves State and BU Ownership
|
||||
|
||||
*For any* finding (open or closed) stored in `ivanti_findings`, querying it back by ID SHALL return the same `state` and `bu_ownership` values it was stored with.
|
||||
|
||||
**Validates: Requirements 3.4, 4.1**
|
||||
|
||||
### Property 3: Count Query Accuracy
|
||||
|
||||
*For any* set of findings in `ivanti_findings` and any BU filter (including empty/no filter), the count query result SHALL equal the actual number of rows matching that filter and state combination.
|
||||
|
||||
**Validates: Requirements 4.2, 4.3, 4.5**
|
||||
|
||||
### Property 4: Migration Data Preservation (Findings)
|
||||
|
||||
*For any* finding in the source `findings_json` blob with associated notes (from `ivanti_finding_notes`) and overrides (from `ivanti_finding_overrides`), the migrated `ivanti_findings` row SHALL contain the finding data with `state = 'open'`, the correct `note` value, and the correct `override_host_name`/`override_dns` values.
|
||||
|
||||
**Validates: Requirements 7.4, 7.5, 7.6**
|
||||
|
||||
### Property 5: Migration Table Copy Preservation
|
||||
|
||||
*For any* table copied from SQLite to Postgres, the row count in Postgres SHALL equal the row count in SQLite, and each row's data SHALL be equivalent (accounting for type conversions: 0/1 → boolean, DATETIME → TIMESTAMPTZ).
|
||||
|
||||
**Validates: Requirements 7.7, 7.8**
|
||||
|
||||
### Property 6: Migration Idempotence
|
||||
|
||||
*For any* initial state of the SQLite and Postgres databases, running the migration script N times (N ≥ 1) SHALL produce the same final state in Postgres as running it exactly once (no duplicate rows, no errors).
|
||||
|
||||
**Validates: Requirements 7.3, 7.9**
|
||||
|
||||
### Property 7: Migration Source Safety
|
||||
|
||||
*For any* execution of the migration script, the SQLite database file SHALL remain byte-for-byte identical before and after (checksum unchanged).
|
||||
|
||||
**Validates: Requirements 7.10**
|
||||
|
||||
### Property 8: Schema Creation Idempotence
|
||||
|
||||
*For any* number of times the schema creation DDL is executed against the same database, the resulting schema SHALL be identical (no errors, same tables, same indexes, same constraints).
|
||||
|
||||
**Validates: Requirements 2.5**
|
||||
|
||||
### Property 9: API Response Shape Preservation
|
||||
|
||||
*For any* valid API request to any endpoint, the response JSON structure (top-level keys and value types) after migration SHALL be identical to the pre-migration response structure.
|
||||
|
||||
**Validates: Requirements 10.1**
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error Scenario | Handling Strategy |
|
||||
|----------------|-------------------|
|
||||
| Pool connection failure | `pool.on('error')` logs error; automatic reconnection on next query attempt |
|
||||
| Pool exhaustion (all 10 busy) | Queries queue internally; warning logged at 8 active connections |
|
||||
| Query timeout | `connectionTimeoutMillis: 5000` — rejects after 5s with error |
|
||||
| Sync failure mid-batch | Transaction rollback; `sync_status = 'error'` with message; previous data preserved |
|
||||
| Migration script failure | Idempotent design — safe to re-run; prints error and exits with code 1 |
|
||||
| Docker container crash | `--restart unless-stopped` auto-recovers; pool reconnects on next query |
|
||||
| Invalid finding data | `NOT NULL DEFAULT ''` columns prevent null constraint violations; CHECK constraints reject invalid state values |
|
||||
| Rollback needed | Stop Postgres backend → revert `.env` → restart SQLite backend; SQLite file always preserved |
|
||||
|
||||
### Error Response Format (Unchanged)
|
||||
|
||||
All error responses maintain the existing format:
|
||||
```json
|
||||
{ "error": "Human-readable error message" }
|
||||
```
|
||||
|
||||
With appropriate HTTP status codes: 400 (validation), 401 (auth), 403 (permission), 404 (not found), 500 (server error).
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests (Example-Based)
|
||||
|
||||
- Verify each route returns correct response shape with known test data
|
||||
- Verify auth middleware rejects expired sessions
|
||||
- Verify parameter placeholder conversion (`?` → `$1`) in all queries
|
||||
- Verify schema DDL executes without errors
|
||||
- Verify migration script handles empty tables gracefully
|
||||
|
||||
### Property-Based Tests
|
||||
|
||||
Property-based testing is appropriate for this feature because the core operations (upsert, count queries, data migration) have clear input/output behavior with universal properties that hold across a wide input space.
|
||||
|
||||
**Library**: [fast-check](https://github.com/dubzzz/fast-check) (JavaScript PBT library)
|
||||
|
||||
**Configuration**: Minimum 100 iterations per property test.
|
||||
|
||||
**Tag format**: `Feature: postgres-migration, Property {number}: {property_text}`
|
||||
|
||||
Each correctness property (1-9) maps to a single property-based test:
|
||||
- Property 1: Generate random findings, upsert each N times, verify exactly one row per ID
|
||||
- Property 2: Generate findings with random state/bu_ownership, store and retrieve, verify equality
|
||||
- Property 3: Generate random finding sets, insert, run count queries with random filters, verify accuracy
|
||||
- Property 4: Generate random findings JSON with notes/overrides, run migration logic, verify merged output
|
||||
- Property 5: Generate random table rows, copy via migration, verify count and data equivalence
|
||||
- Property 6: Run migration logic N times on same input, verify final state equals single-run state
|
||||
- Property 7: Checksum SQLite before/after migration, verify unchanged
|
||||
- Property 8: Run schema DDL N times, verify no errors and same schema
|
||||
- Property 9: Compare response shapes between SQLite and Postgres backends for same requests
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- Docker container health check (port 5433 accessible)
|
||||
- Full sync cycle: trigger sync → verify rows created → verify counts endpoint
|
||||
- Concurrent read during write: start sync, simultaneously query findings, verify no blocking
|
||||
- Performance: GET /findings < 500ms, GET /counts < 100ms with 6000+ rows
|
||||
- Cutover simulation: stop/start backend, verify API responds correctly
|
||||
|
||||
### Development Isolation
|
||||
|
||||
- Test backend runs on port 3003 with `DATABASE_URL` pointing to Postgres
|
||||
- Production backend continues on port 3001 with SQLite (no `DATABASE_URL` set)
|
||||
- Switching is controlled by presence of `DATABASE_URL` environment variable
|
||||
- All work on `feature/multi-tenancy` branch
|
||||
158
.kiro/specs/postgres-migration/requirements.md
Normal file
158
.kiro/specs/postgres-migration/requirements.md
Normal file
@@ -0,0 +1,158 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
Migrate the CVE Dashboard (STEAM Security Dashboard) backend from SQLite to PostgreSQL 16. The current SQLite architecture stores all Ivanti findings as a single 2.6MB JSON blob (`ivanti_findings_cache.findings_json`) that must be parsed on every API request, causing 5-10 second load times. Additionally, SQLite's single-writer lock blocks reads during sync writes, and per-BU closed finding counts are unavailable (only a global count exists). PostgreSQL enables individual finding rows with indexed columns, per-BU open and closed counts, connection pooling for concurrent access, and proper type support. The Postgres instance runs in a dedicated Docker container on port 5433, isolated from the existing Postgres on port 5432 which belongs to another project.
|
||||
|
||||
## Glossary
|
||||
|
||||
- **SQLite**: The current embedded database engine storing all data in `backend/cve_database.db` (13MB total)
|
||||
- **PostgreSQL_16**: The target relational database running in a Docker container (`steam-postgres`) on port 5433
|
||||
- **findings_json**: The current TEXT column in `ivanti_findings_cache` storing all Ivanti findings as a serialized JSON array (2.6MB+)
|
||||
- **ivanti_findings**: The new Postgres table storing each finding as an individual row with indexed columns
|
||||
- **Pool**: A `pg` (node-postgres) connection pool managing up to 10 concurrent database connections
|
||||
- **DATABASE_URL**: Environment variable containing the Postgres connection string (`postgresql://steam:<password>@localhost:5433/cve_dashboard`)
|
||||
- **Migration_Script**: A one-time Node.js script (`backend/scripts/migrate-to-postgres.js`) that reads from SQLite and writes to Postgres
|
||||
- **Cutover**: The moment production switches from SQLite backend on port 3001 to Postgres backend on port 3001
|
||||
- **steam-postgres**: The Docker container name for the CVE Dashboard's dedicated PostgreSQL instance
|
||||
- **steam-pgdata**: The Docker volume providing persistent storage for the Postgres data directory
|
||||
- **bu_ownership**: The Ivanti finding field containing the BU assignment (e.g. "NTS-AEO-STEAM")
|
||||
- **Upsert**: An INSERT that updates the existing row on primary key conflict (`INSERT ... ON CONFLICT DO UPDATE`)
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Dedicated PostgreSQL Docker Container
|
||||
|
||||
**User Story:** As a system operator, I want a dedicated Postgres 16 container for the CVE Dashboard on port 5433, so that it is fully isolated from the existing Postgres instance on port 5432 belonging to another project.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Infrastructure SHALL run PostgreSQL 16 (Alpine) in a Docker container named `steam-postgres`
|
||||
2. THE container SHALL map host port 5433 to container port 5432
|
||||
3. THE container SHALL use a Docker volume named `steam-pgdata` for persistent data storage
|
||||
4. THE container SHALL be configured with `--restart unless-stopped` for automatic recovery after host reboots
|
||||
5. THE container SHALL create a database named `cve_dashboard` with a user named `steam`
|
||||
6. THE Infrastructure SHALL NOT modify or affect the existing Postgres instance on port 5432
|
||||
|
||||
### Requirement 2: Schema Parity with Proper Types
|
||||
|
||||
**User Story:** As a developer, I want all existing SQLite tables recreated in Postgres with proper types and constraints, so that no functionality is lost during migration.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Postgres schema SHALL include all tables from the current SQLite schema: users, sessions, audit_logs, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, compliance_uploads, compliance_items, compliance_notes, ivanti_sync_state, ivanti_finding_notes, ivanti_counts_cache, ivanti_counts_history, ivanti_finding_overrides, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue, atlas_action_plans_cache
|
||||
2. THE Postgres schema SHALL use appropriate types: SERIAL for auto-increment, TIMESTAMPTZ for timestamps, BOOLEAN for booleans, NUMERIC for decimals, TEXT[] for arrays, DATE for date-only fields
|
||||
3. THE Postgres schema SHALL preserve all existing CHECK constraints and foreign key relationships
|
||||
4. THE Postgres schema SHALL include the `bu_teams` column on the users table (required by multi-BU tenancy feature)
|
||||
5. THE schema creation SHALL be idempotent using `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS`
|
||||
|
||||
### Requirement 3: Findings Table Redesign
|
||||
|
||||
**User Story:** As a user, I want findings stored as individual rows with indexed columns, so that filtering by BU ownership and state is instant instead of requiring full JSON blob parsing.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE system SHALL replace the `ivanti_findings_cache.findings_json` blob with an `ivanti_findings` table containing one row per finding
|
||||
2. THE `ivanti_findings` table SHALL include columns: id (TEXT PRIMARY KEY, Ivanti finding ID), host_id (INTEGER), title (TEXT), severity (NUMERIC), vrr_group (TEXT), host_name (TEXT), ip_address (TEXT), dns (TEXT), status (TEXT), sla_status (TEXT), due_date (DATE), last_found_on (DATE), bu_ownership (TEXT), cves (TEXT[] array), workflow_id (TEXT), workflow_state (TEXT), workflow_type (TEXT), state (TEXT with CHECK constraint for 'open' or 'closed'), note (TEXT), override_host_name (TEXT), override_dns (TEXT), synced_at (TIMESTAMPTZ), created_at (TIMESTAMPTZ)
|
||||
3. THE system SHALL create indexes on: state, bu_ownership, severity, and a composite index on (state, bu_ownership)
|
||||
4. THE system SHALL store both open AND closed findings as individual rows with their respective state values
|
||||
5. WHEN findings are synced from Ivanti, THE system SHALL upsert rows using `INSERT ... ON CONFLICT (id) DO UPDATE` rather than replacing a JSON blob
|
||||
|
||||
### Requirement 4: Per-BU Closed Finding Counts
|
||||
|
||||
**User Story:** As a user viewing the Reporting page with a BU scope filter, I want accurate open and closed counts for my selected BUs, so that the dashboard shows meaningful per-team metrics instead of only a global count.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE system SHALL store closed findings with their `bu_ownership` field preserved as individual rows with `state = 'closed'`
|
||||
2. THE counts endpoint SHALL derive per-BU counts using `SELECT COUNT(*) ... WHERE state = ? AND bu_ownership ILIKE ?` queries
|
||||
3. WHEN a teams filter is provided, THE counts endpoint SHALL return open and closed counts scoped to those BUs
|
||||
4. WHEN no filter is applied, THE counts endpoint SHALL return global totals (backward compatible with current behavior)
|
||||
5. THE system SHALL support aggregated counts grouped by bu_ownership and state in a single query
|
||||
|
||||
### Requirement 5: Connection Pooling
|
||||
|
||||
**User Story:** As a system operator, I want the backend to use connection pooling via the pg package, so that multiple concurrent requests are handled efficiently without blocking and reads are never blocked by writes.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE backend SHALL use the `pg` npm package with a Pool instance (maximum pool size: 10 connections)
|
||||
2. THE Pool SHALL read the connection string from the `DATABASE_URL` environment variable
|
||||
3. IF the Pool detects connection errors, THEN THE Pool SHALL attempt automatic reconnection
|
||||
4. THE Pool SHALL log a warning when active connections reach 8 (approaching exhaustion)
|
||||
5. ALL database operations SHALL use async/await with the Pool (replacing all callback-based SQLite patterns)
|
||||
|
||||
### Requirement 6: Backend Route Migration
|
||||
|
||||
**User Story:** As a developer, I want all SQLite-specific code replaced with Postgres equivalents using async/await, so that the codebase uses a single database driver consistently.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE backend SHALL replace all `db.get(sql, params, callback)` calls with `pool.query(sql, params)` returning `rows[0]`
|
||||
2. THE backend SHALL replace all `db.all(sql, params, callback)` calls with `pool.query(sql, params)` returning `rows`
|
||||
3. THE backend SHALL replace all `db.run(sql, params, callback)` calls with `pool.query(sql, params)` using `RETURNING` clauses where the inserted/updated row is needed
|
||||
4. THE backend SHALL replace `?` parameter placeholders with `$1, $2, $3...` Postgres numbered parameter syntax
|
||||
5. THE backend SHALL remove all callback-based database patterns in favor of async/await with try/catch error handling
|
||||
6. THE Ivanti sync logic SHALL write individual finding rows via upsert instead of serializing to a JSON blob
|
||||
|
||||
### Requirement 7: Data Migration Script
|
||||
|
||||
**User Story:** As a system operator, I want a one-time migration script that copies all data from SQLite to Postgres, so that no data is lost during the transition.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Migration_Script SHALL open the SQLite database in read-only mode
|
||||
2. THE Migration_Script SHALL connect to Postgres using the DATABASE_URL connection string
|
||||
3. THE Migration_Script SHALL create all tables idempotently before inserting data
|
||||
4. THE Migration_Script SHALL parse the `findings_json` blob and insert individual finding rows into the `ivanti_findings` table with `state = 'open'`
|
||||
5. THE Migration_Script SHALL merge `ivanti_finding_notes` into the corresponding `ivanti_findings.note` column
|
||||
6. THE Migration_Script SHALL merge `ivanti_finding_overrides` into the corresponding `ivanti_findings.override_*` columns
|
||||
7. THE Migration_Script SHALL copy all other tables preserving their data: users, sessions, cves, documents, jira_tickets, archer_tickets, knowledge_base, audit_logs, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, atlas_action_plans_cache, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue
|
||||
8. THE Migration_Script SHALL verify row counts after migration and print a summary with any discrepancies
|
||||
9. THE Migration_Script SHALL be idempotent (safe to run multiple times using ON CONFLICT/upsert logic)
|
||||
10. THE Migration_Script SHALL never modify the SQLite database file
|
||||
|
||||
### Requirement 8: Zero-Impact Cutover
|
||||
|
||||
**User Story:** As a system operator, I want the cutover from SQLite to Postgres to take under 30 seconds with a clear rollback path, so that downtime is minimal and reversible.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE cutover procedure SHALL consist of: stop old backend, start new backend on the same port (3001)
|
||||
2. THE frontend SHALL NOT require any changes for the cutover (same API contract, same port, same URL)
|
||||
3. IF issues are detected after cutover, THEN THE system SHALL support rollback by reverting the .env configuration and restarting the SQLite-based backend
|
||||
4. THE SQLite database file SHALL be preserved indefinitely as a backup after cutover
|
||||
5. THE cutover downtime SHALL not exceed 30 seconds
|
||||
|
||||
### Requirement 9: Performance Improvement
|
||||
|
||||
**User Story:** As a user, I want the dashboard to load findings in under 1 second regardless of BU scope, so that switching between BU views feels instant compared to the current 5-10 second load times.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE `GET /api/ivanti/findings` endpoint SHALL respond in under 500ms for any BU filter combination
|
||||
2. THE `GET /api/ivanti/findings/counts` endpoint SHALL respond in under 100ms for any BU filter
|
||||
3. THE Ivanti sync process SHALL complete without blocking concurrent read requests (Postgres MVCC)
|
||||
4. THE system SHALL eliminate JSON blob parsing from the findings read path entirely
|
||||
|
||||
### Requirement 10: API Backward Compatibility
|
||||
|
||||
**User Story:** As a developer, I want the API contract to remain unchanged after migration, so that the frontend works identically without code changes.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. ALL existing API endpoints SHALL return the same response shape after migration
|
||||
2. THE `GET /api/ivanti/findings` response SHALL include: findings array, total count, synced_at timestamp, sync_status, and error_message fields
|
||||
3. THE authentication and session system SHALL work identically (cookie-based sessions, same expiration behavior)
|
||||
4. THE frontend SHALL require zero code changes specifically for the database migration (multi-BU filtering changes are a separate feature)
|
||||
|
||||
### Requirement 11: Development and Testing Isolation
|
||||
|
||||
**User Story:** As a developer, I want to test the Postgres backend on port 3003 while production continues on port 3001 with SQLite, so that development does not disrupt the live system.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. WHILE the migration is in development, THE test backend SHALL run on port 3003
|
||||
2. WHILE the migration is in development, THE production backend SHALL continue running on port 3001 with SQLite
|
||||
3. THE system SHALL support switching between SQLite and Postgres via environment variable configuration (DATABASE_URL presence or DB_TYPE flag)
|
||||
4. ALL development work SHALL occur on the existing `feature/multi-tenancy` branch
|
||||
294
.kiro/specs/postgres-migration/tasks.md
Normal file
294
.kiro/specs/postgres-migration/tasks.md
Normal file
@@ -0,0 +1,294 @@
|
||||
# Implementation Plan: PostgreSQL Migration
|
||||
|
||||
## Overview
|
||||
|
||||
Migrate the CVE Dashboard backend from SQLite to PostgreSQL 16. Replace the monolithic `findings_json` blob (2.6MB) with individual indexed rows in `ivanti_findings`, enable per-BU closed counts, and eliminate the single-writer lock. All work on the `feature/multi-tenancy` branch. Docker container `steam-postgres` on port 5433, test backend on port 3003, production on port 3001.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] 1. Infrastructure setup and connection pool
|
||||
- [x] 1.1 Install `pg` dependency and configure environment
|
||||
- Run `npm install pg` in `backend/`
|
||||
- Add `DATABASE_URL=postgresql://steam:<password>@localhost:5433/cve_dashboard` to `backend/.env`
|
||||
- Add `DATABASE_URL` placeholder to `backend/.env.example`
|
||||
- _Requirements: 1.5, 5.1, 5.2_
|
||||
|
||||
- [x] 1.2 Create `backend/db.js` connection pool module
|
||||
- Import `pg` and create a `Pool` instance reading from `DATABASE_URL`
|
||||
- Set `max: 10`, `idleTimeoutMillis: 30000`, `connectionTimeoutMillis: 5000`
|
||||
- Add `pool.on('error')` handler logging unexpected errors
|
||||
- Track active connections; log warning when count reaches 8
|
||||
- Export the pool instance for use by all route files
|
||||
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5_
|
||||
|
||||
- [x] 1.3 Create Docker run command documentation
|
||||
- Document the `docker run` command for `steam-postgres` container (port 5433:5432, volume `steam-pgdata`, `--restart unless-stopped`, Postgres 16 Alpine)
|
||||
- Verify container creates `cve_dashboard` database with `steam` user
|
||||
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6_
|
||||
|
||||
- [x] 2. Schema creation
|
||||
- [x] 2.1 Create `backend/db-schema.sql` with complete Postgres DDL
|
||||
- Define all tables with proper Postgres types: SERIAL, TIMESTAMPTZ, BOOLEAN, NUMERIC, TEXT[], DATE
|
||||
- Include `ivanti_findings` table with TEXT PRIMARY KEY (`id`), all columns per design, CHECK constraint on `state`
|
||||
- Include `ivanti_sync_state` table (single-row pattern, replaces `ivanti_findings_cache` metadata)
|
||||
- Include all other tables: users (with `bu_teams`), sessions, audit_logs, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_cache, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue, atlas_action_plans_cache
|
||||
- Include all indexes: findings (state, bu_ownership, severity, state+bu_ownership composite), sessions (session_id, user_id, expires_at), audit_logs (created_at), todo_queue (user_id+status)
|
||||
- Include all foreign key relationships and CHECK constraints
|
||||
- Use `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS` for idempotence
|
||||
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 3.1, 3.2, 3.3_
|
||||
|
||||
- [ ]* 2.2 Write property test for schema creation idempotence
|
||||
- **Property 8: Schema Creation Idempotence**
|
||||
- Run schema DDL N times against the same database, verify no errors and same resulting schema each time
|
||||
- **Validates: Requirements 2.5**
|
||||
|
||||
- [x] 2.3 Create schema initialization in `backend/setup.js`
|
||||
- Read `db-schema.sql` and execute via pool.query
|
||||
- Make callable on server startup or as standalone script
|
||||
- Seed `ivanti_sync_state` row (id=1) and `ivanti_counts_cache` row (id=1) if not exists
|
||||
- _Requirements: 2.1, 2.5_
|
||||
|
||||
- [x] 3. Checkpoint — Verify infrastructure
|
||||
- Ensure Docker container is running on port 5433, pool connects successfully, schema creates without errors. Ask the user if questions arise.
|
||||
|
||||
- [-] 4. Migrate auth and session system
|
||||
- [x] 4.1 Update `backend/middleware/auth.js`
|
||||
- Replace `db.get()` callback with `pool.query()` async/await
|
||||
- Change `?` placeholders to `$1, $2, ...` numbered params
|
||||
- Join sessions and users tables, check `expires_at > NOW()`
|
||||
- Return 401 for missing/expired sessions, 500 for query errors
|
||||
- _Requirements: 6.1, 6.4, 6.5, 10.3_
|
||||
|
||||
- [x] 4.2 Update `backend/routes/auth.js`
|
||||
- Replace all `db.get/db.all/db.run` with `pool.query`
|
||||
- Login: query user by username, create session with `RETURNING`
|
||||
- Logout: delete session by session_id
|
||||
- Password change: update password_hash
|
||||
- Profile/me endpoint: query user by session
|
||||
- Use `$1, $2...` placeholders throughout
|
||||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 10.3_
|
||||
|
||||
- [x] 4.3 Update `backend/routes/users.js`
|
||||
- Replace all sqlite3 calls with pool.query
|
||||
- CRUD operations: list users, create user, update user, delete user
|
||||
- Use `RETURNING` clause for inserts/updates where row data is needed
|
||||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
|
||||
|
||||
- [x] 4.4 Update `backend/server.js` database initialization
|
||||
- Remove sqlite3 database opening and `db` object creation
|
||||
- Import pool from `backend/db.js`
|
||||
- Remove passing `db` parameter to route factory functions
|
||||
- Update inline CVE/document/vendor routes to use pool.query
|
||||
- _Requirements: 6.1, 6.4, 6.5_
|
||||
|
||||
- [ ] 5. Checkpoint — Verify auth and core routes
|
||||
- Ensure login, logout, session validation, user CRUD, and CVE routes work on port 3003. Ask the user if questions arise.
|
||||
|
||||
- [-] 6. Migrate remaining route files
|
||||
- [x] 6.1 Update `backend/routes/jiraTickets.js`
|
||||
- Replace sqlite3 calls with pool.query, update placeholders
|
||||
- _Requirements: 6.1, 6.4, 6.5_
|
||||
|
||||
- [x] 6.2 Update `backend/routes/archerTickets.js`
|
||||
- Replace sqlite3 calls with pool.query, update placeholders
|
||||
- _Requirements: 6.1, 6.4, 6.5_
|
||||
|
||||
- [x] 6.3 Update `backend/routes/knowledgeBase.js`
|
||||
- Replace sqlite3 calls with pool.query, update placeholders
|
||||
- _Requirements: 6.1, 6.4, 6.5_
|
||||
|
||||
- [x] 6.4 Update `backend/routes/compliance.js`
|
||||
- Replace sqlite3 calls with pool.query, update placeholders
|
||||
- Handle compliance_items, compliance_uploads, compliance_notes queries
|
||||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
|
||||
|
||||
- [x] 6.5 Update `backend/routes/auditLog.js` and `backend/helpers/auditLog.js`
|
||||
- Replace sqlite3 db.run/db.all with pool.query
|
||||
- Update the audit logging helper to use async pool.query
|
||||
- _Requirements: 6.1, 6.4, 6.5_
|
||||
|
||||
- [x] 6.6 Update `backend/routes/ivantiWorkflows.js`
|
||||
- Replace sqlite3 calls with pool.query, update placeholders
|
||||
- _Requirements: 6.1, 6.4, 6.5_
|
||||
|
||||
- [x] 6.7 Update `backend/routes/ivantiFpWorkflow.js`
|
||||
- Replace sqlite3 calls with pool.query, update placeholders
|
||||
- Handle fp_submissions and fp_submission_history tables
|
||||
- _Requirements: 6.1, 6.4, 6.5_
|
||||
|
||||
- [x] 6.8 Update `backend/routes/ivantiTodoQueue.js`
|
||||
- Replace sqlite3 calls with pool.query, update placeholders
|
||||
- _Requirements: 6.1, 6.4, 6.5_
|
||||
|
||||
- [x] 6.9 Update `backend/routes/ivantiArchive.js`
|
||||
- Replace sqlite3 calls with pool.query, update placeholders
|
||||
- Handle archive and transition table queries
|
||||
- _Requirements: 6.1, 6.4, 6.5_
|
||||
|
||||
- [x] 6.10 Update `backend/routes/atlas.js`
|
||||
- Replace sqlite3 calls with pool.query, update placeholders
|
||||
- _Requirements: 6.1, 6.4, 6.5_
|
||||
|
||||
- [x] 6.11 Update `backend/routes/cardApi.js` and `backend/routes/feedback.js`
|
||||
- Replace any sqlite3 calls with pool.query if present
|
||||
- _Requirements: 6.1, 6.4, 6.5_
|
||||
|
||||
- [ ] 7. Checkpoint — Verify all route migrations
|
||||
- Ensure all non-findings routes work correctly on port 3003. Ask the user if questions arise.
|
||||
|
||||
- [-] 8. Rewrite Ivanti findings sync and read logic
|
||||
- [x] 8.1 Rewrite sync logic in `backend/routes/ivantiFindings.js`
|
||||
- Replace JSON blob serialization with batch upsert of individual rows
|
||||
- Use `INSERT INTO ivanti_findings (...) VALUES ... ON CONFLICT (id) DO UPDATE SET ...` in batches of 100
|
||||
- Sync both open and closed findings as individual rows with correct `state` value
|
||||
- Update `ivanti_sync_state` with total count, synced_at, sync_status after sync
|
||||
- Preserve existing note and override values during upsert (do not overwrite user-set fields)
|
||||
- _Requirements: 3.4, 3.5, 6.6, 9.3_
|
||||
|
||||
- [ ]* 8.2 Write property test for upsert idempotence
|
||||
- **Property 1: Upsert Idempotence**
|
||||
- Generate random findings, upsert each N times with same ID, verify exactly one row per ID with most recent data
|
||||
- **Validates: Requirements 3.5, 6.6**
|
||||
|
||||
- [x] 8.3 Rewrite read endpoints in `backend/routes/ivantiFindings.js`
|
||||
- GET /findings: `SELECT * FROM ivanti_findings WHERE state = 'open'` with optional BU filter via `bu_ownership ILIKE`
|
||||
- Return response shape: `{ findings, total, synced_at, sync_status, error_message }`
|
||||
- GET /findings/counts: derive open/closed counts from `ivanti_findings` with optional BU filter
|
||||
- Support `teams` query parameter for per-BU scoped counts using `ILIKE ANY(ARRAY[...])`
|
||||
- GET /findings/counts/history: unchanged (reads from `ivanti_counts_history`)
|
||||
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 9.1, 9.2, 10.1, 10.2_
|
||||
|
||||
- [ ]* 8.4 Write property test for finding state and BU preservation
|
||||
- **Property 2: Finding Storage Preserves State and BU Ownership**
|
||||
- Generate findings with random state/bu_ownership, store and retrieve by ID, verify equality
|
||||
- **Validates: Requirements 3.4, 4.1**
|
||||
|
||||
- [ ]* 8.5 Write property test for count query accuracy
|
||||
- **Property 3: Count Query Accuracy**
|
||||
- Generate random finding sets, insert into DB, run count queries with random BU filters, verify counts match actual row counts
|
||||
- **Validates: Requirements 4.2, 4.3, 4.5**
|
||||
|
||||
- [x] 8.6 Update note and override endpoints
|
||||
- PUT /findings/:id/note: `UPDATE ivanti_findings SET note = $1 WHERE id = $2`
|
||||
- PUT /findings/:id/override: `UPDATE ivanti_findings SET override_host_name = $1, override_dns = $2 WHERE id = $3`
|
||||
- _Requirements: 6.1, 6.4_
|
||||
|
||||
- [ ] 9. Checkpoint — Verify findings redesign
|
||||
- Ensure sync creates individual rows, read endpoints return correct shape, counts work with and without BU filter. Ask the user if questions arise.
|
||||
|
||||
- [-] 10. Data migration script
|
||||
- [x] 10.1 Create `backend/scripts/migrate-to-postgres.js`
|
||||
- Open SQLite database in read-only mode (`OPEN_READONLY`)
|
||||
- Connect to Postgres via `DATABASE_URL`
|
||||
- Run schema creation (idempotent) before inserting data
|
||||
- Copy all simple tables: users, sessions, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, audit_logs, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, atlas_action_plans_cache, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue
|
||||
- Handle type conversions: SQLite 0/1 → Postgres boolean, DATETIME strings → TIMESTAMPTZ
|
||||
- Use batch inserts with `ON CONFLICT` for idempotency (safe to re-run)
|
||||
- _Requirements: 7.1, 7.2, 7.3, 7.7, 7.9, 7.10_
|
||||
|
||||
- [x] 10.2 Implement findings migration with note/override merging
|
||||
- Parse `ivanti_findings_cache.findings_json` blob into individual objects
|
||||
- Insert each finding as a row in `ivanti_findings` with `state = 'open'`
|
||||
- Query `ivanti_finding_notes` and merge each note into the corresponding `ivanti_findings.note` column
|
||||
- Query `ivanti_finding_overrides` and merge into `ivanti_findings.override_host_name` / `override_dns`
|
||||
- _Requirements: 7.4, 7.5, 7.6_
|
||||
|
||||
- [x] 10.3 Add verification and summary reporting
|
||||
- After migration, query row counts for every table in both SQLite and Postgres
|
||||
- Print comparison table showing source count vs destination count
|
||||
- Flag any discrepancies with warnings
|
||||
- Exit with code 0 on success, code 1 on any errors
|
||||
- _Requirements: 7.8, 7.9_
|
||||
|
||||
- [ ]* 10.4 Write property test for migration data preservation (findings)
|
||||
- **Property 4: Migration Data Preservation (Findings)**
|
||||
- Generate random findings JSON with associated notes and overrides, run migration logic, verify merged output matches expected values
|
||||
- **Validates: Requirements 7.4, 7.5, 7.6**
|
||||
|
||||
- [ ]* 10.5 Write property test for migration table copy preservation
|
||||
- **Property 5: Migration Table Copy Preservation**
|
||||
- Generate random table rows, copy via migration logic, verify row counts and data equivalence (accounting for type conversions)
|
||||
- **Validates: Requirements 7.7, 7.8**
|
||||
|
||||
- [ ]* 10.6 Write property test for migration idempotence
|
||||
- **Property 6: Migration Idempotence**
|
||||
- Run migration logic N times on same input data, verify final Postgres state equals single-run state (no duplicates)
|
||||
- **Validates: Requirements 7.3, 7.9**
|
||||
|
||||
- [ ]* 10.7 Write property test for migration source safety
|
||||
- **Property 7: Migration Source Safety**
|
||||
- Checksum SQLite file before and after migration script execution, verify bytes unchanged
|
||||
- **Validates: Requirements 7.10**
|
||||
|
||||
- [ ] 11. Checkpoint — Verify data migration
|
||||
- Run migration script against test Postgres instance, verify all row counts match, verify findings have merged notes/overrides. Ask the user if questions arise.
|
||||
|
||||
- [ ] 12. Frontend updates for per-BU closed counts
|
||||
- [ ] 12.1 Update ReportingPage donut chart to show per-BU closed counts
|
||||
- Remove "N/A" fallback for closed count when BU filter is active
|
||||
- Display actual closed count from the updated `/api/ivanti/findings/counts` endpoint
|
||||
- Pass `teams` parameter to counts endpoint for server-side BU filtering
|
||||
- _Requirements: 4.3, 4.4_
|
||||
|
||||
- [ ]* 12.2 Write property test for API response shape preservation
|
||||
- **Property 9: API Response Shape Preservation**
|
||||
- For various valid API requests, verify response JSON structure (top-level keys and value types) matches expected contract
|
||||
- **Validates: Requirements 10.1**
|
||||
|
||||
- [ ] 13. Testing and verification
|
||||
- [ ] 13.1 Run backend on port 3003 and verify all endpoints
|
||||
- Set `PORT=3003` in test environment with `DATABASE_URL` pointing to Postgres
|
||||
- Verify auth endpoints (login, logout, me, password change)
|
||||
- Verify findings endpoints (list, counts, sync, notes, overrides)
|
||||
- Verify all other routes (CVEs, Jira, Archer, compliance, knowledge base, audit log)
|
||||
- Compare response shapes against production SQLite backend on port 3001
|
||||
- _Requirements: 9.1, 9.2, 10.1, 10.2, 10.3, 10.4, 11.1, 11.2, 11.3_
|
||||
|
||||
- [ ] 13.2 Performance verification
|
||||
- Confirm GET /api/ivanti/findings responds in <500ms with full dataset
|
||||
- Confirm GET /api/ivanti/findings/counts responds in <100ms
|
||||
- Confirm sync does not block concurrent read requests
|
||||
- _Requirements: 9.1, 9.2, 9.3, 9.4_
|
||||
|
||||
- [ ] 13.3 Run existing test suite against Postgres backend
|
||||
- Verify all existing property tests and unit tests pass
|
||||
- _Requirements: 10.1, 10.3_
|
||||
|
||||
- [ ] 14. Cutover to production
|
||||
- [ ] 14.1 Execute cutover procedure
|
||||
- Run final Ivanti sync on SQLite production backend
|
||||
- Run migration script to copy latest data to Postgres
|
||||
- Stop production backend on port 3001
|
||||
- Update production `.env` with `DATABASE_URL`
|
||||
- Start new backend on port 3001 with Postgres
|
||||
- Verify frontend loads and API responds correctly
|
||||
- _Requirements: 8.1, 8.2, 8.3_
|
||||
|
||||
- [ ] 14.2 Preserve SQLite backup and document rollback
|
||||
- Keep `backend/cve_database.db` as permanent backup (do not delete)
|
||||
- Document rollback procedure: stop backend → remove DATABASE_URL from .env → restart
|
||||
- _Requirements: 8.3, 8.4_
|
||||
|
||||
- [ ] 15. Cleanup
|
||||
- [ ] 15.1 Remove SQLite dependency and legacy code
|
||||
- Run `npm uninstall sqlite3` from `backend/`
|
||||
- Remove any remaining sqlite3 imports or `db` parameter passing
|
||||
- Remove the old `ivanti_findings_cache` and `ivanti_finding_notes` / `ivanti_finding_overrides` table references
|
||||
- Update `backend/.env.example` and README with Postgres prerequisites
|
||||
- _Requirements: 6.5, 11.4_
|
||||
|
||||
- [ ] 16. Final checkpoint — Ensure all tests pass
|
||||
- Ensure all tests pass, production is stable on Postgres, and SQLite backup is preserved. Ask the user if questions arise.
|
||||
|
||||
## Notes
|
||||
|
||||
- Tasks marked with `*` are optional and can be skipped for faster MVP
|
||||
- Each task references specific requirements for traceability
|
||||
- Checkpoints ensure incremental validation between major phases
|
||||
- Property tests validate universal correctness properties from the design document
|
||||
- All work on `feature/multi-tenancy` branch
|
||||
- Docker container `steam-postgres` on port 5433 (NOT 5432 — that belongs to another project)
|
||||
- Test backend on port 3003 during development; production stays on port 3001 with SQLite until cutover
|
||||
- The `pg` package connection pool handles concurrent reads during sync writes (MVCC)
|
||||
- Batch upserts in chunks of 100 for sync performance
|
||||
- SQLite file is never modified — opened read-only during migration
|
||||
Reference in New Issue
Block a user