cve-dashboard/docs/guides/postgres-migration-plan.md

Postgres Migration Plan

Overview

Migrate the STEAM Security Dashboard from SQLite (cve_database.db) to PostgreSQL. This eliminates the JSON blob performance bottleneck, enables per-BU closed finding counts, and supports the multi-tenancy feature properly.

Current State

• Database: SQLite 3, single file backend/cve_database.db (13MB)
• Performance bottleneck: ivanti_findings_cache.findings_json — a 2.6MB TEXT column holding all findings as serialized JSON, parsed on every API request
• Limitation: No per-BU closed finding data (only a global count)
• Concurrency: SQLite single-writer lock blocks reads during sync writes

Target State

• Database: PostgreSQL 16 (Docker container on port 5433)
• Findings storage: Individual rows in ivanti_findings table with indexed columns
• Closed findings: Stored as rows with state = 'closed' and bu_ownership column
• Per-BU counts: Simple SELECT COUNT(*) WHERE state = ? AND bu_ownership LIKE ?
• Concurrency: Connection pool (10 connections), reads never blocked by writes

Infrastructure

Port Allocation

Port	Service	Status
3000	Frontend (production)	In use
3001	Backend (production)	In use
3002	Other project (Python)	In use — do not touch
3003	Test backend (temporary, during migration dev)	Available
5000	Other project (Python)	In use — do not touch
5432	Other project (Postgres)	In use — do not touch
5433	CVE Dashboard Postgres (Docker)	Available — ours

Docker Setup

docker run -d --name steam-postgres \
  --restart unless-stopped \
  -e POSTGRES_DB=cve_dashboard \
  -e POSTGRES_USER=steam \
  -e POSTGRES_PASSWORD=<generated-password> \
  -p 5433:5432 \
  -v steam-pgdata:/var/lib/postgresql/data \
  postgres:16-alpine

Connection String

postgresql://steam:<password>@localhost:5433/cve_dashboard

Added to backend/.env as DATABASE_URL.

Migration Strategy

Approach: Blue-Green on Same Box

1. Production stays on SQLite (port 3001) throughout development
2. New Postgres backend tested on port 3003
3. Cutover: stop old backend, start new backend on port 3001
4. Rollback: stop new backend, start old SQLite backend

Branch Strategy

All work happens on feature/multi-tenancy branch (same branch as the multi-BU work). The Postgres migration is the infrastructure that makes multi-BU tenancy performant.

Schema Design

Key Changes from SQLite

SQLite	PostgreSQL
findings_json TEXT blob (2.6MB)	ivanti_findings table — one row per finding
Single ivanti_counts_cache row	Derived from ivanti_findings via queries
TEXT for everything	Proper types: INTEGER, NUMERIC, TIMESTAMPTZ, TEXT[]
No concurrent writes	Connection pool, MVCC
File-based	Docker volume steam-pgdata

New ivanti_findings Table

CREATE TABLE ivanti_findings (
    id TEXT PRIMARY KEY,              -- Ivanti finding ID
    host_id INTEGER,
    title TEXT NOT NULL DEFAULT '',
    severity NUMERIC(4,2) NOT NULL DEFAULT 0,
    vrr_group TEXT NOT NULL DEFAULT '',
    host_name TEXT NOT NULL DEFAULT '',
    ip_address TEXT NOT NULL DEFAULT '',
    dns TEXT NOT NULL DEFAULT '',
    status TEXT NOT NULL DEFAULT '',
    sla_status TEXT NOT NULL DEFAULT '',
    due_date DATE,
    last_found_on DATE,
    bu_ownership TEXT NOT NULL DEFAULT '',
    cves TEXT[] DEFAULT '{}',
    workflow_id TEXT,
    workflow_state TEXT,
    workflow_type TEXT,
    state TEXT NOT NULL DEFAULT 'open' CHECK (state IN ('open', 'closed')),
    note TEXT NOT NULL DEFAULT '',
    override_host_name TEXT,
    override_dns TEXT,
    synced_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_findings_state ON ivanti_findings(state);
CREATE INDEX idx_findings_bu ON ivanti_findings(bu_ownership);
CREATE INDEX idx_findings_severity ON ivanti_findings(severity);
CREATE INDEX idx_findings_state_bu ON ivanti_findings(state, bu_ownership);

Per-BU Counts (No Separate Table Needed)

-- Open count for STEAM
SELECT COUNT(*) FROM ivanti_findings WHERE state = 'open' AND bu_ownership ILIKE '%STEAM%';

-- Closed count for STEAM
SELECT COUNT(*) FROM ivanti_findings WHERE state = 'closed' AND bu_ownership ILIKE '%STEAM%';

-- All BU counts in one query
SELECT
    bu_ownership,
    state,
    COUNT(*) as count
FROM ivanti_findings
GROUP BY bu_ownership, state;

Data Migration Script

A one-time script (backend/scripts/migrate-to-postgres.js) that:

1. Opens the SQLite database (read-only)
2. Connects to Postgres
3. Creates all tables (idempotent — IF NOT EXISTS)
4. Copies data table by table:
   • users → users (direct copy + bu_teams)
   • sessions → sessions
   • cves → cves
   • documents → documents
   • jira_tickets → jira_tickets
   • archer_tickets → archer_tickets
   • knowledge_base → knowledge_base
   • audit_logs → audit_logs
   • compliance_uploads → compliance_uploads
   • compliance_items → compliance_items
   • compliance_notes → compliance_notes
   • ivanti_findings_cache.findings_json → individual rows in ivanti_findings (state='open')
   • ivanti_finding_notes → merged into ivanti_findings.note
   • ivanti_finding_overrides → merged into ivanti_findings.override_*
   • ivanti_counts_history → ivanti_counts_history
   • ivanti_finding_archives → ivanti_finding_archives
   • ivanti_archive_transitions → ivanti_archive_transitions
   • ivanti_sync_anomaly_log → ivanti_sync_anomaly_log
   • ivanti_finding_bu_history → ivanti_finding_bu_history
   • atlas_action_plans_cache → atlas_action_plans_cache
   • ivanti_fp_submissions → ivanti_fp_submissions
   • ivanti_fp_submission_history → ivanti_fp_submission_history
   • ivanti_todo_queue → ivanti_todo_queue
5. Verifies row counts match
6. Prints summary

Code Changes

Backend

1. New dependency: pg (node-postgres) replaces sqlite3
2. Connection pool: backend/db.js — creates and exports a Pool instance
3. Query pattern change:

   // Before (SQLite callback):
   db.get('SELECT * FROM users WHERE id = ?', [id], (err, row) => { ... });
   
   // After (Postgres async):
   const { rows } = await pool.query('SELECT * FROM users WHERE id = $1', [id]);
   const row = rows[0];
   

4. Placeholder syntax: ? → $1, $2, $3...
5. Findings sync: Write individual rows via INSERT ... ON CONFLICT (id) DO UPDATE
6. Closed findings sync: Same pattern — upsert with state = 'closed'
7. Counts: Derived queries instead of a cache table

Frontend

No changes needed — the API contract stays the same. The frontend already does client-side filtering.

Cutover Procedure

# 1. Final sync on production (SQLite) to get latest data
curl -X POST http://localhost:3001/api/ivanti/findings/sync

# 2. Run migration script (copies SQLite → Postgres)
node backend/scripts/migrate-to-postgres.js

# 3. Stop production backend
systemctl stop cve-backend  # or kill the process

# 4. Update .env to use Postgres
# DATABASE_URL=postgresql://steam:<pass>@localhost:5433/cve_dashboard
# DB_TYPE=postgres

# 5. Start new backend on same port
systemctl start cve-backend  # now uses Postgres

# 6. Verify
curl http://localhost:3001/api/auth/me  # should work

Rollback (if needed)

# 1. Stop new backend
systemctl stop cve-backend

# 2. Revert .env
# DB_TYPE=sqlite  (or remove DATABASE_URL)

# 3. Start old backend
systemctl start cve-backend

Timeline Estimate

Phase	Effort	Description
Docker setup	5 min	One command
Schema creation	1 hour	SQL DDL for all tables
DB abstraction layer	2-3 hours	backend/db.js pool + query helpers
Route migration	4-6 hours	Update all routes from sqlite3 callbacks to pg async
Findings redesign	2-3 hours	New sync logic writing individual rows
Closed findings	1-2 hours	Store closed findings, per-BU count queries
Data migration script	1-2 hours	SQLite → Postgres copy
Testing	2-3 hours	Verify all endpoints, sync, UI
Cutover	30 min	Stop/start + verify
Total	~15-20 hours

Risks and Mitigations

Risk	Mitigation
Docker container crashes	--restart unless-stopped flag
Data loss during cutover	SQLite file preserved as backup forever
Postgres disk fills up	Docker volume on main disk; monitor with df
Connection pool exhaustion	Pool max = 10, with queue; log warnings at 8
Migration script bugs	Run against dev DB first; verify row counts

Post-Migration Benefits

• Instant BU filtering: WHERE bu_ownership ILIKE '%STEAM%' on indexed column
• Per-BU closed counts: No more "N/A" — real numbers per team
• No JSON parsing: Findings are rows, not a blob
• Concurrent access: Multiple users can read while sync writes
• Future-proof: Easy to add full-text search, materialized views, partitioning