cve-dashboard/.kiro/specs/postgres-migration/requirements.md

Requirements Document

Introduction

Migrate the CVE Dashboard (STEAM Security Dashboard) backend from SQLite to PostgreSQL 16. The current SQLite architecture stores all Ivanti findings as a single 2.6MB JSON blob (ivanti_findings_cache.findings_json) that must be parsed on every API request, causing 5-10 second load times. Additionally, SQLite's single-writer lock blocks reads during sync writes, and per-BU closed finding counts are unavailable (only a global count exists). PostgreSQL enables individual finding rows with indexed columns, per-BU open and closed counts, connection pooling for concurrent access, and proper type support. The Postgres instance runs in a dedicated Docker container on port 5433, isolated from the existing Postgres on port 5432 which belongs to another project.

Glossary

• SQLite: The current embedded database engine storing all data in backend/cve_database.db (13MB total)
• PostgreSQL_16: The target relational database running in a Docker container (steam-postgres) on port 5433
• findings_json: The current TEXT column in ivanti_findings_cache storing all Ivanti findings as a serialized JSON array (2.6MB+)
• ivanti_findings: The new Postgres table storing each finding as an individual row with indexed columns
• Pool: A pg (node-postgres) connection pool managing up to 10 concurrent database connections
• DATABASE_URL: Environment variable containing the Postgres connection string (postgresql://steam:<password>@localhost:5433/cve_dashboard)
• Migration_Script: A one-time Node.js script (backend/scripts/migrate-to-postgres.js) that reads from SQLite and writes to Postgres
• Cutover: The moment production switches from SQLite backend on port 3001 to Postgres backend on port 3001
• steam-postgres: The Docker container name for the CVE Dashboard's dedicated PostgreSQL instance
• steam-pgdata: The Docker volume providing persistent storage for the Postgres data directory
• bu_ownership: The Ivanti finding field containing the BU assignment (e.g. "NTS-AEO-STEAM")
• Upsert: An INSERT that updates the existing row on primary key conflict (INSERT ... ON CONFLICT DO UPDATE)

Requirements

Requirement 1: Dedicated PostgreSQL Docker Container

User Story: As a system operator, I want a dedicated Postgres 16 container for the CVE Dashboard on port 5433, so that it is fully isolated from the existing Postgres instance on port 5432 belonging to another project.

Acceptance Criteria

1. THE Infrastructure SHALL run PostgreSQL 16 (Alpine) in a Docker container named steam-postgres
2. THE container SHALL map host port 5433 to container port 5432
3. THE container SHALL use a Docker volume named steam-pgdata for persistent data storage
4. THE container SHALL be configured with --restart unless-stopped for automatic recovery after host reboots
5. THE container SHALL create a database named cve_dashboard with a user named steam
6. THE Infrastructure SHALL NOT modify or affect the existing Postgres instance on port 5432

Requirement 2: Schema Parity with Proper Types

User Story: As a developer, I want all existing SQLite tables recreated in Postgres with proper types and constraints, so that no functionality is lost during migration.

Acceptance Criteria

1. THE Postgres schema SHALL include all tables from the current SQLite schema: users, sessions, audit_logs, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, compliance_uploads, compliance_items, compliance_notes, ivanti_sync_state, ivanti_finding_notes, ivanti_counts_cache, ivanti_counts_history, ivanti_finding_overrides, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue, atlas_action_plans_cache
2. THE Postgres schema SHALL use appropriate types: SERIAL for auto-increment, TIMESTAMPTZ for timestamps, BOOLEAN for booleans, NUMERIC for decimals, TEXT[] for arrays, DATE for date-only fields
3. THE Postgres schema SHALL preserve all existing CHECK constraints and foreign key relationships
4. THE Postgres schema SHALL include the bu_teams column on the users table (required by multi-BU tenancy feature)
5. THE schema creation SHALL be idempotent using CREATE TABLE IF NOT EXISTS and CREATE INDEX IF NOT EXISTS

Requirement 3: Findings Table Redesign

User Story: As a user, I want findings stored as individual rows with indexed columns, so that filtering by BU ownership and state is instant instead of requiring full JSON blob parsing.

Acceptance Criteria

1. THE system SHALL replace the ivanti_findings_cache.findings_json blob with an ivanti_findings table containing one row per finding
2. THE ivanti_findings table SHALL include columns: id (TEXT PRIMARY KEY, Ivanti finding ID), host_id (INTEGER), title (TEXT), severity (NUMERIC), vrr_group (TEXT), host_name (TEXT), ip_address (TEXT), dns (TEXT), status (TEXT), sla_status (TEXT), due_date (DATE), last_found_on (DATE), bu_ownership (TEXT), cves (TEXT[] array), workflow_id (TEXT), workflow_state (TEXT), workflow_type (TEXT), state (TEXT with CHECK constraint for 'open' or 'closed'), note (TEXT), override_host_name (TEXT), override_dns (TEXT), synced_at (TIMESTAMPTZ), created_at (TIMESTAMPTZ)
3. THE system SHALL create indexes on: state, bu_ownership, severity, and a composite index on (state, bu_ownership)
4. THE system SHALL store both open AND closed findings as individual rows with their respective state values
5. WHEN findings are synced from Ivanti, THE system SHALL upsert rows using INSERT ... ON CONFLICT (id) DO UPDATE rather than replacing a JSON blob

Requirement 4: Per-BU Closed Finding Counts

User Story: As a user viewing the Reporting page with a BU scope filter, I want accurate open and closed counts for my selected BUs, so that the dashboard shows meaningful per-team metrics instead of only a global count.

Acceptance Criteria

1. THE system SHALL store closed findings with their bu_ownership field preserved as individual rows with state = 'closed'
2. THE counts endpoint SHALL derive per-BU counts using SELECT COUNT(*) ... WHERE state = ? AND bu_ownership ILIKE ? queries
3. WHEN a teams filter is provided, THE counts endpoint SHALL return open and closed counts scoped to those BUs
4. WHEN no filter is applied, THE counts endpoint SHALL return global totals (backward compatible with current behavior)
5. THE system SHALL support aggregated counts grouped by bu_ownership and state in a single query

Requirement 5: Connection Pooling

User Story: As a system operator, I want the backend to use connection pooling via the pg package, so that multiple concurrent requests are handled efficiently without blocking and reads are never blocked by writes.

Acceptance Criteria

1. THE backend SHALL use the pg npm package with a Pool instance (maximum pool size: 10 connections)
2. THE Pool SHALL read the connection string from the DATABASE_URL environment variable
3. IF the Pool detects connection errors, THEN THE Pool SHALL attempt automatic reconnection
4. THE Pool SHALL log a warning when active connections reach 8 (approaching exhaustion)
5. ALL database operations SHALL use async/await with the Pool (replacing all callback-based SQLite patterns)

Requirement 6: Backend Route Migration

User Story: As a developer, I want all SQLite-specific code replaced with Postgres equivalents using async/await, so that the codebase uses a single database driver consistently.

Acceptance Criteria

1. THE backend SHALL replace all db.get(sql, params, callback) calls with pool.query(sql, params) returning rows[0]
2. THE backend SHALL replace all db.all(sql, params, callback) calls with pool.query(sql, params) returning rows
3. THE backend SHALL replace all db.run(sql, params, callback) calls with pool.query(sql, params) using RETURNING clauses where the inserted/updated row is needed
4. THE backend SHALL replace ? parameter placeholders with $1, $2, $3... Postgres numbered parameter syntax
5. THE backend SHALL remove all callback-based database patterns in favor of async/await with try/catch error handling
6. THE Ivanti sync logic SHALL write individual finding rows via upsert instead of serializing to a JSON blob

Requirement 7: Data Migration Script

User Story: As a system operator, I want a one-time migration script that copies all data from SQLite to Postgres, so that no data is lost during the transition.

Acceptance Criteria

1. THE Migration_Script SHALL open the SQLite database in read-only mode
2. THE Migration_Script SHALL connect to Postgres using the DATABASE_URL connection string
3. THE Migration_Script SHALL create all tables idempotently before inserting data
4. THE Migration_Script SHALL parse the findings_json blob and insert individual finding rows into the ivanti_findings table with state = 'open'
5. THE Migration_Script SHALL merge ivanti_finding_notes into the corresponding ivanti_findings.note column
6. THE Migration_Script SHALL merge ivanti_finding_overrides into the corresponding ivanti_findings.override_* columns
7. THE Migration_Script SHALL copy all other tables preserving their data: users, sessions, cves, documents, jira_tickets, archer_tickets, knowledge_base, audit_logs, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, atlas_action_plans_cache, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue
8. THE Migration_Script SHALL verify row counts after migration and print a summary with any discrepancies
9. THE Migration_Script SHALL be idempotent (safe to run multiple times using ON CONFLICT/upsert logic)
10. THE Migration_Script SHALL never modify the SQLite database file

Requirement 8: Zero-Impact Cutover

User Story: As a system operator, I want the cutover from SQLite to Postgres to take under 30 seconds with a clear rollback path, so that downtime is minimal and reversible.

Acceptance Criteria

1. THE cutover procedure SHALL consist of: stop old backend, start new backend on the same port (3001)
2. THE frontend SHALL NOT require any changes for the cutover (same API contract, same port, same URL)
3. IF issues are detected after cutover, THEN THE system SHALL support rollback by reverting the .env configuration and restarting the SQLite-based backend
4. THE SQLite database file SHALL be preserved indefinitely as a backup after cutover
5. THE cutover downtime SHALL not exceed 30 seconds

Requirement 9: Performance Improvement

User Story: As a user, I want the dashboard to load findings in under 1 second regardless of BU scope, so that switching between BU views feels instant compared to the current 5-10 second load times.

Acceptance Criteria

1. THE GET /api/ivanti/findings endpoint SHALL respond in under 500ms for any BU filter combination
2. THE GET /api/ivanti/findings/counts endpoint SHALL respond in under 100ms for any BU filter
3. THE Ivanti sync process SHALL complete without blocking concurrent read requests (Postgres MVCC)
4. THE system SHALL eliminate JSON blob parsing from the findings read path entirely

Requirement 10: API Backward Compatibility

User Story: As a developer, I want the API contract to remain unchanged after migration, so that the frontend works identically without code changes.

Acceptance Criteria

1. ALL existing API endpoints SHALL return the same response shape after migration
2. THE GET /api/ivanti/findings response SHALL include: findings array, total count, synced_at timestamp, sync_status, and error_message fields
3. THE authentication and session system SHALL work identically (cookie-based sessions, same expiration behavior)
4. THE frontend SHALL require zero code changes specifically for the database migration (multi-BU filtering changes are a separate feature)

Requirement 11: Development and Testing Isolation

User Story: As a developer, I want to test the Postgres backend on port 3003 while production continues on port 3001 with SQLite, so that development does not disrupt the live system.

Acceptance Criteria

1. WHILE the migration is in development, THE test backend SHALL run on port 3003
2. WHILE the migration is in development, THE production backend SHALL continue running on port 3001 with SQLite
3. THE system SHALL support switching between SQLite and Postgres via environment variable configuration (DATABASE_URL presence or DB_TYPE flag)
4. ALL development work SHALL occur on the existing feature/multi-tenancy branch