# Implementation Plan: PostgreSQL Migration ## Overview Migrate the CVE Dashboard backend from SQLite to PostgreSQL 16. Replace the monolithic `findings_json` blob (2.6MB) with individual indexed rows in `ivanti_findings`, enable per-BU closed counts, and eliminate the single-writer lock. All work on the `feature/multi-tenancy` branch. Docker container `steam-postgres` on port 5433, test backend on port 3003, production on port 3001. ## Tasks - [x] 1. Infrastructure setup and connection pool - [x] 1.1 Install `pg` dependency and configure environment - Run `npm install pg` in `backend/` - Add `DATABASE_URL=postgresql://steam:@localhost:5433/cve_dashboard` to `backend/.env` - Add `DATABASE_URL` placeholder to `backend/.env.example` - _Requirements: 1.5, 5.1, 5.2_ - [x] 1.2 Create `backend/db.js` connection pool module - Import `pg` and create a `Pool` instance reading from `DATABASE_URL` - Set `max: 10`, `idleTimeoutMillis: 30000`, `connectionTimeoutMillis: 5000` - Add `pool.on('error')` handler logging unexpected errors - Track active connections; log warning when count reaches 8 - Export the pool instance for use by all route files - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5_ - [x] 1.3 Create Docker run command documentation - Document the `docker run` command for `steam-postgres` container (port 5433:5432, volume `steam-pgdata`, `--restart unless-stopped`, Postgres 16 Alpine) - Verify container creates `cve_dashboard` database with `steam` user - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6_ - [x] 2. Schema creation - [x] 2.1 Create `backend/db-schema.sql` with complete Postgres DDL - Define all tables with proper Postgres types: SERIAL, TIMESTAMPTZ, BOOLEAN, NUMERIC, TEXT[], DATE - Include `ivanti_findings` table with TEXT PRIMARY KEY (`id`), all columns per design, CHECK constraint on `state` - Include `ivanti_sync_state` table (single-row pattern, replaces `ivanti_findings_cache` metadata) - Include all other tables: users (with `bu_teams`), sessions, audit_logs, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_cache, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue, atlas_action_plans_cache - Include all indexes: findings (state, bu_ownership, severity, state+bu_ownership composite), sessions (session_id, user_id, expires_at), audit_logs (created_at), todo_queue (user_id+status) - Include all foreign key relationships and CHECK constraints - Use `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS` for idempotence - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 3.1, 3.2, 3.3_ - [ ]* 2.2 Write property test for schema creation idempotence - **Property 8: Schema Creation Idempotence** - Run schema DDL N times against the same database, verify no errors and same resulting schema each time - **Validates: Requirements 2.5** - [x] 2.3 Create schema initialization in `backend/setup.js` - Read `db-schema.sql` and execute via pool.query - Make callable on server startup or as standalone script - Seed `ivanti_sync_state` row (id=1) and `ivanti_counts_cache` row (id=1) if not exists - _Requirements: 2.1, 2.5_ - [x] 3. Checkpoint — Verify infrastructure - Ensure Docker container is running on port 5433, pool connects successfully, schema creates without errors. Ask the user if questions arise. - [-] 4. Migrate auth and session system - [x] 4.1 Update `backend/middleware/auth.js` - Replace `db.get()` callback with `pool.query()` async/await - Change `?` placeholders to `$1, $2, ...` numbered params - Join sessions and users tables, check `expires_at > NOW()` - Return 401 for missing/expired sessions, 500 for query errors - _Requirements: 6.1, 6.4, 6.5, 10.3_ - [x] 4.2 Update `backend/routes/auth.js` - Replace all `db.get/db.all/db.run` with `pool.query` - Login: query user by username, create session with `RETURNING` - Logout: delete session by session_id - Password change: update password_hash - Profile/me endpoint: query user by session - Use `$1, $2...` placeholders throughout - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 10.3_ - [x] 4.3 Update `backend/routes/users.js` - Replace all sqlite3 calls with pool.query - CRUD operations: list users, create user, update user, delete user - Use `RETURNING` clause for inserts/updates where row data is needed - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_ - [x] 4.4 Update `backend/server.js` database initialization - Remove sqlite3 database opening and `db` object creation - Import pool from `backend/db.js` - Remove passing `db` parameter to route factory functions - Update inline CVE/document/vendor routes to use pool.query - _Requirements: 6.1, 6.4, 6.5_ - [ ] 5. Checkpoint — Verify auth and core routes - Ensure login, logout, session validation, user CRUD, and CVE routes work on port 3003. Ask the user if questions arise. - [-] 6. Migrate remaining route files - [x] 6.1 Update `backend/routes/jiraTickets.js` - Replace sqlite3 calls with pool.query, update placeholders - _Requirements: 6.1, 6.4, 6.5_ - [x] 6.2 Update `backend/routes/archerTickets.js` - Replace sqlite3 calls with pool.query, update placeholders - _Requirements: 6.1, 6.4, 6.5_ - [x] 6.3 Update `backend/routes/knowledgeBase.js` - Replace sqlite3 calls with pool.query, update placeholders - _Requirements: 6.1, 6.4, 6.5_ - [x] 6.4 Update `backend/routes/compliance.js` - Replace sqlite3 calls with pool.query, update placeholders - Handle compliance_items, compliance_uploads, compliance_notes queries - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_ - [x] 6.5 Update `backend/routes/auditLog.js` and `backend/helpers/auditLog.js` - Replace sqlite3 db.run/db.all with pool.query - Update the audit logging helper to use async pool.query - _Requirements: 6.1, 6.4, 6.5_ - [x] 6.6 Update `backend/routes/ivantiWorkflows.js` - Replace sqlite3 calls with pool.query, update placeholders - _Requirements: 6.1, 6.4, 6.5_ - [x] 6.7 Update `backend/routes/ivantiFpWorkflow.js` - Replace sqlite3 calls with pool.query, update placeholders - Handle fp_submissions and fp_submission_history tables - _Requirements: 6.1, 6.4, 6.5_ - [x] 6.8 Update `backend/routes/ivantiTodoQueue.js` - Replace sqlite3 calls with pool.query, update placeholders - _Requirements: 6.1, 6.4, 6.5_ - [x] 6.9 Update `backend/routes/ivantiArchive.js` - Replace sqlite3 calls with pool.query, update placeholders - Handle archive and transition table queries - _Requirements: 6.1, 6.4, 6.5_ - [x] 6.10 Update `backend/routes/atlas.js` - Replace sqlite3 calls with pool.query, update placeholders - _Requirements: 6.1, 6.4, 6.5_ - [x] 6.11 Update `backend/routes/cardApi.js` and `backend/routes/feedback.js` - Replace any sqlite3 calls with pool.query if present - _Requirements: 6.1, 6.4, 6.5_ - [ ] 7. Checkpoint — Verify all route migrations - Ensure all non-findings routes work correctly on port 3003. Ask the user if questions arise. - [-] 8. Rewrite Ivanti findings sync and read logic - [x] 8.1 Rewrite sync logic in `backend/routes/ivantiFindings.js` - Replace JSON blob serialization with batch upsert of individual rows - Use `INSERT INTO ivanti_findings (...) VALUES ... ON CONFLICT (id) DO UPDATE SET ...` in batches of 100 - Sync both open and closed findings as individual rows with correct `state` value - Update `ivanti_sync_state` with total count, synced_at, sync_status after sync - Preserve existing note and override values during upsert (do not overwrite user-set fields) - _Requirements: 3.4, 3.5, 6.6, 9.3_ - [ ]* 8.2 Write property test for upsert idempotence - **Property 1: Upsert Idempotence** - Generate random findings, upsert each N times with same ID, verify exactly one row per ID with most recent data - **Validates: Requirements 3.5, 6.6** - [x] 8.3 Rewrite read endpoints in `backend/routes/ivantiFindings.js` - GET /findings: `SELECT * FROM ivanti_findings WHERE state = 'open'` with optional BU filter via `bu_ownership ILIKE` - Return response shape: `{ findings, total, synced_at, sync_status, error_message }` - GET /findings/counts: derive open/closed counts from `ivanti_findings` with optional BU filter - Support `teams` query parameter for per-BU scoped counts using `ILIKE ANY(ARRAY[...])` - GET /findings/counts/history: unchanged (reads from `ivanti_counts_history`) - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 9.1, 9.2, 10.1, 10.2_ - [ ]* 8.4 Write property test for finding state and BU preservation - **Property 2: Finding Storage Preserves State and BU Ownership** - Generate findings with random state/bu_ownership, store and retrieve by ID, verify equality - **Validates: Requirements 3.4, 4.1** - [ ]* 8.5 Write property test for count query accuracy - **Property 3: Count Query Accuracy** - Generate random finding sets, insert into DB, run count queries with random BU filters, verify counts match actual row counts - **Validates: Requirements 4.2, 4.3, 4.5** - [x] 8.6 Update note and override endpoints - PUT /findings/:id/note: `UPDATE ivanti_findings SET note = $1 WHERE id = $2` - PUT /findings/:id/override: `UPDATE ivanti_findings SET override_host_name = $1, override_dns = $2 WHERE id = $3` - _Requirements: 6.1, 6.4_ - [ ] 9. Checkpoint — Verify findings redesign - Ensure sync creates individual rows, read endpoints return correct shape, counts work with and without BU filter. Ask the user if questions arise. - [-] 10. Data migration script - [x] 10.1 Create `backend/scripts/migrate-to-postgres.js` - Open SQLite database in read-only mode (`OPEN_READONLY`) - Connect to Postgres via `DATABASE_URL` - Run schema creation (idempotent) before inserting data - Copy all simple tables: users, sessions, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, audit_logs, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, atlas_action_plans_cache, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue - Handle type conversions: SQLite 0/1 → Postgres boolean, DATETIME strings → TIMESTAMPTZ - Use batch inserts with `ON CONFLICT` for idempotency (safe to re-run) - _Requirements: 7.1, 7.2, 7.3, 7.7, 7.9, 7.10_ - [x] 10.2 Implement findings migration with note/override merging - Parse `ivanti_findings_cache.findings_json` blob into individual objects - Insert each finding as a row in `ivanti_findings` with `state = 'open'` - Query `ivanti_finding_notes` and merge each note into the corresponding `ivanti_findings.note` column - Query `ivanti_finding_overrides` and merge into `ivanti_findings.override_host_name` / `override_dns` - _Requirements: 7.4, 7.5, 7.6_ - [x] 10.3 Add verification and summary reporting - After migration, query row counts for every table in both SQLite and Postgres - Print comparison table showing source count vs destination count - Flag any discrepancies with warnings - Exit with code 0 on success, code 1 on any errors - _Requirements: 7.8, 7.9_ - [ ]* 10.4 Write property test for migration data preservation (findings) - **Property 4: Migration Data Preservation (Findings)** - Generate random findings JSON with associated notes and overrides, run migration logic, verify merged output matches expected values - **Validates: Requirements 7.4, 7.5, 7.6** - [ ]* 10.5 Write property test for migration table copy preservation - **Property 5: Migration Table Copy Preservation** - Generate random table rows, copy via migration logic, verify row counts and data equivalence (accounting for type conversions) - **Validates: Requirements 7.7, 7.8** - [ ]* 10.6 Write property test for migration idempotence - **Property 6: Migration Idempotence** - Run migration logic N times on same input data, verify final Postgres state equals single-run state (no duplicates) - **Validates: Requirements 7.3, 7.9** - [ ]* 10.7 Write property test for migration source safety - **Property 7: Migration Source Safety** - Checksum SQLite file before and after migration script execution, verify bytes unchanged - **Validates: Requirements 7.10** - [ ] 11. Checkpoint — Verify data migration - Run migration script against test Postgres instance, verify all row counts match, verify findings have merged notes/overrides. Ask the user if questions arise. - [ ] 12. Frontend updates for per-BU closed counts - [ ] 12.1 Update ReportingPage donut chart to show per-BU closed counts - Remove "N/A" fallback for closed count when BU filter is active - Display actual closed count from the updated `/api/ivanti/findings/counts` endpoint - Pass `teams` parameter to counts endpoint for server-side BU filtering - _Requirements: 4.3, 4.4_ - [ ]* 12.2 Write property test for API response shape preservation - **Property 9: API Response Shape Preservation** - For various valid API requests, verify response JSON structure (top-level keys and value types) matches expected contract - **Validates: Requirements 10.1** - [ ] 13. Testing and verification - [ ] 13.1 Run backend on port 3003 and verify all endpoints - Set `PORT=3003` in test environment with `DATABASE_URL` pointing to Postgres - Verify auth endpoints (login, logout, me, password change) - Verify findings endpoints (list, counts, sync, notes, overrides) - Verify all other routes (CVEs, Jira, Archer, compliance, knowledge base, audit log) - Compare response shapes against production SQLite backend on port 3001 - _Requirements: 9.1, 9.2, 10.1, 10.2, 10.3, 10.4, 11.1, 11.2, 11.3_ - [ ] 13.2 Performance verification - Confirm GET /api/ivanti/findings responds in <500ms with full dataset - Confirm GET /api/ivanti/findings/counts responds in <100ms - Confirm sync does not block concurrent read requests - _Requirements: 9.1, 9.2, 9.3, 9.4_ - [ ] 13.3 Run existing test suite against Postgres backend - Verify all existing property tests and unit tests pass - _Requirements: 10.1, 10.3_ - [ ] 14. Cutover to production - [ ] 14.1 Execute cutover procedure - Run final Ivanti sync on SQLite production backend - Run migration script to copy latest data to Postgres - Stop production backend on port 3001 - Update production `.env` with `DATABASE_URL` - Start new backend on port 3001 with Postgres - Verify frontend loads and API responds correctly - _Requirements: 8.1, 8.2, 8.3_ - [ ] 14.2 Preserve SQLite backup and document rollback - Keep `backend/cve_database.db` as permanent backup (do not delete) - Document rollback procedure: stop backend → remove DATABASE_URL from .env → restart - _Requirements: 8.3, 8.4_ - [ ] 15. Cleanup - [ ] 15.1 Remove SQLite dependency and legacy code - Run `npm uninstall sqlite3` from `backend/` - Remove any remaining sqlite3 imports or `db` parameter passing - Remove the old `ivanti_findings_cache` and `ivanti_finding_notes` / `ivanti_finding_overrides` table references - Update `backend/.env.example` and README with Postgres prerequisites - _Requirements: 6.5, 11.4_ - [ ] 16. Final checkpoint — Ensure all tests pass - Ensure all tests pass, production is stable on Postgres, and SQLite backup is preserved. Ask the user if questions arise. ## Notes - Tasks marked with `*` are optional and can be skipped for faster MVP - Each task references specific requirements for traceability - Checkpoints ensure incremental validation between major phases - Property tests validate universal correctness properties from the design document - All work on `feature/multi-tenancy` branch - Docker container `steam-postgres` on port 5433 (NOT 5432 — that belongs to another project) - Test backend on port 3003 during development; production stays on port 3001 with SQLite until cutover - The `pg` package connection pool handles concurrent reads during sync writes (MVCC) - Batch upserts in chunks of 100 for sync performance - SQLite file is never modified — opened read-only during migration