cve-dashboard/.kiro/specs/postgres-migration/tasks.md

# Implementation Plan: PostgreSQL Migration

## Overview

Migrate the CVE Dashboard backend from SQLite to PostgreSQL 16. Replace the monolithic `findings_json` blob (2.6MB) with individual indexed rows in `ivanti_findings`, enable per-BU closed counts, and eliminate the single-writer lock. All work on the `feature/multi-tenancy` branch. Docker container `steam-postgres` on port 5433, test backend on port 3003, production on port 3001.

## Tasks

- [x] 1. Infrastructure setup and connection pool
  - [x] 1.1 Install `pg` dependency and configure environment
    - Run `npm install pg` in `backend/`
    - Add `DATABASE_URL=postgresql://steam:<password>@localhost:5433/cve_dashboard` to `backend/.env`
    - Add `DATABASE_URL` placeholder to `backend/.env.example`
    - _Requirements: 1.5, 5.1, 5.2_

  - [x] 1.2 Create `backend/db.js` connection pool module
    - Import `pg` and create a `Pool` instance reading from `DATABASE_URL`
    - Set `max: 10`, `idleTimeoutMillis: 30000`, `connectionTimeoutMillis: 5000`
    - Add `pool.on('error')` handler logging unexpected errors
    - Track active connections; log warning when count reaches 8
    - Export the pool instance for use by all route files
    - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5_

  - [x] 1.3 Create Docker run command documentation
    - Document the `docker run` command for `steam-postgres` container (port 5433:5432, volume `steam-pgdata`, `--restart unless-stopped`, Postgres 16 Alpine)
    - Verify container creates `cve_dashboard` database with `steam` user
    - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6_

- [x] 2. Schema creation
  - [x] 2.1 Create `backend/db-schema.sql` with complete Postgres DDL
    - Define all tables with proper Postgres types: SERIAL, TIMESTAMPTZ, BOOLEAN, NUMERIC, TEXT[], DATE
    - Include `ivanti_findings` table with TEXT PRIMARY KEY (`id`), all columns per design, CHECK constraint on `state`
    - Include `ivanti_sync_state` table (single-row pattern, replaces `ivanti_findings_cache` metadata)
    - Include all other tables: users (with `bu_teams`), sessions, audit_logs, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_cache, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue, atlas_action_plans_cache
    - Include all indexes: findings (state, bu_ownership, severity, state+bu_ownership composite), sessions (session_id, user_id, expires_at), audit_logs (created_at), todo_queue (user_id+status)
    - Include all foreign key relationships and CHECK constraints
    - Use `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS` for idempotence
    - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 3.1, 3.2, 3.3_

  - [ ]* 2.2 Write property test for schema creation idempotence
    - **Property 8: Schema Creation Idempotence**
    - Run schema DDL N times against the same database, verify no errors and same resulting schema each time
    - **Validates: Requirements 2.5**

  - [x] 2.3 Create schema initialization in `backend/setup.js`
    - Read `db-schema.sql` and execute via pool.query
    - Make callable on server startup or as standalone script
    - Seed `ivanti_sync_state` row (id=1) and `ivanti_counts_cache` row (id=1) if not exists
    - _Requirements: 2.1, 2.5_

- [x] 3. Checkpoint — Verify infrastructure
  - Ensure Docker container is running on port 5433, pool connects successfully, schema creates without errors. Ask the user if questions arise.

- [-] 4. Migrate auth and session system
  - [x] 4.1 Update `backend/middleware/auth.js`
    - Replace `db.get()` callback with `pool.query()` async/await
    - Change `?` placeholders to `$1, $2, ...` numbered params
    - Join sessions and users tables, check `expires_at > NOW()`
    - Return 401 for missing/expired sessions, 500 for query errors
    - _Requirements: 6.1, 6.4, 6.5, 10.3_

  - [x] 4.2 Update `backend/routes/auth.js`
    - Replace all `db.get/db.all/db.run` with `pool.query`
    - Login: query user by username, create session with `RETURNING`
    - Logout: delete session by session_id
    - Password change: update password_hash
    - Profile/me endpoint: query user by session
    - Use `$1, $2...` placeholders throughout
    - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 10.3_

  - [x] 4.3 Update `backend/routes/users.js`
    - Replace all sqlite3 calls with pool.query
    - CRUD operations: list users, create user, update user, delete user
    - Use `RETURNING` clause for inserts/updates where row data is needed
    - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_

  - [x] 4.4 Update `backend/server.js` database initialization
    - Remove sqlite3 database opening and `db` object creation
    - Import pool from `backend/db.js`
    - Remove passing `db` parameter to route factory functions
    - Update inline CVE/document/vendor routes to use pool.query
    - _Requirements: 6.1, 6.4, 6.5_

- [ ] 5. Checkpoint — Verify auth and core routes
  - Ensure login, logout, session validation, user CRUD, and CVE routes work on port 3003. Ask the user if questions arise.

- [-] 6. Migrate remaining route files
  - [x] 6.1 Update `backend/routes/jiraTickets.js`
    - Replace sqlite3 calls with pool.query, update placeholders
    - _Requirements: 6.1, 6.4, 6.5_

  - [x] 6.2 Update `backend/routes/archerTickets.js`
    - Replace sqlite3 calls with pool.query, update placeholders
    - _Requirements: 6.1, 6.4, 6.5_

  - [x] 6.3 Update `backend/routes/knowledgeBase.js`
    - Replace sqlite3 calls with pool.query, update placeholders
    - _Requirements: 6.1, 6.4, 6.5_

  - [x] 6.4 Update `backend/routes/compliance.js`
    - Replace sqlite3 calls with pool.query, update placeholders
    - Handle compliance_items, compliance_uploads, compliance_notes queries
    - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_

  - [x] 6.5 Update `backend/routes/auditLog.js` and `backend/helpers/auditLog.js`
    - Replace sqlite3 db.run/db.all with pool.query
    - Update the audit logging helper to use async pool.query
    - _Requirements: 6.1, 6.4, 6.5_

  - [x] 6.6 Update `backend/routes/ivantiWorkflows.js`
    - Replace sqlite3 calls with pool.query, update placeholders
    - _Requirements: 6.1, 6.4, 6.5_

  - [x] 6.7 Update `backend/routes/ivantiFpWorkflow.js`
    - Replace sqlite3 calls with pool.query, update placeholders
    - Handle fp_submissions and fp_submission_history tables
    - _Requirements: 6.1, 6.4, 6.5_

  - [x] 6.8 Update `backend/routes/ivantiTodoQueue.js`
    - Replace sqlite3 calls with pool.query, update placeholders
    - _Requirements: 6.1, 6.4, 6.5_

  - [x] 6.9 Update `backend/routes/ivantiArchive.js`
    - Replace sqlite3 calls with pool.query, update placeholders
    - Handle archive and transition table queries
    - _Requirements: 6.1, 6.4, 6.5_

  - [x] 6.10 Update `backend/routes/atlas.js`
    - Replace sqlite3 calls with pool.query, update placeholders
    - _Requirements: 6.1, 6.4, 6.5_

  - [x] 6.11 Update `backend/routes/cardApi.js` and `backend/routes/feedback.js`
    - Replace any sqlite3 calls with pool.query if present
    - _Requirements: 6.1, 6.4, 6.5_

- [ ] 7. Checkpoint — Verify all route migrations
  - Ensure all non-findings routes work correctly on port 3003. Ask the user if questions arise.

- [-] 8. Rewrite Ivanti findings sync and read logic
  - [x] 8.1 Rewrite sync logic in `backend/routes/ivantiFindings.js`
    - Replace JSON blob serialization with batch upsert of individual rows
    - Use `INSERT INTO ivanti_findings (...) VALUES ... ON CONFLICT (id) DO UPDATE SET ...` in batches of 100
    - Sync both open and closed findings as individual rows with correct `state` value
    - Update `ivanti_sync_state` with total count, synced_at, sync_status after sync
    - Preserve existing note and override values during upsert (do not overwrite user-set fields)
    - _Requirements: 3.4, 3.5, 6.6, 9.3_

  - [ ]* 8.2 Write property test for upsert idempotence
    - **Property 1: Upsert Idempotence**
    - Generate random findings, upsert each N times with same ID, verify exactly one row per ID with most recent data
    - **Validates: Requirements 3.5, 6.6**

  - [x] 8.3 Rewrite read endpoints in `backend/routes/ivantiFindings.js`
    - GET /findings: `SELECT * FROM ivanti_findings WHERE state = 'open'` with optional BU filter via `bu_ownership ILIKE`
    - Return response shape: `{ findings, total, synced_at, sync_status, error_message }`
    - GET /findings/counts: derive open/closed counts from `ivanti_findings` with optional BU filter
    - Support `teams` query parameter for per-BU scoped counts using `ILIKE ANY(ARRAY[...])`
    - GET /findings/counts/history: unchanged (reads from `ivanti_counts_history`)
    - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 9.1, 9.2, 10.1, 10.2_

  - [ ]* 8.4 Write property test for finding state and BU preservation
    - **Property 2: Finding Storage Preserves State and BU Ownership**
    - Generate findings with random state/bu_ownership, store and retrieve by ID, verify equality
    - **Validates: Requirements 3.4, 4.1**

  - [ ]* 8.5 Write property test for count query accuracy
    - **Property 3: Count Query Accuracy**
    - Generate random finding sets, insert into DB, run count queries with random BU filters, verify counts match actual row counts
    - **Validates: Requirements 4.2, 4.3, 4.5**

  - [x] 8.6 Update note and override endpoints
    - PUT /findings/:id/note: `UPDATE ivanti_findings SET note = $1 WHERE id = $2`
    - PUT /findings/:id/override: `UPDATE ivanti_findings SET override_host_name = $1, override_dns = $2 WHERE id = $3`
    - _Requirements: 6.1, 6.4_

- [ ] 9. Checkpoint — Verify findings redesign
  - Ensure sync creates individual rows, read endpoints return correct shape, counts work with and without BU filter. Ask the user if questions arise.

- [-] 10. Data migration script
  - [x] 10.1 Create `backend/scripts/migrate-to-postgres.js`
    - Open SQLite database in read-only mode (`OPEN_READONLY`)
    - Connect to Postgres via `DATABASE_URL`
    - Run schema creation (idempotent) before inserting data
    - Copy all simple tables: users, sessions, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, audit_logs, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, atlas_action_plans_cache, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue
    - Handle type conversions: SQLite 0/1 → Postgres boolean, DATETIME strings → TIMESTAMPTZ
    - Use batch inserts with `ON CONFLICT` for idempotency (safe to re-run)
    - _Requirements: 7.1, 7.2, 7.3, 7.7, 7.9, 7.10_

  - [x] 10.2 Implement findings migration with note/override merging
    - Parse `ivanti_findings_cache.findings_json` blob into individual objects
    - Insert each finding as a row in `ivanti_findings` with `state = 'open'`
    - Query `ivanti_finding_notes` and merge each note into the corresponding `ivanti_findings.note` column
    - Query `ivanti_finding_overrides` and merge into `ivanti_findings.override_host_name` / `override_dns`
    - _Requirements: 7.4, 7.5, 7.6_

  - [x] 10.3 Add verification and summary reporting
    - After migration, query row counts for every table in both SQLite and Postgres
    - Print comparison table showing source count vs destination count
    - Flag any discrepancies with warnings
    - Exit with code 0 on success, code 1 on any errors
    - _Requirements: 7.8, 7.9_

  - [ ]* 10.4 Write property test for migration data preservation (findings)
    - **Property 4: Migration Data Preservation (Findings)**
    - Generate random findings JSON with associated notes and overrides, run migration logic, verify merged output matches expected values
    - **Validates: Requirements 7.4, 7.5, 7.6**

  - [ ]* 10.5 Write property test for migration table copy preservation
    - **Property 5: Migration Table Copy Preservation**
    - Generate random table rows, copy via migration logic, verify row counts and data equivalence (accounting for type conversions)
    - **Validates: Requirements 7.7, 7.8**

  - [ ]* 10.6 Write property test for migration idempotence
    - **Property 6: Migration Idempotence**
    - Run migration logic N times on same input data, verify final Postgres state equals single-run state (no duplicates)
    - **Validates: Requirements 7.3, 7.9**

  - [ ]* 10.7 Write property test for migration source safety
    - **Property 7: Migration Source Safety**
    - Checksum SQLite file before and after migration script execution, verify bytes unchanged
    - **Validates: Requirements 7.10**

- [ ] 11. Checkpoint — Verify data migration
  - Run migration script against test Postgres instance, verify all row counts match, verify findings have merged notes/overrides. Ask the user if questions arise.

- [ ] 12. Frontend updates for per-BU closed counts
  - [ ] 12.1 Update ReportingPage donut chart to show per-BU closed counts
    - Remove "N/A" fallback for closed count when BU filter is active
    - Display actual closed count from the updated `/api/ivanti/findings/counts` endpoint
    - Pass `teams` parameter to counts endpoint for server-side BU filtering
    - _Requirements: 4.3, 4.4_

  - [ ]* 12.2 Write property test for API response shape preservation
    - **Property 9: API Response Shape Preservation**
    - For various valid API requests, verify response JSON structure (top-level keys and value types) matches expected contract
    - **Validates: Requirements 10.1**

- [ ] 13. Testing and verification
  - [ ] 13.1 Run backend on port 3003 and verify all endpoints
    - Set `PORT=3003` in test environment with `DATABASE_URL` pointing to Postgres
    - Verify auth endpoints (login, logout, me, password change)
    - Verify findings endpoints (list, counts, sync, notes, overrides)
    - Verify all other routes (CVEs, Jira, Archer, compliance, knowledge base, audit log)
    - Compare response shapes against production SQLite backend on port 3001
    - _Requirements: 9.1, 9.2, 10.1, 10.2, 10.3, 10.4, 11.1, 11.2, 11.3_

  - [ ] 13.2 Performance verification
    - Confirm GET /api/ivanti/findings responds in <500ms with full dataset
    - Confirm GET /api/ivanti/findings/counts responds in <100ms
    - Confirm sync does not block concurrent read requests
    - _Requirements: 9.1, 9.2, 9.3, 9.4_

  - [ ] 13.3 Run existing test suite against Postgres backend
    - Verify all existing property tests and unit tests pass
    - _Requirements: 10.1, 10.3_

- [ ] 14. Cutover to production
  - [ ] 14.1 Execute cutover procedure
    - Run final Ivanti sync on SQLite production backend
    - Run migration script to copy latest data to Postgres
    - Stop production backend on port 3001
    - Update production `.env` with `DATABASE_URL`
    - Start new backend on port 3001 with Postgres
    - Verify frontend loads and API responds correctly
    - _Requirements: 8.1, 8.2, 8.3_

  - [ ] 14.2 Preserve SQLite backup and document rollback
    - Keep `backend/cve_database.db` as permanent backup (do not delete)
    - Document rollback procedure: stop backend → remove DATABASE_URL from .env → restart
    - _Requirements: 8.3, 8.4_

- [ ] 15. Cleanup
  - [ ] 15.1 Remove SQLite dependency and legacy code
    - Run `npm uninstall sqlite3` from `backend/`
    - Remove any remaining sqlite3 imports or `db` parameter passing
    - Remove the old `ivanti_findings_cache` and `ivanti_finding_notes` / `ivanti_finding_overrides` table references
    - Update `backend/.env.example` and README with Postgres prerequisites
    - _Requirements: 6.5, 11.4_

- [ ] 16. Final checkpoint — Ensure all tests pass
  - Ensure all tests pass, production is stable on Postgres, and SQLite backup is preserved. Ask the user if questions arise.

## Notes

- Tasks marked with `*` are optional and can be skipped for faster MVP
- Each task references specific requirements for traceability
- Checkpoints ensure incremental validation between major phases
- Property tests validate universal correctness properties from the design document
- All work on `feature/multi-tenancy` branch
- Docker container `steam-postgres` on port 5433 (NOT 5432 — that belongs to another project)
- Test backend on port 3003 during development; production stays on port 3001 with SQLite until cutover
- The `pg` package connection pool handles concurrent reads during sync writes (MVCC)
- Batch upserts in chunks of 100 for sync performance
- SQLite file is never modified — opened read-only during migration