jramos/cve-dashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a61d254ff998e6243295bf5a413f75ab8e54c5d8
cve-dashboard/.kiro/specs/postgres-migration/tasks.md

16 KiB
Raw Blame History

Implementation Plan: PostgreSQL Migration

Overview

Migrate the CVE Dashboard backend from SQLite to PostgreSQL 16. Replace the monolithic findings_json blob (2.6MB) with individual indexed rows in ivanti_findings, enable per-BU closed counts, and eliminate the single-writer lock. All work on the feature/multi-tenancy branch. Docker container steam-postgres on port 5433, test backend on port 3003, production on port 3001.

Tasks

  • 1. Infrastructure setup and connection pool

    • 1.1 Install pg dependency and configure environment

      • Run npm install pg in backend/
      • Add DATABASE_URL=postgresql://steam:<password>@localhost:5433/cve_dashboard to backend/.env
      • Add DATABASE_URL placeholder to backend/.env.example
      • Requirements: 1.5, 5.1, 5.2

    • 1.2 Create backend/db.js connection pool module

      • Import pg and create a Pool instance reading from DATABASE_URL
      • Set max: 10, idleTimeoutMillis: 30000, connectionTimeoutMillis: 5000
      • Add pool.on('error') handler logging unexpected errors
      • Track active connections; log warning when count reaches 8
      • Export the pool instance for use by all route files
      • Requirements: 5.1, 5.2, 5.3, 5.4, 5.5

    • 1.3 Create Docker run command documentation

      • Document the docker run command for steam-postgres container (port 5433:5432, volume steam-pgdata, --restart unless-stopped, Postgres 16 Alpine)
      • Verify container creates cve_dashboard database with steam user
      • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6

  • 2. Schema creation

    • 2.1 Create backend/db-schema.sql with complete Postgres DDL

      • Define all tables with proper Postgres types: SERIAL, TIMESTAMPTZ, BOOLEAN, NUMERIC, TEXT[], DATE
      • Include ivanti_findings table with TEXT PRIMARY KEY (id), all columns per design, CHECK constraint on state
      • Include ivanti_sync_state table (single-row pattern, replaces ivanti_findings_cache metadata)
      • Include all other tables: users (with bu_teams), sessions, audit_logs, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_cache, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue, atlas_action_plans_cache
      • Include all indexes: findings (state, bu_ownership, severity, state+bu_ownership composite), sessions (session_id, user_id, expires_at), audit_logs (created_at), todo_queue (user_id+status)
      • Include all foreign key relationships and CHECK constraints
      • Use CREATE TABLE IF NOT EXISTS and CREATE INDEX IF NOT EXISTS for idempotence
      • Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 3.1, 3.2, 3.3

    • * 2.2 Write property test for schema creation idempotence

      • Property 8: Schema Creation Idempotence
      • Run schema DDL N times against the same database, verify no errors and same resulting schema each time
      • Validates: Requirements 2.5

    • 2.3 Create schema initialization in backend/setup.js

      • Read db-schema.sql and execute via pool.query
      • Make callable on server startup or as standalone script
      • Seed ivanti_sync_state row (id=1) and ivanti_counts_cache row (id=1) if not exists
      • Requirements: 2.1, 2.5

  • 3. Checkpoint — Verify infrastructure

    • Ensure Docker container is running on port 5433, pool connects successfully, schema creates without errors. Ask the user if questions arise.

  • [-] 4. Migrate auth and session system

    • 4.1 Update backend/middleware/auth.js

      • Replace db.get() callback with pool.query() async/await
      • Change ? placeholders to $1, $2, ... numbered params
      • Join sessions and users tables, check expires_at > NOW()
      • Return 401 for missing/expired sessions, 500 for query errors
      • Requirements: 6.1, 6.4, 6.5, 10.3

    • 4.2 Update backend/routes/auth.js

      • Replace all db.get/db.all/db.run with pool.query
      • Login: query user by username, create session with RETURNING
      • Logout: delete session by session_id
      • Password change: update password_hash
      • Profile/me endpoint: query user by session
      • Use $1, $2... placeholders throughout
      • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 10.3

    • 4.3 Update backend/routes/users.js

      • Replace all sqlite3 calls with pool.query
      • CRUD operations: list users, create user, update user, delete user
      • Use RETURNING clause for inserts/updates where row data is needed
      • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5

    • 4.4 Update backend/server.js database initialization

      • Remove sqlite3 database opening and db object creation
      • Import pool from backend/db.js
      • Remove passing db parameter to route factory functions
      • Update inline CVE/document/vendor routes to use pool.query
      • Requirements: 6.1, 6.4, 6.5

  • 5. Checkpoint — Verify auth and core routes

    • Ensure login, logout, session validation, user CRUD, and CVE routes work on port 3003. Ask the user if questions arise.

  • [-] 6. Migrate remaining route files

    • 6.1 Update backend/routes/jiraTickets.js

      • Replace sqlite3 calls with pool.query, update placeholders
      • Requirements: 6.1, 6.4, 6.5

    • 6.2 Update backend/routes/archerTickets.js

      • Replace sqlite3 calls with pool.query, update placeholders
      • Requirements: 6.1, 6.4, 6.5

    • 6.3 Update backend/routes/knowledgeBase.js

      • Replace sqlite3 calls with pool.query, update placeholders
      • Requirements: 6.1, 6.4, 6.5

    • 6.4 Update backend/routes/compliance.js

      • Replace sqlite3 calls with pool.query, update placeholders
      • Handle compliance_items, compliance_uploads, compliance_notes queries
      • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5

    • 6.5 Update backend/routes/auditLog.js and backend/helpers/auditLog.js

      • Replace sqlite3 db.run/db.all with pool.query
      • Update the audit logging helper to use async pool.query
      • Requirements: 6.1, 6.4, 6.5

    • 6.6 Update backend/routes/ivantiWorkflows.js

      • Replace sqlite3 calls with pool.query, update placeholders
      • Requirements: 6.1, 6.4, 6.5

    • 6.7 Update backend/routes/ivantiFpWorkflow.js

      • Replace sqlite3 calls with pool.query, update placeholders
      • Handle fp_submissions and fp_submission_history tables
      • Requirements: 6.1, 6.4, 6.5

    • 6.8 Update backend/routes/ivantiTodoQueue.js

      • Replace sqlite3 calls with pool.query, update placeholders
      • Requirements: 6.1, 6.4, 6.5

    • 6.9 Update backend/routes/ivantiArchive.js

      • Replace sqlite3 calls with pool.query, update placeholders
      • Handle archive and transition table queries
      • Requirements: 6.1, 6.4, 6.5

    • 6.10 Update backend/routes/atlas.js

      • Replace sqlite3 calls with pool.query, update placeholders
      • Requirements: 6.1, 6.4, 6.5

    • 6.11 Update backend/routes/cardApi.js and backend/routes/feedback.js

      • Replace any sqlite3 calls with pool.query if present
      • Requirements: 6.1, 6.4, 6.5

  • 7. Checkpoint — Verify all route migrations

    • Ensure all non-findings routes work correctly on port 3003. Ask the user if questions arise.

  • [-] 8. Rewrite Ivanti findings sync and read logic

    • 8.1 Rewrite sync logic in backend/routes/ivantiFindings.js

      • Replace JSON blob serialization with batch upsert of individual rows
      • Use INSERT INTO ivanti_findings (...) VALUES ... ON CONFLICT (id) DO UPDATE SET ... in batches of 100
      • Sync both open and closed findings as individual rows with correct state value
      • Update ivanti_sync_state with total count, synced_at, sync_status after sync
      • Preserve existing note and override values during upsert (do not overwrite user-set fields)
      • Requirements: 3.4, 3.5, 6.6, 9.3

    • * 8.2 Write property test for upsert idempotence

      • Property 1: Upsert Idempotence
      • Generate random findings, upsert each N times with same ID, verify exactly one row per ID with most recent data
      • Validates: Requirements 3.5, 6.6

    • 8.3 Rewrite read endpoints in backend/routes/ivantiFindings.js

      • GET /findings: SELECT * FROM ivanti_findings WHERE state = 'open' with optional BU filter via bu_ownership ILIKE
      • Return response shape: { findings, total, synced_at, sync_status, error_message }
      • GET /findings/counts: derive open/closed counts from ivanti_findings with optional BU filter
      • Support teams query parameter for per-BU scoped counts using ILIKE ANY(ARRAY[...])
      • GET /findings/counts/history: unchanged (reads from ivanti_counts_history)
      • Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 9.1, 9.2, 10.1, 10.2

    • * 8.4 Write property test for finding state and BU preservation

      • Property 2: Finding Storage Preserves State and BU Ownership
      • Generate findings with random state/bu_ownership, store and retrieve by ID, verify equality
      • Validates: Requirements 3.4, 4.1

    • * 8.5 Write property test for count query accuracy

      • Property 3: Count Query Accuracy
      • Generate random finding sets, insert into DB, run count queries with random BU filters, verify counts match actual row counts
      • Validates: Requirements 4.2, 4.3, 4.5

    • 8.6 Update note and override endpoints

      • PUT /findings/:id/note: UPDATE ivanti_findings SET note = $1 WHERE id = $2
      • PUT /findings/:id/override: UPDATE ivanti_findings SET override_host_name = $1, override_dns = $2 WHERE id = $3
      • Requirements: 6.1, 6.4

  • 9. Checkpoint — Verify findings redesign

    • Ensure sync creates individual rows, read endpoints return correct shape, counts work with and without BU filter. Ask the user if questions arise.

  • [-] 10. Data migration script

    • 10.1 Create backend/scripts/migrate-to-postgres.js

      • Open SQLite database in read-only mode (OPEN_READONLY)
      • Connect to Postgres via DATABASE_URL
      • Run schema creation (idempotent) before inserting data
      • Copy all simple tables: users, sessions, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, audit_logs, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, atlas_action_plans_cache, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue
      • Handle type conversions: SQLite 0/1 → Postgres boolean, DATETIME strings → TIMESTAMPTZ
      • Use batch inserts with ON CONFLICT for idempotency (safe to re-run)
      • Requirements: 7.1, 7.2, 7.3, 7.7, 7.9, 7.10

    • 10.2 Implement findings migration with note/override merging

      • Parse ivanti_findings_cache.findings_json blob into individual objects
      • Insert each finding as a row in ivanti_findings with state = 'open'
      • Query ivanti_finding_notes and merge each note into the corresponding ivanti_findings.note column
      • Query ivanti_finding_overrides and merge into ivanti_findings.override_host_name / override_dns
      • Requirements: 7.4, 7.5, 7.6

    • 10.3 Add verification and summary reporting

      • After migration, query row counts for every table in both SQLite and Postgres
      • Print comparison table showing source count vs destination count
      • Flag any discrepancies with warnings
      • Exit with code 0 on success, code 1 on any errors
      • Requirements: 7.8, 7.9

    • * 10.4 Write property test for migration data preservation (findings)

      • Property 4: Migration Data Preservation (Findings)
      • Generate random findings JSON with associated notes and overrides, run migration logic, verify merged output matches expected values
      • Validates: Requirements 7.4, 7.5, 7.6

    • * 10.5 Write property test for migration table copy preservation

      • Property 5: Migration Table Copy Preservation
      • Generate random table rows, copy via migration logic, verify row counts and data equivalence (accounting for type conversions)
      • Validates: Requirements 7.7, 7.8

    • * 10.6 Write property test for migration idempotence

      • Property 6: Migration Idempotence
      • Run migration logic N times on same input data, verify final Postgres state equals single-run state (no duplicates)
      • Validates: Requirements 7.3, 7.9

    • * 10.7 Write property test for migration source safety

      • Property 7: Migration Source Safety
      • Checksum SQLite file before and after migration script execution, verify bytes unchanged
      • Validates: Requirements 7.10

  • 11. Checkpoint — Verify data migration

    • Run migration script against test Postgres instance, verify all row counts match, verify findings have merged notes/overrides. Ask the user if questions arise.

  • 12. Frontend updates for per-BU closed counts

    • 12.1 Update ReportingPage donut chart to show per-BU closed counts

      • Remove "N/A" fallback for closed count when BU filter is active
      • Display actual closed count from the updated /api/ivanti/findings/counts endpoint
      • Pass teams parameter to counts endpoint for server-side BU filtering
      • Requirements: 4.3, 4.4

    • * 12.2 Write property test for API response shape preservation

      • Property 9: API Response Shape Preservation
      • For various valid API requests, verify response JSON structure (top-level keys and value types) matches expected contract
      • Validates: Requirements 10.1

  • 13. Testing and verification

    • 13.1 Run backend on port 3003 and verify all endpoints

      • Set PORT=3003 in test environment with DATABASE_URL pointing to Postgres
      • Verify auth endpoints (login, logout, me, password change)
      • Verify findings endpoints (list, counts, sync, notes, overrides)
      • Verify all other routes (CVEs, Jira, Archer, compliance, knowledge base, audit log)
      • Compare response shapes against production SQLite backend on port 3001
      • Requirements: 9.1, 9.2, 10.1, 10.2, 10.3, 10.4, 11.1, 11.2, 11.3

    • 13.2 Performance verification

      • Confirm GET /api/ivanti/findings responds in <500ms with full dataset
      • Confirm GET /api/ivanti/findings/counts responds in <100ms
      • Confirm sync does not block concurrent read requests
      • Requirements: 9.1, 9.2, 9.3, 9.4

    • 13.3 Run existing test suite against Postgres backend

      • Verify all existing property tests and unit tests pass
      • Requirements: 10.1, 10.3

  • 14. Cutover to production

    • 14.1 Execute cutover procedure

      • Run final Ivanti sync on SQLite production backend
      • Run migration script to copy latest data to Postgres
      • Stop production backend on port 3001
      • Update production .env with DATABASE_URL
      • Start new backend on port 3001 with Postgres
      • Verify frontend loads and API responds correctly
      • Requirements: 8.1, 8.2, 8.3

    • 14.2 Preserve SQLite backup and document rollback

      • Keep backend/cve_database.db as permanent backup (do not delete)
      • Document rollback procedure: stop backend → remove DATABASE_URL from .env → restart
      • Requirements: 8.3, 8.4

  • 15. Cleanup

    • 15.1 Remove SQLite dependency and legacy code
      • Run npm uninstall sqlite3 from backend/
      • Remove any remaining sqlite3 imports or db parameter passing
      • Remove the old ivanti_findings_cache and ivanti_finding_notes / ivanti_finding_overrides table references
      • Update backend/.env.example and README with Postgres prerequisites
      • Requirements: 6.5, 11.4

  • 16. Final checkpoint — Ensure all tests pass

    • Ensure all tests pass, production is stable on Postgres, and SQLite backup is preserved. Ask the user if questions arise.

Notes

  • Tasks marked with * are optional and can be skipped for faster MVP
  • Each task references specific requirements for traceability
  • Checkpoints ensure incremental validation between major phases
  • Property tests validate universal correctness properties from the design document
  • All work on feature/multi-tenancy branch
  • Docker container steam-postgres on port 5433 (NOT 5432 — that belongs to another project)
  • Test backend on port 3003 during development; production stays on port 3001 with SQLite until cutover
  • The pg package connection pool handles concurrent reads during sync writes (MVCC)
  • Batch upserts in chunks of 100 for sync performance
  • SQLite file is never modified — opened read-only during migration
Reference in New Issue View Git Blame Copy Permalink