jramos/cve-dashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update .kiro: remove SQLite hooks, add PostgreSQL migration hook, add workflow steering, sync specs

Browse Source
This commit is contained in:
Jordan Ramos
2026-05-12 14:45:58 -06:00
parent 3ee8487286
commit 1bb8ec1658
35 changed files with 4645 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

158
.kiro/specs/postgres-migration/requirements.md Normal file
View File

@@ -0,0 +1,158 @@
# Requirements Document
## Introduction
Migrate the CVE Dashboard (STEAM Security Dashboard) backend from SQLite to PostgreSQL 16. The current SQLite architecture stores all Ivanti findings as a single 2.6MB JSON blob (`ivanti_findings_cache.findings_json`) that must be parsed on every API request, causing 5-10 second load times. Additionally, SQLite's single-writer lock blocks reads during sync writes, and per-BU closed finding counts are unavailable (only a global count exists). PostgreSQL enables individual finding rows with indexed columns, per-BU open and closed counts, connection pooling for concurrent access, and proper type support. The Postgres instance runs in a dedicated Docker container on port 5433, isolated from the existing Postgres on port 5432 which belongs to another project.
## Glossary
- **SQLite**: The current embedded database engine storing all data in `backend/cve_database.db` (13MB total)
- **PostgreSQL_16**: The target relational database running in a Docker container (`steam-postgres`) on port 5433
- **findings_json**: The current TEXT column in `ivanti_findings_cache` storing all Ivanti findings as a serialized JSON array (2.6MB+)
- **ivanti_findings**: The new Postgres table storing each finding as an individual row with indexed columns
- **Pool**: A `pg` (node-postgres) connection pool managing up to 10 concurrent database connections
- **DATABASE_URL**: Environment variable containing the Postgres connection string (`postgresql://steam:<password>@localhost:5433/cve_dashboard`)
- **Migration_Script**: A one-time Node.js script (`backend/scripts/migrate-to-postgres.js`) that reads from SQLite and writes to Postgres
- **Cutover**: The moment production switches from SQLite backend on port 3001 to Postgres backend on port 3001
- **steam-postgres**: The Docker container name for the CVE Dashboard's dedicated PostgreSQL instance
- **steam-pgdata**: The Docker volume providing persistent storage for the Postgres data directory
- **bu_ownership**: The Ivanti finding field containing the BU assignment (e.g. "NTS-AEO-STEAM")
- **Upsert**: An INSERT that updates the existing row on primary key conflict (`INSERT ... ON CONFLICT DO UPDATE`)
## Requirements
### Requirement 1: Dedicated PostgreSQL Docker Container
**User Story:** As a system operator, I want a dedicated Postgres 16 container for the CVE Dashboard on port 5433, so that it is fully isolated from the existing Postgres instance on port 5432 belonging to another project.
#### Acceptance Criteria
1. THE Infrastructure SHALL run PostgreSQL 16 (Alpine) in a Docker container named `steam-postgres`
2. THE container SHALL map host port 5433 to container port 5432
3. THE container SHALL use a Docker volume named `steam-pgdata` for persistent data storage
4. THE container SHALL be configured with `--restart unless-stopped` for automatic recovery after host reboots
5. THE container SHALL create a database named `cve_dashboard` with a user named `steam`
6. THE Infrastructure SHALL NOT modify or affect the existing Postgres instance on port 5432
### Requirement 2: Schema Parity with Proper Types
**User Story:** As a developer, I want all existing SQLite tables recreated in Postgres with proper types and constraints, so that no functionality is lost during migration.
#### Acceptance Criteria
1. THE Postgres schema SHALL include all tables from the current SQLite schema: users, sessions, audit_logs, cves, documents, required_documents, jira_tickets, archer_tickets, knowledge_base, compliance_uploads, compliance_items, compliance_notes, ivanti_sync_state, ivanti_finding_notes, ivanti_counts_cache, ivanti_counts_history, ivanti_finding_overrides, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue, atlas_action_plans_cache
2. THE Postgres schema SHALL use appropriate types: SERIAL for auto-increment, TIMESTAMPTZ for timestamps, BOOLEAN for booleans, NUMERIC for decimals, TEXT[] for arrays, DATE for date-only fields
3. THE Postgres schema SHALL preserve all existing CHECK constraints and foreign key relationships
4. THE Postgres schema SHALL include the `bu_teams` column on the users table (required by multi-BU tenancy feature)
5. THE schema creation SHALL be idempotent using `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS`
### Requirement 3: Findings Table Redesign
**User Story:** As a user, I want findings stored as individual rows with indexed columns, so that filtering by BU ownership and state is instant instead of requiring full JSON blob parsing.
#### Acceptance Criteria
1. THE system SHALL replace the `ivanti_findings_cache.findings_json` blob with an `ivanti_findings` table containing one row per finding
2. THE `ivanti_findings` table SHALL include columns: id (TEXT PRIMARY KEY, Ivanti finding ID), host_id (INTEGER), title (TEXT), severity (NUMERIC), vrr_group (TEXT), host_name (TEXT), ip_address (TEXT), dns (TEXT), status (TEXT), sla_status (TEXT), due_date (DATE), last_found_on (DATE), bu_ownership (TEXT), cves (TEXT[] array), workflow_id (TEXT), workflow_state (TEXT), workflow_type (TEXT), state (TEXT with CHECK constraint for 'open' or 'closed'), note (TEXT), override_host_name (TEXT), override_dns (TEXT), synced_at (TIMESTAMPTZ), created_at (TIMESTAMPTZ)
3. THE system SHALL create indexes on: state, bu_ownership, severity, and a composite index on (state, bu_ownership)
4. THE system SHALL store both open AND closed findings as individual rows with their respective state values
5. WHEN findings are synced from Ivanti, THE system SHALL upsert rows using `INSERT ... ON CONFLICT (id) DO UPDATE` rather than replacing a JSON blob
### Requirement 4: Per-BU Closed Finding Counts
**User Story:** As a user viewing the Reporting page with a BU scope filter, I want accurate open and closed counts for my selected BUs, so that the dashboard shows meaningful per-team metrics instead of only a global count.
#### Acceptance Criteria
1. THE system SHALL store closed findings with their `bu_ownership` field preserved as individual rows with `state = 'closed'`
2. THE counts endpoint SHALL derive per-BU counts using `SELECT COUNT(*) ... WHERE state = ? AND bu_ownership ILIKE ?` queries
3. WHEN a teams filter is provided, THE counts endpoint SHALL return open and closed counts scoped to those BUs
4. WHEN no filter is applied, THE counts endpoint SHALL return global totals (backward compatible with current behavior)
5. THE system SHALL support aggregated counts grouped by bu_ownership and state in a single query
### Requirement 5: Connection Pooling
**User Story:** As a system operator, I want the backend to use connection pooling via the pg package, so that multiple concurrent requests are handled efficiently without blocking and reads are never blocked by writes.
#### Acceptance Criteria
1. THE backend SHALL use the `pg` npm package with a Pool instance (maximum pool size: 10 connections)
2. THE Pool SHALL read the connection string from the `DATABASE_URL` environment variable
3. IF the Pool detects connection errors, THEN THE Pool SHALL attempt automatic reconnection
4. THE Pool SHALL log a warning when active connections reach 8 (approaching exhaustion)
5. ALL database operations SHALL use async/await with the Pool (replacing all callback-based SQLite patterns)
### Requirement 6: Backend Route Migration
**User Story:** As a developer, I want all SQLite-specific code replaced with Postgres equivalents using async/await, so that the codebase uses a single database driver consistently.
#### Acceptance Criteria
1. THE backend SHALL replace all `db.get(sql, params, callback)` calls with `pool.query(sql, params)` returning `rows[0]`
2. THE backend SHALL replace all `db.all(sql, params, callback)` calls with `pool.query(sql, params)` returning `rows`
3. THE backend SHALL replace all `db.run(sql, params, callback)` calls with `pool.query(sql, params)` using `RETURNING` clauses where the inserted/updated row is needed
4. THE backend SHALL replace `?` parameter placeholders with `$1, $2, $3...` Postgres numbered parameter syntax
5. THE backend SHALL remove all callback-based database patterns in favor of async/await with try/catch error handling
6. THE Ivanti sync logic SHALL write individual finding rows via upsert instead of serializing to a JSON blob
### Requirement 7: Data Migration Script
**User Story:** As a system operator, I want a one-time migration script that copies all data from SQLite to Postgres, so that no data is lost during the transition.
#### Acceptance Criteria
1. THE Migration_Script SHALL open the SQLite database in read-only mode
2. THE Migration_Script SHALL connect to Postgres using the DATABASE_URL connection string
3. THE Migration_Script SHALL create all tables idempotently before inserting data
4. THE Migration_Script SHALL parse the `findings_json` blob and insert individual finding rows into the `ivanti_findings` table with `state = 'open'`
5. THE Migration_Script SHALL merge `ivanti_finding_notes` into the corresponding `ivanti_findings.note` column
6. THE Migration_Script SHALL merge `ivanti_finding_overrides` into the corresponding `ivanti_findings.override_*` columns
7. THE Migration_Script SHALL copy all other tables preserving their data: users, sessions, cves, documents, jira_tickets, archer_tickets, knowledge_base, audit_logs, compliance_uploads, compliance_items, compliance_notes, ivanti_counts_history, ivanti_finding_archives, ivanti_archive_transitions, ivanti_sync_anomaly_log, ivanti_finding_bu_history, atlas_action_plans_cache, ivanti_fp_submissions, ivanti_fp_submission_history, ivanti_todo_queue
8. THE Migration_Script SHALL verify row counts after migration and print a summary with any discrepancies
9. THE Migration_Script SHALL be idempotent (safe to run multiple times using ON CONFLICT/upsert logic)
10. THE Migration_Script SHALL never modify the SQLite database file
### Requirement 8: Zero-Impact Cutover
**User Story:** As a system operator, I want the cutover from SQLite to Postgres to take under 30 seconds with a clear rollback path, so that downtime is minimal and reversible.
#### Acceptance Criteria
1. THE cutover procedure SHALL consist of: stop old backend, start new backend on the same port (3001)
2. THE frontend SHALL NOT require any changes for the cutover (same API contract, same port, same URL)
3. IF issues are detected after cutover, THEN THE system SHALL support rollback by reverting the .env configuration and restarting the SQLite-based backend
4. THE SQLite database file SHALL be preserved indefinitely as a backup after cutover
5. THE cutover downtime SHALL not exceed 30 seconds
### Requirement 9: Performance Improvement
**User Story:** As a user, I want the dashboard to load findings in under 1 second regardless of BU scope, so that switching between BU views feels instant compared to the current 5-10 second load times.
#### Acceptance Criteria
1. THE `GET /api/ivanti/findings` endpoint SHALL respond in under 500ms for any BU filter combination
2. THE `GET /api/ivanti/findings/counts` endpoint SHALL respond in under 100ms for any BU filter
3. THE Ivanti sync process SHALL complete without blocking concurrent read requests (Postgres MVCC)
4. THE system SHALL eliminate JSON blob parsing from the findings read path entirely
### Requirement 10: API Backward Compatibility
**User Story:** As a developer, I want the API contract to remain unchanged after migration, so that the frontend works identically without code changes.
#### Acceptance Criteria
1. ALL existing API endpoints SHALL return the same response shape after migration
2. THE `GET /api/ivanti/findings` response SHALL include: findings array, total count, synced_at timestamp, sync_status, and error_message fields
3. THE authentication and session system SHALL work identically (cookie-based sessions, same expiration behavior)
4. THE frontend SHALL require zero code changes specifically for the database migration (multi-BU filtering changes are a separate feature)
### Requirement 11: Development and Testing Isolation
**User Story:** As a developer, I want to test the Postgres backend on port 3003 while production continues on port 3001 with SQLite, so that development does not disrupt the live system.
#### Acceptance Criteria
1. WHILE the migration is in development, THE test backend SHALL run on port 3003
2. WHILE the migration is in development, THE production backend SHALL continue running on port 3001 with SQLite
3. THE system SHALL support switching between SQLite and Postgres via environment variable configuration (DATABASE_URL presence or DB_TYPE flag)
4. ALL development work SHALL occur on the existing `feature/multi-tenancy` branch