docs: update README and reference manual for PostgreSQL migration and systemd scripts

2026-05-08 09:17:38 -06:00
parent f657351219
commit 86fdd084ac
2 changed files with 75 additions and 60 deletions
--- a/README.md
+++ b/README.md
@@ -7,6 +7,7 @@ A self-hosted vulnerability management dashboard for the NTS-AEO-STEAM and NTS-A
 ### Prerequisites

 - Node.js 18+
+- Docker (for PostgreSQL 16 container)
 - Python 3 with `python3-pandas` and `python3-openpyxl` (for compliance xlsx parsing)

 ### Install
@@ -29,19 +30,22 @@ apt install -y python3-pandas python3-openpyxl

 ```bash
 cp backend/.env.example backend/.env
-# Edit backend/.env — at minimum set SESSION_SECRET:
+# Edit backend/.env — at minimum set SESSION_SECRET and DATABASE_URL:
 #   openssl rand -base64 32
 ```

-See `backend/.env.example` for all available options including Ivanti API, Jira, and Atlas integration keys.
+See `backend/.env.example` for all available options including `DATABASE_URL`, Ivanti API, Jira, and Atlas integration keys.

-### Initialize Database
+### Start PostgreSQL
+
+The deploy script handles the full Postgres setup — container, schema, dependencies, and data migration from SQLite:

 ```bash
-node backend/setup.js
+chmod +x scripts/deploy-postgres.sh
+./scripts/deploy-postgres.sh
 ```

-Creates the database with the complete schema and prints a one-time admin password. Save it.
+For fresh installs without an existing SQLite database, the script creates the schema and skips migration.

 ### Build and Run

@@ -55,7 +59,7 @@ cd frontend && npm run build && cd ..

 Dashboard: http://localhost:3000 · API: http://localhost:3001

-For persistent deployments, use the systemd services in `systemd/`. See the full manual for setup instructions.
+The helper scripts use `systemctl` under the hood — the systemd units in `systemd/` must be installed first. See the full manual for setup instructions.

 ## Features

@@ -80,11 +84,13 @@ For persistent deployments, use the systemd services in `systemd/`. See the full
 cve-dashboard/
 ├── backend/
 │   ├── server.js              # Express API server
-│   ├── setup.js               # Database initialization (run once)
+│   ├── db.js                  # PostgreSQL connection pool (pg)
+│   ├── db-schema.sql          # Complete DDL for fresh Postgres setup
+│   ├── setup-postgres.js      # Schema initializer (runs db-schema.sql)
 │   ├── routes/                # API route handlers
 │   ├── helpers/               # API clients (Ivanti, Jira, Atlas, CARD)
 │   ├── middleware/            # Auth middleware
-│   ├── migrations/            # Schema migrations (for existing deployments)
+│   ├── migrations/            # Schema migrations (legacy SQLite deployments)
 │   └── scripts/               # Compliance parser, data import utilities
 ├── frontend/
 │   ├── src/
@@ -99,6 +105,9 @@ cve-dashboard/
 │   ├── security/              # Security audits and remediation plans
 │   ├── testing/               # Test plans and scripts
 │   └── troubleshooting/       # Investigation scripts and reports
+├── docker-compose.yml         # PostgreSQL 16 container definition
+├── scripts/
+│   └── deploy-postgres.sh     # One-time deployment: container, schema, migration
 ├── systemd/                   # systemd service files
 ├── start-servers.sh
 └── stop-servers.sh
@@ -108,7 +117,8 @@ cve-dashboard/

 | Layer | Technology |
 |-------|------------|
-| Backend | Node.js 18+, Express 5, SQLite3 |
+| Backend | Node.js 18+, Express 5 |
+| Database | PostgreSQL 16 (Docker, port 5433) |
 | Frontend | React 19, Recharts, Lucide React |
 | Auth | bcryptjs, cookie-based sessions, express-rate-limit |
 | Compliance | Python 3, pandas, openpyxl |
@@ -116,6 +126,7 @@ cve-dashboard/
 ## Documentation

 - **[Full Reference Manual](docs/guides/full-reference-manual.md)** — comprehensive feature documentation, API reference, database schema, security model, and configuration details
+- **[Postgres Migration Plan](docs/guides/postgres-migration-plan.md)** — architecture decisions, schema design, and cutover procedure for the SQLite to PostgreSQL migration
 - **[Migration Guide](backend/migrations/README.md)** — schema migration scripts for upgrading existing deployments
 - **[Design System](docs/design/design-system.md)** — UI component patterns and color system
 - **[Ivanti API Reference](docs/api/ivanti-api-reference.md)** — Ivanti/RiskSense API integration details
--- a/docs/guides/full-reference-manual.md
+++ b/docs/guides/full-reference-manual.md
@@ -58,7 +58,7 @@ The application provides:
 | Layer | Technology |
 |---|---|
 | Backend | Node.js 18+, Express 5 |
-| Database | SQLite3 |
+| Database | PostgreSQL 16 (Docker container on port 5433, `pg` driver) |
 | File uploads | Multer 2 |
 | Auth | bcryptjs, cookie-based sessions, express-rate-limit |
 | Frontend | React 19, lucide-react, recharts, xlsx, react-markdown, rehype-sanitize, mermaid |
@@ -70,6 +70,7 @@ The application provides:

 - Node.js 18 or later
 - npm
+- Docker (for the PostgreSQL 16 container)
 - Python 3 with `python3-pandas` and `python3-openpyxl` apt packages (required for compliance xlsx parsing)

 ---
@@ -121,17 +122,26 @@ cp .env.example .env

 See [Configuration](#configuration) for all available options.

-### 6. Initialize the database
+### 6. Deploy PostgreSQL and initialize the database

-Run once from the project root to create the SQLite database, all tables, indexes, triggers, and a default admin user:
+The deploy script handles the full setup — starts the Postgres container, creates the schema, installs the `pg` dependency, migrates data from SQLite (if present), and builds the frontend:

 ```bash
-node backend/setup.js
+chmod +x scripts/deploy-postgres.sh
+./scripts/deploy-postgres.sh
 ```

-This creates `backend/cve_database.db` with the complete v1.0.0 schema and generates a random admin password printed to stdout. **Save the password — it is only shown once.**
+This starts a PostgreSQL 16 container (`steam-postgres`) on port 5433 with a persistent Docker volume, then runs `backend/db-schema.sql` to create all tables, indexes, and views.

-> **Existing deployments:** If upgrading from a pre-v1.0.0 database, run the individual migration scripts in `backend/migrations/` instead. See `backend/migrations/README.md` for the full list and order.
+For manual setup or troubleshooting, the individual steps are:
+
+```bash
+docker compose up -d                          # Start Postgres container
+node backend/setup-postgres.js                # Run schema DDL
+node backend/scripts/migrate-to-postgres.js   # Migrate data from SQLite (if upgrading)
+```
+
+> **Existing deployments:** If upgrading from SQLite, the deploy script automatically runs the data migration. The original `backend/cve_database.db` file is preserved as a backup. See [Postgres Migration Plan](docs/guides/postgres-migration-plan.md) for full details.

 ### 7. Build the frontend

@@ -157,6 +167,9 @@ CORS_ORIGINS=http://YOUR_IP:3000
 SESSION_SECRET=<generate with: openssl rand -base64 32>
 # NODE_ENV=production  — see note below

+# PostgreSQL connection (required)
+DATABASE_URL=postgresql://steam:<password>@localhost:5433/cve_dashboard
+
 # Optional: NVD API key for higher rate limits (50 req/30s vs 5 req/30s)
 # Register at https://nvd.nist.gov/developers/request-an-api-key
 NVD_API_KEY=your-key-here
@@ -187,6 +200,8 @@ JIRA_SKIP_TLS=false

 **`SESSION_SECRET` is required.** The server will exit on startup if it is not set. Generate one with `openssl rand -base64 32`.

+**`DATABASE_URL` is required.** The backend connects to PostgreSQL via this connection string. Format: `postgresql://user:password@host:port/database`. The deploy script adds this automatically.
+
 **`NODE_ENV` and the Secure cookie flag:** When `NODE_ENV=production`, session cookies are set with the `Secure` flag, which means the browser will only send them over HTTPS connections. If you are running the application over plain HTTP (no TLS/SSL), you **must** leave `NODE_ENV` unset or set it to `development` — otherwise login will succeed but every subsequent API request will return 401 because the browser silently drops the cookie. Only set `NODE_ENV=production` when the application is served behind HTTPS (e.g., via a reverse proxy with TLS termination).

 ### Frontend: `frontend/.env`
@@ -209,11 +224,11 @@ Replace `YOUR_IP` with the machine's IP address or hostname. Use `localhost` for
 From the project root:

 ```bash
-./start-servers.sh   # Starts backend and frontend in the background
-./stop-servers.sh    # Stops all servers
+./start-servers.sh   # Starts backend and frontend via systemd
+./stop-servers.sh    # Stops both services
 ```

-The start script saves PIDs to `backend.pid` and `frontend.pid`. Logs are written to `backend/backend.log` and `frontend/frontend.log`.
+Both scripts call `systemctl start` / `systemctl stop` on the `cve-backend` and `cve-frontend` services. The systemd units must be installed first — see [Running as systemd services](#running-as-systemd-services-auto-start-on-reboot) for setup.

 ### Running manually

@@ -261,7 +276,7 @@ journalctl -u cve-backend -f                  # Follow backend journal
 journalctl -u cve-frontend -f                 # Follow frontend journal
 ```

-> The helper scripts (`start-servers.sh` / `stop-servers.sh`) still work for ad-hoc use, but systemd is the recommended approach for persistent deployments. If switching to systemd, stop any script-launched processes first with `./stop-servers.sh` to avoid port conflicts.
+> The helper scripts (`start-servers.sh` / `stop-servers.sh`) are thin wrappers around `systemctl start` / `systemctl stop`. They require the systemd units to be installed and enabled as described above.

 ### Default ports

@@ -347,7 +362,7 @@ Click **Sync** (top right) to pull the latest findings from Ivanti. Sync require
 1. Fetches all open host findings matching your BU filters and severity range (8.5–9.9 VRR)
 2. Fetches the closed finding count separately
 3. Sweeps closed findings to capture FP workflow states (including Approved FPs now closed)
-4. Stores everything in the local SQLite cache
+4. Stores findings as individual rows in the PostgreSQL `ivanti_findings` table

 Findings are also auto-synced on a 24-hour schedule. The last sync timestamp is shown at the top of the page.

@@ -751,17 +766,21 @@ All endpoints are prefixed with `/api`. All endpoints except `/api/auth/login` a

 ```
 cve-dashboard/
-├── start-servers.sh          # Start backend + frontend in background
-├── stop-servers.sh           # Stop all servers
+├── start-servers.sh          # Start backend + frontend via systemd
+├── stop-servers.sh           # Stop both systemd services
+├── docker-compose.yml        # PostgreSQL 16 container definition
 ├── package.json              # Root package.json (backend dependencies)
+├── scripts/
+│   └── deploy-postgres.sh    # One-time deployment: container, schema, migration
 ├── systemd/                  # systemd unit files for auto-start on boot
 │   ├── cve-backend.service
 │   └── cve-frontend.service
 │
 ├── backend/
 │   ├── server.js             # Express app — routes, middleware, security headers
-│   ├── setup.js              # One-time DB initialization and default admin creation
-│   ├── cve_database.db       # SQLite database (gitignored)
+│   ├── db.js                 # PostgreSQL connection pool (pg)
+│   ├── db-schema.sql         # Complete DDL for fresh Postgres setup
+│   ├── setup-postgres.js     # Schema initializer (runs db-schema.sql)
 │   ├── uploads/              # File storage root (gitignored)
 │   │   ├── <CVE-ID>/<vendor>/    # CVE documents
 │   │   ├── knowledge_base/       # Knowledge base documents
@@ -786,8 +805,9 @@ cve-dashboard/
 │   │   ├── driftChecker.js       # Schema drift detection: compareSchemaToDrift(), loadConfig(), reconcileConfig()
 │   │   ├── ivantiApi.js          # Ivanti API HTTP helpers (multipart, JSON, form POST)
 │   │   └── jiraApi.js            # Jira Data Center REST API helpers (Basic/PAT auth, rate limiting)
-│   ├── migrations/               # Sequential migration scripts (run manually with node)
+│   ├── migrations/               # Legacy SQLite migration scripts (not needed for Postgres)
 │   └── scripts/
+│       ├── migrate-to-postgres.js        # One-time SQLite → Postgres data migration
 │       ├── compliance_config.json        # Shared parser config (metric_categories, core_cols, skip_sheets)
 │       ├── extract_xlsx_schema.py        # Extracts xlsx structure as JSON for drift checking
 │       ├── parse_compliance_xlsx.py      # Parses NTS_AEO xlsx compliance reports
@@ -831,7 +851,9 @@ cve-dashboard/

 ## Database Schema

-### Core tables (created by `setup.js`)
+All tables are defined in `backend/db-schema.sql` and created by `setup-postgres.js`. The database runs in a PostgreSQL 16 Docker container (`steam-postgres`) on port 5433.
+
+### Core tables

 **`cves`** — One row per CVE/vendor pair. `UNIQUE(cve_id, vendor)`. Includes `created_by` column for ownership tracking.

@@ -839,13 +861,13 @@ cve-dashboard/

 **`required_documents`** — Vendor-specific document requirements.

-**`users`** — Accounts with group-based access control. `user_group` column with values: `Admin`, `Standard_User`, `Leadership`, `Read_Only`. Enforced by INSERT/UPDATE triggers. Legacy `role` column retained for rollback safety.
+**`users`** — Accounts with group-based access control. `user_group` column with values: `Admin`, `Standard_User`, `Leadership`, `Read_Only`. Includes `bu_teams` column for multi-BU tenancy scoping.

 **`sessions`** — Active sessions with 24-hour expiry.

 **`audit_logs`** — Append-only log of all state-changing actions.

-### Feature tables (added by migrations)
+### Feature tables

 **`knowledge_base`** — Document library entries with title, slug, category, description, file metadata, and `created_by`.

@@ -855,11 +877,9 @@ cve-dashboard/

 **`ivanti_sync_state`** — Single-row cache for Ivanti workflow batch data.

-**`ivanti_findings_cache`** — Single-row cache for Ivanti host findings.
+**`ivanti_findings`** — One row per Ivanti host finding. Indexed on `state`, `bu_ownership`, `severity`, and `(state, bu_ownership)`. Replaces the old single-row JSON blob with queryable individual rows. Includes `state` (`open`/`closed`), `workflow_id`, `workflow_state`, `note`, and `override_host_name`/`override_dns` columns.

-**`ivanti_finding_notes`** — Persistent per-finding notes keyed by finding ID. Survives cache refreshes. `UNIQUE(finding_id)`.
-
-**`ivanti_counts_cache`** — Single-row cache for finding metrics: open/closed counts, FP workflow state breakdowns by finding and by unique ticket ID.
+**`ivanti_counts_history`** — Historical open/closed/FP counts per sync for trend charts.

 **`ivanti_finding_overrides`** — Editor-applied overrides for `hostName` and `dns` fields. `UNIQUE(finding_id, field)`.

@@ -931,7 +951,7 @@ Standard_User delete restrictions are enforced at the API level: ownership check
 - Finding override field must be one of: `hostName`, `dns`
 - User group validated against: `Admin`, `Standard_User`, `Leadership`, `Read_Only` (enforced by DB triggers and app-level validation)
 - Hostname format validated with `/^[a-zA-Z0-9._-]+$/` in compliance notes
- All database operations use prepared statements — no string interpolation in SQL
+- All database operations use parameterized queries (`$1, $2, ...` placeholders) — no string interpolation in SQL

 ### Security headers

@@ -947,7 +967,7 @@ Applied to all responses:

 ## Upgrading an Existing Deployment

-This procedure updates the application code and schema while preserving all existing data. The database file (`backend/cve_database.db`) is never overwritten by `git pull` — it is gitignored.
+This procedure updates the application code and schema while preserving all existing data.

 ```bash
 # 1. Stop the running servers
@@ -965,31 +985,13 @@ cd frontend
 npm install
 cd ..

-# 5. Ensure SESSION_SECRET is set in backend/.env
+# 5. Ensure SESSION_SECRET and DATABASE_URL are set in backend/.env
 #    If missing:
 #    echo "SESSION_SECRET=$(openssl rand -base64 32)" >> backend/.env
+#    echo "DATABASE_URL=postgresql://steam:<password>@localhost:5433/cve_dashboard" >> backend/.env

-# 6. Run all migrations (idempotent — safe to re-run, skips already-applied changes)
-cd backend
-node migrations/add_knowledge_base_table.js
-node migrations/add_archer_tickets_table.js
-node migrations/add_ivanti_sync_table.js
-node migrations/add_ivanti_findings_tables.js
-node migrations/add_ivanti_todo_queue_table.js
-node migrations/add_card_workflow_type.js
-node migrations/add_todo_queue_ip_address.js
-node migrations/add_todo_queue_hostname.js
-node migrations/add_compliance_tables.js
-node migrations/add_finding_archive_tables.js
-node migrations/add_archer_tickets_timestamps.js
-node migrations/add_ivanti_counts_history_table.js
-node migrations/add_fp_submissions_table.js
-node migrations/add_user_groups.js
-node migrations/add_created_by_columns.js
-node migrations/add_fp_submission_editing.js
-node migrations/add_granite_workflow_type.js
-node migrations/add_compliance_notes_group_id.js
-cd ..
+# 6. Apply any schema changes (idempotent — safe to re-run)
+node backend/setup-postgres.js

 # 7. Rebuild the frontend
 cd frontend
@@ -998,13 +1000,13 @@ cd ..

 # 8. Start servers
 ./start-servers.sh
-# Or, if using systemd services:
-#   systemctl restart cve-backend cve-frontend
 ```

 After upgrading, clear your browser cookies and log in fresh — session format changes between versions will invalidate old sessions.

-> **Do not re-run `node setup.js`** on an existing deployment. It is only for first-time initialization. Re-running it will not destroy data (it checks for existing tables/users), but it is unnecessary and may create a duplicate admin account.
+> **Migrating from SQLite:** If this is the first upgrade after the Postgres migration, run `scripts/deploy-postgres.sh` instead of the manual steps above. It handles the full cutover including data migration. See [Postgres Migration Plan](docs/guides/postgres-migration-plan.md) for details.
+
+> **Do not re-run `node setup.js`** on an existing deployment. The legacy SQLite setup script is retained for reference only. Use `setup-postgres.js` for schema initialization.

 > **NODE_ENV reminder:** If you are running over plain HTTP (no TLS), make sure `NODE_ENV` is **not** set to `production` in `backend/.env`. See [Troubleshooting](#troubleshooting) for details.

@@ -1012,7 +1014,9 @@ After upgrading, clear your browser cookies and log in fresh — session format

 ## Migrations

-Migrations are standalone Node.js scripts. Run them in the listed order on a fresh install. All are idempotent and safe to re-run.
+> **Note:** The migration scripts in `backend/migrations/` are legacy SQLite migrations. They are not needed for PostgreSQL deployments — the complete schema is defined in `backend/db-schema.sql` and applied by `setup-postgres.js`. These scripts are retained for reference and for any remaining SQLite-based environments.
+
+For deployments still on SQLite, run them in the listed order. All are idempotent and safe to re-run.

 ```bash
 cd backend