cve-dashboard/README.md

# STEAM Security Dashboard v2.2.0

A self-hosted vulnerability management dashboard for the NTS-AEO-STEAM, NTS-AEO-ACCESS-ENG, NTS-AEO-ACCESS-OPS, and NTS-AEO-INTELDEV business units. Centralises CVE tracking, Ivanti host finding triage, AEO compliance posture monitoring, CCP Metrics cross-org compliance reporting, FP/Archer/CARD/GRANITE/DECOM exception workflows, CARD asset ownership management, Granite Loader Sheet generation, Jira ticket management, and internal documentation in a single interface.

---

## Table of Contents

- [Quick Start](#quick-start)
- [Features](#features)
  - [Home — CVE Management](#home--cve-management)
  - [Reporting — Ivanti Host Findings](#reporting--ivanti-host-findings)
  - [Ivanti Queue — Workflow Staging](#ivanti-queue--workflow-staging)
  - [FP Workflow Submission](#fp-workflow-submission)
  - [Archer Tickets](#archer-tickets)
  - [Archer Template Library](#archer-template-library)
  - [AEO Compliance](#aeo-compliance)
  - [CCP Metrics — Multi-Vertical Compliance](#ccp-metrics--multi-vertical-compliance)
  - [CARD Asset Ownership](#card-asset-ownership)
  - [Granite Loader Sheet](#granite-loader-sheet)
  - [Atlas Action Plans](#atlas-action-plans)
  - [Jira Integration](#jira-integration)
  - [Finding Archive Tracking](#finding-archive-tracking)
  - [Findings Trend](#findings-trend)
  - [Knowledge Base](#knowledge-base)
  - [Exports](#exports)
  - [In-App Notifications](#in-app-notifications)
  - [Feedback — GitLab Integration](#feedback--gitlab-integration)
  - [Access Control](#access-control)
- [Project Structure](#project-structure)
- [Tech Stack](#tech-stack)
- [Configuration](#configuration)
- [Database Schema](#database-schema)
- [Migrations](#migrations)
- [API Reference](#api-reference)
- [CI/CD Pipeline](#cicd-pipeline)
- [Documentation](#documentation)
- [Troubleshooting](#troubleshooting)
- [License](#license)

---

## Quick Start

### Prerequisites

- Node.js 18+
- Docker (for PostgreSQL 16 container)
- Python 3 with `python3-pandas` and `python3-openpyxl` (for compliance xlsx parsing)

### Install

```bash
git clone <repo-url>
cd cve-dashboard

# Backend dependencies
npm install

# Frontend dependencies
cd frontend && npm install && cd ..

# Python dependencies (Ubuntu/Debian)
apt install -y python3-pandas python3-openpyxl
```

### Configure

```bash
cp backend/.env.example backend/.env
# Edit backend/.env — at minimum set SESSION_SECRET and DATABASE_URL:
#   openssl rand -base64 32
```

See `backend/.env.example` for all available options including `DATABASE_URL`, Ivanti API, Jira, Atlas, CARD, and GitLab integration keys.

### Start PostgreSQL

The deploy script handles the full Postgres setup — container, schema, dependencies, and data migration from SQLite:

```bash
chmod +x scripts/deploy-postgres.sh
./scripts/deploy-postgres.sh
```

For fresh installs without an existing SQLite database, the script creates the schema and skips migration.

### Run Migrations

```bash
cd backend && node migrations/run-all.js
```

Migrations are idempotent and safe to re-run.

### Build and Run

```bash
# Build frontend
cd frontend && npm run build && cd ..

# Start servers
./start-servers.sh
```

Dashboard: http://localhost:3001

The helper scripts use `systemctl` under the hood — the systemd units in `systemd/` must be installed first. See the full reference manual for setup instructions.

### Interactive Configuration Wizard

For first-time deployments, the wizard walks through all required settings:

```bash
node configure.js
```

---

## Features

### Home — CVE Management

Searchable CVE list with per-vendor tracking and document storage. NVD API integration auto-populates severity, description, and publication date from the National Vulnerability Database.

**Capabilities:**
- Create, edit, and delete CVE entries with vendor association
- Upload advisory documents (PDF, email, screenshot, patch notes) per CVE/vendor pair
- NVD auto-lookup on CVE ID entry (rate-limited, API key optional for higher throughput)
- Filter by vendor, severity, status, and free-text search
- Required document tracking per vendor with mandatory/optional flags
- Calendar widget with SLA date highlighting

### Reporting — Ivanti Host Findings

Ivanti/RiskSense host finding triage with donut charts, advanced filtering, inline editing, CSV/XLSX export, and multi-BU scope.

**Capabilities:**
- Sync open host findings from Ivanti API (24-hour cadence, configurable via `IVANTI_BU_FILTER`)
- Group by Host toggle — collapses duplicate assets (same hostname + IP) with multiple findings into expandable rows
- CARD ownership tooltip on IP hover — displays confirmed/unconfirmed/candidate teams from CARD API
- CARD direct action modal — confirm, decline, or redirect ownership without a queue item
- Advanced column filtering, severity sorting, and free-text search
- Inline note editing with override hostname/DNS fields
- Per-user Ivanti identity for filtered FP workflow views
- Multi-select BU picker for scoping visible findings
- CSV and XLSX export with current filter state
- Add findings to the todo queue for batch workflow processing

### Ivanti Queue — Workflow Staging

Personal staging list for batch-processing FP, Archer, CARD, GRANITE, DECOM, and Remediate workflows.

**Capabilities:**
- Add individual findings or bulk-select from the Reporting page
- Six workflow types: FP (False Positive), Archer (Risk Acceptance), CARD (Ownership), GRANITE (Loader Sheet), DECOM (Decommission), Remediate
- Collapsible sections per workflow type
- Multi-item Jira ticket creation (consolidation modal)
- Ticket link display on completed items
- Clear Completed with FK-safe deletion
- Redirect pending items between workflow types without duplication
- DECOM items auto-note and auto-hide the finding on completion

### FP Workflow Submission

Submit False Positive workflows directly to the Ivanti API with attachments and lifecycle tracking.

**Capabilities:**
- Batch-select queue items for FP submission
- Attach supporting documents (10MB limit per file)
- Configurable expiration date, scope override, and reason
- Lifecycle tracking: submitted, approved, rejected, rework, resubmitted
- Edit and re-submit rejected workflows
- Re-queue findings from rejected submissions
- Auto-clear approved submissions, dismiss rejected
- Per-user workflow history with attachment results
- Collapsible submissions panel

### Archer Tickets

Track Archer risk acceptance exceptions (EXC numbers) linked to CVE/vendor pairs.

**Capabilities:**
- Create Archer ticket records with EXC number, URL, and status
- Status workflow: Draft, Open, Under Review, Accepted
- Link tickets to specific CVE/vendor combinations
- Full CRUD with audit logging

### Archer Template Library

Template management system for Archer Risk Acceptance forms. Stores static content (Environment Overview, Segmentation, Mitigating Controls) organised by Vendor, Platform, and Model.

**Capabilities:**
- Full CRUD with clone, search/filter, and per-section copy-to-clipboard
- Inline view panel with per-section copy buttons
- Template selector integrated into the Ivanti Queue for Archer workflow items
- Organised hierarchy: Vendor > Platform > Model
- Accessible from nav drawer (Template Mgr) and Ivanti Queue Archer items

### AEO Compliance

Weekly AEO compliance xlsx upload with diff preview, drift detection, per-team metric health cards, and device-level violation tracking.

**Capabilities:**
- Upload weekly compliance spreadsheets (xlsx) with automated parsing
- Diff preview showing new, resolved, and recurring non-compliant items
- Per-team metric health cards with pass/fail indicators
- Device-level violation detail panel with notes per hostname/metric pair
- Bulk notes import from CSV
- Compliance history tracking across uploads
- Per-metric estimated resolution date display
- Notes with group_id for batch operations

### CCP Metrics — Multi-Vertical Compliance

Cross-organisational VCL (Vulnerability Compliance Level) upload and reporting for multiple verticals. The CCP Metrics page provides executive-level visibility into compliance posture across teams.

**Capabilities:**
- Multi-vertical VCL xlsx upload with Summary and Detail sheet parsing
- Metric-first hierarchy with per-metric remediation plans
- Per-metric forecast burndown chart
- Aggregated burndown forecast on overview page
- Sub-team drill-down with intermediate view and per-team breakdowns
- Non-Compliant stat clickable with metric breakdown buttons
- Compliant/total counts on metric summary cards
- LIVE and LAST REPORT badges showing data freshness
- Data management panel — delete vertical, rollback upload, reset all
- Exec report page with exportable summaries
- Inline-editable team fields on vertical metadata
- Device metadata fields for asset context

### CARD Asset Ownership

CARD API integration for asset ownership management — confirm, decline, and redirect ownership for network assets.

**Capabilities:**
- Owner lookup by IP address or Ivanti Host ID (asset-search)
- Confirm, decline, and redirect ownership actions via API
- Suffix-guessing fallback when no host_id available (CTEC, NATL, CHTR, COML, RESI, WIFI, VOIP)
- IP address validation before mutation operations
- Update_token handling for safe concurrent operations
- Cached tooltip results per session
- Team assets endpoint for batch enrichment
- Quick mode (CTEC suffix only, 15s timeout) for tooltip performance

### Granite Loader Sheet

Generate Granite Loader Sheets with CARD enrichment, searchable picklists, per-row editing, and XLSX export.

**Capabilities:**
- Generate loader sheets from queue items or the Reporting page
- CARD enrichment — pull NCIM, Qualys, Netops Granite data per asset
- Searchable picklists for teams, statuses, operation types
- Per-row inline editing before export
- Column groups with configurable visibility
- XLSX export with formatting

### Atlas Action Plans

Atlas InfoSec action plan tracking with per-host vulnerability mapping and local cache for badge rendering.

**Capabilities:**
- View action plans linked to Ivanti host findings via host_id
- Create remediation, risk acceptance, and compensating control plans
- AtlasBadge component on findings rows indicating plan existence
- Slide-out detail panel with plan metadata
- Local cache (`atlas_action_plans_cache` table) for instant badge rendering
- Manual cache refresh triggers re-fetch from Atlas API
- Qualys vulnerability mapping per host

### Jira Integration

Create, sync, and track Jira Data Center tickets linked to CVE/vendor pairs and Ivanti queue items.

**Capabilities:**
- Create tickets from CVE records or Ivanti queue items
- Multi-item Jira ticket creation from the consolidation modal
- Flexible ticket creation — CVE/Vendor fields optional, source context tracking
- Vendor-specific issue type dropdown with per-vendor project keys
- JQL-based ticket lookup and sync
- Raw Jira status display (no status mapping)
- Save to Dashboard from Jira lookup results
- Dedicated Jira page for managing tickets
- Rate limiting with configurable window and burst limits
- Blocked dangerous endpoints (bulk delete, user management)

### Finding Archive Tracking

Automatic detection of disappeared and returned findings with BU reassignment classification and anomaly logging.

**Capabilities:**
- Detect findings that disappear between syncs — classify as ARCHIVED, RETURNED, CLOSED, or CLOSED_GONE
- BU reassignment tracking with history log
- Sync anomaly detection with significance thresholds
- Anomaly banner component on the Reporting page
- Archive summary bar with state distribution
- Transition history with severity-at-transition recording
- Return classification (original finding restored vs. new duplicate)
- Configurable `IVANTI_MANAGED_BUS` for drift classification scope

### Findings Trend

Historical open vs closed findings chart with per-BU trend lines, archive activity sparkline, and shift reason tooltips.

**Capabilities:**
- Ivanti counts history chart (open/closed over time)
- Per-BU trend lines via `ivanti_counts_history_by_bu`
- Archive activity sparkline overlay
- Shift reason tooltips from anomaly log

### Knowledge Base

Internal document library for policies, guides, and reference material.

**Capabilities:**
- Upload documents (PDF, Markdown, Word, Excel)
- Inline PDF and Markdown viewing
- Category-based organisation with search
- Slug-based URL routing

### Exports

Dedicated Exports page with pre-built export cards for common data pulls.

**Capabilities:**
- Jira Tickets export
- CCP Metrics export
- Remediation Status export
- CSV and XLSX format options

### In-App Notifications

Native notification system replacing the previous Webex bot integration.

**Capabilities:**
- Per-user notification bell with unread count
- Notification types for sync events, workflow completions, and system alerts
- Mark as read / dismiss actions
- Persistent storage in `notifications` table

### Feedback — GitLab Integration

In-app bug reports and feature requests submitted directly to the GitLab project as issues.

**Capabilities:**
- Feedback modal accessible from the nav drawer
- Bug report and feature request templates
- Submitted as GitLab issues via PAT authentication
- GitLab webhook receiver for issue lifecycle events (label changes, closes)
- Webhook secret validation for security

### Access Control

Four user groups with role-based permissions and full audit trail.

| Group | Permissions |
|-------|------------|
| Admin | Full CRUD, user management, audit log access, system configuration, data management |
| Standard_User | Create/update operations, FP workflow submission, queue management |
| Leadership | Read access to all data plus compliance and export views |
| Read_Only | Read-only access to all data |

**Additional capabilities:**
- Per-user BU team assignments (multi-BU tenancy)
- Per-user Ivanti identity (first/last name) for workflow filtering
- Cookie-based sessions with 24-hour expiry (httpOnly)
- Login rate limiting (20 attempts per 15-minute window)
- Full audit trail for all state-changing operations
- User profile panel with password change

---

## Project Structure

```
cve-dashboard/
├── backend/
│   ├── server.js              # Express API — middleware, CVE/document routes inline
│   ├── db.js                  # PostgreSQL connection pool (pg)
│   ├── db-schema.sql          # Complete DDL for fresh Postgres setup
│   ├── setup.js               # One-time DB init + default admin creation
│   ├── routes/
│   │   ├── auth.js            # Login, logout, session validation
│   │   ├── users.js           # User CRUD, role/group management
│   │   ├── auditLog.js        # Audit log queries
│   │   ├── nvdLookup.js       # NVD API proxy
│   │   ├── knowledgeBase.js   # Document library CRUD
│   │   ├── archerTickets.js   # Archer EXC ticket tracking
│   │   ├── archerTemplates.js # Archer template library CRUD + clone
│   │   ├── ivantiWorkflows.js # Ivanti FP workflow batch queries
│   │   ├── ivantiFindings.js  # Ivanti findings sync, query, inline edit
│   │   ├── ivantiTodoQueue.js # Per-user queue staging
│   │   ├── ivantiArchive.js   # Finding archive tracking + anomaly log
│   │   ├── ivantiFpWorkflow.js# FP workflow submission to Ivanti API
│   │   ├── compliance.js      # AEO compliance upload + items + notes
│   │   ├── vclMultiVertical.js# VCL/CCP multi-vertical compliance
│   │   ├── atlas.js           # Atlas action plan proxy + cache
│   │   ├── jiraTickets.js     # Jira CRUD + REST API integration
│   │   ├── cardApi.js         # CARD ownership proxy + asset-search
│   │   ├── notifications.js   # In-app notification system
│   │   ├── feedback.js        # GitLab issue creation for bug/feature
│   │   └── webhooks.js        # GitLab webhook receiver
│   ├── helpers/
│   │   ├── auditLog.js        # logAudit() — fire-and-forget DB insert
│   │   ├── cardApi.js         # CARD API — OAuth token, owner lookup, asset-search
│   │   ├── ivantiApi.js       # Ivanti/RiskSense HTTP helpers
│   │   ├── atlasApi.js        # Atlas action plan API
│   │   ├── jiraApi.js         # Jira ticket creation + rate limiter
│   │   ├── driftChecker.js    # BU drift detection between syncs
│   │   ├── vclHelpers.js      # VCL metric calculation helpers
│   │   └── teams.js           # Team validation helpers
│   ├── middleware/
│   │   └── auth.js            # requireAuth(), requireGroup(...groups)
│   ├── migrations/            # Sequential migration scripts (idempotent)
│   │   └── run-all.js         # Run all migrations in order
│   └── scripts/               # Python utilities (compliance parsing)
│
├── frontend/
│   └── src/
│       ├── App.js             # Main app — routing, CVE list, filters, modals
│       ├── App.css            # Global styles and CSS variables
│       ├── contexts/
│       │   └── AuthContext.js # Auth state provider (login, logout, role helpers)
│       ├── utils/
│       │   ├── graniteLoaderConfig.js   # Granite column definitions and groups
│       │   ├── graniteLoaderExport.js   # XLSX generation logic
│       │   └── graniteLoaderPicklists.js # Searchable dropdown options
│       └── components/
│           ├── LoginForm.js
│           ├── NavDrawer.js
│           ├── UserMenu.js
│           ├── UserProfilePanel.js
│           ├── CalendarWidget.js
│           ├── UserManagement.js
│           ├── AuditLog.js
│           ├── NvdSyncModal.js
│           ├── KnowledgeBaseModal.js
│           ├── KnowledgeBaseViewer.js
│           ├── CardOwnerTooltip.js
│           ├── CardDetailModal.js
│           ├── CardActionModal.js
│           ├── RedirectModal.js
│           ├── LoaderModal.js
│           ├── SearchableSelect.js
│           ├── AtlasBadge.js
│           ├── AtlasIcon.js
│           ├── AtlasSlideOutPanel.js
│           ├── AdminScopeToggle.js
│           ├── ConfirmModal.js
│           ├── ConsolidationModal.js
│           ├── CveTooltip.js
│           ├── DeleteConfirmModal.js
│           ├── FeedbackModal.js
│           ├── NotificationBell.js
│           ├── RemediationModal.js
│           ├── TemplateFormModal.js
│           ├── TemplateSelector.js
│           └── pages/
│               ├── AdminPage.js
│               ├── ReportingPage.js
│               ├── IvantiTodoQueuePage.js
│               ├── CompliancePage.js
│               ├── ComplianceUploadModal.js
│               ├── ComplianceDetailPanel.js
│               ├── ComplianceChartsPanel.js
│               ├── CCPMetricsPage.js
│               ├── VCLReportPage.js
│               ├── MetricInfoPanel.js
│               ├── BulkUploadModal.js
│               ├── MultiVerticalUploadModal.js
│               ├── IvantiCountsChart.js
│               ├── AnomalyBanner.js
│               ├── ArchiveSummaryBar.js
│               ├── ArcherPage.js
│               ├── ArcherTemplatePage.js
│               ├── JiraPage.js
│               ├── KnowledgeBasePage.js
│               └── ExportsPage.js
│
├── docs/
│   ├── api/                   # API specs (Ivanti, Atlas, Jira)
│   ├── architecture/          # Architecture proposals (AD/SAML, split)
│   ├── design/                # Design system, workflow colour codes
│   ├── guides/                # User guides, full reference manual, VCL calculations
│   ├── operations/            # Operational logs and connectivity tests
│   ├── security/              # Security audits and remediation plans
│   ├── testing/               # Test plans and scripts
│   └── troubleshooting/       # Investigation scripts and reports
│
├── scripts/
│   ├── deploy-postgres.sh     # One-time: container, schema, migration
│   └── reset-and-migrate.sh   # Dev utility: reset DB and re-run migrations
├── deploy/
│   ├── cve-backend-production.service
│   ├── cve-backend-staging.service
│   └── setup-staging.sh
├── systemd/
│   ├── cve-backend.service
│   └── cve-frontend.service
├── configure.js               # Interactive configuration wizard
├── docker-compose.yml         # PostgreSQL 16 container definition
├── start-servers.sh           # Start backend (serves API + frontend build)
├── stop-servers.sh            # Stop backend
└── CHANGELOG.md
```

---

## Tech Stack

| Layer | Technology |
|-------|------------|
| Backend | Node.js 18+, Express 5 |
| Database | PostgreSQL 16 (Docker, port 5433) |
| Auth | bcryptjs, cookie-based sessions (httpOnly, 24h expiry), express-rate-limit |
| File uploads | Multer 2 (10MB limit) |
| Frontend | React 19 (Create React App / react-scripts 5) |
| Frontend serving | Express serves `frontend/build/` as static files on port 3001 |
| UI Icons | lucide-react |
| Charts | recharts |
| Spreadsheet parsing | xlsx (frontend), pandas + openpyxl (backend Python scripts) |
| Markdown rendering | react-markdown + rehype-sanitize |
| Diagrams | mermaid |
| Testing | Jest 30 (backend property tests), React Testing Library (frontend) |

---

## Configuration

All configuration lives in `backend/.env`. Copy `backend/.env.example` as a starting point.

| Variable | Purpose | Default |
|---|---|---|
| `PORT` | Backend server port | `3001` |
| `API_HOST` | Backend bind address | `localhost` |
| `CORS_ORIGINS` | Comma-separated allowed origins | `http://localhost:3000` |
| `SESSION_SECRET` | **Required.** Session signing key. Generate with `openssl rand -base64 32` | — |
| `DATABASE_URL` | PostgreSQL connection string | — |
| `NVD_API_KEY` | NVD API key (increases rate limit from 5 to 50 req/30s) | — |
| `IVANTI_API_KEY` | Ivanti/RiskSense API key | — |
| `IVANTI_CLIENT_ID` | Ivanti client ID | `1550` |
| `IVANTI_FIRST_NAME` | Fallback Ivanti identity (first name) | — |
| `IVANTI_LAST_NAME` | Fallback Ivanti identity (last name) | — |
| `IVANTI_BU_FILTER` | Comma-separated BU values to sync | `NTS-AEO-ACCESS-ENG,NTS-AEO-STEAM` |
| `IVANTI_MANAGED_BUS` | BUs considered "managed" for drift classification | `NTS-AEO-ACCESS-ENG,NTS-AEO-STEAM` |
| `IVANTI_SKIP_TLS` | Skip TLS verification for SSL inspection proxies | `false` |
| `ATLAS_API_URL` | Atlas InfoSec API base URL | — |
| `ATLAS_API_USER` | Atlas Basic Auth username | — |
| `ATLAS_API_PASS` | Atlas Basic Auth password | — |
| `ATLAS_SKIP_TLS` | Skip Atlas TLS verification | `false` |
| `JIRA_BASE_URL` | Jira Data Center REST API base URL | — |
| `JIRA_AUTH_METHOD` | Auth method: `basic` or `pat` | `basic` |
| `JIRA_API_USER` | Jira Basic Auth username | — |
| `JIRA_API_TOKEN` | Jira Basic Auth token | — |
| `JIRA_PAT` | Jira Personal Access Token (when `JIRA_AUTH_METHOD=pat`) | — |
| `JIRA_PROJECT_KEY` | Default Jira project key | — |
| `JIRA_ISSUE_TYPE` | Default issue type | `Task` |
| `JIRA_SKIP_TLS` | Skip Jira TLS verification | `false` |
| `CARD_API_URL` | CARD API base URL | — |
| `CARD_API_USER` | CARD OAuth username | — |
| `CARD_API_PASS` | CARD OAuth password | — |
| `CARD_SKIP_TLS` | Skip CARD TLS verification | `false` |
| `GITLAB_URL` | GitLab instance URL for feedback integration | `http://steam-gitlab.charterlab.com` |
| `GITLAB_PROJECT_ID` | GitLab project numeric ID | — |
| `GITLAB_PAT` | GitLab project access token (api scope) | — |
| `GITLAB_WEBHOOK_SECRET` | Shared secret for webhook validation | — |

> `SESSION_SECRET` and `DATABASE_URL` are required for the backend to start. All integration keys are optional — features degrade gracefully when their keys are absent.

---

## Database Schema

PostgreSQL 16 with the following tables:

| Table | Purpose |
|-------|---------|
| `cves` | CVE records with vendor association, severity, status |
| `documents` | Uploaded advisory documents linked to CVE/vendor pairs |
| `required_documents` | Per-vendor required document types |
| `users` | User accounts with role, group, BU teams, Ivanti identity |
| `sessions` | Active session tokens with expiry |
| `audit_logs` | Immutable audit trail for all state-changing operations |
| `jira_tickets` | Locally tracked Jira tickets linked to CVE/vendor |
| `archer_tickets` | Archer risk acceptance exceptions (EXC numbers) |
| `knowledge_base` | Internal document library entries |
| `ivanti_findings` | Individual Ivanti host findings (synced from API) |
| `ivanti_sync_state` | Sync metadata — last run time, status, error |
| `ivanti_counts_cache` | Cached open/closed counts and FP workflow counts |
| `ivanti_counts_history` | Historical open/closed counts over time |
| `ivanti_counts_history_by_bu` | Per-BU historical counts |
| `ivanti_fp_submissions` | FP workflow submissions with lifecycle tracking |
| `ivanti_fp_submission_history` | Edit history for FP submissions |
| `ivanti_todo_queue` | Per-user workflow staging queue |
| `ivanti_finding_archives` | Archived/returned/closed finding records |
| `ivanti_archive_transitions` | State transition history for archived findings |
| `ivanti_sync_anomaly_log` | Sync anomaly detection log |
| `ivanti_finding_bu_history` | BU reassignment history per finding |
| `atlas_action_plans_cache` | Cached Atlas action plan data for badge rendering |
| `compliance_uploads` | Compliance xlsx upload metadata |
| `compliance_items` | Individual non-compliant items per upload |
| `compliance_notes` | Per-hostname/metric notes |

The complete DDL is in `backend/db-schema.sql`. For fresh installs, `scripts/deploy-postgres.sh` applies it automatically.

---

## Migrations

Run all migrations (idempotent):

```bash
cd backend && node migrations/run-all.js
```

Migration files in execution order:

| Migration | Purpose |
|-----------|---------|
| `add_user_groups.js` | User group column and check constraint |
| `add_user_ivanti_identity.js` | Per-user Ivanti first/last name |
| `add_user_bu_teams.js` | Per-user BU team assignments |
| `add_knowledge_base_table.js` | Knowledge base document library |
| `add_ivanti_sync_table.js` | Ivanti sync state (single-row) |
| `add_ivanti_findings_tables.js` | Individual findings + counts cache |
| `add_ivanti_findings_ipv6_columns.js` | IPv6 fallback columns |
| `add_ivanti_counts_history_table.js` | Historical counts + per-BU |
| `add_ivanti_todo_queue_table.js` | Per-user workflow staging queue |
| `add_todo_queue_hostname.js` | Hostname column on queue items |
| `add_todo_queue_ip_address.js` | IP address column on queue items |
| `add_granite_workflow_type.js` | GRANITE workflow type |
| `add_card_workflow_type.js` | CARD workflow type |
| `add_decom_workflow_type.js` | DECOM workflow type |
| `add_remediate_workflow_type.js` | Remediate workflow type |
| `add_compliance_tables.js` | Compliance uploads, items, notes |
| `add_compliance_notes_group_id.js` | Group ID for batch notes |
| `add_compliance_history_metric_id.js` | Metric ID in compliance history |
| `add_compliance_item_history.js` | Compliance item history tracking |
| `add_fp_submissions_table.js` | FP workflow submissions |
| `add_fp_submission_editing.js` | Submission edit history |
| `add_fp_submissions_dismissed.js` | Dismissed/lifecycle status |
| `add_fp_submissions_requeued_at.js` | Requeue timestamp |
| `add_archer_tickets_table.js` | Archer EXC tickets |
| `add_archer_tickets_timestamps.js` | Timestamp columns |
| `add_archer_templates_table.js` | Archer template library |
| `add_atlas_action_plans_cache.js` | Atlas plan cache |
| `add_atlas_known_column.js` | Atlas known flag |
| `add_finding_archive_tables.js` | Archive tracking tables |
| `add_closed_gone_state.js` | CLOSED_GONE archive state |
| `add_return_classification.js` | Return classification field |
| `add_sync_anomaly_tables.js` | Anomaly detection log |
| `add_flexible_jira_ticket_creation.js` | Optional CVE/vendor on Jira tickets |
| `add_multi_item_jira_ticket.js` | Multi-item ticket references |
| `add_jira_sync_columns.js` | Jira sync metadata (SQLite) |
| `add_jira_sync_columns_pg.js` | Jira sync metadata (PostgreSQL) |
| `drop_jira_status_check_constraint.js` | Remove status enum for raw display |
| `add_notifications_table.js` | In-app notification system |
| `add_created_by_columns.js` | created_by on archer_tickets |
| `add_queue_remediation_notes_table.js` | Remediation notes on queue items |
| `add_vcl_multi_vertical.js` | VCL multi-vertical tables |
| `add_vcl_reporting_columns.js` | VCL reporting metadata |
| `add_vcl_vertical_metadata.js` | VCL vertical team fields |
| `backfill_anomaly_log.js` | Backfill anomaly classification data |
| `backfill_return_classification.js` | Backfill return classification |
| `reclassify_bu_roundtrips.js` | Reclassify BU roundtrip transitions |

---

## API Reference

All routes are prefixed with `/api`. All endpoints except login, logout, health check, and webhooks require a valid session cookie.

### Public

| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/health` | Health check (used by CI/CD verification) |
| POST | `/api/auth/login` | Authenticate and create session |
| POST | `/api/auth/logout` | Destroy session |
| POST | `/api/webhooks/gitlab` | GitLab webhook receiver (validated by secret) |

### CVE Management

| Method | Path | Description | Group |
|--------|------|-------------|-------|
| GET | `/api/cves` | List CVEs with filtering | Any |
| POST | `/api/cves` | Create CVE | Standard_User+ |
| PUT | `/api/cves/:id` | Update CVE | Standard_User+ |
| DELETE | `/api/cves/:id` | Delete CVE | Admin |
| POST | `/api/cves/:cveId/:vendor/documents` | Upload document | Standard_User+ |
| GET | `/api/cves/:cveId/:vendor/documents` | List documents | Any |
| DELETE | `/api/documents/:id` | Delete document | Admin |

### Users and Auth

| Method | Path | Description | Group |
|--------|------|-------------|-------|
| GET | `/api/auth/me` | Current session user | Any |
| GET | `/api/users` | List all users | Admin |
| POST | `/api/users` | Create user | Admin |
| PUT | `/api/users/:id` | Update user (role, group, teams) | Admin |
| DELETE | `/api/users/:id` | Delete user | Admin |
| PUT | `/api/users/:id/password` | Change password | Owner or Admin |

### Ivanti Findings

| Method | Path | Description | Group |
|--------|------|-------------|-------|
| GET | `/api/ivanti/findings` | Query synced findings | Any |
| POST | `/api/ivanti/findings/sync` | Trigger manual sync | Standard_User+ |
| PUT | `/api/ivanti/findings/:id/note` | Update finding note | Standard_User+ |
| PUT | `/api/ivanti/findings/:id/override` | Override hostname/DNS | Standard_User+ |

### Ivanti Queue

| Method | Path | Description | Group |
|--------|------|-------------|-------|
| GET | `/api/ivanti/todo-queue` | Get user's queue | Any |
| POST | `/api/ivanti/todo-queue` | Add item(s) to queue | Standard_User+ |
| PUT | `/api/ivanti/todo-queue/:id` | Update queue item | Standard_User+ |
| DELETE | `/api/ivanti/todo-queue/:id` | Remove queue item | Standard_User+ |
| DELETE | `/api/ivanti/todo-queue/completed` | Clear completed items | Standard_User+ |

### Ivanti FP Workflow

| Method | Path | Description | Group |
|--------|------|-------------|-------|
| POST | `/api/ivanti/fp-workflow` | Submit FP workflow to Ivanti | Standard_User+ |
| GET | `/api/ivanti/fp-workflow/submissions` | List user's submissions | Any |
| PUT | `/api/ivanti/fp-workflow/submissions/:id` | Edit/resubmit | Standard_User+ |

### Ivanti Archive

| Method | Path | Description | Group |
|--------|------|-------------|-------|
| GET | `/api/ivanti/archive` | Query archived findings | Any |
| GET | `/api/ivanti/archive/anomalies` | Sync anomaly log | Any |
| GET | `/api/ivanti/archive/transitions/:id` | Transition history | Any |

### Compliance (AEO)

| Method | Path | Description | Group |
|--------|------|-------------|-------|
| POST | `/api/compliance/upload` | Upload xlsx | Standard_User+ |
| GET | `/api/compliance/items` | Query non-compliant items | Any |
| GET | `/api/compliance/trends` | Compliance trend data | Any |
| POST | `/api/compliance/notes` | Add note to hostname/metric | Standard_User+ |

### VCL Multi-Vertical (CCP Metrics)

| Method | Path | Description | Group |
|--------|------|-------------|-------|
| POST | `/api/compliance/vcl-multi/upload` | Upload VCL xlsx | Standard_User+ |
| GET | `/api/compliance/vcl-multi/verticals` | List verticals | Any |
| GET | `/api/compliance/vcl-multi/metrics` | Query metrics | Any |
| GET | `/api/compliance/vcl-multi/forecast` | Burndown forecast | Any |
| DELETE | `/api/compliance/vcl-multi/verticals/:id` | Delete vertical | Admin |

### Atlas

| Method | Path | Description | Group |
|--------|------|-------------|-------|
| GET | `/api/atlas/hosts/:hostId/plans` | Get plans for host | Any |
| PUT | `/api/atlas/hosts/:hostId/plans` | Create plan | Standard_User+ |
| PATCH | `/api/atlas/hosts/:hostId/plans` | Update plan | Standard_User+ |
| POST | `/api/atlas/hosts/:hostId/refresh` | Refresh cache | Standard_User+ |

### Jira

| Method | Path | Description | Group |
|--------|------|-------------|-------|
| GET | `/api/jira-tickets` | List local tickets | Any |
| POST | `/api/jira-tickets` | Create ticket (local + Jira API) | Standard_User+ |
| GET | `/api/jira-tickets/lookup` | JQL search against Jira | Standard_User+ |
| POST | `/api/jira-tickets/save` | Save from Jira lookup | Standard_User+ |

### CARD

| Method | Path | Description | Group |
|--------|------|-------------|-------|
| GET | `/api/card/owner/:assetId` | Lookup owner | Any |
| GET | `/api/card/asset-search/:hostId` | Search by Ivanti host ID | Any |
| POST | `/api/card/owner/:assetId/confirm` | Confirm ownership | Standard_User+ |
| POST | `/api/card/owner/:assetId/decline` | Decline ownership | Standard_User+ |
| POST | `/api/card/owner/:assetId/redirect` | Redirect ownership | Standard_User+ |
| GET | `/api/card/teams` | List all CARD teams | Any |

### Other

| Method | Path | Description | Group |
|--------|------|-------------|-------|
| GET | `/api/audit-logs` | Query audit trail | Admin |
| GET | `/api/nvd/:cveId` | NVD metadata lookup | Any |
| GET/POST | `/api/knowledge-base` | Document library CRUD | Any / Standard_User+ |
| GET/POST | `/api/archer-tickets` | Archer ticket CRUD | Any / Standard_User+ |
| GET/POST | `/api/archer-templates` | Template library CRUD | Any / Standard_User+ |
| GET | `/api/notifications` | User notifications | Any |
| POST | `/api/feedback` | Submit bug/feature to GitLab | Any |

---

## CI/CD Pipeline

Defined in `.gitlab-ci.yml`. Stages: install, lint, test, build, deploy, verify.

| Stage | What it does |
|-------|-------------|
| install | `npm ci` for root + frontend |
| lint | ESLint on frontend (warning threshold: 25) |
| test | Jest backend property tests + frontend unit tests |
| build | `npm run build` in frontend/ |
| deploy-staging | rsync to 71.85.90.9, run migrations, restart service (auto on master) |
| deploy-production | rsync to 71.85.90.6, run migrations, restart service (manual trigger) |
| verify | Hit `/api/health` endpoint, post issue comments for `Closes #N` references |

**Runner:** Docker executor on LXC 108 (71.85.90.8), Runner #6, `node:18` image for build stages, `alpine:latest` for deploy.

**Required CI/CD variables:** `DATABASE_URL`, `SSH_PRIVATE_KEY` (base64), `GITLAB_PAT`.

---

## Documentation

- **[Full Reference Manual](docs/guides/full-reference-manual.md)** — comprehensive feature documentation, API reference, database schema, security model, and configuration details
- **[VCL Metric Calculations](docs/guides/vcl-metric-calculations.md)** — formula reference for CCP Metrics compliance calculations
- **[Postgres Migration Plan](docs/guides/postgres-migration-plan.md)** — architecture decisions, schema design, and cutover procedure for the SQLite to PostgreSQL migration
- **[Migration Guide](backend/migrations/README.md)** — schema migration scripts for upgrading existing deployments
- **[Design System](docs/design/design-system.md)** — UI component patterns and colour system
- **[Ivanti API Reference](docs/api/ivanti-api-reference.md)** — Ivanti/RiskSense API integration details
- **[Jira API Use Cases](docs/api/jira-api-use-cases.md)** — Jira Data Center API compliance summary
- **[AD/SAML Integration Architecture](docs/architecture/ad-saml-integration.md)** — planned Active Directory / SAML SSO integration
- **[Security Audit Tracker](docs/security/security-audit-tracker.md)** — living security audit document

---

## Troubleshooting

### Backend fails to start with "FATAL: SESSION_SECRET"

**Symptom:** Server exits immediately with `FATAL: SESSION_SECRET environment variable must be set`.

**Cause:** The `SESSION_SECRET` env var is not configured in `backend/.env`.

**Fix:** Generate a secret and add it to `backend/.env`:
```bash
echo "SESSION_SECRET=$(openssl rand -base64 32)" >> backend/.env
```

### Database connection refused

**Symptom:** Backend logs `Error: connect ECONNREFUSED` on startup.

**Cause:** PostgreSQL container is not running or `DATABASE_URL` is misconfigured.

**Fix:**
```bash
docker compose up -d    # Start the Postgres container
# Verify DATABASE_URL in backend/.env matches: postgresql://steam:<password>@localhost:5433/cve_dashboard
```

### CARD API timeouts in production

**Symptom:** CARD tooltip shows timeout errors or takes minutes to respond.

**Cause:** IPv6 AAAA records for `card.charter.com` are unreachable from this network. Node.js defaults to trying IPv6 first.

**Fix:** The backend sets `dns.setDefaultResultOrder('ipv4first')` globally. If you still see issues, verify the setting is at the top of `server.js` before any network imports.

### Compliance upload fails with "No items parsed"

**Symptom:** Upload completes but reports 0 new, 0 resolved, 0 recurring items.

**Cause:** The xlsx file structure does not match the expected format (sheet names, column headers).

**Fix:** Ensure the compliance xlsx has the expected sheet layout. Check `backend/scripts/parse_compliance_xlsx.py` for the expected column mappings.

### Missing migrations after deploy

**Symptom:** API returns 500 errors referencing missing columns or tables.

**Cause:** Migrations were not run after pulling new code.

**Fix:**
```bash
cd backend && node migrations/run-all.js
```

### Frontend changes not visible after deploy

**Symptom:** UI still shows old version after code changes.

**Cause:** The frontend must be rebuilt after any source changes — Express serves the static build.

**Fix:**
```bash
cd frontend && npm run build
# Restart backend or refresh the browser
```

---

## License

Internal use only — Charter Communications / NTS-AEO.

---

*Designed and built by Jordan Ramos (jordan.ramos@spectrum.com)*