diff --git a/README.md b/README.md index 41cb894..797a029 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,44 @@ -# STEAM Security Dashboard v1.0.0 +# STEAM Security Dashboard v2.2.0 -A self-hosted vulnerability management dashboard for the NTS-AEO-STEAM and NTS-AEO-ACCESS-ENG business units. Centralises CVE tracking, Ivanti host finding triage, AEO compliance posture, FP/Archer/CARD exception workflows, and internal documentation in a single interface. +A self-hosted vulnerability management dashboard for the NTS-AEO-STEAM, NTS-AEO-ACCESS-ENG, NTS-AEO-ACCESS-OPS, and NTS-AEO-INTELDEV business units. Centralises CVE tracking, Ivanti host finding triage, AEO compliance posture monitoring, CCP Metrics cross-org compliance reporting, FP/Archer/CARD/GRANITE/DECOM exception workflows, CARD asset ownership management, Granite Loader Sheet generation, Jira ticket management, and internal documentation in a single interface. + +--- + +## Table of Contents + +- [Quick Start](#quick-start) +- [Features](#features) + - [Home — CVE Management](#home--cve-management) + - [Reporting — Ivanti Host Findings](#reporting--ivanti-host-findings) + - [Ivanti Queue — Workflow Staging](#ivanti-queue--workflow-staging) + - [FP Workflow Submission](#fp-workflow-submission) + - [Archer Tickets](#archer-tickets) + - [Archer Template Library](#archer-template-library) + - [AEO Compliance](#aeo-compliance) + - [CCP Metrics — Multi-Vertical Compliance](#ccp-metrics--multi-vertical-compliance) + - [CARD Asset Ownership](#card-asset-ownership) + - [Granite Loader Sheet](#granite-loader-sheet) + - [Atlas Action Plans](#atlas-action-plans) + - [Jira Integration](#jira-integration) + - [Finding Archive Tracking](#finding-archive-tracking) + - [Findings Trend](#findings-trend) + - [Knowledge Base](#knowledge-base) + - [Exports](#exports) + - [In-App Notifications](#in-app-notifications) + - [Feedback — GitLab Integration](#feedback--gitlab-integration) + - [Access Control](#access-control) +- [Project Structure](#project-structure) +- [Tech Stack](#tech-stack) +- [Configuration](#configuration) +- [Database Schema](#database-schema) +- [Migrations](#migrations) +- [API Reference](#api-reference) +- [CI/CD Pipeline](#cicd-pipeline) +- [Documentation](#documentation) +- [Troubleshooting](#troubleshooting) +- [License](#license) + +--- ## Quick Start @@ -34,7 +72,7 @@ cp backend/.env.example backend/.env # openssl rand -base64 32 ``` -See `backend/.env.example` for all available options including `DATABASE_URL`, Ivanti API, Jira, and Atlas integration keys. +See `backend/.env.example` for all available options including `DATABASE_URL`, Ivanti API, Jira, Atlas, CARD, and GitLab integration keys. ### Start PostgreSQL @@ -47,6 +85,14 @@ chmod +x scripts/deploy-postgres.sh For fresh installs without an existing SQLite database, the script creates the schema and skips migration. +### Run Migrations + +```bash +cd backend && node migrations/run-all.js +``` + +Migrations are idempotent and safe to re-run. + ### Build and Run ```bash @@ -57,80 +103,801 @@ cd frontend && npm run build && cd .. ./start-servers.sh ``` -Dashboard: http://localhost:3000 · API: http://localhost:3001 +Dashboard: http://localhost:3001 -The helper scripts use `systemctl` under the hood — the systemd units in `systemd/` must be installed first. See the full manual for setup instructions. +The helper scripts use `systemctl` under the hood — the systemd units in `systemd/` must be installed first. See the full reference manual for setup instructions. + +### Interactive Configuration Wizard + +For first-time deployments, the wizard walks through all required settings: + +```bash +node configure.js +``` + +--- ## Features -| Feature | Description | -|---------|-------------| -| **CVE Management** | Track CVEs across multiple vendors with document storage and NVD auto-fill | -| **Reporting** | Ivanti host finding triage with donut charts, inline editing, advanced filtering, CSV/XLSX export | -| **Ivanti Queue** | Personal staging list for batch FP, Archer, CARD, and Granite workflows | -| **FP Workflow** | Submit false positive workflows directly to Ivanti API with attachments | -| **Compliance** | Weekly AEO xlsx upload with diff preview, drift detection, per-team metric health cards | -| **Archive Tracking** | Automatic detection of disappeared/returned findings with BU reassignment classification | -| **Findings Trend** | Historical open vs closed chart with archive activity sparkline and shift reason tooltips | -| **Jira Integration** | Create, sync, and track Jira Data Center tickets linked to CVE/vendor pairs | -| **Archer Tickets** | Track risk acceptance exceptions (EXC numbers) linked to findings | -| **CARD API** | Granite/CARD asset lookup integration for network device workflows | -| **Knowledge Base** | Internal document library with inline PDF/Markdown viewing | -| **Access Control** | Four user groups (Admin, Standard_User, Leadership, Read_Only) with full audit trail | +### Home — CVE Management + +Searchable CVE list with per-vendor tracking and document storage. NVD API integration auto-populates severity, description, and publication date from the National Vulnerability Database. + +**Capabilities:** +- Create, edit, and delete CVE entries with vendor association +- Upload advisory documents (PDF, email, screenshot, patch notes) per CVE/vendor pair +- NVD auto-lookup on CVE ID entry (rate-limited, API key optional for higher throughput) +- Filter by vendor, severity, status, and free-text search +- Required document tracking per vendor with mandatory/optional flags +- Calendar widget with SLA date highlighting + +### Reporting — Ivanti Host Findings + +Ivanti/RiskSense host finding triage with donut charts, advanced filtering, inline editing, CSV/XLSX export, and multi-BU scope. + +**Capabilities:** +- Sync open host findings from Ivanti API (24-hour cadence, configurable via `IVANTI_BU_FILTER`) +- Group by Host toggle — collapses duplicate assets (same hostname + IP) with multiple findings into expandable rows +- CARD ownership tooltip on IP hover — displays confirmed/unconfirmed/candidate teams from CARD API +- CARD direct action modal — confirm, decline, or redirect ownership without a queue item +- Advanced column filtering, severity sorting, and free-text search +- Inline note editing with override hostname/DNS fields +- Per-user Ivanti identity for filtered FP workflow views +- Multi-select BU picker for scoping visible findings +- CSV and XLSX export with current filter state +- Add findings to the todo queue for batch workflow processing + +### Ivanti Queue — Workflow Staging + +Personal staging list for batch-processing FP, Archer, CARD, GRANITE, DECOM, and Remediate workflows. + +**Capabilities:** +- Add individual findings or bulk-select from the Reporting page +- Six workflow types: FP (False Positive), Archer (Risk Acceptance), CARD (Ownership), GRANITE (Loader Sheet), DECOM (Decommission), Remediate +- Collapsible sections per workflow type +- Multi-item Jira ticket creation (consolidation modal) +- Ticket link display on completed items +- Clear Completed with FK-safe deletion +- Redirect pending items between workflow types without duplication +- DECOM items auto-note and auto-hide the finding on completion + +### FP Workflow Submission + +Submit False Positive workflows directly to the Ivanti API with attachments and lifecycle tracking. + +**Capabilities:** +- Batch-select queue items for FP submission +- Attach supporting documents (10MB limit per file) +- Configurable expiration date, scope override, and reason +- Lifecycle tracking: submitted, approved, rejected, rework, resubmitted +- Edit and re-submit rejected workflows +- Re-queue findings from rejected submissions +- Auto-clear approved submissions, dismiss rejected +- Per-user workflow history with attachment results +- Collapsible submissions panel + +### Archer Tickets + +Track Archer risk acceptance exceptions (EXC numbers) linked to CVE/vendor pairs. + +**Capabilities:** +- Create Archer ticket records with EXC number, URL, and status +- Status workflow: Draft, Open, Under Review, Accepted +- Link tickets to specific CVE/vendor combinations +- Full CRUD with audit logging + +### Archer Template Library + +Template management system for Archer Risk Acceptance forms. Stores static content (Environment Overview, Segmentation, Mitigating Controls) organised by Vendor, Platform, and Model. + +**Capabilities:** +- Full CRUD with clone, search/filter, and per-section copy-to-clipboard +- Inline view panel with per-section copy buttons +- Template selector integrated into the Ivanti Queue for Archer workflow items +- Organised hierarchy: Vendor > Platform > Model +- Accessible from nav drawer (Template Mgr) and Ivanti Queue Archer items + +### AEO Compliance + +Weekly AEO compliance xlsx upload with diff preview, drift detection, per-team metric health cards, and device-level violation tracking. + +**Capabilities:** +- Upload weekly compliance spreadsheets (xlsx) with automated parsing +- Diff preview showing new, resolved, and recurring non-compliant items +- Per-team metric health cards with pass/fail indicators +- Device-level violation detail panel with notes per hostname/metric pair +- Bulk notes import from CSV +- Compliance history tracking across uploads +- Per-metric estimated resolution date display +- Notes with group_id for batch operations + +### CCP Metrics — Multi-Vertical Compliance + +Cross-organisational VCL (Vulnerability Compliance Level) upload and reporting for multiple verticals. The CCP Metrics page provides executive-level visibility into compliance posture across teams. + +**Capabilities:** +- Multi-vertical VCL xlsx upload with Summary and Detail sheet parsing +- Metric-first hierarchy with per-metric remediation plans +- Per-metric forecast burndown chart +- Aggregated burndown forecast on overview page +- Sub-team drill-down with intermediate view and per-team breakdowns +- Non-Compliant stat clickable with metric breakdown buttons +- Compliant/total counts on metric summary cards +- LIVE and LAST REPORT badges showing data freshness +- Data management panel — delete vertical, rollback upload, reset all +- Exec report page with exportable summaries +- Inline-editable team fields on vertical metadata +- Device metadata fields for asset context + +### CARD Asset Ownership + +CARD API integration for asset ownership management — confirm, decline, and redirect ownership for network assets. + +**Capabilities:** +- Owner lookup by IP address or Ivanti Host ID (asset-search) +- Confirm, decline, and redirect ownership actions via API +- Suffix-guessing fallback when no host_id available (CTEC, NATL, CHTR, COML, RESI, WIFI, VOIP) +- IP address validation before mutation operations +- Update_token handling for safe concurrent operations +- Cached tooltip results per session +- Team assets endpoint for batch enrichment +- Quick mode (CTEC suffix only, 15s timeout) for tooltip performance + +### Granite Loader Sheet + +Generate Granite Loader Sheets with CARD enrichment, searchable picklists, per-row editing, and XLSX export. + +**Capabilities:** +- Generate loader sheets from queue items or the Reporting page +- CARD enrichment — pull NCIM, Qualys, Netops Granite data per asset +- Searchable picklists for teams, statuses, operation types +- Per-row inline editing before export +- Column groups with configurable visibility +- XLSX export with formatting + +### Atlas Action Plans + +Atlas InfoSec action plan tracking with per-host vulnerability mapping and local cache for badge rendering. + +**Capabilities:** +- View action plans linked to Ivanti host findings via host_id +- Create remediation, risk acceptance, and compensating control plans +- AtlasBadge component on findings rows indicating plan existence +- Slide-out detail panel with plan metadata +- Local cache (`atlas_action_plans_cache` table) for instant badge rendering +- Manual cache refresh triggers re-fetch from Atlas API +- Qualys vulnerability mapping per host + +### Jira Integration + +Create, sync, and track Jira Data Center tickets linked to CVE/vendor pairs and Ivanti queue items. + +**Capabilities:** +- Create tickets from CVE records or Ivanti queue items +- Multi-item Jira ticket creation from the consolidation modal +- Flexible ticket creation — CVE/Vendor fields optional, source context tracking +- Vendor-specific issue type dropdown with per-vendor project keys +- JQL-based ticket lookup and sync +- Raw Jira status display (no status mapping) +- Save to Dashboard from Jira lookup results +- Dedicated Jira page for managing tickets +- Rate limiting with configurable window and burst limits +- Blocked dangerous endpoints (bulk delete, user management) + +### Finding Archive Tracking + +Automatic detection of disappeared and returned findings with BU reassignment classification and anomaly logging. + +**Capabilities:** +- Detect findings that disappear between syncs — classify as ARCHIVED, RETURNED, CLOSED, or CLOSED_GONE +- BU reassignment tracking with history log +- Sync anomaly detection with significance thresholds +- Anomaly banner component on the Reporting page +- Archive summary bar with state distribution +- Transition history with severity-at-transition recording +- Return classification (original finding restored vs. new duplicate) +- Configurable `IVANTI_MANAGED_BUS` for drift classification scope + +### Findings Trend + +Historical open vs closed findings chart with per-BU trend lines, archive activity sparkline, and shift reason tooltips. + +**Capabilities:** +- Ivanti counts history chart (open/closed over time) +- Per-BU trend lines via `ivanti_counts_history_by_bu` +- Archive activity sparkline overlay +- Shift reason tooltips from anomaly log + +### Knowledge Base + +Internal document library for policies, guides, and reference material. + +**Capabilities:** +- Upload documents (PDF, Markdown, Word, Excel) +- Inline PDF and Markdown viewing +- Category-based organisation with search +- Slug-based URL routing + +### Exports + +Dedicated Exports page with pre-built export cards for common data pulls. + +**Capabilities:** +- Jira Tickets export +- CCP Metrics export +- Remediation Status export +- CSV and XLSX format options + +### In-App Notifications + +Native notification system replacing the previous Webex bot integration. + +**Capabilities:** +- Per-user notification bell with unread count +- Notification types for sync events, workflow completions, and system alerts +- Mark as read / dismiss actions +- Persistent storage in `notifications` table + +### Feedback — GitLab Integration + +In-app bug reports and feature requests submitted directly to the GitLab project as issues. + +**Capabilities:** +- Feedback modal accessible from the nav drawer +- Bug report and feature request templates +- Submitted as GitLab issues via PAT authentication +- GitLab webhook receiver for issue lifecycle events (label changes, closes) +- Webhook secret validation for security + +### Access Control + +Four user groups with role-based permissions and full audit trail. + +| Group | Permissions | +|-------|------------| +| Admin | Full CRUD, user management, audit log access, system configuration, data management | +| Standard_User | Create/update operations, FP workflow submission, queue management | +| Leadership | Read access to all data plus compliance and export views | +| Read_Only | Read-only access to all data | + +**Additional capabilities:** +- Per-user BU team assignments (multi-BU tenancy) +- Per-user Ivanti identity (first/last name) for workflow filtering +- Cookie-based sessions with 24-hour expiry (httpOnly) +- Login rate limiting (20 attempts per 15-minute window) +- Full audit trail for all state-changing operations +- User profile panel with password change + +--- ## Project Structure ``` cve-dashboard/ ├── backend/ -│ ├── server.js # Express API server +│ ├── server.js # Express API — middleware, CVE/document routes inline │ ├── db.js # PostgreSQL connection pool (pg) │ ├── db-schema.sql # Complete DDL for fresh Postgres setup -│ ├── setup-postgres.js # Schema initializer (runs db-schema.sql) -│ ├── routes/ # API route handlers -│ ├── helpers/ # API clients (Ivanti, Jira, Atlas, CARD) -│ ├── middleware/ # Auth middleware -│ ├── migrations/ # Schema migrations (legacy SQLite deployments) -│ └── scripts/ # Compliance parser, data import utilities +│ ├── setup.js # One-time DB init + default admin creation +│ ├── routes/ +│ │ ├── auth.js # Login, logout, session validation +│ │ ├── users.js # User CRUD, role/group management +│ │ ├── auditLog.js # Audit log queries +│ │ ├── nvdLookup.js # NVD API proxy +│ │ ├── knowledgeBase.js # Document library CRUD +│ │ ├── archerTickets.js # Archer EXC ticket tracking +│ │ ├── archerTemplates.js # Archer template library CRUD + clone +│ │ ├── ivantiWorkflows.js # Ivanti FP workflow batch queries +│ │ ├── ivantiFindings.js # Ivanti findings sync, query, inline edit +│ │ ├── ivantiTodoQueue.js # Per-user queue staging +│ │ ├── ivantiArchive.js # Finding archive tracking + anomaly log +│ │ ├── ivantiFpWorkflow.js# FP workflow submission to Ivanti API +│ │ ├── compliance.js # AEO compliance upload + items + notes +│ │ ├── vclMultiVertical.js# VCL/CCP multi-vertical compliance +│ │ ├── atlas.js # Atlas action plan proxy + cache +│ │ ├── jiraTickets.js # Jira CRUD + REST API integration +│ │ ├── cardApi.js # CARD ownership proxy + asset-search +│ │ ├── notifications.js # In-app notification system +│ │ ├── feedback.js # GitLab issue creation for bug/feature +│ │ └── webhooks.js # GitLab webhook receiver +│ ├── helpers/ +│ │ ├── auditLog.js # logAudit() — fire-and-forget DB insert +│ │ ├── cardApi.js # CARD API — OAuth token, owner lookup, asset-search +│ │ ├── ivantiApi.js # Ivanti/RiskSense HTTP helpers +│ │ ├── atlasApi.js # Atlas action plan API +│ │ ├── jiraApi.js # Jira ticket creation + rate limiter +│ │ ├── driftChecker.js # BU drift detection between syncs +│ │ ├── vclHelpers.js # VCL metric calculation helpers +│ │ └── teams.js # Team validation helpers +│ ├── middleware/ +│ │ └── auth.js # requireAuth(), requireGroup(...groups) +│ ├── migrations/ # Sequential migration scripts (idempotent) +│ │ └── run-all.js # Run all migrations in order +│ └── scripts/ # Python utilities (compliance parsing) +│ ├── frontend/ -│ ├── src/ -│ │ ├── App.js # Main app with routing -│ │ ├── components/ # React components -│ │ └── contexts/ # Auth context -│ └── public/ +│ └── src/ +│ ├── App.js # Main app — routing, CVE list, filters, modals +│ ├── App.css # Global styles and CSS variables +│ ├── contexts/ +│ │ └── AuthContext.js # Auth state provider (login, logout, role helpers) +│ ├── utils/ +│ │ ├── graniteLoaderConfig.js # Granite column definitions and groups +│ │ ├── graniteLoaderExport.js # XLSX generation logic +│ │ └── graniteLoaderPicklists.js # Searchable dropdown options +│ └── components/ +│ ├── LoginForm.js +│ ├── NavDrawer.js +│ ├── UserMenu.js +│ ├── UserProfilePanel.js +│ ├── CalendarWidget.js +│ ├── UserManagement.js +│ ├── AuditLog.js +│ ├── NvdSyncModal.js +│ ├── KnowledgeBaseModal.js +│ ├── KnowledgeBaseViewer.js +│ ├── CardOwnerTooltip.js +│ ├── CardDetailModal.js +│ ├── CardActionModal.js +│ ├── RedirectModal.js +│ ├── LoaderModal.js +│ ├── SearchableSelect.js +│ ├── AtlasBadge.js +│ ├── AtlasIcon.js +│ ├── AtlasSlideOutPanel.js +│ ├── AdminScopeToggle.js +│ ├── ConfirmModal.js +│ ├── ConsolidationModal.js +│ ├── CveTooltip.js +│ ├── DeleteConfirmModal.js +│ ├── FeedbackModal.js +│ ├── NotificationBell.js +│ ├── RemediationModal.js +│ ├── TemplateFormModal.js +│ ├── TemplateSelector.js +│ └── pages/ +│ ├── AdminPage.js +│ ├── ReportingPage.js +│ ├── IvantiTodoQueuePage.js +│ ├── CompliancePage.js +│ ├── ComplianceUploadModal.js +│ ├── ComplianceDetailPanel.js +│ ├── ComplianceChartsPanel.js +│ ├── CCPMetricsPage.js +│ ├── VCLReportPage.js +│ ├── MetricInfoPanel.js +│ ├── BulkUploadModal.js +│ ├── MultiVerticalUploadModal.js +│ ├── IvantiCountsChart.js +│ ├── AnomalyBanner.js +│ ├── ArchiveSummaryBar.js +│ ├── ArcherPage.js +│ ├── ArcherTemplatePage.js +│ ├── JiraPage.js +│ ├── KnowledgeBasePage.js +│ └── ExportsPage.js +│ ├── docs/ │ ├── api/ # API specs (Ivanti, Atlas, Jira) -│ ├── design/ # Design system, workflow diagrams -│ ├── guides/ # User guides, full reference manual +│ ├── architecture/ # Architecture proposals (AD/SAML, split) +│ ├── design/ # Design system, workflow colour codes +│ ├── guides/ # User guides, full reference manual, VCL calculations +│ ├── operations/ # Operational logs and connectivity tests │ ├── security/ # Security audits and remediation plans │ ├── testing/ # Test plans and scripts │ └── troubleshooting/ # Investigation scripts and reports -├── docker-compose.yml # PostgreSQL 16 container definition +│ ├── scripts/ -│ └── deploy-postgres.sh # One-time deployment: container, schema, migration -├── systemd/ # systemd service files -├── start-servers.sh -└── stop-servers.sh +│ ├── deploy-postgres.sh # One-time: container, schema, migration +│ └── reset-and-migrate.sh # Dev utility: reset DB and re-run migrations +├── deploy/ +│ ├── cve-backend-production.service +│ ├── cve-backend-staging.service +│ └── setup-staging.sh +├── systemd/ +│ ├── cve-backend.service +│ └── cve-frontend.service +├── configure.js # Interactive configuration wizard +├── docker-compose.yml # PostgreSQL 16 container definition +├── start-servers.sh # Start backend (serves API + frontend build) +├── stop-servers.sh # Stop backend +└── CHANGELOG.md ``` +--- + ## Tech Stack | Layer | Technology | |-------|------------| | Backend | Node.js 18+, Express 5 | | Database | PostgreSQL 16 (Docker, port 5433) | -| Frontend | React 19, Recharts, Lucide React | -| Auth | bcryptjs, cookie-based sessions, express-rate-limit | -| Compliance | Python 3, pandas, openpyxl | +| Auth | bcryptjs, cookie-based sessions (httpOnly, 24h expiry), express-rate-limit | +| File uploads | Multer 2 (10MB limit) | +| Frontend | React 19 (Create React App / react-scripts 5) | +| Frontend serving | Express serves `frontend/build/` as static files on port 3001 | +| UI Icons | lucide-react | +| Charts | recharts | +| Spreadsheet parsing | xlsx (frontend), pandas + openpyxl (backend Python scripts) | +| Markdown rendering | react-markdown + rehype-sanitize | +| Diagrams | mermaid | +| Testing | Jest 30 (backend property tests), React Testing Library (frontend) | + +--- + +## Configuration + +All configuration lives in `backend/.env`. Copy `backend/.env.example` as a starting point. + +| Variable | Purpose | Default | +|---|---|---| +| `PORT` | Backend server port | `3001` | +| `API_HOST` | Backend bind address | `localhost` | +| `CORS_ORIGINS` | Comma-separated allowed origins | `http://localhost:3000` | +| `SESSION_SECRET` | **Required.** Session signing key. Generate with `openssl rand -base64 32` | — | +| `DATABASE_URL` | PostgreSQL connection string | — | +| `NVD_API_KEY` | NVD API key (increases rate limit from 5 to 50 req/30s) | — | +| `IVANTI_API_KEY` | Ivanti/RiskSense API key | — | +| `IVANTI_CLIENT_ID` | Ivanti client ID | `1550` | +| `IVANTI_FIRST_NAME` | Fallback Ivanti identity (first name) | — | +| `IVANTI_LAST_NAME` | Fallback Ivanti identity (last name) | — | +| `IVANTI_BU_FILTER` | Comma-separated BU values to sync | `NTS-AEO-ACCESS-ENG,NTS-AEO-STEAM` | +| `IVANTI_MANAGED_BUS` | BUs considered "managed" for drift classification | `NTS-AEO-ACCESS-ENG,NTS-AEO-STEAM` | +| `IVANTI_SKIP_TLS` | Skip TLS verification for SSL inspection proxies | `false` | +| `ATLAS_API_URL` | Atlas InfoSec API base URL | — | +| `ATLAS_API_USER` | Atlas Basic Auth username | — | +| `ATLAS_API_PASS` | Atlas Basic Auth password | — | +| `ATLAS_SKIP_TLS` | Skip Atlas TLS verification | `false` | +| `JIRA_BASE_URL` | Jira Data Center REST API base URL | — | +| `JIRA_AUTH_METHOD` | Auth method: `basic` or `pat` | `basic` | +| `JIRA_API_USER` | Jira Basic Auth username | — | +| `JIRA_API_TOKEN` | Jira Basic Auth token | — | +| `JIRA_PAT` | Jira Personal Access Token (when `JIRA_AUTH_METHOD=pat`) | — | +| `JIRA_PROJECT_KEY` | Default Jira project key | — | +| `JIRA_ISSUE_TYPE` | Default issue type | `Task` | +| `JIRA_SKIP_TLS` | Skip Jira TLS verification | `false` | +| `CARD_API_URL` | CARD API base URL | — | +| `CARD_API_USER` | CARD OAuth username | — | +| `CARD_API_PASS` | CARD OAuth password | — | +| `CARD_SKIP_TLS` | Skip CARD TLS verification | `false` | +| `GITLAB_URL` | GitLab instance URL for feedback integration | `http://steam-gitlab.charterlab.com` | +| `GITLAB_PROJECT_ID` | GitLab project numeric ID | — | +| `GITLAB_PAT` | GitLab project access token (api scope) | — | +| `GITLAB_WEBHOOK_SECRET` | Shared secret for webhook validation | — | + +> `SESSION_SECRET` and `DATABASE_URL` are required for the backend to start. All integration keys are optional — features degrade gracefully when their keys are absent. + +--- + +## Database Schema + +PostgreSQL 16 with the following tables: + +| Table | Purpose | +|-------|---------| +| `cves` | CVE records with vendor association, severity, status | +| `documents` | Uploaded advisory documents linked to CVE/vendor pairs | +| `required_documents` | Per-vendor required document types | +| `users` | User accounts with role, group, BU teams, Ivanti identity | +| `sessions` | Active session tokens with expiry | +| `audit_logs` | Immutable audit trail for all state-changing operations | +| `jira_tickets` | Locally tracked Jira tickets linked to CVE/vendor | +| `archer_tickets` | Archer risk acceptance exceptions (EXC numbers) | +| `knowledge_base` | Internal document library entries | +| `ivanti_findings` | Individual Ivanti host findings (synced from API) | +| `ivanti_sync_state` | Sync metadata — last run time, status, error | +| `ivanti_counts_cache` | Cached open/closed counts and FP workflow counts | +| `ivanti_counts_history` | Historical open/closed counts over time | +| `ivanti_counts_history_by_bu` | Per-BU historical counts | +| `ivanti_fp_submissions` | FP workflow submissions with lifecycle tracking | +| `ivanti_fp_submission_history` | Edit history for FP submissions | +| `ivanti_todo_queue` | Per-user workflow staging queue | +| `ivanti_finding_archives` | Archived/returned/closed finding records | +| `ivanti_archive_transitions` | State transition history for archived findings | +| `ivanti_sync_anomaly_log` | Sync anomaly detection log | +| `ivanti_finding_bu_history` | BU reassignment history per finding | +| `atlas_action_plans_cache` | Cached Atlas action plan data for badge rendering | +| `compliance_uploads` | Compliance xlsx upload metadata | +| `compliance_items` | Individual non-compliant items per upload | +| `compliance_notes` | Per-hostname/metric notes | + +The complete DDL is in `backend/db-schema.sql`. For fresh installs, `scripts/deploy-postgres.sh` applies it automatically. + +--- + +## Migrations + +Run all migrations (idempotent): + +```bash +cd backend && node migrations/run-all.js +``` + +Migration files in execution order: + +| Migration | Purpose | +|-----------|---------| +| `add_user_groups.js` | User group column and check constraint | +| `add_user_ivanti_identity.js` | Per-user Ivanti first/last name | +| `add_user_bu_teams.js` | Per-user BU team assignments | +| `add_knowledge_base_table.js` | Knowledge base document library | +| `add_ivanti_sync_table.js` | Ivanti sync state (single-row) | +| `add_ivanti_findings_tables.js` | Individual findings + counts cache | +| `add_ivanti_findings_ipv6_columns.js` | IPv6 fallback columns | +| `add_ivanti_counts_history_table.js` | Historical counts + per-BU | +| `add_ivanti_todo_queue_table.js` | Per-user workflow staging queue | +| `add_todo_queue_hostname.js` | Hostname column on queue items | +| `add_todo_queue_ip_address.js` | IP address column on queue items | +| `add_granite_workflow_type.js` | GRANITE workflow type | +| `add_card_workflow_type.js` | CARD workflow type | +| `add_decom_workflow_type.js` | DECOM workflow type | +| `add_remediate_workflow_type.js` | Remediate workflow type | +| `add_compliance_tables.js` | Compliance uploads, items, notes | +| `add_compliance_notes_group_id.js` | Group ID for batch notes | +| `add_compliance_history_metric_id.js` | Metric ID in compliance history | +| `add_compliance_item_history.js` | Compliance item history tracking | +| `add_fp_submissions_table.js` | FP workflow submissions | +| `add_fp_submission_editing.js` | Submission edit history | +| `add_fp_submissions_dismissed.js` | Dismissed/lifecycle status | +| `add_fp_submissions_requeued_at.js` | Requeue timestamp | +| `add_archer_tickets_table.js` | Archer EXC tickets | +| `add_archer_tickets_timestamps.js` | Timestamp columns | +| `add_archer_templates_table.js` | Archer template library | +| `add_atlas_action_plans_cache.js` | Atlas plan cache | +| `add_atlas_known_column.js` | Atlas known flag | +| `add_finding_archive_tables.js` | Archive tracking tables | +| `add_closed_gone_state.js` | CLOSED_GONE archive state | +| `add_return_classification.js` | Return classification field | +| `add_sync_anomaly_tables.js` | Anomaly detection log | +| `add_flexible_jira_ticket_creation.js` | Optional CVE/vendor on Jira tickets | +| `add_multi_item_jira_ticket.js` | Multi-item ticket references | +| `add_jira_sync_columns.js` | Jira sync metadata (SQLite) | +| `add_jira_sync_columns_pg.js` | Jira sync metadata (PostgreSQL) | +| `drop_jira_status_check_constraint.js` | Remove status enum for raw display | +| `add_notifications_table.js` | In-app notification system | +| `add_created_by_columns.js` | created_by on archer_tickets | +| `add_queue_remediation_notes_table.js` | Remediation notes on queue items | +| `add_vcl_multi_vertical.js` | VCL multi-vertical tables | +| `add_vcl_reporting_columns.js` | VCL reporting metadata | +| `add_vcl_vertical_metadata.js` | VCL vertical team fields | +| `backfill_anomaly_log.js` | Backfill anomaly classification data | +| `backfill_return_classification.js` | Backfill return classification | +| `reclassify_bu_roundtrips.js` | Reclassify BU roundtrip transitions | + +--- + +## API Reference + +All routes are prefixed with `/api`. All endpoints except login, logout, health check, and webhooks require a valid session cookie. + +### Public + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/health` | Health check (used by CI/CD verification) | +| POST | `/api/auth/login` | Authenticate and create session | +| POST | `/api/auth/logout` | Destroy session | +| POST | `/api/webhooks/gitlab` | GitLab webhook receiver (validated by secret) | + +### CVE Management + +| Method | Path | Description | Group | +|--------|------|-------------|-------| +| GET | `/api/cves` | List CVEs with filtering | Any | +| POST | `/api/cves` | Create CVE | Standard_User+ | +| PUT | `/api/cves/:id` | Update CVE | Standard_User+ | +| DELETE | `/api/cves/:id` | Delete CVE | Admin | +| POST | `/api/cves/:cveId/:vendor/documents` | Upload document | Standard_User+ | +| GET | `/api/cves/:cveId/:vendor/documents` | List documents | Any | +| DELETE | `/api/documents/:id` | Delete document | Admin | + +### Users and Auth + +| Method | Path | Description | Group | +|--------|------|-------------|-------| +| GET | `/api/auth/me` | Current session user | Any | +| GET | `/api/users` | List all users | Admin | +| POST | `/api/users` | Create user | Admin | +| PUT | `/api/users/:id` | Update user (role, group, teams) | Admin | +| DELETE | `/api/users/:id` | Delete user | Admin | +| PUT | `/api/users/:id/password` | Change password | Owner or Admin | + +### Ivanti Findings + +| Method | Path | Description | Group | +|--------|------|-------------|-------| +| GET | `/api/ivanti/findings` | Query synced findings | Any | +| POST | `/api/ivanti/findings/sync` | Trigger manual sync | Standard_User+ | +| PUT | `/api/ivanti/findings/:id/note` | Update finding note | Standard_User+ | +| PUT | `/api/ivanti/findings/:id/override` | Override hostname/DNS | Standard_User+ | + +### Ivanti Queue + +| Method | Path | Description | Group | +|--------|------|-------------|-------| +| GET | `/api/ivanti/todo-queue` | Get user's queue | Any | +| POST | `/api/ivanti/todo-queue` | Add item(s) to queue | Standard_User+ | +| PUT | `/api/ivanti/todo-queue/:id` | Update queue item | Standard_User+ | +| DELETE | `/api/ivanti/todo-queue/:id` | Remove queue item | Standard_User+ | +| DELETE | `/api/ivanti/todo-queue/completed` | Clear completed items | Standard_User+ | + +### Ivanti FP Workflow + +| Method | Path | Description | Group | +|--------|------|-------------|-------| +| POST | `/api/ivanti/fp-workflow` | Submit FP workflow to Ivanti | Standard_User+ | +| GET | `/api/ivanti/fp-workflow/submissions` | List user's submissions | Any | +| PUT | `/api/ivanti/fp-workflow/submissions/:id` | Edit/resubmit | Standard_User+ | + +### Ivanti Archive + +| Method | Path | Description | Group | +|--------|------|-------------|-------| +| GET | `/api/ivanti/archive` | Query archived findings | Any | +| GET | `/api/ivanti/archive/anomalies` | Sync anomaly log | Any | +| GET | `/api/ivanti/archive/transitions/:id` | Transition history | Any | + +### Compliance (AEO) + +| Method | Path | Description | Group | +|--------|------|-------------|-------| +| POST | `/api/compliance/upload` | Upload xlsx | Standard_User+ | +| GET | `/api/compliance/items` | Query non-compliant items | Any | +| GET | `/api/compliance/trends` | Compliance trend data | Any | +| POST | `/api/compliance/notes` | Add note to hostname/metric | Standard_User+ | + +### VCL Multi-Vertical (CCP Metrics) + +| Method | Path | Description | Group | +|--------|------|-------------|-------| +| POST | `/api/compliance/vcl-multi/upload` | Upload VCL xlsx | Standard_User+ | +| GET | `/api/compliance/vcl-multi/verticals` | List verticals | Any | +| GET | `/api/compliance/vcl-multi/metrics` | Query metrics | Any | +| GET | `/api/compliance/vcl-multi/forecast` | Burndown forecast | Any | +| DELETE | `/api/compliance/vcl-multi/verticals/:id` | Delete vertical | Admin | + +### Atlas + +| Method | Path | Description | Group | +|--------|------|-------------|-------| +| GET | `/api/atlas/hosts/:hostId/plans` | Get plans for host | Any | +| PUT | `/api/atlas/hosts/:hostId/plans` | Create plan | Standard_User+ | +| PATCH | `/api/atlas/hosts/:hostId/plans` | Update plan | Standard_User+ | +| POST | `/api/atlas/hosts/:hostId/refresh` | Refresh cache | Standard_User+ | + +### Jira + +| Method | Path | Description | Group | +|--------|------|-------------|-------| +| GET | `/api/jira-tickets` | List local tickets | Any | +| POST | `/api/jira-tickets` | Create ticket (local + Jira API) | Standard_User+ | +| GET | `/api/jira-tickets/lookup` | JQL search against Jira | Standard_User+ | +| POST | `/api/jira-tickets/save` | Save from Jira lookup | Standard_User+ | + +### CARD + +| Method | Path | Description | Group | +|--------|------|-------------|-------| +| GET | `/api/card/owner/:assetId` | Lookup owner | Any | +| GET | `/api/card/asset-search/:hostId` | Search by Ivanti host ID | Any | +| POST | `/api/card/owner/:assetId/confirm` | Confirm ownership | Standard_User+ | +| POST | `/api/card/owner/:assetId/decline` | Decline ownership | Standard_User+ | +| POST | `/api/card/owner/:assetId/redirect` | Redirect ownership | Standard_User+ | +| GET | `/api/card/teams` | List all CARD teams | Any | + +### Other + +| Method | Path | Description | Group | +|--------|------|-------------|-------| +| GET | `/api/audit-logs` | Query audit trail | Admin | +| GET | `/api/nvd/:cveId` | NVD metadata lookup | Any | +| GET/POST | `/api/knowledge-base` | Document library CRUD | Any / Standard_User+ | +| GET/POST | `/api/archer-tickets` | Archer ticket CRUD | Any / Standard_User+ | +| GET/POST | `/api/archer-templates` | Template library CRUD | Any / Standard_User+ | +| GET | `/api/notifications` | User notifications | Any | +| POST | `/api/feedback` | Submit bug/feature to GitLab | Any | + +--- + +## CI/CD Pipeline + +Defined in `.gitlab-ci.yml`. Stages: install, lint, test, build, deploy, verify. + +| Stage | What it does | +|-------|-------------| +| install | `npm ci` for root + frontend | +| lint | ESLint on frontend (warning threshold: 25) | +| test | Jest backend property tests + frontend unit tests | +| build | `npm run build` in frontend/ | +| deploy-staging | rsync to 71.85.90.9, run migrations, restart service (auto on master) | +| deploy-production | rsync to 71.85.90.6, run migrations, restart service (manual trigger) | +| verify | Hit `/api/health` endpoint, post issue comments for `Closes #N` references | + +**Runner:** Docker executor on LXC 108 (71.85.90.8), Runner #6, `node:18` image for build stages, `alpine:latest` for deploy. + +**Required CI/CD variables:** `DATABASE_URL`, `SSH_PRIVATE_KEY` (base64), `GITLAB_PAT`. + +--- ## Documentation - **[Full Reference Manual](docs/guides/full-reference-manual.md)** — comprehensive feature documentation, API reference, database schema, security model, and configuration details +- **[VCL Metric Calculations](docs/guides/vcl-metric-calculations.md)** — formula reference for CCP Metrics compliance calculations - **[Postgres Migration Plan](docs/guides/postgres-migration-plan.md)** — architecture decisions, schema design, and cutover procedure for the SQLite to PostgreSQL migration - **[Migration Guide](backend/migrations/README.md)** — schema migration scripts for upgrading existing deployments -- **[Design System](docs/design/design-system.md)** — UI component patterns and color system +- **[Design System](docs/design/design-system.md)** — UI component patterns and colour system - **[Ivanti API Reference](docs/api/ivanti-api-reference.md)** — Ivanti/RiskSense API integration details - **[Jira API Use Cases](docs/api/jira-api-use-cases.md)** — Jira Data Center API compliance summary +- **[AD/SAML Integration Architecture](docs/architecture/ad-saml-integration.md)** — planned Active Directory / SAML SSO integration +- **[Security Audit Tracker](docs/security/security-audit-tracker.md)** — living security audit document + +--- + +## Troubleshooting + +### Backend fails to start with "FATAL: SESSION_SECRET" + +**Symptom:** Server exits immediately with `FATAL: SESSION_SECRET environment variable must be set`. + +**Cause:** The `SESSION_SECRET` env var is not configured in `backend/.env`. + +**Fix:** Generate a secret and add it to `backend/.env`: +```bash +echo "SESSION_SECRET=$(openssl rand -base64 32)" >> backend/.env +``` + +### Database connection refused + +**Symptom:** Backend logs `Error: connect ECONNREFUSED` on startup. + +**Cause:** PostgreSQL container is not running or `DATABASE_URL` is misconfigured. + +**Fix:** +```bash +docker compose up -d # Start the Postgres container +# Verify DATABASE_URL in backend/.env matches: postgresql://steam:@localhost:5433/cve_dashboard +``` + +### CARD API timeouts in production + +**Symptom:** CARD tooltip shows timeout errors or takes minutes to respond. + +**Cause:** IPv6 AAAA records for `card.charter.com` are unreachable from this network. Node.js defaults to trying IPv6 first. + +**Fix:** The backend sets `dns.setDefaultResultOrder('ipv4first')` globally. If you still see issues, verify the setting is at the top of `server.js` before any network imports. + +### Compliance upload fails with "No items parsed" + +**Symptom:** Upload completes but reports 0 new, 0 resolved, 0 recurring items. + +**Cause:** The xlsx file structure does not match the expected format (sheet names, column headers). + +**Fix:** Ensure the compliance xlsx has the expected sheet layout. Check `backend/scripts/parse_compliance_xlsx.py` for the expected column mappings. + +### Missing migrations after deploy + +**Symptom:** API returns 500 errors referencing missing columns or tables. + +**Cause:** Migrations were not run after pulling new code. + +**Fix:** +```bash +cd backend && node migrations/run-all.js +``` + +### Frontend changes not visible after deploy + +**Symptom:** UI still shows old version after code changes. + +**Cause:** The frontend must be rebuilt after any source changes — Express serves the static build. + +**Fix:** +```bash +cd frontend && npm run build +# Restart backend or refresh the browser +``` + +--- ## License