Commit Graph

216 Commits

Author SHA1 Message Date
Jordan Ramos
ae2b7e0433 Fix forecast deduplication for multi-vertical metrics
Devices appearing in multiple verticals were counted multiple times,
causing non_compliant > totalAssets and negative compliance percentages.
Deduplicate by hostname before passing to the forecast helper.
2026-05-20 17:53:29 -06:00
Jordan Ramos
e45deccdb7 Fix forecast burndown chart data issues
- Fix Date object handling for resolution_date from PostgreSQL
- Fix totalAssets using per-metric summary (vcl_multi_vertical_summary)
  instead of vertical-level compliance_snapshots total_devices
- Fix duplicate current month in chart (forecast starts from next month)
- Fix multi-vertical metrics summing across all relevant verticals
- Fix bar stacking: orange (non-compliant) on bottom, blue (compliant)
  on top, both sharing same baseline (stacked to total)
- Add fill props to Bar components for correct legend colors
- Backfill historical snapshots with per-metric totalAssets
2026-05-20 17:28:20 -06:00
Jordan Ramos
f9770872ba Add Jira production UAT test script, update CHANGELOG
- Jira UAT test script for production API validation (all 10 use cases)
- CHANGELOG updates for recent features and fixes
2026-05-20 16:15:37 -06:00
Jordan Ramos
f9b96e9040 Add per-metric forecast burndown chart to CCP Metrics page
New feature: combined historical + forecast burndown chart with metric
selector on the CCP Metrics page. Shows stacked bars (total assets vs
non-compliant) with a compliance percentage trend line. A bold divider
separates actual historical data from projected future remediation.
Forecast assumes constant asset count and on-schedule remediation plans.

Backend:
- computeMetricForecastBurndown helper in vclHelpers.js (pure function)
- GET /api/compliance/vcl-multi/metrics-list endpoint
- GET /api/compliance/vcl-multi/metric/:metricId/forecast-burndown endpoint

Frontend:
- MetricSelector dropdown with device counts per metric
- ForecastBurndownChart using recharts ComposedChart (Bar + Line + ReferenceLine)
- Forecast bars render at 50% opacity to distinguish from actuals
- Race condition handling for rapid metric switching
- Queue panel width increased from 420px to 600px

Closes #18
2026-05-20 16:15:21 -06:00
Jordan Ramos
df31cc3c79 Update JQL property test to reflect cross-project sync fix
The project filter was intentionally removed from searchIssuesByKeys() to
fix cross-project ticket sync. Update the property test to no longer assert
the presence of 'project =' in the generated JQL.
2026-05-20 14:01:28 -06:00
Jordan Ramos
56bd5ca148 Restructure CCP Metrics to metric-first hierarchy, fix Jira cross-project sync
CCP Metrics View Restructure:
- Add GET /metrics endpoint (aggregated across verticals)
- Add GET /metric/:id/verticals endpoint (per-vertical breakdown)
- Replace VerticalTable with MetricTable on overview (one row per metric)
- Add MetricDetailView for metric-first drill-down
- Restructure navigation: Metric → Vertical → Subteam → Devices
- Remove By Vertical table from AggregatedBurndownChart

Jira Sync Fix:
- Remove hardcoded project filter from getIssue() and searchIssuesByKeys()
- Issue keys are globally unique; project filter broke cross-project tickets
- Fixes 502 Bad Gateway when syncing tickets from non-STEAM projects
2026-05-20 13:30:22 -06:00
Jordan Ramos
f00a1ce7bb Replace Webex bot with in-app notification system
Org blocks external Webex bots, so replaced the DM approach with an in-app
notification bell. GitLab webhook still fires on issue close, but now writes
to a notifications table instead of calling Webex API.

- New: notifications table + migration
- New: GET/PATCH/POST /api/notifications endpoints
- New: NotificationBell component (bell icon + badge + dropdown)
- Removed: backend/helpers/webexBot.js (org-blocked)
- Removed: WEBEX_BOT_TOKEN from .env
2026-05-18 17:15:05 -06:00
Jordan Ramos
00bf92a2a1 Add screenshot uploads to feedback modal, Webex bot DM on issue close
- Feedback modal now supports up to 3 image attachments (PNG/JPG/GIF/WebP, 5MB
  each) with thumbnail previews. Images are uploaded to GitLab project uploads
  and embedded as markdown in the issue description.
- New webhook endpoint (POST /api/webhooks/gitlab) receives issue close events,
  parses the submitter from the description, looks up their email, and sends a
  Webex DM via the Patches O'Houlihan bot.
- New helper: backend/helpers/webexBot.js (fire-and-forget DM sender).
- Requires WEBEX_BOT_TOKEN and GITLAB_WEBHOOK_SECRET in backend/.env.
2026-05-18 16:54:00 -06:00
Jordan Ramos
520f50fbbf Fix duplicate failing metrics on same asset across compliance endpoints
Deduplicate (hostname, metric_id) rows across verticals using DISTINCT ON in
GET /items, GET /items/:hostname, GET /vcl/stats (heavy-hitters + forecast),
GET /mttr, and persistUpload() snapshot block. Add defensive groupByHostname
Set and hostname_status CTE for snapshot classification.

Includes 38 property-based tests (11 exploration + 27 preservation) covering
all six affected sites.

Closes #13
2026-05-18 15:57:10 -06:00
Jordan Ramos
3814de5845 Fix duplicate chart entries on compliance page when multiple verticals share a report_date
Aggregate /trends, /top-recurring, /category-trend by report_date instead of
per-upload row. Add sibling-upload disclosure to /summary. Filter persistUpload
snapshot query by the upload's vertical to prevent cross-vertical contamination.

Fixes GitLab #12 (reported by nkapur — STEAM active findings chart showed 3
entries for 5/11 after uploading three vertical data sets for that date).

Includes 30 property-based tests covering bug condition and preservation.
2026-05-18 15:00:53 -06:00
Jordan Ramos
487489e26c Add unified setup script (configure.js) merging deploy + config wizard
Single-file Node.js CLI that orchestrates the full setup lifecycle:
- Interactive env var configuration with validation and smart defaults
- Postgres provisioning via Docker Compose with readiness polling
- Schema initialization (psql with docker exec fallback)
- npm dependency installation with 120s timeout
- Optional SQLite-to-Postgres data migration with retry logic
- Frontend build with smart skip on reconfiguration

Includes 84 tests: 50 property-based (fast-check) covering 19 correctness
properties, and 34 integration tests for filesystem and parsing flows.
2026-05-18 11:58:21 -06:00
Jordan Ramos
3643c123b4 Fix requeue inserting Postgres array literal instead of JSON into cves_json 2026-05-15 17:41:38 -06:00
Jordan Ramos
be1d357692 Fix todo queue crash on malformed cves_json data 2026-05-15 17:31:19 -06:00
Jordan Ramos
492780fd90 Add aggregated burndown forecast to CCP Metrics overview page 2026-05-15 17:08:55 -06:00
Jordan Ramos
1fe6c1f84c Add remediation plan and resolution date history tracking
New table compliance_item_history stores an append-only audit trail of
changes to resolution_date and remediation_plan. The current values remain
on compliance_items for fast VCL reporting queries (no double-counting).

Backend:
- Migration: creates compliance_item_history with indexes
- PATCH /items/:hostname/metadata: records old→new in history before updating,
  accepts optional change_reason field (max 500 chars)
- GET /items/:hostname: returns history array (last 10 entries, newest first)
- POST /vcl/bulk-commit: records history for each changed field per hostname

Frontend:
- ComplianceDetailPanel: added change reason input below Save button
- Added Change History section showing field changes with timestamps,
  usernames, old→new values, and reasons
- Re-fetches detail after save to show updated history immediately

Tests updated to match new transaction-based PATCH flow.
2026-05-15 10:53:14 -06:00
Jordan Ramos
97e5d68d8e Fix AEO compliance page not showing metric health cards on dev
The /summary endpoint was fetching the most recent upload regardless of
vertical, which on dev was a PRDCT_VSO multi-vertical upload. Now it
looks for AEO uploads (vertical IS NULL) first, then falls back to the
NTS_AEO multi-vertical upload.

The /items endpoint now includes items from both vertical IS NULL and
vertical = 'NTS_AEO' so the AEO compliance page shows devices uploaded
through either flow.
2026-05-14 15:39:25 -06:00
Jordan Ramos
7577ab1219 Make Non-Compliant stat clickable — reveals metric breakdown buttons
Clicking the Non-Compliant card on the CCP Metrics overview now toggles a
panel of metric buttons below it, each showing the metric ID, category,
non-compliant count, and compliance % vs target. Styled like the compliance
page's MetricHealthCard pattern.

Backend: added metric_breakdown to the /stats response — aggregated
cross-vertical metric totals (ALL: rows only, grouped by metric_id).

Also updated tech steering file to document the single-port Express
architecture and the requirement to run npm run build after frontend changes.
2026-05-14 15:24:10 -06:00
Jordan Ramos
a2bc1ff564 Add metric sub-team intermediate drill-down view
Clicking a metric now shows a sub-team breakdown page with totals per team
(compliant, non-compliant, total, %) instead of jumping directly to a flat
device list. Clicking a sub-team then shows the device list filtered to
that team only.

Navigation flow: Overview → Vertical → Metric (sub-team totals) → Team (devices)

Backend: added optional ?team= query param to the device list endpoint for
filtered queries.

Frontend: added MetricSubTeamView component with metric-level stats bar and
clickable sub-team table. Updated navigation state to include selectedTeam.

Also updated design brief to reflect the new drill-down hierarchy.
2026-05-14 14:53:41 -06:00
Jordan Ramos
61d7e00d4f Add sub-team level display to CCP Metrics vertical drill-down
Backend: restructured /vertical/:code/metrics endpoint to return metrics
with nested sub_teams arrays. Each metric now has the ALL: rollup as the
primary row and individual team breakdowns (ACCESS-OPS, STEAM, etc.) as
sub_teams. Also returns a teams array for the filter UI.

Frontend: VerticalDetailView now supports two interaction modes:
- Expand/collapse: click the arrow on any metric row to reveal sub-team
  breakdown inline (teal-highlighted rows beneath the parent)
- Team filter: click a team button to filter the entire table to show
  only that team's numbers per metric

Both modes avoid double-counting by using the ALL: rollup for totals
and only showing sub-team data as supplementary detail.
2026-05-14 12:27:46 -06:00
Jordan Ramos
ebaf4cd18c Fix double-counting in VCL multi-vertical stats — use only ALL: rollup rows
The Summary sheet in each vertical spreadsheet contains both sub-team rows
(ACCESS-OPS, STEAM, INTELDEV, etc.) AND a rollup row (ALL: NTS-AEO) per
metric. The rollup row already includes all sub-team totals, so summing
all rows was double-counting every device.

Fixed in three places:
- GET /stats endpoint: added AND team LIKE 'ALL:%' filter
- persistMultiVerticalUpload snapshot creation: only sum ALL: entries
- GET /vertical/:code/metrics category aggregation: only use ALL: rows

Also ran a one-time data fix to correct existing compliance_snapshots.
2026-05-14 12:09:44 -06:00
Jordan Ramos
55238ec71e Fix compliance stats to use Summary sheet data instead of item counts
The compliance_items table only contains non-compliant devices (detail
sheet rows). Compliant devices are never inserted — they only exist in
the Summary sheet totals. This caused Compliant to show 0 and
Compliance % to show 0% for all verticals.

Fix: stats endpoint now reads from vcl_multi_vertical_summary (parsed
Summary sheet data) for total/compliant/non-compliant counts. Snapshot
creation also uses summary data for accurate trend charting.

The compliance_items table is still used for:
- Donut chart (blocked vs in-progress based on resolution_date)
- Burndown forecast (devices with/without resolution dates)
- Device drill-down (actual non-compliant device list)
2026-05-14 12:01:19 -06:00
Jordan Ramos
408aaa7012 Add data management panel with delete vertical, rollback upload, and reset all
Backend:
- DELETE /api/compliance/vcl-multi/vertical/:code — wipe a single vertical
- DELETE /api/compliance/vcl-multi/upload/:uploadId — rollback most recent upload
- DELETE /api/compliance/vcl-multi/all — nuclear reset of all multi-vertical data
- All delete operations are Admin-only and audit-logged

Frontend:
- Manage button (red, Admin-only) in CCP Metrics header
- DataManagementPanel modal showing upload history grouped by vertical
- Per-vertical delete button
- Per-upload rollback button (most recent only)
- Reset All button with confirmation dialog
- Success/error messaging
2026-05-14 11:54:58 -06:00
Jordan Ramos
1eb8eab76f Fix route mount order: vcl-multi must precede general compliance router 2026-05-14 10:15:15 -06:00
Jordan Ramos
04360cc4bc Add CCP Metrics page with multi-vertical VCL upload and cross-org reporting
New feature: multi-file per-vertical compliance xlsx upload with scoped
resolution logic, executive-level aggregated reporting, and drill-down
by vertical and metric. Supports daily upload cadence and batch commit.

Backend:
- Migration: add vertical column to compliance_items/uploads, create
  vcl_multi_vertical_summary table
- New route module: routes/vclMultiVertical.js with preview, commit,
  stats, trend, metric drill-down, device list, and burndown endpoints
- New helpers: parseVerticalFilename(), computeVerticalBurndown()
- Vertical-scoped resolution: uploading one vertical never resolves
  items from other verticals

Frontend:
- CCPMetricsPage with stats bar, trend chart, donut, vertical table
- Drill-down: vertical -> metrics by category -> device list
- Per-vertical burndown forecast chart
- MultiVerticalUploadModal: multi-file drag-drop, batch preview, commit
- Nav entry: CCP Metrics (Building2 icon)

Docs:
- Design brief for stakeholder meeting (docs/vcl-multi-vertical-design-brief.md)
2026-05-14 09:49:59 -06:00
Jordan Ramos
808625dab4 Fix requeue: fallback to finding_ids_json when queue items are deleted or absent
The requeue endpoint now handles three scenarios:
1. Original queue items still exist — uses their finding data (ideal case)
2. Queue items deleted (Clear Completed) — looks up findings from
   ivanti_findings table using finding_ids_json
3. FP created outside dashboard (no queue_item_ids) — same fallback
   to finding_ids_json and ivanti_findings lookup
4. Last resort — creates queue items with just finding IDs if the
   findings aren't in ivanti_findings either
2026-05-13 16:57:57 -06:00
Jordan Ramos
0fefd2a707 Add re-queue findings from rejected FP submissions
New feature: users can re-queue findings from a rejected FP submission
back into the Ivanti todo queue under a different workflow type (FP,
Archer, CARD, GRANITE, or DECOM). Primary use case is when an FP is
rejected with a recommendation to submit an Archer risk acceptance.

Backend:
- New migration: add requeued_at column to ivanti_fp_submissions
- New endpoint: POST /api/ivanti/fp-workflow/submissions/:id/requeue
  - Validates workflow_type and vendor (required for FP/Archer/DECOM)
  - Creates new pending queue items from original finding data
  - Marks submission as requeued (prevents double re-queue)
  - Audit logs the action

Frontend (ReportingPage.js):
- RequeueConfirmDialog component with workflow type selector and vendor input
- Re-queue Findings button in Edit FP Modal header (rejected submissions only)
- Already re-queued label when submission.requeued_at is set
- Success notification on completion
2026-05-13 16:46:49 -06:00
Jordan Ramos
828e7cc45d Sync FP submission lifecycle_status from Ivanti currentState on fetch
When GET /submissions enriches submissions with Ivanti API data, it now
checks if batch.currentState (APPROVED, REJECTED, REWORK) differs from
the local lifecycle_status and updates the DB accordingly. This ensures
approved submissions get filtered out of the queue panel as intended.

Also changed safeText() to return null for non-string Ivanti note values
(arrays/objects) instead of JSON-stringifying them. The notes array
filters nulls via .filter(Boolean) so non-string data is simply hidden.
2026-05-13 14:36:05 -06:00
Jordan Ramos
5126ccc6ae Fix History tab crash: coerce Ivanti note fields to strings before rendering
PostgreSQL + Ivanti API enrichment can return non-string values
(objects/arrays) for currentStateUserNotes and similar fields.
React crashes silently (blank page, no console error) when trying
to render non-string values as children. Same root cause pattern
as Bug 3 in ivanti-panel-bugs-2026-05-12.

Added safeText() wrapper that coerces any non-string truthy value
to a JSON string before rendering in the History tab notes section.

Also fixed flaky property test: fc.date() could generate invalid
dates causing RangeError on .toISOString(). Added .filter() guard
and explicit UTC date bounds.
2026-05-13 14:24:16 -06:00
Jordan Ramos
671894ff5f Fix vcl-compliance-reporting test: stats.total → stats.total_devices 2026-05-13 09:56:30 -06:00
Jordan Ramos
9eec63ea42 Add VCL vertical metadata: inline-editable team fields, JSDoc on compliance routes, stats query rewrite 2026-05-13 07:57:41 -06:00
Jordan Ramos
0d29a1b84e Document IVANTI_MANAGED_BUS env var in .env.example, reference manual, and API docs 2026-05-13 07:56:03 -06:00
Jordan Ramos
4416f6a25d Make EXPECTED_BUS configurable via IVANTI_MANAGED_BUS env var for multi-tenant drift classification 2026-05-12 15:27:58 -06:00
Jordan Ramos
97d378033b Revert EXPECTED_BUS to STEAM+ACCESS-ENG: findings leaving managed teams are BU reassignments 2026-05-12 15:22:52 -06:00
Jordan Ramos
537cf96a0a Fix BU drift checker: derive EXPECTED_BUS from IVANTI_BU_FILTER env var instead of hardcoded 2 BUs 2026-05-12 15:18:44 -06:00
Jordan Ramos
8c93e86fe0 Fix Ivanti panel bugs: Invalid Date, wrong workflow count, crash on archive click, BU scope filtering 2026-05-12 14:21:46 -06:00
Jordan Ramos
d093a3d113 Add VCL compliance reporting: exec report page, device metadata fields, bulk upload 2026-05-11 15:48:10 -06:00
Jordan Ramos
955036145d Fix property test CI failure: mock db module before importing route 2026-05-11 14:51:16 -06:00
Jordan Ramos
7245352496 Add FP submissions cleanup: auto-clear approved, dismiss rejected, collapsible section 2026-05-11 14:29:50 -06:00
Jordan Ramos
cda1eaadc9 Add DECOM workflow type, auto-note/hide on decom, show CVEs on CARD queue items, auto-run migrations in pipeline
- Add DECOM to queue workflow types (red badge, inventory-style display)
- When findings are added as DECOM, auto-set note to 'DECOM' and hide row
- Hidden rows are excluded from donut charts (removes from pending count)
- Show CVEs on CARD/GRANITE/DECOM queue items (was previously omitted)
- Add backend/migrations/run-all.js for CI/CD auto-migration execution
- Pipeline now runs migrations before service restart on both staging and prod
- Add add_decom_workflow_type.js migration (updates CHECK constraint)
2026-05-08 14:51:05 -06:00
Jordan Ramos
890d7b82dc Fix CI: exclude test files from lint, mock db.js in jira test for runner env 2026-05-08 13:11:06 -06:00
Jordan Ramos
de2c5f245e Add CI/CD pipeline, feedback modal, Atlas qualys_id fallback, and health endpoint
- Rewrite .gitlab-ci.yml with proper stages, blocking tests, staging
  environment on dev box, and SSH-based production deploy to 71.85.90.6
- Add POST /api/health endpoint for pipeline verification
- Add POST /atlas/hosts/:hostId/refresh-cache for Atlas cache staleness
- AtlasSlideOutPanel: auto-resolve qualys_id from Atlas vulnerabilities,
  prefer qualys_id over active_host_findings_id, retry on failure
- Add FeedbackModal component with bug report button in header and
  feature request in UserMenu, creates GitLab issues via /api/feedback
- Fix all frontend test failures (ESM transforms, TextDecoder polyfill,
  fast-check resolution, App.test.js boilerplate replacement)
- Fix root package.json test script to run jest
- Add deploy/ directory with staging systemd service and setup script
2026-05-08 12:50:11 -06:00
Jordan Ramos
3db84a377b Fix null bu_teams in postgres migration, add retry logic to deploy script 2026-05-07 13:28:19 -06:00
Jordan Ramos
1b8790ff16 fix: add missing created_by column to archer_tickets table 2026-05-06 15:29:20 -06:00
Jordan Ramos
cf43e85c38 fix: scope FP workflow counts donut by BU
- Rewrite /fp-workflow-counts endpoint to query ivanti_findings table
  directly with optional teams ILIKE filter (replaces pre-computed JSON blob)
- Frontend passes getActiveTeamsParam() to FP counts fetch
- FP counts refresh on scope toggle change alongside open/closed counts
- Both FP Finding Status and FP Workflow Status donuts now respect BU scope
2026-05-06 15:19:34 -06:00
Jordan Ramos
6163be626e ops: add docker-compose.yml and deploy-postgres.sh for production cutover
- docker-compose.yml: Postgres 16 Alpine on port 5433 with healthcheck
- scripts/deploy-postgres.sh: one-shot deployment script that handles
  container startup, schema creation, npm install, data migration, and
  frontend build
- Backup SQLite database as cve_database.db.pre-postgres-backup
2026-05-06 15:07:06 -06:00
Jordan Ramos
573903a885 feat: per-BU trend lines in counts history chart
- Create ivanti_counts_history_by_bu table (bu_ownership, state, count per sync)
- Sync writes per-BU snapshot alongside global history on each sync
- Seed table with current counts for immediate first data point
- GET /counts/history accepts ?teams param — queries per-BU table when filtered
- IvantiCountsChart accepts teamsParam prop, re-fetches on scope change
- ReportingPage passes getActiveTeamsParam() to the chart
- Historical per-BU data accumulates from this point forward
- Global history (no filter) still uses the original aggregate table
2026-05-06 13:38:38 -06:00
Jordan Ramos
77f113e9ae fix: load dotenv in db.js so DATABASE_URL is available on import 2026-05-06 12:30:45 -06:00
Jordan Ramos
8cd73c126e feat(postgres): data migration + per-BU closed counts in frontend
- Create backend/scripts/migrate-to-postgres.js (one-time SQLite→Postgres copy)
- Successfully migrated: 6 users, 21 CVEs, 6307 findings, 20965 compliance items,
  138 archives, 67 atlas plans, all notes/overrides merged
- All 22 tables verified with matching row counts
- Frontend StatusDonut now uses server-provided per-BU counts (no more N/A)
- Counts endpoint called with teams param on scope change
- Re-fetch counts when admin scope toggle changes
2026-05-06 12:26:54 -06:00
Jordan Ramos
e30ad79f2a feat(postgres): rewrite Ivanti findings to individual rows
- Replace 2.6MB JSON blob with individual rows in ivanti_findings table
- Batch upsert via INSERT ... ON CONFLICT in chunks of 100
- Sync stores both open AND closed findings as rows with state column
- Per-BU closed counts now possible via SQL GROUP BY
- GET /findings queries indexed table with optional ILIKE BU filter
- GET /counts returns per-BU open+closed via GROUP BY state
- Notes and overrides are columns on ivanti_findings (no separate tables)
- Removed: readState, readStateWithNotes, _findingsCache, initTables
- Preserved: extractFinding, archive detection, FP workflow counts, anomaly log
- Response shape unchanged — frontend works without modification
2026-05-06 12:12:34 -06:00
Jordan Ramos
33927b150b feat(postgres): migrate all route files from SQLite to pg pool
- All 16 route files now import pool from ../db directly
- Removed db parameter from all factory functions
- All callbacks replaced with async/await pool.query()
- All ? placeholders converted to $1, $2... numbered params
- datetime('now') → NOW(), INSERT OR IGNORE → ON CONFLICT DO NOTHING
- LIKE → ILIKE for case-insensitive searches
- Error detection: err.code === '23505' for unique violations
- server.js no longer passes pool/db/requireAuth to route factories
- Only ivantiFindings.js still receives pool (pending task 8 rewrite)
2026-05-06 11:44:17 -06:00