diff --git a/CHANGELOG.md b/CHANGELOG.md index 18fe9cb..cb46e9d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,59 +1,76 @@ # Changelog -## v1.0.0 — 2026-05-01 +All notable changes to the STEAM Security Dashboard are documented in this file. -First official release. Consolidates all features developed since initial commit into a stable, documented, deployment-ready package. +Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -### Core Platform -- CVE tracking with multi-vendor support, document storage, and NVD API auto-fill -- Session-based authentication with four user groups (Admin, Standard_User, Leadership, Read_Only) -- Full audit logging of all state-changing actions -- Dark tactical intelligence UI theme with monospace typography +--- -### Ivanti Integration -- Live sync of open host findings from Ivanti/RiskSense API (auto-sync every 24h) -- Reporting page with donut metric charts, advanced per-column filtering, inline editing -- FP workflow submission directly to Ivanti API with file attachments -- Ivanti Queue — personal staging list for batch FP, Archer, CARD, and Granite workflows -- Queue item redirect between workflow types after completion -- Row visibility controls with localStorage persistence +## [2.0.0] — 2026-05-19 -### Archive and Anomaly Tracking -- Automatic detection of disappeared and returned findings across syncs -- BU drift checker — classifies archived findings by reason (BU reassignment, severity drift, closed on platform, decommissioned) -- Return classification — explains why findings came back (BU reassigned back, severity re-escalated, etc.) -- Findings Trend chart with archive activity sparkline and shift reason tooltips -- Anomaly banner for significant archive events +### Breaking Changes -### Compliance (AEO Posture) -- Weekly NTS_AEO xlsx upload with diff preview (new, resolved, recurring) -- Schema drift detection with breaking/silent-miss/cosmetic classification -- Admin config reconciliation for parser updates -- Per-team metric health cards with grouped categories and variant pills -- Device-level violation tracking with timestamped notes history -- Multi-metric note grouping -- Upload rollback support +- **PostgreSQL migration** — database engine switched from SQLite to PostgreSQL. Requires running `deploy-postgres.sh`, data migration, and `DATABASE_URL` env var. SQLite is no longer supported. +- **Multi-BU tenancy** — data is now scoped per business unit with per-user team assignments. Replaces the previous binary scope toggle. -### Integrations -- Jira Data Center — create, sync, and track tickets linked to CVE/vendor pairs -- Archer — risk acceptance exception tracking (EXC numbers) -- Atlas InfoSec — action plan cache, bulk creation from row selection, metrics reporting -- CARD API — Granite/CARD asset lookup for network device workflows -- NVD API — auto-fill CVE metadata with bulk sync support +### Features -### Knowledge Base -- Internal document library with inline PDF and Markdown rendering -- Category-based browsing and search +- **In-app notification system** — replaces Webex bot integration with native notifications +- **Screenshot uploads** in feedback modal, Webex bot DM on issue close +- **CCP Metrics page** — multi-vertical VCL upload and cross-org compliance reporting +- **VCL compliance reporting** — exec report page, device metadata fields, bulk upload +- **Aggregated burndown forecast** on CCP Metrics overview page +- **Sub-team drill-down** — metric sub-team intermediate view with per-team breakdowns +- **Metric breakdown panel** — Non-Compliant stat clickable, reveals metric breakdown buttons, compact grid with top 8 and show-all toggle +- **Remediation plan and resolution date history tracking** +- **Data management panel** — delete vertical, rollback upload, and reset all +- **VCL vertical metadata** — inline-editable team fields on compliance routes +- **Re-queue findings** from rejected FP submissions +- **FP submissions cleanup** — auto-clear approved, dismiss rejected, collapsible section +- **DECOM workflow type** — auto-note/hide on decom, show CVEs on CARD queue items +- **Interactive configuration wizard** for deployment setup +- **Unified setup script** (`configure.js`) merging deploy + config wizard +- **Per-BU trend lines** in Ivanti counts history chart +- **Multi-select BU picker** replacing binary scope toggle +- **Configurable IVANTI_MANAGED_BUS** env var for multi-tenant drift classification +- **Pipeline-to-issue traceability** via `after_script` comments in CI/CD +- **CI/CD pipeline** with feedback modal, Atlas `qualys_id` fallback, and health endpoint +- **Docker Compose** and `deploy-postgres.sh` for production cutover +- **Systemd service scripts** for start/stop management -### Admin -- Full-page admin panel with user management, audit log, and system info tabs -- Themed confirm modals replacing browser dialogs -- User profile panel with self-service password change +### Bug Fixes + +- Fix duplicate failing metrics on same asset across compliance endpoints +- Fix duplicate chart entries on compliance page when multiple verticals share a report_date +- Fix requeue inserting Postgres array literal instead of JSON into `cves_json` +- Fix todo queue crash on malformed `cves_json` data +- Fix AEO compliance page not showing metric health cards on dev +- Fix double-counting in VCL multi-vertical stats — use only `ALL:` rollup rows +- Fix compliance stats to use Summary sheet data instead of item counts +- Fix route mount order: `vcl-multi` must precede general compliance router +- Fix requeue: fallback to `finding_ids_json` when queue items are deleted or absent +- Sync FP submission `lifecycle_status` from Ivanti `currentState` on fetch +- Fix History tab crash: coerce Ivanti note fields to strings before rendering +- Fix archive bar chart: `fmtDate` now handles ISO datetime strings from PostgreSQL date columns +- Fix Ivanti panel bugs: Invalid Date, wrong workflow count, crash on archive click, BU scope filtering +- Fix BU drift checker: derive `EXPECTED_BUS` from `IVANTI_BU_FILTER` env var +- Fix null `bu_teams` in postgres migration, add retry logic to deploy script +- Fix missing `created_by` column in `archer_tickets` table +- Fix FP workflow counts donut scoped by BU +- Fix `dotenv` loading in `db.js` so `DATABASE_URL` is available on import +- Fix property test CI failure: mock db module before importing route + +### Maintenance + +- Track `package-lock.json` files for deterministic CI installs +- Remove unused icon imports and unused imports to satisfy ESLint thresholds +- CI pipeline fixes: dependency installation, lint thresholds, test isolation +- Auto-run migrations in pipeline +- Documentation updates for PostgreSQL migration, systemd scripts, and reference manual + +--- + +## [1.0.0] — 2026-05-01 + +Initial release of the STEAM Security Dashboard. -### Infrastructure -- Consolidated `setup.js` with complete database schema (27 tables, all indexes and triggers) -- systemd service files for persistent deployment -- GitLab CI/CD pipeline (install, lint, test, build, deploy) -- GPG-signed commits for code provenance -- Organized documentation structure (api, design, guides, security, testing, troubleshooting) -- Migration scripts documented and retained for existing deployment upgrades diff --git a/backend/scripts/jira-uat-test.js b/backend/scripts/jira-uat-test.js new file mode 100644 index 0000000..ea3ba7d --- /dev/null +++ b/backend/scripts/jira-uat-test.js @@ -0,0 +1,412 @@ +#!/usr/bin/env node +// ========================================================================== +// Jira UAT Test Script +// ========================================================================== +// Exercises every Jira REST API use case the STEAM Dashboard will run in +// production. Run this against the UAT instance before submitting the +// ATLSUP Rest API Approval ticket. +// +// Usage: +// cd backend +// node scripts/jira-uat-test.js +// +// Note: The JQL search test uses a 72-hour window (updated >= -72h) to +// match the production bulk-sync behavior and account for weekend gaps. +// +// Prerequisites: +// - backend/.env has JIRA_BASE_URL pointing to UAT +// - JIRA_API_USER / JIRA_API_TOKEN set to service account credentials +// - JIRA_PROJECT_KEY set to a UAT project your service account can access +// - Service account has been granted access to the target space by space owners +// +// The script logs every API call, response status, and timing to both +// console and a log file at backend/scripts/jira-uat-test.log for the +// ATLSUP reviewers. +// ========================================================================== + +require('dotenv').config({ path: require('path').join(__dirname, '..', '.env') }); + +const fs = require('fs'); +const path = require('path'); +const jiraApi = require('../helpers/jiraApi'); + +const LOG_FILE = path.join(__dirname, 'jira-uat-test.log'); +const results = []; +let createdIssueKey = null; + +// --------------------------------------------------------------------------- +// Logging +// --------------------------------------------------------------------------- +function log(level, message, data) { + const timestamp = new Date().toISOString(); + const entry = { timestamp, level, message }; + if (data !== undefined) entry.data = data; + results.push(entry); + + const line = `[${timestamp}] ${level.toUpperCase().padEnd(5)} ${message}`; + console.log(line); + if (data) { + const dataStr = typeof data === 'string' ? data : JSON.stringify(data, null, 2); + // Truncate long data to keep logs readable (HTML error pages can be 50KB+) + const truncated = dataStr.length > 2000 ? dataStr.substring(0, 2000) + '\n ... [truncated — ' + dataStr.length + ' chars total]' : dataStr; + console.log(' ' + truncated.split('\n').join('\n ')); + } +} + +function logPass(testName, data) { log('pass', `PASS: ${testName}`, data); } +function logFail(testName, data) { log('fail', `FAIL: ${testName}`, data); } +function logInfo(message, data) { log('info', message, data); } +function logWarn(message, data) { log('warn', message, data); } + +// --------------------------------------------------------------------------- +// Test runner +// --------------------------------------------------------------------------- +async function runTest(name, fn) { + logInfo(`--- Running: ${name} ---`); + const start = Date.now(); + try { + await fn(); + logPass(name, { durationMs: Date.now() - start }); + return true; + } catch (err) { + logFail(name, { error: err.message, durationMs: Date.now() - start }); + return false; + } +} + +function assert(condition, message) { + if (!condition) throw new Error('Assertion failed: ' + message); +} + +// --------------------------------------------------------------------------- +// Use Case 1: Connection Test (GET /rest/api/2/myself) +// Production use: Admin clicks "Test Connection" button on Jira settings panel +// --------------------------------------------------------------------------- +async function testConnection() { + const result = await jiraApi.testConnection(); + assert(result.ok, 'Connection test should succeed. Got: ' + JSON.stringify(result)); + assert(result.user && result.user.name, 'Should return authenticated user name'); + logInfo('Authenticated as:', result.user); +} + +// --------------------------------------------------------------------------- +// Use Case 2: Create Issue (POST /rest/api/2/issue) +// Production use: User clicks "Create in Jira" from CVE detail panel +// --------------------------------------------------------------------------- +async function testCreateIssue() { + const projectKey = jiraApi.JIRA_PROJECT_KEY; + assert(projectKey, 'JIRA_PROJECT_KEY must be set in .env'); + + // Discover available issue types for this project + const projRes = await jiraApi.jiraGet('/rest/api/2/project/' + encodeURIComponent(projectKey)); + assert(projRes.status === 200, 'Should be able to fetch project metadata. Got HTTP ' + projRes.status + ': ' + (projRes.body || '').substring(0, 300)); + + const projData = JSON.parse(projRes.body); + const availableTypes = (projData.issueTypes || []).filter(t => !t.subtask); + logInfo('Available issue types:', availableTypes.map(t => t.name)); + + // Determine which issue type to use: configured type first, then fallback order + const configuredType = jiraApi.JIRA_ISSUE_TYPE || 'Task'; + const fallbackOrder = [configuredType, 'Story', 'Task', 'Bug']; + let issueTypeName = null; + + for (const candidate of fallbackOrder) { + if (availableTypes.some(t => t.name === candidate)) { + issueTypeName = candidate; + break; + } + } + + // If none of the preferred types exist, use the first available non-subtask type + if (!issueTypeName && availableTypes.length > 0) { + issueTypeName = availableTypes[0].name; + } + + assert(issueTypeName, 'No usable issue type found in project ' + projectKey); + + if (issueTypeName !== configuredType) { + logWarn('Configured JIRA_ISSUE_TYPE "' + configuredType + '" not available — falling back to "' + issueTypeName + '"'); + } + + const fields = { + project: { key: projectKey }, + summary: '[UAT TEST] STEAM Dashboard - CVE-2025-0001 - TestVendor - ' + new Date().toISOString(), + issuetype: { name: issueTypeName }, + description: 'UAT test issue created by STEAM Security Dashboard Jira integration test script. This issue can be safely deleted after review.' + }; + + // Epic type requires an Epic Name field — add it if creating an Epic + if (issueTypeName === 'Epic') { + fields.customfield_10004 = fields.summary; // Epic Name (standard Jira field ID) + } + + logInfo('Creating issue with fields:', { project: fields.project, summary: fields.summary, issuetype: fields.issuetype }); + + let result = await jiraApi.createIssue(fields); + + // If the first attempt fails with 400, try without description (some screens don't have it) + if (!result.ok && result.status === 400) { + const errBody = (result.body || '').substring(0, 500); + logWarn('Create failed with 400, retrying without description. Error: ' + errBody); + + const retryFields = { ...fields }; + delete retryFields.description; + result = await jiraApi.createIssue(retryFields); + } + + // If still failing with 400 and we used Epic, try without the customfield_10004 + // (Epic Name field ID varies across Jira instances) + if (!result.ok && result.status === 400 && issueTypeName === 'Epic') { + const errBody = (result.body || '').substring(0, 500); + logWarn('Epic create failed, retrying with alternate Epic Name field. Error: ' + errBody); + + const retryFields = { ...fields }; + delete retryFields.customfield_10004; + // Try common alternate Epic Name field IDs + retryFields.customfield_10011 = fields.summary; + result = await jiraApi.createIssue(retryFields); + } + + assert(result.ok, 'Create issue should succeed. Got HTTP ' + result.status + ': ' + (result.body || '').substring(0, 500)); + assert(result.data && result.data.key, 'Should return issue key'); + + createdIssueKey = result.data.key; + logInfo('Created issue:', { key: createdIssueKey, id: result.data.id, self: result.data.self, issueType: issueTypeName }); +} + +// --------------------------------------------------------------------------- +// Use Case 3: Get Single Issue (GET /rest/api/2/issue/{key}?fields=...) +// Production use: User clicks "Sync" on a single Jira ticket row +// --------------------------------------------------------------------------- +async function testGetIssue() { + assert(createdIssueKey, 'Need a created issue key from previous test'); + + const result = await jiraApi.getIssue(createdIssueKey); + assert(result.ok, 'Get issue should succeed. Got HTTP ' + (result.status || '') + ': ' + ((result.body || '').substring(0, 500))); + + const issue = result.data; + assert(issue.key === createdIssueKey, 'Returned key should match'); + assert(issue.fields && issue.fields.summary, 'Should have summary field'); + assert(issue.fields.status, 'Should have status field'); + + logInfo('Fetched issue:', { + key: issue.key, + summary: issue.fields.summary, + status: issue.fields.status.name, + issuetype: issue.fields.issuetype ? issue.fields.issuetype.name : null + }); +} + +// --------------------------------------------------------------------------- +// Use Case 4: Update Issue (PUT /rest/api/2/issue/{key}) +// Production use: Local ticket edits synced back to Jira (future feature) +// --------------------------------------------------------------------------- +async function testUpdateIssue() { + assert(createdIssueKey, 'Need a created issue key from previous test'); + + const result = await jiraApi.updateIssue(createdIssueKey, { + summary: `[UAT TEST] STEAM Dashboard - UPDATED - ${new Date().toISOString()}` + }); + assert(result.ok, 'Update issue should succeed (204). Got HTTP ' + (result.status || '') + ': ' + ((result.body || '').substring(0, 500))); + logInfo('Updated issue summary successfully'); +} + +// --------------------------------------------------------------------------- +// Use Case 5: Add Comment (POST /rest/api/2/issue/{key}/comment) +// Production use: Dashboard adds audit trail comments to linked Jira tickets +// --------------------------------------------------------------------------- +async function testAddComment() { + assert(createdIssueKey, 'Need a created issue key from previous test'); + + const commentBody = `STEAM Dashboard UAT test comment.\nTimestamp: ${new Date().toISOString()}\nThis comment was created by the automated test script.`; + + const result = await jiraApi.addComment(createdIssueKey, commentBody); + assert(result.ok, 'Add comment should succeed. Got HTTP ' + (result.status || '') + ': ' + ((result.body || '').substring(0, 500))); + assert(result.data && result.data.id, 'Should return comment ID'); + + logInfo('Added comment:', { commentId: result.data.id }); +} + +// --------------------------------------------------------------------------- +// Use Case 6: Get Transitions (GET /rest/api/2/issue/{key}/transitions) +// Production use: Dashboard checks available workflow transitions before +// attempting to move a ticket to a new status +// --------------------------------------------------------------------------- +async function testGetTransitions() { + assert(createdIssueKey, 'Need a created issue key from previous test'); + + const result = await jiraApi.getTransitions(createdIssueKey); + assert(result.ok, 'Get transitions should succeed. Got HTTP ' + (result.status || '') + ': ' + ((result.body || '').substring(0, 500))); + + const transitions = result.data.transitions || []; + logInfo('Available transitions:', transitions.map(t => ({ id: t.id, name: t.name, to: t.to ? t.to.name : null }))); + + // Store for the transition test + return transitions; +} + +// --------------------------------------------------------------------------- +// Use Case 7: Transition Issue (POST /rest/api/2/issue/{key}/transitions) +// Production use: Dashboard moves ticket status (e.g., Open → In Progress) +// --------------------------------------------------------------------------- +async function testTransitionIssue(transitions) { + assert(createdIssueKey, 'Need a created issue key from previous test'); + + if (!transitions || transitions.length === 0) { + logWarn('No transitions available — skipping transition test. This may be expected depending on the project workflow.'); + return; + } + + // Pick the first available transition + const transition = transitions[0]; + logInfo(`Transitioning to: ${transition.name} (id: ${transition.id})`); + + const result = await jiraApi.transitionIssue(createdIssueKey, transition.id); + assert(result.ok, 'Transition should succeed (204). Got HTTP ' + (result.status || '') + ': ' + ((result.body || '').substring(0, 500))); + logInfo('Transition successful'); +} + +// --------------------------------------------------------------------------- +// Use Case 8: JQL Search (POST /rest/api/2/search) +// Production use: Bulk sync — fetches all tracked tickets in one request +// instead of one GET per ticket (Charter-compliant) +// --------------------------------------------------------------------------- +async function testJqlSearch() { + const projectKey = jiraApi.JIRA_PROJECT_KEY; + assert(projectKey, 'JIRA_PROJECT_KEY must be set'); + + // Use a 72-hour window to account for weekend gaps between syncs + const jql = `project = ${projectKey} AND updated >= -72h ORDER BY updated DESC`; + logInfo('Searching with JQL:', jql); + + const result = await jiraApi.searchIssues(jql, { maxResults: 10 }); + assert(result.ok, 'Search should succeed. Got HTTP ' + (result.status || '') + ': ' + ((result.body || '').substring(0, 500))); + + const data = result.data; + logInfo('Search results:', { + total: data.total, + returned: (data.issues || []).length, + issues: (data.issues || []).slice(0, 5).map(i => ({ + key: i.key, + summary: i.fields.summary, + status: i.fields.status ? i.fields.status.name : null + })) + }); +} + +// --------------------------------------------------------------------------- +// Use Case 9: Bulk Key Search (searchIssuesByKeys) +// Production use: sync-all endpoint — fetches multiple tickets by key +// in a single JQL query +// --------------------------------------------------------------------------- +async function testBulkKeySearch() { + assert(createdIssueKey, 'Need a created issue key from previous test'); + + // Search for the issue we created plus a fake key to test partial results + const keys = [createdIssueKey, 'FAKE-99999']; + logInfo('Bulk searching keys:', keys); + + const result = await jiraApi.searchIssuesByKeys(keys); + assert(result.ok, 'Bulk key search should succeed. Got HTTP ' + (result.status || '') + ': ' + ((result.body || '').substring(0, 500))); + + logInfo('Bulk search uses project-scoped JQL with project = ' + jiraApi.JIRA_PROJECT_KEY); + + const found = (result.data.issues || []).map(i => i.key); + logInfo('Found issues:', found); + assert(found.includes(createdIssueKey), 'Should find the created issue'); +} + +// --------------------------------------------------------------------------- +// Use Case 10: Rate Limit Status Check +// Production use: Admin views rate limit usage on the Jira settings panel +// --------------------------------------------------------------------------- +async function testRateLimitStatus() { + const status = jiraApi.getRateLimitStatus(); + assert(status.daily && typeof status.daily.used === 'number', 'Should have daily usage'); + assert(status.burst && typeof status.burst.used === 'number', 'Should have burst usage'); + logInfo('Rate limit status after all tests:', status); +} + +// --------------------------------------------------------------------------- +// Main +// --------------------------------------------------------------------------- +async function main() { + logInfo('=== STEAM Dashboard — Jira UAT Test Run ==='); + logInfo('Timestamp: ' + new Date().toISOString()); + logInfo('JIRA_BASE_URL: ' + (process.env.JIRA_BASE_URL || '(not set)')); + logInfo('JIRA_AUTH_METHOD: ' + (process.env.JIRA_AUTH_METHOD || 'basic')); + logInfo('JIRA_API_USER: ' + (process.env.JIRA_API_USER || '(not set)')); + logInfo('JIRA_PROJECT_KEY: ' + (jiraApi.JIRA_PROJECT_KEY || '(not set)')); + logInfo('JIRA_ISSUE_TYPE: ' + (jiraApi.JIRA_ISSUE_TYPE || 'Task')); + logInfo('JIRA_SKIP_TLS: ' + (process.env.JIRA_SKIP_TLS || 'false')); + logInfo('isConfigured: ' + jiraApi.isConfigured); + logInfo(''); + + if (!jiraApi.isConfigured) { + logFail('Pre-flight check', 'Jira API is not configured. Set JIRA_BASE_URL, JIRA_API_USER, and JIRA_API_TOKEN in backend/.env'); + writeLog(); + process.exit(1); + } + + let passed = 0; + let failed = 0; + let transitions = []; + + // Run tests in order — later tests depend on the created issue + if (await runTest('1. Connection Test (GET /myself)', testConnection)) passed++; else failed++; + if (await runTest('2. Create Issue (POST /issue)', testCreateIssue)) passed++; else failed++; + if (await runTest('3. Get Single Issue (JQL search)', testGetIssue)) passed++; else failed++; + if (await runTest('4. Update Issue (PUT /issue/{key})', testUpdateIssue)) passed++; else failed++; + if (await runTest('5. Add Comment (POST /issue/{key}/comment)', testAddComment)) passed++; else failed++; + + if (await runTest('6. Get Transitions (GET /issue/{key}/transitions)', async () => { + transitions = await testGetTransitions(); + })) passed++; else failed++; + + if (await runTest('7. Transition Issue (POST /issue/{key}/transitions)', async () => { + await testTransitionIssue(transitions); + })) passed++; else failed++; + + if (await runTest('8. JQL Search (GET /search)', testJqlSearch)) passed++; else failed++; + if (await runTest('9. Bulk Key Search (searchIssuesByKeys)', testBulkKeySearch)) passed++; else failed++; + if (await runTest('10. Rate Limit Status', testRateLimitStatus)) passed++; else failed++; + + logInfo(''); + logInfo('=== Summary ==='); + logInfo(`Passed: ${passed} | Failed: ${failed} | Total: ${passed + failed}`); + if (createdIssueKey) { + logInfo(`Test issue created: ${createdIssueKey} — delete manually after ATLSUP review if desired.`); + } + logInfo('Rate limit usage:', jiraApi.getRateLimitStatus()); + + writeLog(); + + if (failed > 0) { + console.log('\nSome tests failed. Review the log above and jira-uat-test.log for details.'); + process.exit(1); + } else { + console.log('\nAll tests passed. Log saved to backend/scripts/jira-uat-test.log'); + console.log('Next steps:'); + console.log(' 1. Attach or reference backend/scripts/jira-uat-test.log in the ATLSUP ticket'); + console.log(' 2. Click "Script ran - Review Logs" on the ATLSUP ticket'); + process.exit(0); + } +} + +function writeLog() { + const lines = results.map(r => { + let line = `[${r.timestamp}] ${r.level.toUpperCase().padEnd(5)} ${r.message}`; + if (r.data) { + const dataStr = (typeof r.data === 'string' ? r.data : JSON.stringify(r.data, null, 2)); + const truncated = dataStr.length > 2000 ? dataStr.substring(0, 2000) + '\n ... [truncated — ' + dataStr.length + ' chars total]' : dataStr; + line += '\n ' + truncated.split('\n').join('\n '); + } + return line; + }); + fs.writeFileSync(LOG_FILE, lines.join('\n') + '\n', 'utf8'); +} + +main().catch(err => { + console.error('Unhandled error:', err); + process.exit(1); +});