cve-dashboard/.kiro/specs/sync-anomaly-detection/design.md

Design Document: Sync Anomaly Detection and BU Drift Monitoring

Overview

This feature extends the Ivanti sync pipeline to automatically classify why findings disappear from filtered sync results. The current archive system detects disappearances but labels them all as severity_score_drift — a default that proved incorrect during the April 2026 incident where 109 findings silently disappeared due to a bulk BU reassignment.

The design adds three capabilities to the existing ivantiFindings.js sync pipeline:

1. BU Drift Checker — a post-sync step that queries the Ivanti API without BU/severity filters for newly archived finding IDs, classifying each disappearance as bu_reassignment, severity_drift, closed_on_platform, or decommissioned.
2. Sync Anomaly Summary — a structured report computed after each sync that breaks down count changes by cause and stores the result in a new ivanti_sync_anomaly_log table.
3. Finding-Level BU Tracking — per-finding BU comparison during syncFindings() that detects BU changes across syncs and records them in a new ivanti_finding_bu_history table.

The approach formalizes the ad-hoc diagnostic patterns from drift-check.js and bu-reassignment-check.js into the automated sync pipeline, with results surfaced through new API endpoints and an anomaly banner on the Vulnerability Triage page.

---

Architecture

The feature integrates into the existing sync pipeline as post-sync steps, keeping the core sync logic unchanged. No new route modules are created — all new endpoints and logic live within the existing ivantiFindings.js module and its factory-pattern router.

flowchart TD
    A[syncFindings - fetch all pages] --> B[Compare previous vs current findings]
    B --> C[detectArchiveChanges - existing]
    B --> D[BU comparison - new]
    D --> E[Insert BU changes into ivanti_finding_bu_history]
    C --> F[syncClosedCount - existing]
    F --> G[detectClosedFindings - existing]
    G --> H[detectClosedGoneFindings - existing]
    H --> I[runBUDriftChecker - new]
    I --> J[Batch unfiltered queries for newly archived IDs]
    J --> K[Classify each: bu_reassignment / severity_drift / closed_on_platform / decommissioned]
    K --> L[Update archive transition reasons]
    L --> M[computeAnomalySummary - new]
    M --> N[Insert row into ivanti_sync_anomaly_log]

    style I fill:#F59E0B,color:#000
    style D fill:#F59E0B,color:#000
    style M fill:#F59E0B,color:#000

Key design decisions:

• Post-sync, not inline: The BU drift checker runs after all existing sync steps complete. This means a sync failure does not block drift checking of previously archived findings, and drift checking failures do not block the sync.
• Same module, no new route file: The anomaly and BU history endpoints are added to the existing createIvantiFindingsRouter. This keeps the Ivanti findings API surface in one place and avoids a new factory-pattern module for four endpoints.
• Batched unfiltered queries: Finding IDs are chunked into groups of 50 for the unfiltered Ivanti API call, matching the pattern proven in bu-reassignment-check.js. This stays within API limits while keeping the number of HTTP calls manageable.
• BU comparison in syncFindings: The per-finding BU comparison happens during the existing previous-vs-current comparison in syncFindings(), before the cache is overwritten. This is the only point where both the old and new BU values are available in memory.

---

Components and Interfaces

1. BU Drift Checker (runBUDriftChecker)

A new async function added to ivantiFindings.js that runs after detectClosedGoneFindings() in the sync pipeline.

Signature:

async function runBUDriftChecker(db, newlyArchivedIds, apiKey, clientId, skipTls)

Parameters:

• db — SQLite database instance
• newlyArchivedIds — array of finding ID strings that were newly archived in this sync cycle (from detectArchiveChanges)
• apiKey, clientId, skipTls — Ivanti API credentials (same as existing sync functions)

Behavior:

1. If newlyArchivedIds is empty, return immediately (no API calls).
2. Chunk the IDs into batches of 50.
3. For each batch, call ivantiPost() with a filter on id field only (no BU, severity, or state filters) — the same unfiltered query pattern used in bu-reassignment-check.js.
4. For each finding ID, classify the result:
   • Found, BU differs from expected → bu_reassignment
   • Found, BU matches, severity < 8.5 → severity_drift
   • Found, BU matches, state is Closed → closed_on_platform
   • Not found → decommissioned
5. Update the corresponding ivanti_archive_transitions row's reason field with the classification.
6. Return a classification summary object: { bu_reassignment: N, severity_drift: N, closed_on_platform: N, decommissioned: N }.

Expected BUs are the same values used in FINDINGS_FILTERS: NTS-AEO-ACCESS-ENG and NTS-AEO-STEAM.

Error handling: If an individual batch API call fails, log the error and skip that batch. The findings in the failed batch retain their default severity_score_drift reason. The function never throws — it returns whatever partial results it collected.

2. Anomaly Summary Computation (computeAnomalySummary)

A new async function that runs after the BU drift checker completes.

Signature:

async function computeAnomalySummary(db, openCountDelta, closedCountDelta, newlyArchivedCount, returnedCount, classificationBreakdown)

Parameters:

• db — SQLite database instance
• openCountDelta — integer, current open count minus previous open count
• closedCountDelta — integer, current closed count minus previous closed count
• newlyArchivedCount — integer, number of findings archived in this sync
• returnedCount — integer, number of findings that returned in this sync
• classificationBreakdown — object from runBUDriftChecker, e.g. { bu_reassignment: 38, severity_drift: 5, ... }

Behavior:

1. Determine is_significant: true if newlyArchivedCount > 5.
2. Insert a row into ivanti_sync_anomaly_log with all fields.
3. Log the summary to console.

3. Finding-Level BU Comparison

Integrated into syncFindings() between reading previous findings and writing the new cache. Uses the existing previousFindings and allFindings arrays.

Logic:

for each finding in allFindings:
    previousFinding = previousMap.get(finding.id)
    if previousFinding exists AND previousFinding.buOwnership !== finding.buOwnership
       AND both values are non-empty:
        INSERT into ivanti_finding_bu_history

The buOwnership field is already extracted by extractFinding() from assetCustomAttributes['1550_host_1']. No changes to extractFinding() are needed — it already stores buOwnership on each finding object.

4. New API Endpoints

All endpoints are added to the existing createIvantiFindingsRouter and require authentication via requireAuth(db).

Method	Path	Description
GET	/api/ivanti/findings/anomaly/latest	Returns the most recent anomaly summary row
GET	/api/ivanti/findings/anomaly/history	Returns anomaly history (last 30 or date-filtered)
GET	/api/ivanti/findings/bu-changes	Returns all BU change events, newest first
GET	/api/ivanti/findings/:findingId/bu-history	Returns BU change history for a specific finding

GET /anomaly/latest response:

{
  "anomaly": {
    "id": 1,
    "sync_timestamp": "2026-04-24T12:00:00",
    "open_count_delta": -45,
    "closed_count_delta": -94,
    "newly_archived_count": 45,
    "returned_count": 0,
    "classification": {
      "bu_reassignment": 38,
      "severity_drift": 1,
      "closed_on_platform": 4,
      "decommissioned": 2
    },
    "is_significant": true
  }
}

Returns { anomaly: null } if no anomaly records exist.

GET /anomaly/history query parameters:

• from (optional) — ISO date string, inclusive start
• to (optional) — ISO date string, inclusive end
• If neither provided, returns last 30 rows

GET /bu-changes response:

{
  "changes": [
    {
      "id": 1,
      "finding_id": "2687687777",
      "finding_title": "OpenSSH regreSSHion",
      "host_name": "syn-098-120-000-078",
      "previous_bu": "NTS-AEO-STEAM",
      "new_bu": "SDIT-CSD-ITLS-PIES",
      "detected_at": "2026-04-24T12:00:00"
    }
  ]
}

GET /:findingId/bu-history response:

{
  "finding_id": "2687687777",
  "history": [
    {
      "previous_bu": "NTS-AEO-STEAM",
      "new_bu": "SDIT-CSD-ITLS-PIES",
      "detected_at": "2026-04-24T12:00:00"
    }
  ]
}

5. Anomaly Banner Component (AnomalyBanner.js)

A new React component placed in frontend/src/components/pages/AnomalyBanner.js, rendered on the Vulnerability Triage page above the IvantiCountsChart.

Props: None — fetches its own data from /api/ivanti/findings/anomaly/latest.

Behavior:

1. On mount, fetch the latest anomaly summary.
2. If is_significant is false or no anomaly exists, render nothing.
3. If is_significant is true, render a warning banner with:
   • Amber background tint (rgba(245, 158, 11, 0.15)) with amber border (rgba(245, 158, 11, 0.3))
   • AlertTriangle icon from lucide-react
   • Summary text: "45 findings archived — 38 BU reassignment, 5 severity drift, 2 decommissioned"
   • Expandable detail section (click to toggle) showing affected findings grouped by classification
   • Dismiss button (X icon) that hides the banner for the current session via useState
4. Uses monospace typography and dark theme colors per DESIGN_SYSTEM.md.

Session dismiss: Uses React state only — no localStorage. The banner reappears on page reload, which is appropriate since the anomaly data persists until the next sync produces a non-significant result.

stateDiagram-v2
    [*] --> Loading: Component mounts
    Loading --> Hidden: No anomaly or not significant
    Loading --> Visible: Significant anomaly
    Visible --> Expanded: Click breakdown text
    Expanded --> Visible: Click breakdown text
    Visible --> Dismissed: Click dismiss
    Expanded --> Dismissed: Click dismiss
    Dismissed --> [*]

6. Migration Script

Located at backend/migrations/add_sync_anomaly_tables.js. Uses the same pattern as existing migrations (add_closed_gone_state.js): standalone Node script, opens the database directly, uses CREATE TABLE IF NOT EXISTS and CREATE INDEX IF NOT EXISTS for idempotency.

---

Data Models

New Table: ivanti_sync_anomaly_log

Stores one row per sync cycle with the anomaly summary.

Column	Type	Constraints	Description
id	INTEGER	PRIMARY KEY AUTOINCREMENT	Unique row identifier
sync_timestamp	DATETIME	NOT NULL DEFAULT CURRENT_TIMESTAMP	When the sync completed
open_count_delta	INTEGER	NOT NULL DEFAULT 0	Current open count minus previous open count
closed_count_delta	INTEGER	NOT NULL DEFAULT 0	Current closed count minus previous closed count
newly_archived_count	INTEGER	NOT NULL DEFAULT 0	Number of findings archived in this sync
returned_count	INTEGER	NOT NULL DEFAULT 0	Number of findings that returned in this sync
classification_json	TEXT	NOT NULL DEFAULT '{}'	JSON object: { bu_reassignment, severity_drift, closed_on_platform, decommissioned }
is_significant	INTEGER	NOT NULL DEFAULT 0	1 if newly_archived_count > 5, else 0
created_at	DATETIME	DEFAULT CURRENT_TIMESTAMP	Row creation timestamp

Indexes:

• idx_anomaly_sync_timestamp on sync_timestamp — for efficient latest-record and date-range queries

New Table: ivanti_finding_bu_history

Stores BU change events detected during sync.

Column	Type	Constraints	Description
id	INTEGER	PRIMARY KEY AUTOINCREMENT	Unique row identifier
finding_id	TEXT	NOT NULL	Ivanti finding identifier
finding_title	TEXT	NOT NULL DEFAULT ''	Finding title at time of detection
host_name	TEXT	NOT NULL DEFAULT ''	Host name at time of detection
previous_bu	TEXT	NOT NULL	BU value from previous sync
new_bu	TEXT	NOT NULL	BU value from current sync
detected_at	DATETIME	NOT NULL DEFAULT CURRENT_TIMESTAMP	When the change was detected
created_at	DATETIME	DEFAULT CURRENT_TIMESTAMP	Row creation timestamp

Indexes:

• idx_bu_history_finding_id on finding_id — for per-finding history lookups
• idx_bu_history_detected_at on detected_at — for chronological queries

Modified: ivanti_archive_transitions.reason field

No schema change needed — the reason column is already TEXT NOT NULL DEFAULT ''. The change is in the values written:

Previous values	New values
severity_score_drift	bu_reassignment:<new_bu>
reappeared_in_sync	severity_drift:<new_severity>
remediated_in_ivanti	closed_on_platform
disappeared_from_closed_set	decommissioned

Existing rows with severity_score_drift are not modified — the enhanced reasons apply only to transitions created after deployment.

Existing: ivanti_findings_cache.findings_json

No schema change. The buOwnership field is already present in each finding object within the JSON array, extracted by extractFinding() from assetCustomAttributes['1550_host_1'].

---

Correctness Properties

A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.

Property 1: Classification correctness

For any finding returned by an unfiltered Ivanti API query, the BU drift classifier SHALL produce the correct classification based on the combination of BU value, severity, and state:

• BU not in {NTS-AEO-ACCESS-ENG, NTS-AEO-STEAM} → bu_reassignment
• BU matches expected, severity < 8.5 → severity_drift
• BU matches expected, severity >= 8.5, state is Closed → closed_on_platform
• Finding not returned by API → decommissioned

Validates: Requirements 1.2, 1.3, 1.4, 1.5

Property 2: Archive transition reason formatting

For any classification result, the archive transition reason field SHALL be formatted correctly:

• bu_reassignment classification with BU value B → reason is bu_reassignment:B
• severity_drift classification with severity S → reason is severity_drift:S
• closed_on_platform → reason is closed_on_platform
• decommissioned → reason is decommissioned

Validates: Requirements 6.1, 6.2, 6.3, 6.4

Property 3: Batch size constraint

For any list of finding IDs of length N, the BU drift checker SHALL partition them into ceil(N/50) batches where each batch contains at most 50 IDs and the union of all batches equals the original list.

Validates: Requirements 1.7

Property 4: Significance threshold

For any non-negative integer newly_archived_count, the anomaly summary's is_significant flag SHALL be true if and only if newly_archived_count > 5.

Validates: Requirements 2.7

Property 5: Count delta computation

For any pair of non-negative integers (previous_count, current_count), the anomaly summary SHALL compute the delta as current_count - previous_count for both open and closed counts.

Validates: Requirements 2.1

Property 6: BU extraction preservation

For any raw Ivanti finding object with a non-empty assetCustomAttributes['1550_host_1'] array, extractFinding SHALL produce a finding object whose buOwnership field equals the first element of that array.

Validates: Requirements 3.1

Property 7: BU change detection and recording

For any finding that appears in both the previous and current sync results with different non-empty buOwnership values, the sync pipeline SHALL insert exactly one row into ivanti_finding_bu_history with the correct finding_id, previous_bu, and new_bu. For any finding that appears for the first time (no previous entry) or has the same BU value, no history row SHALL be inserted.

Validates: Requirements 3.2, 3.3, 3.6

Property 8: Latest anomaly returns most recent

For any non-empty sequence of anomaly summary rows with distinct timestamps, the /anomaly/latest endpoint SHALL return the row with the maximum sync_timestamp.

Validates: Requirements 2.5

Property 9: Anomaly history ordering and limit

For any set of N anomaly summary rows, the /anomaly/history endpoint (without date parameters) SHALL return min(N, 30) rows ordered by sync_timestamp descending.

Validates: Requirements 2.6, 7.2

Property 10: Date-range filtering with complete response shape

For any date range [from, to] and set of anomaly summary rows, the /anomaly/history endpoint SHALL return only rows whose sync_timestamp falls within the range (inclusive), ordered by sync_timestamp descending. Each returned row SHALL include sync_timestamp, open_count_delta, closed_count_delta, newly_archived_count, returned_count, classification (parsed as an object from classification_json), and is_significant.

Validates: Requirements 7.1, 7.4

Property 11: BU changes endpoint ordering

For any set of BU change history rows, the /bu-changes endpoint SHALL return all rows ordered by detected_at descending.

Validates: Requirements 3.4

Property 12: Per-finding BU history filtering

For any finding ID F and set of BU history rows across multiple findings, the /:findingId/bu-history endpoint SHALL return only rows where finding_id = F, ordered by detected_at descending.

Validates: Requirements 3.5

---

Error Handling

BU Drift Checker Errors

• Individual batch API failure: Log the error with the batch range, skip the batch, continue with remaining batches. Findings in the failed batch retain the default severity_score_drift reason. The function returns partial results.
• All batches fail: The classification breakdown will be all zeros. The anomaly summary is still written with newly_archived_count reflecting the archive detection results (which don't depend on the drift checker).
• API timeout: The existing 15-second timeout in ivantiPost() applies. Timed-out batches are treated as failed batches.
• Malformed API response: If JSON.parse fails on the response body, treat the batch as failed. Log the raw response length for debugging.

Anomaly Summary Errors

• Database write failure: Log the error. The sync itself has already completed successfully — the anomaly summary is informational. Do not retry.
• Missing previous counts: If no previous anomaly row exists (first sync after deployment), use 0 for previous counts. The first anomaly row will have deltas equal to the current counts.

BU Comparison Errors

• Database insert failure: Log the error for the specific finding, continue processing remaining findings. BU comparison failures are non-fatal.
• Missing buOwnership field: If either the previous or current finding has an empty/undefined buOwnership, skip the comparison for that finding (per requirement 3.6).

API Endpoint Errors

• Database read failure: Return 500 with a generic error message. Do not expose internal error details.
• Invalid date parameters: If from or to are not valid ISO date strings, ignore them and fall back to the default last-30 behavior. Log a warning.
• Authentication failure: Handled by existing requireAuth(db) middleware — returns 401.

Migration Errors

• Table already exists: CREATE TABLE IF NOT EXISTS handles this silently.
• Index already exists: CREATE INDEX IF NOT EXISTS handles this silently.
• Database locked: The migration script opens its own connection. If the server is running, SQLite's WAL mode allows concurrent reads. If a write lock conflict occurs, the migration will fail with a clear error message and can be retried.

---

Testing Strategy

Property-Based Tests

Property-based testing is appropriate for this feature because the core logic involves classification functions, data transformations, and query behaviors that have clear input/output relationships and universal properties.

Library: fast-check — the standard PBT library for JavaScript/Node.js.

Configuration:

• Minimum 100 iterations per property test
• Each test tagged with: Feature: sync-anomaly-detection, Property {N}: {title}

Properties to implement:

Property	Test approach
1: Classification correctness	Generate random {bu, severity, state, found} tuples, verify classifier output
2: Reason formatting	Generate random classification results, verify reason string format
3: Batch size constraint	Generate random-length ID arrays, verify chunking
4: Significance threshold	Generate random integers, verify is_significant flag
5: Delta computation	Generate random count pairs, verify subtraction
6: BU extraction	Generate random raw finding objects, verify buOwnership extraction
7: BU change detection	Generate random previous/current finding pairs, verify history insertion
8–12: API query properties	Generate random DB state, verify endpoint responses

Unit Tests (Example-Based)

Unit tests cover specific scenarios, edge cases, and integration points not suited for PBT:

• Migration idempotency: Run migration twice, verify no errors on second run (Req 4.6)
• API error resilience: Mock ivantiPost to return errors, verify drift checker doesn't throw (Req 1.6)
• Anomaly banner rendering: Mock API response, verify banner shows/hides based on is_significant (Req 5.2, 5.3)
• Banner dismiss: Click dismiss button, verify banner hidden (Req 5.4)
• Banner expand/collapse: Click breakdown text, verify detail section toggles (Req 5.7)
• Authentication enforcement: Unauthenticated requests return 401 (Req 7.3)
• Fixed reason strings: Verify decommissioned and closed_on_platform are exact strings (Req 6.3, 6.4)
• Backward compatibility: Existing severity_score_drift rows are not modified (Req 6.5)

Integration Tests

• End-to-end sync with drift checker: Mock Ivanti API, run full sync pipeline, verify anomaly log and BU history tables are populated correctly
• API endpoint responses: Seed database, call each endpoint, verify response shape and content

Test File Locations

• backend/__tests__/bu-drift-classification.property.test.js — Properties 1–6
• backend/__tests__/anomaly-api.property.test.js — Properties 7–12
• backend/__tests__/sync-anomaly-detection.test.js — Unit and integration tests
• frontend/src/components/pages/__tests__/AnomalyBanner.test.js — UI component tests