jramos/cve-dashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
cve-dashboard/docs/troubleshooting/findings-count-investigation-2026-04-24.md

24 KiB
Raw Permalink Blame History

Findings Count Drop Investigation — 2026-04-24

Summary

On 2026-04-24, the Findings Trend chart showed a sharp drop in both open and closed counts. The total (open + closed) fell from ~170 to 31, which is inconsistent with normal finding lifecycle behavior where findings move between open and closed but the total remains roughly stable.

Timeline

Date Open Closed Total Notes
04/02 127 52 179 Baseline
04/11 116 51 167 Normal fluctuation
04/19 114 50 164 Normal fluctuation
04/20 86 84 170 Batch of findings closed — total stable
04/23 60 110 170 Continued closure — total stable
04/24 15 16 31 Anomalous drop

The 04/20 and 04/23 snapshots show the expected pattern: open decreases, closed increases, total stays at ~170. The 04/24 snapshot breaks this pattern — both open and closed dropped simultaneously.

Root Cause Analysis

What the dashboard queries

The Ivanti sync fetches findings using two API calls, both filtered to:

  • BU: NTS-AEO-ACCESS-ENG, NTS-AEO-STEAM
  • Severity range: 8.5–9.9 VRR
  • State: Open (first call) or Closed (second call)

Any finding that no longer matches all three criteria will not appear in the results.

What happened

A re-test of the Ivanti API on 04/24 confirmed the API itself is returning only 15 open and 16 closed findings (totalElements field). This is not a pagination bug or partial response — the API is reporting these as the complete result sets.

Likely explanation: VRR rescore

The most probable cause is a bulk VRR (Vulnerability Risk Rating) rescore on the Ivanti platform. If Ivanti recalculated severity scores and a large number of findings dropped below the 8.5 threshold, they would vanish from both the open and closed query results.

Key detail: The archive table stores last_severity — the score at the time the finding was last seen in our sync, not the current score in Ivanti. Archived findings show severities of 9.0–9.9, but this reflects their pre-rescore values. After a rescore, these same findings could now be rated below 8.5, which is why they no longer appear in our filtered queries.

This explains why:

  • Open findings dropped from 60 to 15 — rescored findings fell below 8.5
  • Closed findings dropped from 110 to 16 — the same rescore affected closed findings too
  • Archive caught 67 disappearances from the open set, but did not previously track disappearances from the closed set

Alternative explanations

  • BU reassignment: Findings moved out of NTS-AEO-ACCESS-ENG or NTS-AEO-STEAM would also disappear. Less likely at this scale.
  • Ivanti platform issue: Temporary data availability problem. Can be ruled out if the counts remain low on subsequent syncs.
  • Finding decommission: Hosts removed from Ivanti entirely. Possible for some findings but unlikely for ~140 at once.

Accounting

As of 04/24:

Category Count Description
Open (API) 15 Currently in Ivanti open set, severity 8.5–9.9
Closed (API) 16 Currently in Ivanti closed set, severity 8.5–9.9
Archived 67 Disappeared from open set, not found in closed set
Archive-Closed 63 Were archived, then confirmed in Ivanti closed set
Returned 1 Was archived, then reappeared in open set
Tracked total 162
Expected total ~170
Unaccounted ~8 Normal churn (decommissions, new findings offsetting)

The 63 archive-closed findings were previously part of the ~110 closed count on 04/23. They have since disappeared from the closed API results (likely rescored below 8.5). Before this fix, disappearances from the closed set were not tracked.

Fixes Applied

1. Bad data point removed

The 04/24 history row (15 open / 16 closed) was deleted from ivanti_counts_history to prevent it from skewing the trend chart.

2. Drift guard added

Before writing to ivanti_counts_history, the sync now compares the new total (open + closed) against the most recent history entry. If the new total drops below 50% of the previous total, the history write is skipped and a warning is logged. The live cache (ivanti_counts_cache) is still updated so current counts remain accurate.

3. Closed-set disappearance tracking (CLOSED_GONE)

A new archive state CLOSED_GONE was added. On each sync, findings previously marked as CLOSED in the archive are checked against the current closed API results. If a finding is no longer in the closed set, it transitions to CLOSED_GONE with reason disappeared_from_closed_set. This closes the visibility gap where findings could vanish from the closed API results without being tracked.

Migration required: node backend/migrations/add_closed_gone_state.js

Archive state machine (updated)

NONE ──→ ARCHIVED ──→ RETURNED ──→ ARCHIVED (cycle)
              │              │
              ▼              ▼
           CLOSED ──→ CLOSED_GONE
State Meaning
ARCHIVED Disappeared from the open findings set; not found in closed set
RETURNED Was archived but reappeared in the open set
CLOSED Confirmed present in the Ivanti closed findings set
CLOSED_GONE Was confirmed closed, then disappeared from the closed set

4. Automated sync anomaly detection

The manual diagnostic work from this investigation was formalized into an automated feature in the sync pipeline (backend/routes/ivantiFindings.js). After each sync, the system now:

  • Classifies disappearances — queries Ivanti without BU/severity filters for newly archived finding IDs and labels each as bu_reassignment, severity_drift, closed_on_platform, or decommissioned. The classification is stored on the archive transition record, replacing the generic severity_score_drift default.
  • Logs anomaly summaries — writes a breakdown of count changes to ivanti_sync_anomaly_log after each sync, flagging syncs where more than 5 findings are archived as significant.
  • Tracks BU changes per finding — compares each finding's BU against the previous sync and records changes in ivanti_finding_bu_history.
  • Surfaces anomalies in the UI — an amber warning banner on the Vulnerability Triage page displays the latest anomaly summary when a significant count change is detected.

API endpoints for anomaly data: GET /api/ivanti/findings/anomaly/latest, GET /api/ivanti/findings/anomaly/history, GET /api/ivanti/findings/bu-changes, GET /api/ivanti/findings/:findingId/bu-history.

Migration required: node backend/migrations/add_sync_anomaly_tables.js

Recommended Follow-Up

  1. Check with Ivanti platform team whether a bulk VRR rescore occurred around 04/23–04/24.
  2. Monitor the next few syncs to see if counts stabilize at the new level or recover.
  3. Consider querying without the severity filter as a one-time diagnostic to see the true total of findings across all severities for the two BUs. This would confirm whether the findings still exist at lower severity scores.

Appendix: Cached Data Analysis

A cross-reference of the 04/22 findings export against the current cached data and archive was performed to test the score drift hypothesis.

Export reconciliation (04/22 export — ~80 open findings)

Current status Count
Still open in API 8
Archived (disappeared from open set) 44
Closed (confirmed in Ivanti closed set) 26
Untracked 0
Total 78

Every finding from the export is accounted for. Zero findings are untracked.

What disappeared on 04/24 (43 findings archived that day)

Vulnerability Count Last-seen severity
OpenSSH regreSSHion (CVE-2024-6387) 36 9.38
OpenSSH Multiple Security Vulnerabilities 3 9.9
Rocky Linux sudo update (RLSA-2025:9978) 2 9.06
Rocky Linux sqlite update (RLSA-2025:20936) 1 9.9
Rocky Linux sqlite update (RLSA-2025:11992) 1 9.9

What survived (15 findings still in API)

Vulnerability Count Current severity
OpenSSH Multiple Security Vulnerabilities 9 9.9
OpenSSH regreSSHion (CVE-2024-6387) 4 9.38
OpenSSH 7.4 Not Installed Multiple Vulnerabilities 1 9.18
Rocky Linux sudo update (RLSA-2025:9978) 1 9.06

Conclusion: host-level VRR drift

The pattern is consistent with host-level VRR score drift, not a blanket CVE rescore. Key evidence:

  • Selective disappearance within the same CVE: 36 of 40 regreSSHion findings disappeared, but 4 survived at the same last-seen severity (9.38). If the CVE itself were rescored, all would be affected equally.
  • Same pattern for OpenSSH Multiple: 3 of 12 disappeared at 9.9, while 9 survived at 9.9.
  • High last-seen severities: All disappeared findings had severities well above the 8.5 threshold (9.06–9.9), but last_severity reflects the score at time of last sync, not the current Ivanti score. A host-level rescore could move individual findings below 8.5 while leaving others on different hosts unchanged.

Ivanti calculates VRR per host-finding combination using factors like network exposure, asset criticality, and compensating controls. A platform-side recalculation of these host-level factors would produce exactly this pattern — some hosts for the same CVE drop below threshold while others remain above it.

To fully confirm: Query Ivanti without the severity filter for the disappeared finding IDs and check their current VRR scores. If they now show scores below 8.5, host-level drift is confirmed.

Appendix B: Unfiltered API Query Results (04/24)

A follow-up diagnostic queried Ivanti without the severity filter to check whether the disappeared findings still exist at lower severity scores.

Unfiltered totals

State Count (no severity filter) Count (8.5–9.9 filter)
Open 1,404 15
Closed 280 16
Total 1,684 31

The BUs have 1,684 total findings across all severities. The severity filter narrows this to 31.

Cross-reference against 130 archived/closed findings

Category Count Meaning
Completely gone from API 124 Not in Ivanti at any severity, open or closed
Confirmed score drift 1 Juniper finding dropped from 9.0 to 7.57
Still high severity (>= 8.5) 5 Still in Ivanti closed set at original scores

Verdict: Score drift hypothesis DISPROVED

Only 1 of 130 findings actually drifted below the severity threshold. 124 findings are completely absent from the Ivanti API at any severity in any state. They were not rescored — they were removed from the platform entirely.

This rules out VRR score drift as the primary cause and points to one of:

  • Host decommission / asset removal — the hosts were removed from Ivanti's asset inventory
  • BU reassignment — the hosts were moved out of NTS-AEO-ACCESS-ENG / NTS-AEO-STEAM to a different business unit
  • Platform-side data cleanup — findings were purged or merged on the Ivanti side

Given the scale (124 findings disappearing simultaneously), a bulk operation on the Ivanti platform is the most likely explanation. This should be raised with the Ivanti platform administrators to determine what changed.

Diagnostic script

The unfiltered query was originally performed using backend/scripts/drift-check.js. This logic has since been automated by the sync anomaly detection feature — the BU drift checker in backend/routes/ivantiFindings.js now runs these checks automatically after each sync. See the anomaly API endpoints (/api/ivanti/findings/anomaly/latest, /api/ivanti/findings/bu-changes) for current data.

Appendix C: BU Reassignment Confirmation (04/24)

A follow-up query searched for the disappeared finding IDs with no filters at all (no BU, no severity, no state) to determine whether the findings still exist in Ivanti under a different business unit.

Results

Category Count Detail
Reassigned to SDIT-CSD-ITLS-PIES 109 Hosts moved to different BU
Still same BU (STEAM/ACCESS-ENG) 6 5 closed (timing), 1 severity drift (7.57)
Completely gone from platform 15 Not found at any BU, severity, or state

Verdict: BU REASSIGNMENT CONFIRMED

109 of 130 disappeared findings were reassigned from NTS-AEO-STEAM / NTS-AEO-ACCESS-ENG to SDIT-CSD-ITLS-PIES. The severity scores are unchanged — the findings still exist at 9.38 and 9.9 — but they no longer match the dashboard's BU filter.

This is not score drift, not a platform bug, and not a data purge. It is a deliberate (or accidental) bulk BU reassignment on the Ivanti platform.

FP workflow impact

69 of the 109 reassigned findings have FP workflows attached, predominantly FP#0000459 (Approved). These are false positive approvals that were submitted by the STEAM/ACCESS-ENG team. The FP workflows followed the findings to the new BU. This should be reviewed with the team that performed the reassignment to determine whether the FP approvals are still valid under the new BU context.

15 truly gone findings

15 findings are not found in Ivanti at any BU, severity, or state. These are likely decommissioned hosts. All 15 are OpenSSH Remote Unauthenticated Code Execution Vulnerability (regreSSHion) at severity 9.30–9.38.

Reassigned findings — 109 findings moved to SDIT-CSD-ITLS-PIES

With approved FP workflows (58 findings):

Finding ID Severity FP Workflow Host IP Address
2687687777 9.38 FP#0000459 (Approved) syn-098-120-000-078 98.120.0.78
2687714078 9.38 FP#0000459 (Approved) syn-098-120-032-185 98.120.32.185
2561784254 9.38 FP#0000459 (Approved) mon15-agg-sw 10.240.78.177
2561788625 9.38 FP#0000459 (Approved) mon16-agg-sw 10.240.78.176
2689641701 9.38 FP#0000459 (Approved) mon15-sw14 10.240.78.133
2689642036 9.38 FP#0000459 (Approved) mon15-sw11 10.240.78.130
2689642107 9.38 FP#0000459 (Approved) mon19-sw3 10.240.78.150
2689642299 9.38 FP#0000459 (Approved) mon16-sw2 10.240.78.107
2689643552 9.38 FP#0000459 (Approved) mon16-sw5 10.240.78.110
2689645817 9.38 FP#0000459 (Approved) mon16-sw1 10.240.78.106
2689646279 9.38 FP#0000459 (Approved) mon19-sw2 10.240.78.149
2689647223 9.38 FP#0000459 (Approved) mon19-sw7 10.240.78.154
2689647732 9.38 FP#0000459 (Approved) mon16-sw6 10.240.78.111
2689662078 9.38 FP#0000459 (Approved) mon19-sw6 10.240.78.153
2689662169 9.38 FP#0000459 (Approved) mon15-sw13 10.240.78.132
2689667727 9.38 FP#0000459 (Approved) mon16-sw10 10.240.78.115
2689674347 9.38 FP#0000459 (Approved) mon16-sw4 10.240.78.109
2689680179 9.38 FP#0000459 (Approved) mon16-sw7 10.240.78.112
2689687694 9.38 FP#0000459 (Approved) mon16-sw14 10.240.78.119
2689703211 9.38 FP#0000459 (Approved) mon16-sw9 10.240.78.114
2689704574 9.38 FP#0000459 (Approved) mon16-sw13 10.240.78.118
2689707099 9.38 FP#0000459 (Approved) mon16-sw12 10.240.78.117
2689711822 9.38 FP#0000459 (Approved) mon16-sw3 10.240.78.108
2689712725 9.38 FP#0000459 (Approved) mon19-sw8 10.240.78.155
2689715642 9.38 FP#0000459 (Approved) mon19-sw10 10.240.78.157
2689717728 9.38 FP#0000459 (Approved) mon19-sw4 10.240.78.151
2689721708 9.38 FP#0000459 (Approved) mon16-sw11 10.240.78.116
2689722995 9.38 FP#0000459 (Approved) mon19-sw5 10.240.78.152
2689723147 9.38 FP#0000459 (Approved) mon19-sw14 10.240.78.161
2689723478 9.38 FP#0000459 (Approved) mon19-sw13 10.240.78.160
2689723840 9.38 FP#0000459 (Approved) mon19-sw12 10.240.78.159
2697106042 9.38 FP#0000459 (Approved) mon19-sw11 10.240.78.158
2697107537 9.38 FP#0000459 (Approved) mon15-sw4 10.240.78.123
2697108314 9.38 FP#0000459 (Approved) mon20-sw4 10.240.78.137
2726771499 9.38 FP#0000459 (Approved) mon19-sw1 10.240.78.148
2726805076 9.38 FP#0000459 (Approved) mon15-sw6 10.240.78.125
2726863413 9.38 FP#0000459 (Approved) mon19-sw9 10.240.78.156
2283414173 9.38 FP#0000459 (Approved) 10.241.0.63
2283664248 9.38 FP#0000459 (Approved) apc01se1shcc-n01-bmc 10.244.11.51
2460786621 9.38 FP#0000459 (Approved) 172.27.72.1
2521773008 9.38 FP#0000459 (Approved) 96.37.185.145
2663675680 9.38 FP#0000459 (Approved) mon17-sw9 10.240.78.170
2663676188 9.38 FP#0000459 (Approved) mon17-sw11 10.240.78.172
2663676366 9.38 FP#0000459 (Approved) mon17-sw8 10.240.78.169
2663676895 9.38 FP#0000459 (Approved) mon17-sw5 10.240.78.166
2663677778 9.38 FP#0000459 (Approved) mon17-sw13 10.240.78.174
2663677987 9.38 FP#0000459 (Approved) mon17-sw12 10.240.78.173
2663681315 9.38 FP#0000459 (Approved) mon17-sw6 10.240.78.167
2663683699 9.38 FP#0000459 (Approved) mon17-sw14 10.240.78.175
2663685466 9.38 FP#0000459 (Approved) mon17-sw7 10.240.78.168
2663695383 9.38 FP#0000459 (Approved) mon17-sw10 10.240.78.171
2744240319 9.38 FP#0000459 (Approved) syn-066-061-128-010 66.61.128.10
2744252609 9.38 FP#0000459 (Approved) apa01se1shcc-bvi101-secondary 66.61.128.233
2744261786 9.38 FP#0000459 (Approved) syn-066-061-128-049 66.61.128.49
2744295544 9.38 FP#0000459 (Approved) syn-066-061-128-018 66.61.128.18
2312013545 9.90 FP#0000459 (Approved) 10.244.4.26
2329805541 9.90 FP#0000459 (Approved) 10.244.11.5
2329818159 9.90 FP#0000459 (Approved) 10.244.11.6

With rejected FP workflows (8 findings):

Finding ID Severity FP Workflow Host IP Address
2281232044 9.38 FP#0000460 (Rejected) apc15se1shcc-n03 10.244.4.55
2281440017 9.38 FP#0000460 (Rejected) apc01se1shcc-n03-bmc 10.244.11.53
2282142049 9.38 FP#0000460 (Rejected) 10.244.4.30
2282338246 9.38 FP#0000460 (Rejected) apc04se1shcc-n01-cimc 10.244.11.63
2283364439 9.90 FP#0000470 (Rejected) 24.28.208.125
2283577805 9.90 FP#0000470 (Rejected) syn-024-028-210-101 24.28.210.101
2283734550 9.90 FP#0000452 (Rejected) 10.244.11.27
2286607835 9.90 FP#0000452 (Rejected) 10.240.1.203

Without FP workflows (43 findings):

Finding ID Severity Host IP Address Title
2289169183 9.90 10.240.78.20 IPMI 2.0 RAKP Authentication
2458498036 9.90 eon-node-dhcp OpenSSH Multiple Security Vulnerabilities
2352647807 9.90 localhost Rocky Linux sqlite update (RLSA-2025:20936)
2312562977 9.90 rphy-runner-falconv Rocky Linux sqlite update (RLSA-2025:11992)
2352629939 9.90 rphy-runner-falconv Rocky Linux sqlite update (RLSA-2025:20936)
2281281250 9.38 172.16.1.229 OpenSSH regreSSHion
2282419417 9.38 10.244.11.96 OpenSSH regreSSHion
2282688566 9.38 apc02se1shcc-n01-cimc 10.244.11.54 OpenSSH regreSSHion
2283112486 9.38 apc14se1shcc-n02 10.244.4.51 OpenSSH regreSSHion
2283720427 9.38 10.244.11.86 OpenSSH regreSSHion
2283873511 9.38 apc02se1shcc-n02-cimc 10.244.11.55 OpenSSH regreSSHion
2284154592 9.38 syn-024-028-208-105 24.28.208.105 OpenSSH regreSSHion
2284337626 9.38 apc14se1shcc-n01 10.244.4.50 OpenSSH regreSSHion
2284372435 9.38 apc15se1shcc-n01 10.244.4.53 OpenSSH regreSSHion
2284395753 9.38 apc07se1shcc-n02-cimc 10.244.11.73 OpenSSH regreSSHion
2284622624 9.38 apc04se1shcc-n02-cimc 10.244.11.64 OpenSSH regreSSHion
2284681286 9.38 apc15se1shcc-n02 10.244.4.54 OpenSSH regreSSHion
2285988119 9.38 10.244.4.28 OpenSSH regreSSHion
2286255181 9.38 10.244.11.94 OpenSSH regreSSHion
2286422988 9.38 c220-wzp27340ss5 10.241.0.43 OpenSSH regreSSHion
2286541484 9.38 apc02se1shcc-n03-cimc 10.244.11.56 OpenSSH regreSSHion
2286589497 9.38 apc05se1shcc-n01-bmc 10.244.11.66 OpenSSH regreSSHion
2287156417 9.38 apc13se1shcc-n01 10.244.4.47 OpenSSH regreSSHion
2287168608 9.38 apc13se1shcc-n03 10.244.4.49 OpenSSH regreSSHion
2287400005 9.38 apc14se1shcc-n03 10.244.4.52 OpenSSH regreSSHion
2287503960 9.38 apc07se1shcc-n01-cimc 10.244.11.72 OpenSSH regreSSHion
2287822934 9.38 apc02ctsbcom7-n03-cimc 10.244.4.25 OpenSSH regreSSHion
2287849796 9.38 10.244.4.29 OpenSSH regreSSHion
2287917789 9.38 apc07se1shcc-n03-cimc 10.244.11.74 OpenSSH regreSSHion
2287954330 9.38 apc13se1shcc-n02 10.244.4.48 OpenSSH regreSSHion
2288500154 9.38 apc04se1shcc-n03-cimc 10.244.11.65 OpenSSH regreSSHion
2288545686 9.38 apc02ctsbcom7-n02-cimc 10.244.4.24 OpenSSH regreSSHion
2288829837 9.38 10.244.11.87 OpenSSH regreSSHion
2288874420 9.38 apc05se1shcc-n03-bmc 10.244.11.68 OpenSSH regreSSHion
2289487733 9.38 apc05se1shcc-n02-bmc 10.244.11.67 OpenSSH regreSSHion
2289651084 9.38 apc02ctsbcom7-n01-cimc 10.244.4.23 OpenSSH regreSSHion
2289802898 9.38 10.244.11.57 OpenSSH regreSSHion
2454510043 9.38 10.244.11.95 OpenSSH regreSSHion
2687702557 9.38 syn-098-120-032-145 98.120.32.145 OpenSSH regreSSHion
2687710954 9.38 syn-098-120-000-129 98.120.0.129 OpenSSH regreSSHion
2284209398 9.06 rphy-runner-vecima 68.114.184.84 Rocky Linux sudo update (RLSA-2025:9978)
2288585418 9.06 rphy-runner-falconv Rocky Linux sudo update (RLSA-2025:9978)
2728824329 8.50 localhost Rocky Linux kernel update (RLSA-2026:6570)

Still same BU — 6 findings

Finding ID Severity Current State BU Host IP Address
2359379898 9.06 Closed NTS-AEO-STEAM aeo-bpa-app-01-lab
2286639694 9.38 Closed NTS-AEO-STEAM syn-024-024-116-183 24.24.116.183
2744295322 7.57 Open NTS-AEO-STEAM ana01pongcoc1 96.37.185.81
2687694321 9.38 Closed NTS-AEO-ACCESS-ENG asa04chaococ1 98.120.32.167
2687701818 9.38 Closed NTS-AEO-ACCESS-ENG asr01chaococ1 98.120.32.180
2687702475 9.38 Closed NTS-AEO-ACCESS-ENG asr02chaococ1 98.120.32.181

Finding 2744295322 is the only confirmed score drift case — dropped from 9.0 to 7.57. The other 5 are in the Closed state and still match the BU and severity filters; they were likely closed between syncs.

Completely gone from platform — 15 findings

These findings are not found in Ivanti at any BU, severity, or state. All are OpenSSH regreSSHion (CVE-2024-6387).

Finding ID Last Severity Host IP Address
2283426805 9.38 10.244.3.136
2284481283 9.38 10.244.3.165
2285495688 9.38 10.244.3.134
2285658756 9.38 10.244.3.137
2285828688 9.38 10.244.3.133
2286763965 9.38 10.244.3.135
2286932880 9.38 10.244.3.166
2288594216 9.38 10.244.3.164
2289475366 9.38 10.244.3.132
2662566450 9.38 syn-065-185-198-071 65.185.198.71
2662633263 9.38 syn-065-185-198-070 65.185.198.70
2687700013 9.38 syn-098-120-032-166 98.120.32.166
2687707862 9.38 syn-098-120-032-182 98.120.32.182
2613547630 9.30 096-037-187-009 96.37.187.9
2613548575 9.30 096-037-187-017 96.37.187.17

The 10.244.3.x subnet (9 findings) suggests a cluster of hosts that were decommissioned or removed from Ivanti's asset inventory entirely.

Diagnostic scripts

The drift-check.js and bu-reassignment-check.js scripts used during this investigation have been removed from the repository. Their logic is now automated by the sync anomaly detection feature in backend/routes/ivantiFindings.js, which classifies disappearances as BU reassignment, severity drift, closure, or decommission after each sync.

Reference in New Issue View Git Blame Copy Permalink