cve-dashboard/docs/guides/vcl-metric-calculations.md

VCL Metric Calculations — Database Reference

Overview

This document describes how every percentage, total, and forecast number on the VCL Report and CCP Metrics pages is computed from the underlying database. It is the single reference for verifying that what you see on the page matches what is in the data.

Each section answers four questions:

• What it shows — the field name on screen and the data path
• What feeds it — the table(s) and columns the value is computed from
• How it is calculated — the exact SQL or formula, plus any rounding rules
• Why it can drift — known sources of inaccuracy and how the dashboard guards against them

---

Table of Contents

• Data Sources
  • compliance_items
  • compliance_uploads
  • compliance_snapshots
  • vcl_multi_vertical_summary
• VCL Report Page (Single-Vertical / Legacy AEO)
  • Stats Bar
  • Donut Chart — Status of Non-Compliant Assets
  • Heavy Hitters Table
  • Vertical Breakdown Table
  • Compliance Trend Chart
• CCP Metrics Page (Multi-Vertical)
  • Aggregated Stats Bar
  • Donut — Blocked vs In-Progress
  • Trend Chart
  • Aggregated Burndown Forecast
  • Metric Table (Cross-Vertical)
  • Metric Detail View — Per-Vertical Breakdown
  • Per-Metric Forecast Burndown Chart
  • Per-Vertical Detail and Burndown
• Forecast Algorithms
  • Linear Regression Forecast (Trend)
  • Resolution-Date Burndown Forecast
  • Per-Metric Forecast (Historical + Projected)
• Cross-Cutting Correctness Rules
• Verifying Values by Hand
• Worked Example — Vulns_Aging vs 7.1.1 Forecast Charts

---

Data Sources

The VCL pages read from four tables. Knowing what each one stores is the prerequisite for understanding the calculations.

compliance_items

One row per (hostname, metric_id, vertical) combination per upload. Only non-compliant findings are stored here. Compliant devices never appear in this table — they are inferred from the Summary sheet's totals minus what is in compliance_items.

Column	Type	Notes
hostname	TEXT	Device hostname
metric_id	TEXT	Compliance metric identifier (e.g., 2.3.5, 7.1.1)
team	TEXT	Sub-team responsible (STEAM, ACCESS-ENG, etc.)
vertical	TEXT	Vertical code (NTS_AEO, SDIT_CISO, TSI); NULL for legacy AEO uploads
status	TEXT	'active' if currently failing, 'resolved' once the device drops off the next upload
resolution_date	DATE	Target remediation date (manual entry)
seen_count	INTEGER	Number of consecutive uploads this finding has appeared on
first_seen_upload_id / upload_id / resolved_upload_id	INTEGER	Upload references for first appearance, latest, and resolution

A device is "compliant" when no active row exists for it in this table.

compliance_uploads

One row per uploaded xlsx. A multi-vertical upload day produces multiple rows that share the same report_date.

Column	Type	Notes
report_date	TEXT	The reporting period the file covers (YYYY-MM-DD)
vertical	TEXT	Same vertical code as compliance_items.vertical; NULL for legacy AEO
new_count / recurring_count / resolved_count	INTEGER	Per-upload deltas (vertical-scoped)
summary_json	TEXT	The raw parsed Summary sheet — used as a fallback by /summary

compliance_snapshots

Monthly aggregated snapshot keyed by (snapshot_month, vertical). The trend chart reads exclusively from here. Snapshots are written automatically inside the upload commit transaction.

Column	Type	Notes
snapshot_month	TEXT	YYYY-MM
vertical	TEXT	Vertical code or team name (legacy)
total_devices / compliant / non_compliant	INTEGER	Counts at month end
compliance_pct	NUMERIC(5,2)	Pre-computed for that month

The (snapshot_month, vertical) pair is UNIQUE. Re-uploading inside the same calendar month overwrites the row via ON CONFLICT DO UPDATE.

vcl_multi_vertical_summary

One row per (metric_id, team) pair per upload, populated from the Summary sheet of a multi-vertical xlsx. This is the source of truth for compliant counts — compliance_items only has non-compliant rows.

Column	Type	Notes
upload_id	INTEGER	FK → compliance_uploads
vertical	TEXT	Vertical code
metric_id	TEXT	Metric identifier
team	TEXT	Either an ALL: <vertical> rollup row or a sub-team row (STEAM, ACCESS-ENG)
non_compliant / compliant / total	INTEGER	From the Summary sheet
compliance_pct	NUMERIC(5,2)	From the Summary sheet (decimal — 0.95 = 95%)
target	NUMERIC(5,2)	Per-metric target from the spreadsheet

Critical aggregation rule: rows where team LIKE 'ALL:%' are vertical-level rollups that already include their sub-teams. Aggregating both rollup and sub-team rows would double-count. Every cross-vertical query in this codebase filters with WHERE team LIKE 'ALL:%'.

---

VCL Report Page (Single-Vertical / Legacy AEO)

This is the original VCL Report at /api/compliance/vcl/.... It aggregates across whatever data exists in compliance_items regardless of vertical, and is primarily used for the AEO single-team view.

Source: backend/routes/compliance.js (router.get('/vcl/stats', ...) and router.get('/vcl/trend', ...))

Stats Bar

What it shows: Total Devices, In-Scope, Compliant, Non-Compliant, Remediations Required, Current %, Target %.

What feeds it: compliance_items — every distinct hostname.

How it is calculated:

SELECT
    COUNT(DISTINCT hostname) AS total_devices,
    COUNT(DISTINCT hostname) AS in_scope,
    COUNT(DISTINCT CASE
        WHEN hostname NOT IN (SELECT DISTINCT hostname FROM compliance_items WHERE status = 'active')
        THEN hostname END) AS compliant,
    COUNT(DISTINCT CASE
        WHEN hostname IN (SELECT DISTINCT hostname FROM compliance_items WHERE status = 'active')
        THEN hostname END) AS non_compliant
FROM compliance_items;

Then in JavaScript:

compliance_pct = in_scope > 0 ? Math.round((compliant / in_scope) * 100) : 0
remediations_required = non_compliant
target_pct = process.env.VCL_TARGET_PCT || 95

Field-by-field:

Field	Definition
Total Devices	Count of unique hostnames that have ever appeared in any upload
In-Scope	Same as Total Devices — every tracked device is in-scope by definition
Compliant	Hostnames with zero rows where status = 'active'
Non-Compliant	Hostnames with at least one row where status = 'active'
Remediations Required	Equals Non-Compliant — every non-compliant device needs at least one fix
Current %	ROUND((Compliant / In-Scope) * 100) — whole-number percent
Target %	VCL_TARGET_PCT env var on the backend, default 95

Why it can drift:

• If a hostname has both an active row in one vertical and a resolved row in another, the IN/NOT IN subqueries above already classify correctly — active wins because the IN subquery includes any active row.
• Compliant devices are inferred. If compliance_items is missing rows that the Summary sheet reported (e.g., truncated upload), the count silently undercounts.

Donut Chart — Status of Non-Compliant Assets

What it shows: Two slices — Blocked (red) and In-Progress (amber) — with counts and percentages.

What feeds it: compliance_items rows where status = 'active', deduplicated to one row per hostname using MAX(resolution_date).

How it is calculated:

SELECT hostname, MAX(resolution_date) AS resolution_date
FROM compliance_items
WHERE status = 'active'
GROUP BY hostname;

Then categorizeNonCompliant() partitions:

blocked    = items.filter(i => i.resolution_date == null)
in_progress = items.filter(i => i.resolution_date != null)
blocked.pct     = Math.round((blocked.count    / total) * 100)
in_progress.pct = Math.round((in_progress.count / total) * 100)

A device with any resolution date set on any of its active findings is considered In-Progress. Only when every active finding lacks a date is the device counted as Blocked.

Why it can drift:

• The MAX(resolution_date) clause means a device with one dated finding and one undated finding is classified as In-Progress, not Blocked. This is intentional — once one team commits to a date, the device is no longer fully blocked.
• Rounding to whole numbers means blocked.pct + in_progress.pct may total 99 or 101 in edge cases. The chart still displays the correct underlying counts.

Heavy Hitters Table

What it shows: One row per team, sorted by non-compliant device count descending. Columns: Vertical/Team, Non-Compliant, Compliance Date, Notes.

What feeds it: compliance_items deduplicated to one team per hostname, plus vcl_vertical_metadata for manual fields (Notes, Compliance Date, RAs).

How it is calculated:

WITH device_team AS (
    SELECT DISTINCT ON (hostname)
           hostname,
           COALESCE(team, 'Unknown') AS team,
           resolution_date
    FROM compliance_items
    WHERE status = 'active'
    ORDER BY hostname, seen_count DESC, upload_id DESC
)
SELECT team,
       COUNT(DISTINCT hostname)::int AS non_compliant,
       MAX(resolution_date) AS compliance_date
FROM device_team
GROUP BY team
ORDER BY COUNT(DISTINCT hostname) DESC;

The CTE picks one representative row per hostname using the (seen_count DESC, upload_id DESC) rule — the longest-running, most recently seen team assignment wins. This guarantees SUM(heavy_hitters[*].non_compliant) == stats.non_compliant.

Why it can drift:

• Before the fix tracked under spec compliance-duplicate-failing-metrics, a hostname that appeared with different team values across verticals was double-counted. The CTE above is the fix — confirmed by Property 3 of that spec.
• compliance_date here is the latest resolution date across the team's devices, used as a default. The team's manually entered Compliance Date in vcl_vertical_metadata overrides it when present.

Vertical Breakdown Table

What it shows: Same teams as Heavy Hitters, plus per-team Compliance %, Forecast Burndown columns, Blockers count, RAs, Notes.

What feeds it:

• Per-team total devices: same device_team CTE as Heavy Hitters but without the status = 'active' filter.
• Forecast: compliance_items with non-null resolution_date.
• Manual fields: vcl_vertical_metadata.

How it is calculated (per team):

-- Total devices for the team
WITH device_team AS (
    SELECT DISTINCT ON (hostname)
           hostname,
           COALESCE(team, 'Unknown') AS team
    FROM compliance_items
    ORDER BY hostname, seen_count DESC, upload_id DESC
)
SELECT COUNT(*)::int AS total FROM device_team WHERE team = $1;

-- Forecast resolution dates for the team
SELECT DISTINCT ON (hostname, metric_id) resolution_date
FROM compliance_items
WHERE status = 'active'
  AND COALESCE(team, 'Unknown') = $1
  AND resolution_date IS NOT NULL
ORDER BY hostname, metric_id, seen_count DESC, upload_id DESC;

In JavaScript:

team_compliant     = team_total - team_non_compliant
compliance_pct     = team_total > 0 ? Math.round((team_compliant / team_total) * 100) : 0
forecast_burndown  = computeForecastBurndown(forecastItems)   // YYYY-MM → count
blockers           = Math.max(team_non_compliant - forecastItems.length, 0)

computeForecastBurndown buckets each device's resolution date into a YYYY-MM key. The result is { "2026-06": 12, "2026-07": 8, ... } — the count of devices expected to resolve each month.

Why it can drift:

• The DISTINCT ON (hostname, metric_id) in the forecast query was added by the duplicate-failing-metrics fix. Without it, a device failing the same metric in two verticals would have its resolution date counted twice and blockers would go negative (the Math.max clamp protects the UI but masks the inconsistency).
• The team total uses all rows in compliance_items (active and resolved), so a team's total here is "every device that has ever been part of this team," not just current devices.

Compliance Trend Chart

What it shows: Bar chart of compliant device count per month, plus a solid line (actual %) and a dashed line (forecasted %) on a secondary axis. A horizontal reference line marks the target.

What feeds it: compliance_snapshots, aggregated across all verticals.

How it is calculated:

SELECT snapshot_month,
       SUM(compliant)::int AS compliant_count,
       CASE WHEN SUM(total_devices) > 0
            THEN ROUND((SUM(compliant)::numeric / SUM(total_devices)::numeric) * 100, 1)
            ELSE 0 END AS compliance_pct
FROM compliance_snapshots
GROUP BY snapshot_month
ORDER BY snapshot_month ASC;

The forecast logic is described in Linear Regression Forecast. Snapshots are persisted in persistUpload() using the upload's report_date month so historical uploads land in the correct bucket. See Cross-Cutting Correctness Rules for the snapshot upsert behavior.

Why it can drift:

• Snapshots are keyed (snapshot_month, vertical), so re-uploading the same month overwrites — only the latest upload's totals are preserved per month.
• Pre-fix snapshots from before the duplicate-failing-metrics correction may have compliant + non_compliant > total_devices if a hostname had both active and resolved rows across verticals. The fix uses MIN(status) inside a CTE so each hostname is classified once. Older snapshots written before the fix should be regenerated by re-running the affected uploads.

---

CCP Metrics Page (Multi-Vertical)

The CCP Metrics page is the executive cross-vertical view. It uses the multi-vertical Summary sheet data as the source of truth for totals (since compliance_items only contains non-compliant devices).

Source: backend/routes/vclMultiVertical.js. Mounted at /api/compliance/vcl-multi/....

Aggregated Stats Bar

What it shows: Total Devices, Compliant, Non-Compliant, Current %, Target % across every vertical's latest upload.

What feeds it: vcl_multi_vertical_summary filtered to ALL: rollup rows of the latest upload per vertical.

How it is calculated:

-- 1. Find the latest upload ID per vertical
SELECT DISTINCT ON (vertical) id, vertical
FROM compliance_uploads
WHERE vertical IS NOT NULL
ORDER BY vertical, id DESC;

-- 2. Sum totals from rollup rows only (avoids double-counting sub-teams)
SELECT vertical,
       SUM(total)::int AS total_devices,
       SUM(compliant)::int AS compliant,
       SUM(non_compliant)::int AS non_compliant
FROM vcl_multi_vertical_summary
WHERE upload_id = ANY($1) AND team LIKE 'ALL:%'
GROUP BY vertical;

In JavaScript:

agg_total         = SUM(vertical.total_devices for each vertical)
agg_compliant     = SUM(vertical.compliant     for each vertical)
agg_non_compliant = SUM(vertical.non_compliant for each vertical)
compliance_pct    = agg_total > 0 ? Math.round((agg_compliant / agg_total) * 100) : 0

The team LIKE 'ALL:%' filter is the most important rule on this page. Each Summary sheet contains one rollup row per metric (ALL: NTS-AEO) plus one row per sub-team (STEAM, ACCESS-ENG). Summing both rollup and sub-team rows would double the totals. Every cross-vertical query enforces this filter.

Why it can drift:

• If a Summary sheet ever omits the ALL: rollup row for a metric, that metric's totals will be missing from the aggregate. The Python parser does not fabricate rollup rows, so this is a function of what the upstream xlsx contains.
• Verticals with no rows in vcl_multi_vertical_summary (e.g., the legacy AEO data) do not contribute. Their data is visible only on the original VCL Report, not the CCP Metrics page.

Donut — Blocked vs In-Progress

What it shows: Same as the legacy donut, scoped to multi-vertical data.

What feeds it: compliance_items where vertical IS NOT NULL, deduplicated by hostname.

How it is calculated: Identical formula to the legacy donut, but the filter is WHERE vertical IS NOT NULL AND status = 'active'.

SELECT hostname, MAX(resolution_date) AS resolution_date
FROM compliance_items
WHERE vertical IS NOT NULL AND status = 'active'
GROUP BY hostname;

Trend Chart

What it shows: Cross-vertical monthly trend of compliant device count and compliance percentage with a 3-month forecast.

What feeds it: compliance_snapshots where vertical IS NOT NULL AND vertical != ''.

How it is calculated:

SELECT snapshot_month,
       SUM(total_devices)::int AS total_devices,
       SUM(compliant)::int     AS compliant,
       SUM(non_compliant)::int AS non_compliant
FROM compliance_snapshots
WHERE vertical IS NOT NULL AND vertical != ''
GROUP BY snapshot_month
ORDER BY snapshot_month ASC;

Each row's compliance_pct is ROUND((compliant / total_devices) * 100, 1) — one decimal place. The forecast then uses the Linear Regression Forecast logic.

Aggregated Burndown Forecast

What it shows: A bar chart of expected device remediations per month across all verticals, plus stat cards for In-Progress, Blockers, and Projected Clear date.

What feeds it: compliance_items where vertical IS NOT NULL AND status = 'active'.

How it is calculated:

SELECT hostname, resolution_date, vertical
FROM compliance_items
WHERE vertical IS NOT NULL AND status = 'active';

Then the rows pass through two pure helpers:

1. deduplicateByHostname(rows) — collapses each hostname to one entry, keeping the earliest non-null resolution_date. A device that fails three metrics with different planned dates is bucketed by its earliest commitment.

2. computeAggregatedBurndown(devices) — computes:

   total       = devices.length
   blockers    = devices.filter(d => d.resolution_date == null).length
   with_dates  = total - blockers
   monthly[m]  = count of devices whose resolution_date falls in month m   // YYYY-MM
   projection[m] = { remediated: monthly[m], remaining: running_remainder }
   projected_clear_date = (blockers === 0 && monthly is non-empty)
       ? last_month_in_monthly_keys
       : null
   

Projected Clear is only computed when blockers === 0. Any device without a resolution date prevents the projection from showing — the dashboard is honest about the fact that an open-ended commitment cannot be projected.

Why it can drift:

• Devices with multiple compliance_items rows (one per failing metric) are deduplicated by hostname before bucketing. Without deduplication, a device with three failing metrics and one resolution date would count three times.
• A resolution date in the past still buckets into its actual month — computeAggregatedBurndown does not roll past-due dates forward. The per-metric chart does roll them forward; see Per-Metric Forecast for that distinction.

Metric Table (Cross-Vertical)

What it shows: One row per metric, with non-compliant, compliant, total, compliance %, and target % aggregated across every vertical's latest upload. Sorted by non-compliant descending.

What feeds it: vcl_multi_vertical_summary rollup rows from the latest upload per vertical.

How it is calculated:

SELECT metric_id,
       MAX(metric_desc) AS metric_desc,
       MAX(category)    AS category,
       SUM(non_compliant)::int   AS non_compliant,
       SUM(compliant)::int       AS compliant,
       SUM(total)::int           AS total,
       ROUND(AVG(target::numeric), 4) AS target
FROM vcl_multi_vertical_summary
WHERE upload_id = ANY($1) AND team LIKE 'ALL:%'
GROUP BY metric_id
ORDER BY non_compliant DESC;

Then for each row:

compliance_pct = total > 0 ? compliant / total : 0   // stored as decimal

The frontend renders the percentage with one decimal: (compliance_pct * 100).toFixed(1) + '%'.

target is the arithmetic mean across verticals, not the worst or best. If two verticals report a target of 0.90 and 0.95 for the same metric, the cross-vertical target is 0.925. This is a deliberate choice — the page shows a fleet-wide composite target, not the strictest individual one.

Why it can drift:

• MAX(metric_desc) and MAX(category) rely on every Summary sheet using the same description for the same metric_id. If two verticals describe the same metric differently, the alphabetically-last description wins.

Metric Detail View — Per-Vertical Breakdown

What it shows: For a selected metric, one row per vertical with that metric's numbers, plus a sub_teams array per vertical.

What feeds it: vcl_multi_vertical_summary for the selected metric, latest upload per vertical, both rollup and sub-team rows.

How it is calculated:

SELECT vertical, metric_desc, category, team,
       non_compliant, compliant, total, compliance_pct, target
FROM vcl_multi_vertical_summary
WHERE upload_id = ANY($1) AND metric_id = $2
ORDER BY vertical, team;

The handler then separates rollup rows (team LIKE 'ALL:%') from sub-team rows in JavaScript:

• The rollup row for each vertical becomes the primary entry.
• Each sub-team row is attached to its vertical's sub_teams array.
• Rows where team = '(Other)' are skipped — they are catch-all rows already counted in the rollup.

compliance_pct is read directly from the table (already a decimal — 0.95 = 95%).

Why it can drift:

• A sub-team named (Other) is used by the spreadsheet for unassignable devices — it is intentionally excluded from the sub-team breakdown to avoid duplication.
• The vertical-level compliance_pct is what was in the Summary sheet at upload time. It is not recomputed from compliant / total. If those numbers ever disagree (Summary rounded differently), the table shows the spreadsheet's number.

Per-Metric Forecast Burndown Chart

What it shows: A combined chart with up to 4 historical monthly snapshots (left of the divider) and up to 12 forecast months (right of the divider). Each data point shows total assets, non-compliant count, and compliance %.

What feeds it: Three sources combined:

1. compliance_snapshots for historical totals (3 months back).
2. vcl_multi_vertical_summary for the metric's total (used as total_assets).
3. compliance_items for current devices and their resolution dates.

How it is calculated:

-- Active devices for this metric across every vertical it spans
SELECT hostname, resolution_date, vertical
FROM compliance_items
WHERE metric_id = $1 AND status = 'active' AND vertical IS NOT NULL;

-- Historical snapshots for those verticals (3 months back)
SELECT snapshot_month AS month,
       SUM(total_devices)::int AS total_assets,
       SUM(non_compliant)::int AS non_compliant,
       ROUND((SUM(compliant)::numeric / NULLIF(SUM(total_devices), 0)) * 100, 1) AS compliance_pct
FROM compliance_snapshots
WHERE vertical = ANY($1) AND snapshot_month >= $2 AND snapshot_month < $3
GROUP BY snapshot_month
ORDER BY snapshot_month ASC;

-- The metric's per-metric total assets (from latest summary)
SELECT SUM(total)::int AS total
FROM vcl_multi_vertical_summary
WHERE metric_id = $1 AND team LIKE 'ALL:%'
  AND upload_id IN (
    SELECT id FROM compliance_uploads
    WHERE vertical IS NOT NULL
    ORDER BY id DESC LIMIT 20
  );

Historical Computation — Ratio Method

The compliance_snapshots table stores vertical-level totals, not per-metric. To estimate this metric's historical non-compliant count for a past month, the handler uses a ratio:

metric_nc_for_month = ROUND(
    snapshot.non_compliant_for_vertical * (current_metric_nc / current_vertical_total_nc)
)

In words: "the metric's share of the vertical's current non-compliant load is assumed constant — apply that ratio to historical snapshot non-compliant counts."

The ratio method is an approximation. It assumes the metric's contribution to the vertical's non-compliance is steady over the last three months. If the metric load shifts dramatically month-to-month, the historical bars will not match the actual past.

The current month is always computed from live data (not the ratio):

current_month_nc = number of distinct active hostnames for this metric
current_compliance_pct = ROUND((total_assets - current_month_nc) / total_assets * 1000) / 10   // 1 decimal

Forecast Computation — Resolution Date Bucketing

Forward projections are handled by computeMetricForecastBurndown. The algorithm:

1. Partition active devices into blockers (no resolution date) and with_dates.
2. Bucket each dated device by its resolution_date month (YYYY-MM).
3. Past-due dates roll into the current month. A device whose date is March when today is May counts as remediating in May, not March.
4. Walk forward up to 12 months, decrementing remaining_non_compliant by each month's bucket.
5. Stop early if remaining_non_compliant <= blockers — meaning every device with a date is projected to be done and only blockers remain.

The compliance percentage at each forecast point is:

compliance_pct = ROUND((total_assets - remaining_non_compliant) / total_assets * 1000) / 10

Correctness Properties of the Forecast

These properties hold for any input (verified by property-based tests in backend/__tests__/compliance-duplicate-chart-entries.property.test.js and the forecast spec):

1. Partition invariant: blockers + with_dates == non_compliant.
2. Compliance formula: compliance_pct == ROUND((total - nc) / total * 1000) / 10 (or 0 when total is 0).
3. Monotonic non-increasing: each month's non_compliant is less than or equal to the previous month's.
4. Horizon bound: at most 12 forecast points; terminates early when only blockers remain.
5. Past-due treated as current month: dates in already-passed months are bucketed into the current month for projection purposes.

Why it can drift:

• The ratio method for historical data is an estimate. Verify by hand using the actual upload's Summary sheet for that month if precise historical numbers matter.
• The fallback totalAssets = metricNcCount triggers when no Summary data exists for the metric. This produces a compliance_pct of 0 because every "asset" is non-compliant. This is correct for the data we have — the chart cannot show compliance percentages for metrics that have only been observed as failures.

Per-Vertical Detail and Burndown

What it shows: Stats and burndown for a single vertical (e.g., NTS_AEO).

What feeds it:

• Stats: vcl_multi_vertical_summary for that vertical, latest upload, with sub-team breakouts.
• Burndown: compliance_items for that vertical, deduplicated by hostname.

How it is calculated:

The vertical-level burndown deduplicates per hostname using the first non-null resolution date (any one is enough to mark the device In-Progress):

// In the route handler, after fetching compliance_items for the vertical:
const deviceMap = {};
for (const row of rows) {
    if (!deviceMap[row.hostname]) {
        deviceMap[row.hostname] = { hostname: row.hostname, resolution_date: row.resolution_date };
    } else if (row.resolution_date && !deviceMap[row.hostname].resolution_date) {
        // Promote a null entry to In-Progress when any other row has a date
        deviceMap[row.hostname].resolution_date = row.resolution_date;
    }
}
const devices = Object.values(deviceMap);
const burndown = computeVerticalBurndown(devices);

computeVerticalBurndown returns the same shape as computeAggregatedBurndown but scoped to one vertical.

---

Forecast Algorithms

The dashboard uses three different forecasting approaches depending on the data being projected.

Linear Regression Forecast (Trend)

Used by: Trend chart on the VCL Report and CCP Metrics pages.

Inputs: Monthly compliance percentages from compliance_snapshots (one decimal place).

Algorithm: Least-squares linear regression on the time series.

// X = month index (0, 1, 2, ...), Y = compliance_pct
slope     = (n * SUM(X*Y) - SUM(X) * SUM(Y)) / (n * SUM(X^2) - SUM(X)^2)
intercept = (SUM(Y) - slope * SUM(X)) / n

For each future month i (1, 2, 3 — three months out):

forecast_pct = ROUND((slope * (n + i - 1) + intercept) * 10) / 10
forecast_pct = Math.min(100, Math.max(0, forecast_pct))   // clamp to [0, 100]

Activation: Forecast appears only when 3 or more historical months exist. With fewer points, the regression is unreliable, so the dashed line is omitted.

Why it works for compliance trends: Compliance is bounded [0, 100] and changes slowly across months. A linear fit captures the directional trajectory ("we're trending up two points per month") accurately enough to inform planning, though it can over-predict near the boundaries (the clamp prevents impossible values).

Resolution-Date Burndown Forecast

Used by: Aggregated burndown, per-vertical burndown, the deprecated team-level forecast in the legacy VCL Report.

Inputs: Active non-compliant devices and their resolution_date values.

Algorithm: Simple monthly bucketing — no math beyond grouping and counting.

buckets = {}                                  // YYYY-MM → count
for each device with a non-null resolution_date:
    month = first 7 chars of resolution_date  // 'YYYY-MM-DD' → 'YYYY-MM'
    buckets[month] += 1

remaining = total_with_dates
projection = {}
for each month (sorted ascending):
    remaining -= buckets[month]
    projection[month] = { remediated: buckets[month], remaining }

Projected Clear date: the last month in projection only if blockers === 0. If any device lacks a date, no projection is shown — there is no honest way to forecast something with no commitment.

Why this is preferred over regression for burndown: Resolution dates are explicit human commitments. Linear regression on past remediation rates would project an average pace that ignores what teams have actually committed to. The bucketing approach reports exactly what has been promised and nothing more.

Per-Metric Forecast (Historical + Projected)

Used by: The CCP Metrics per-metric forecast burndown chart.

Inputs: Historical snapshots (vertical-level) + current metric devices (per-metric) + the metric's total_assets from the Summary sheet.

Algorithm: Two separate parts joined at the current month.

Historical part — Ratio Method:

// For each historical month from compliance_snapshots:
metric_share        = current_metric_nc / current_vertical_total_nc
month.non_compliant = ROUND(snapshot.non_compliant * metric_share)
month.compliance_pct = ROUND((total_assets - month.non_compliant) / total_assets * 1000) / 10

The ratio method assumes the metric's share of vertical non-compliance is stable. If a metric was recently introduced or recently fixed at scale, the historical bars will be off. Validate against the source Summary sheet for that month if you need precision.

Forecast part — Resolution-Date Bucketing:

Identical to Resolution-Date Burndown Forecast, with one extra rule: past-due dates roll into the current month. A device with resolution_date = '2026-02-15' when today is May is bucketed into May, not February. Empirically, past-due dates are commitments that slipped — projecting them as remediating "now" reflects reality (the team is overdue and has to act this month) better than leaving them stuck in the past.

Termination: the loop exits as soon as remaining_non_compliant <= blockers. Once every dated device is projected to be done, continuing would just show flat blocker count.

---

Cross-Cutting Correctness Rules

These rules apply across every metric on both pages. Violations indicate a real bug — the dashboard's tests verify these properties hold.

Rule 1: Rollup-only aggregation across verticals. Every cross-vertical query that touches vcl_multi_vertical_summary filters with WHERE team LIKE 'ALL:%'. Aggregating both rollup rows and sub-team rows would double-count. (Validated by the ccp-metrics-view-restructure spec, Property 1.)

Rule 2: Latest upload per vertical. Cross-vertical queries select DISTINCT ON (vertical) from compliance_uploads ORDER BY vertical, id DESC to take only the most recent upload per vertical. Older uploads contribute to compliance_snapshots but not to current totals.

Rule 3: Snapshot deduplication. compliance_snapshots is keyed UNIQUE(snapshot_month, vertical) and updated via ON CONFLICT DO UPDATE. Re-uploading the same month for a vertical overwrites the earlier snapshot. Snapshots are upserted using the upload's report_date month, not the current calendar month, so a backfilled upload for March lands in 2026-03 even if it is uploaded in May.

Rule 4: Status classification on duplicate hostnames. When a hostname has rows in multiple verticals (one active, one resolved), the snapshot logic uses MIN(status) inside a CTE — 'active' lexicographically wins over 'resolved', so the device is classified as non-compliant. This guarantees compliant + non_compliant <= total_devices for every snapshot row.

Rule 5: Per-(hostname, metric_id) deduplication. Queries that bucket or count active findings use DISTINCT ON (hostname, metric_id) with the canonical ORDER BY hostname, metric_id, seen_count DESC, upload_id DESC. A device failing the same metric in two verticals contributes one entry, not two. (Validated by the compliance-duplicate-failing-metrics spec, Properties 1–5.)

Rule 6: Aggregation by report_date, not upload ID. Trends, top-recurring, and category-trend queries GROUP BY report_date rather than GROUP BY id. A multi-vertical day produces multiple compliance_uploads rows sharing one report_date. (Validated by the compliance-duplicate-chart-entries spec, Properties 1–3.)

Rule 7: Decimal vs whole-number percentages. Two conventions coexist:

Source	Format	Example
vcl_multi_vertical_summary.compliance_pct	Decimal	0.95
compliance_snapshots.compliance_pct	Whole-number	95.00
/vcl/stats response stats.compliance_pct	Whole-number integer	95
/vcl-multi/stats response stats.compliance_pct	Whole-number integer	95
/vcl-multi/metrics response compliance_pct	Decimal	0.95

The frontend handles both — multiply decimals by 100 and call toFixed(1) for the metric-table view, or display the whole number directly for stats bars. Mismatching the formats is a common source of "values look 100x off" bugs.

---

Verifying Values by Hand

When you suspect a number is wrong, work through this checklist before opening a bug.

1. Check whether the value comes from compliance_items or vcl_multi_vertical_summary.

• compliance_items only has non-compliant rows. If a "compliant count" is wrong, the bug is in the Summary sheet path, not item counting.
• vcl_multi_vertical_summary is the source of truth for both compliant and non-compliant totals on the CCP Metrics page.

2. Check the upload date.

• Cross-vertical numbers use the latest upload per vertical. If you uploaded NTS_AEO yesterday but TSI two weeks ago, the aggregate uses today's NTS_AEO and the two-week-old TSI.
• The "Last Upload" column in the vertical breakdown shows the report_date of each vertical's most recent upload.

3. Check the snapshot.

• The trend chart reads from compliance_snapshots, not from current compliance_items. If you fixed a hostname today, it will not appear in the trend until the next upload writes a new snapshot.
• Historical months are frozen — only the current month's snapshot updates on re-upload.

4. Check for ALL: rollup vs sub-team aggregation.

• If a vertical or cross-vertical total looks roughly 2x too high, you are probably summing rollup AND sub-team rows. Confirm with:

  SELECT team, COUNT(*) FROM vcl_multi_vertical_summary
  WHERE upload_id = <latest> AND metric_id = '<metric>'
  GROUP BY team;
  

  You should see one ALL: <vertical> row plus one row per sub-team. Use only the ALL: row for cross-vertical totals.

5. Check for cross-vertical hostname collisions.

• A hostname appearing in two verticals (e.g., it was migrated between teams) needs the deduplication rules in Cross-Cutting Correctness Rules to count once. Confirm with:

  SELECT hostname, vertical, team, status, seen_count
  FROM compliance_items
  WHERE hostname = '<hostname>'
  ORDER BY hostname, vertical;
  

  If you see two rows with different team values, the device is counted under the team from its representative row (highest seen_count, then most recent upload_id).

6. Reconcile against the Summary sheet.

• Open the source xlsx for the upload, navigate to the Summary tab, and find the ALL: <vertical> row for the metric in question. The Total, Compliant, and Non-Compliant columns should match vcl_multi_vertical_summary exactly (the parser does not transform these numbers — it copies them verbatim).

If after these steps the displayed value still does not match the source data, file an issue with the SQL output from steps 4–5 and the relevant Summary sheet rows attached.

---

Worked Example — Vulns_Aging vs 7.1.1 Forecast Charts

This example walks the playbook through two side-by-side per-metric forecast burndown charts that look very different despite using the same code path. It is the canonical case study for "the chart looks weird but the data is internally consistent."

What the charts show

Vulns_Aging — 17,628 devices. Every bar is full-height, predominantly orange. The compliance line reads 0.1% for March, jumps to 27.3% in April, then back to 0.0% for May. The total bar height is 17628 on every month.

7.1.1 — 6,149 non-compliant out of ~66,674 total. Every bar is mostly blue (compliant) with a thin orange slice at the top. The compliance line stays between 90.8% and 93.3%. The total bar height is 66674 on every month.

Both charts are produced by the same endpoint (GET /api/compliance/vcl-multi/metric/:metricId/forecast-burndown) and the same helper (computeMetricForecastBurndown). The difference is entirely in the input data.

Walking the playbook

Step 1 — Where does the value come from?

Both charts blend two sources: compliance_items for active non-compliant devices, and vcl_multi_vertical_summary for the metric's total_assets. The historical bars also use compliance_snapshots (vertical-level totals) reshaped via the ratio method.

The differing input here is total_assets, sourced from this query:

SELECT SUM(total)::int AS total
FROM vcl_multi_vertical_summary
WHERE metric_id = $1 AND team LIKE 'ALL:%'
  AND upload_id IN (
      SELECT id FROM compliance_uploads
      WHERE vertical IS NOT NULL ORDER BY id DESC LIMIT 20
  );

For 7.1.1 this returns ~66,674. For Vulns_Aging it returns 0 or null, and the route handler falls back to totalAssets = metricNcCount — the device count from compliance_items instead.

Step 2 — Check the upload date.

Both metrics are read from the latest 20 uploads, so this is not the cause of the difference. Skip.

Step 3 — Check the snapshot.

compliance_snapshots is vertical-level only — it does not store per-metric totals. Both metrics use the same snapshot rows scaled by the ratio method:

metricNc = ROUND(snapshot.non_compliant * (currentMetricNc / currentVerticalTotalNc))

For Vulns_Aging the ratio is large (most of the vertical's non-compliant load is aging vulnerabilities), so historical metricNc values are sizeable. April happened to have a snapshot where the vertical's total was smaller — the ratio method produced a metricNc of roughly 17628 * 0.727 ≈ 12810, leaving (17628 - 12810) / 17628 * 100 ≈ 27.3% compliance for that bar.

That April spike is not a real compliance gain. It is an artifact of the historical snapshot's vertical-level non-compliant count being lower that month, multiplied by the metric's current share.

Step 4 — ALL: rollup vs sub-team aggregation.

Run the diagnostic query for each metric:

SELECT team, total, compliant, non_compliant
FROM vcl_multi_vertical_summary
WHERE metric_id = '7.1.1' AND team LIKE 'ALL:%'
ORDER BY upload_id DESC LIMIT 5;

For 7.1.1 this returns rows like ('ALL: NTS-AEO', 66674, 60525, 6149) — a clean, complete rollup.

SELECT team, total, compliant, non_compliant
FROM vcl_multi_vertical_summary
WHERE metric_id = 'Vulns_Aging' AND team LIKE 'ALL:%'
ORDER BY upload_id DESC LIMIT 5;

For Vulns_Aging this returns zero rows. The Summary sheet of the source xlsx does not contain Vulns_Aging as a tracked metric. The metric exists only as a detail sheet inside the workbook, never as a Summary row.

Step 5 — Cross-vertical hostname collisions.

Confirm the device counts come from compliance_items:

SELECT COUNT(DISTINCT hostname) FROM compliance_items
WHERE metric_id = 'Vulns_Aging' AND status = 'active' AND vertical IS NOT NULL;
-- Returns 17628

This matches the chart's total_assets exactly, which is the smoking gun. The chart is showing 17,628 in scope and 17,628 non-compliant — compliance_pct = 0% is mathematically correct for the data the chart received.

Step 6 — Reconcile against the Summary sheet.

Open any recent xlsx upload for any vertical and inspect the Summary tab. Search for Vulns_Aging in the Metric column — it is not there. The Summary sheet enumerates the standard metric IDs (2.3.5, 5.2.4, 7.1.1, etc.) and has no row for the aging vulnerability dashboard.

Now search for 7.1.1 — it appears with rows for both the rollup (ALL: NTS-AEO) and each sub-team. The Summary's Total, Compliant, and Non-Compliant columns match vcl_multi_vertical_summary exactly.

Why the charts look the way they do

Vulns_Aging is a tracked-but-uncategorized detail metric. The Python parser walks every detail sheet of the xlsx and writes one compliance_items row per non-compliant device, using the sheet name as the metric_id. Vulns_Aging is one such sheet. But because Vulns_Aging does not appear in the workbook's Summary sheet, no row is written to vcl_multi_vertical_summary — there is no rollup, no compliant count, no total population.

When the per-metric forecast endpoint asks for the metric's total, the query returns null. The handler falls back to totalAssets = metricNcCount, which is the count of non-compliant devices. The chart's denominator is therefore identical to its numerator, which forces every "current" compliance percentage to 0%. The April 27.3% spike is the historical ratio method projecting a smaller non-compliant count for that month against the same fallback denominator. May returns to 0% because May is the current month and uses the live device count, which always equals the fallback total by construction.

The chart is internally consistent with the data the database has. It is not internally consistent with what an executive expects "compliance" to mean for an aging dashboard, because the source xlsx never reported a population total for that detail. The fix is upstream — either the xlsx Summary sheet needs to include a Vulns_Aging row with a population total, or the handler needs a special case to mark metrics with no Summary data as "not measurable for compliance percentage" and render the chart differently (e.g., counts only, no percentage line).

7.1.1 is a fully populated standard metric. It has a sheet for the failing devices, a metric_categories mapping (Logging & Monitoring), and a Summary row with Total, Compliant, and Non-Compliant numbers. The route gets a real totalAssets value (~66,674), the compliance percentage is (66674 - 6149) / 66674 ≈ 90.8%, and the bar visualizes the actual ratio of compliant to non-compliant devices. Historical and forecast bars track real population data, not a fallback.

What this tells you about the dashboard

The forecast burndown chart is honest about what it knows. When the source data lacks a population total, the chart degrades gracefully to "every device in scope is non-compliant" — which is a literal reading of the rows that exist. It does not fabricate a denominator. The cost is that metrics without Summary entries look catastrophically non-compliant on the chart even when the underlying business reality may be different.

The diagnostic flow above is the canonical procedure: when a chart looks wrong, walk down to the Summary sheet rows and ask whether the metric is even represented as a tracked compliance metric in the source file. If the answer is no, the chart is reflecting an upstream data shape, not a calculation bug.

---

Worked Example — Vulns_Aging vs 7.1.1 Forecast Charts

This example walks the playbook through two side-by-side per-metric forecast burndown charts that look very different despite using the same code path. It is the canonical case study for "the chart looks weird but the data is internally consistent."

What the charts show

Vulns_Aging — 17,628 devices. Every bar is full-height, predominantly orange. The compliance line reads 0.1% for March, jumps to 27.3% in April, then back to 0.0% for May. The total bar height is 17628 on every month.

7.1.1 — 6,149 non-compliant out of ~66,674 total. Every bar is mostly blue (compliant) with a thin orange slice at the top. The compliance line stays between 90.8% and 93.3%. The total bar height is 66674 on every month.

Both charts are produced by the same endpoint (GET /api/compliance/vcl-multi/metric/:metricId/forecast-burndown) and the same helper (computeMetricForecastBurndown). The difference is entirely in the input data.

Walking the playbook

Step 1 — Where does the value come from?

Both charts blend two sources: compliance_items for active non-compliant devices, and vcl_multi_vertical_summary for the metric's total_assets. The historical bars also use compliance_snapshots (vertical-level totals) reshaped via the ratio method.

The differing input here is total_assets, sourced from this query:

SELECT SUM(total)::int AS total
FROM vcl_multi_vertical_summary
WHERE metric_id = $1 AND team LIKE 'ALL:%'
  AND upload_id IN (
      SELECT id FROM compliance_uploads
      WHERE vertical IS NOT NULL ORDER BY id DESC LIMIT 20
  );

For 7.1.1 this returns ~66,674. For Vulns_Aging it returns 0 or null, and the route handler falls back to totalAssets = metricNcCount — the device count from compliance_items instead.

Step 2 — Check the upload date.

Both metrics are read from the latest 20 uploads, so this is not the cause of the difference. Skip.

Step 3 — Check the snapshot.

compliance_snapshots is vertical-level only — it does not store per-metric totals. Both metrics use the same snapshot rows scaled by the ratio method:

metricNc = ROUND(snapshot.non_compliant * (currentMetricNc / currentVerticalTotalNc))

For Vulns_Aging the ratio is large (most of the vertical's non-compliant load is aging vulnerabilities), so historical metricNc values are sizeable. April happened to have a snapshot where the vertical's total non-compliant count was lower — the ratio method produced a metricNc of roughly 17628 * 0.727 ≈ 12810, leaving (17628 - 12810) / 17628 * 100 ≈ 27.3% compliance for that bar.

That April spike is not a real compliance gain. It is an artifact of the historical snapshot's vertical-level non-compliant count being lower that month, multiplied by the metric's current share.

Step 4 — ALL: rollup vs sub-team aggregation.

Run the diagnostic query for each metric:

SELECT team, total, compliant, non_compliant
FROM vcl_multi_vertical_summary
WHERE metric_id = '7.1.1' AND team LIKE 'ALL:%'
ORDER BY upload_id DESC LIMIT 5;

For 7.1.1 this returns rows like ('ALL: NTS-AEO', 66674, 60525, 6149) — a clean, complete rollup.

SELECT team, total, compliant, non_compliant
FROM vcl_multi_vertical_summary
WHERE metric_id = 'Vulns_Aging' AND team LIKE 'ALL:%'
ORDER BY upload_id DESC LIMIT 5;

For Vulns_Aging this returns zero rows. The Summary sheet of the source xlsx does not contain Vulns_Aging as a tracked metric. The metric exists only as a detail sheet inside the workbook, never as a Summary row.

Step 5 — Cross-vertical hostname collisions.

Confirm the device counts come from compliance_items:

SELECT COUNT(DISTINCT hostname) FROM compliance_items
WHERE metric_id = 'Vulns_Aging' AND status = 'active' AND vertical IS NOT NULL;
-- Returns 17628

This matches the chart's total_assets exactly, which is the smoking gun. The chart is showing 17,628 in scope and 17,628 non-compliant — compliance_pct = 0% is mathematically correct for the data the chart received.

Step 6 — Reconcile against the Summary sheet.

Open any recent xlsx upload for any vertical and inspect the Summary tab. Search for Vulns_Aging in the Metric column — it is not there. The Summary sheet enumerates the standard metric IDs (2.3.4i, 5.2.4, 7.1.1, etc.) and has no row for the aging vulnerability dashboard.

Now search for 7.1.1 — it appears with rows for both the rollup (ALL: NTS-AEO) and each sub-team. The Summary's Total, Compliant, and Non-Compliant columns match vcl_multi_vertical_summary exactly.

Why the charts look the way they do

Vulns_Aging is a tracked-but-uncategorized detail metric. The Python parser walks every detail sheet of the xlsx and writes one compliance_items row per non-compliant device, using the sheet name as the metric_id. Vulns_Aging is one such sheet. But because Vulns_Aging does not appear in the workbook's Summary sheet, no row is written to vcl_multi_vertical_summary — there is no rollup, no compliant count, no total population.

When the per-metric forecast endpoint asks for the metric's total, the query returns null. The handler falls back to totalAssets = metricNcCount, which is the count of non-compliant devices. The chart's denominator is therefore identical to its numerator, which forces every "current" compliance percentage to 0%. The April 27.3% spike is the historical ratio method projecting a smaller non-compliant count for that month against the same fallback denominator. May returns to 0% because May is the current month and uses the live device count, which always equals the fallback total by construction.

The chart is internally consistent with the data the database has. It is not internally consistent with what an executive expects "compliance" to mean for an aging dashboard, because the source xlsx never reported a population total for that detail. The fix is upstream — either the xlsx Summary sheet needs to include a Vulns_Aging row with a population total, or the handler needs a special case to mark metrics with no Summary data as "not measurable for compliance percentage" and render the chart differently (e.g., counts only, no percentage line).

7.1.1 is a fully populated standard metric. It has a sheet for the failing devices, a metric_categories mapping (Logging & Monitoring), and a Summary row with Total, Compliant, and Non-Compliant numbers. The route gets a real totalAssets value (~66,674), the compliance percentage is (66674 - 6149) / 66674 ≈ 90.8%, and the bar visualizes the actual ratio of compliant to non-compliant devices. Historical and forecast bars track real population data, not a fallback.

What this tells you about the dashboard

The forecast burndown chart is honest about what it knows. When the source data lacks a population total, the chart degrades gracefully to "every device in scope is non-compliant" — which is a literal reading of the rows that exist. It does not fabricate a denominator. The cost is that metrics without Summary entries look catastrophically non-compliant on the chart even when the underlying business reality may be different.

The diagnostic flow above is the canonical procedure: when a chart looks wrong, walk down to the Summary sheet rows and ask whether the metric is even represented as a tracked compliance metric in the source file. If the answer is no, the chart is reflecting an upstream data shape, not a calculation bug.