Add metrics calculation explainer and sub-team drill-down docs to design brief

This commit is contained in:
Jordan Ramos
2026-05-14 13:00:09 -06:00
parent 61d7e00d4f
commit 682ee9417f

View File

@@ -93,28 +93,120 @@ There is no automated way to:
``` ```
Executive Overview (all verticals aggregated) Executive Overview (all verticals aggregated)
├── Stats: 4200 devices, 90% compliant, target 95% ├── Stats: 2.1M devices, 97% compliant, target 95%
├── Trend: monthly compliance % with forecast ├── Trend: monthly compliance % with forecast
├── Donut: 30% blocked, 70% in-progress ├── Donut: blocked vs in-progress (non-compliant devices)
└── Vertical Breakdown Table └── Vertical Breakdown Table
├── NTS_AEO — 90% — 80 non-compliant — click to drill down ├── NTS_AEO — 99% — 2,163 non-compliant — click to drill down
│ │ │ │
│ ├── Metric Breakdown │ ├── Team Filter: [All (Rollup)] [ACCESS-ENG] [ACCESS-OPS] [INTELDEV] [STEAM]
│ │ ├── 5.2.4 (Access & MFA) — 98.1% — 15 non-compliant │ │
│ ├── 1.1.1 (Logging) — 85% — 120 non-compliant → click ├── Metric Breakdown (expandable rows)
│ │ │ └── Device list: hostname, IP, type, team, seen_count │ │ ├── ▸ 5.5.4i (Vulnerability Mgmt) — 97.0% — 1,762 NC — target 80%
│ │ │ ├── └ ACCESS-ENG: 7 compliant, 1 NC, 8 total — 88.0%
│ │ │ ├── └ ACCESS-OPS: 64,051 compliant, 1,746 NC, 65,797 total — 97.0%
│ │ │ ├── └ INTELDEV: 233 compliant, 11 NC, 244 total — 95.0%
│ │ │ └── └ STEAM: 123 compliant, 4 NC, 127 total — 97.0%
│ │ │
│ │ ├── ▸ 7.1.1 (Logging & Monitoring) — 72.0% — 149 NC → click metric
│ │ │ └── Device list: hostname, IP, type, team, seen_count, resolution_date
│ │ └── ... │ │ └── ...
│ │ │ │
│ └── Burndown: 25 blockers, 55 with dates, projected clear Q3 2026 │ └── Burndown: blockers, with dates, projected clear date
├── SDIT_CISO — 92% — 45 non-compliant ├── SDIT_CISO — 72% — 68 non-compliant
└── ... └── ...
``` ```
--- ---
## How Metrics Are Calculated
### Data Sources
Each vertical's xlsx file contains two types of data:
1. **Summary sheet** — one row per metric per sub-team, with pre-calculated totals (compliant, non-compliant, total, compliance %, target). This is the source of truth for aggregate numbers.
2. **Detail sheets** — one sheet per metric, listing individual non-compliant devices (hostname, IP, device type, team). These feed the device-level drill-down.
### The Double-Counting Problem (and How We Solve It)
The Summary sheet contains **two levels of rows** for each metric:
| Row Type | Example | Purpose |
|---|---|---|
| Sub-team rows | ACCESS-OPS, STEAM, INTELDEV | Individual team breakdown |
| Rollup row | ALL: NTS-AEO | Sum of all sub-teams for that metric |
The rollup row already includes all sub-team totals. If you sum all rows naively, you count every device twice.
**Solution:** All aggregate calculations (stats bar, vertical breakdown, category totals, snapshots) use **only the ALL: rollup rows**. Sub-team rows are stored for drill-down display but never included in totals.
### What Each Number Means
| Metric | Source | Meaning |
|---|---|---|
| **Total Devices** | Sum of `total` from ALL: rows across all metrics for a vertical | Total device-metric pairs evaluated (a device appears once per metric it's measured against) |
| **Compliant** | Sum of `compliant` from ALL: rows | Device-metric pairs that pass the compliance check |
| **Non-Compliant** | Sum of `non_compliant` from ALL: rows | Device-metric pairs that fail |
| **Compliance %** | `compliant / total * 100` | Percentage of device-metric pairs passing |
| **Target %** | Per-metric value from the spreadsheet (e.g., 95%, 80%, 75%) | The threshold set by the compliance program |
| **Blockers** | Non-compliant devices in `compliance_items` with no `resolution_date` | Devices with no committed remediation timeline |
| **In-Progress** | Non-compliant devices with a `resolution_date` set | Devices with a planned fix date |
### Important: "Total Devices" Is Not Unique Devices
A single physical device (hostname) can appear in multiple metrics. For example, one router might be measured against metric 5.5.4i (vulnerability scanning), 7.1.1 (logging), and 2.3.6i (patching). The "Total Devices" count is the sum of all device-metric evaluations, not unique hostnames.
This matches how CyberMetrics reports — each metric has its own scope of applicable devices, and the overall compliance percentage reflects performance across all metrics.
### Per-Metric Compliance Percentage
Each metric row shows its own compliance percentage, which comes directly from the Summary sheet's "Current Compliance" column. This is a decimal between 0 and 1 (displayed as 0100% in the UI). The target is also per-metric — some metrics have a 95% target, others 80% or 75%, depending on the compliance program's priorities.
### Category Aggregation
Metrics are grouped into categories (Logging & Monitoring, Vulnerability Management, Access & MFA, Endpoint Protection, etc.) based on a static mapping in `compliance_config.json`. The category cards in the drill-down view show the aggregate compliance % across all metrics in that category, using only rollup rows.
---
## Sub-Team Drill-Down
### How It Works
When you click into a vertical (e.g., NTS_AEO), the metrics table shows the **rollup totals** by default — one row per metric with the ALL: numbers. Two mechanisms expose sub-team data:
**1. Expand/Collapse (▸ arrow)**
Click the arrow on any metric row to reveal sub-team rows inline beneath it. Each sub-team row shows that team's compliant/non-compliant/total/% for that specific metric. The sub-team rows are visually indented and teal-highlighted.
This is useful for: "Which team is dragging down metric 5.5.4i?"
**2. Team Filter Buttons**
A row of filter buttons appears above the metrics table showing all teams in that vertical (e.g., ACCESS-ENG, ACCESS-OPS, INTELDEV, STEAM). Click one to filter the entire table to show only that team's numbers per metric. The "All (Rollup)" button returns to the aggregated view.
This is useful for: "Show me STEAM's compliance across all metrics."
### What "(Other)" Means
Some metrics have a team value of `(Other)` in the Summary sheet. This represents devices that don't map to a named sub-team. These are included in the ALL: rollup total but are not shown as a separate sub-team in the UI — they're noise for the compliance team's purposes.
### Device-Level Drill-Down
Clicking a metric ID navigates to the device list — individual non-compliant hostnames for that vertical + metric combination. This data comes from the detail sheets (not the Summary sheet) and shows:
- Hostname, IP address, device type, team
- Seen count (how many consecutive uploads this device has been non-compliant)
- First seen / last seen dates
- Resolution date (if set)
- Remediation plan (if documented)
---
## Burndown Forecast ## Burndown Forecast
The burndown forecast answers: "When will this vertical reach compliance?" The burndown forecast answers: "When will this vertical reach compliance?"