# Bugfix Requirements Document

## Introduction

Multiple compliance endpoints incorrectly key their queries by `compliance_uploads.id` (or by individual upload row) instead of by `compliance_uploads.report_date`. The compliance pipeline accepts one xlsx file per vertical (e.g., NTS_AEO, SDIT_CISO, TSI), so a single calendar date typically produces several `compliance_uploads` rows. Any query, aggregation, or "pick latest" logic that treats each upload as a distinct date — instead of grouping all uploads sharing a `report_date` — produces duplicated, fragmented, or silently dropped data.

The originally reported defect (GitLab issue #12, reported by nkapur) was the "Active Findings Over Time" chart on the Compliance page showing 3 entries for 5/11 after STEAM uploaded three vertical data sets that day. Investigation found that the same root cause — keying by `upload_id` instead of `report_date` — affects `GET /trends`, `GET /waterfall` (route handler `GET /top-recurring`), `GET /category-trend`, `GET /summary`, and the `compliance_snapshots` block in `persistUpload()`. This spec covers fixes for all five.

## Bug Analysis

### Current Behavior (Defect)

#### /trends (originally reported)

1.1 WHEN multiple compliance uploads exist with the same `report_date` (due to per-vertical uploads) THEN the system returns one trend data point per upload row, producing duplicate x-axis entries on the chart

1.2 WHEN the chart renders multiple entries for the same date THEN the x-axis displays repeated date labels (e.g., three "05/11/25" entries) making the trend line misleading and unreadable

1.3 WHEN per-team counts are computed for duplicate-date uploads THEN the system counts items per individual `upload_id` rather than aggregating across all uploads sharing that date, resulting in fragmented per-team totals

#### /waterfall (route handler `GET /top-recurring`)

1.4 WHEN multiple compliance uploads exist with the same `report_date` THEN the underlying query `SELECT id, report_date, ... FROM compliance_uploads ORDER BY report_date ASC` returns one row per upload and `computeWaterfall()` emits one bar per row, producing multiple bars stacked under the same date label

1.5 WHEN `computeWaterfall()` carries `start` forward across multiple rows that share a `report_date` THEN each per-vertical row's `new_count`/`recurring_count`/`resolved_count` deltas are applied sequentially as if they were separate cycles, so the running `start` and `end` totals for that date are wrong (they reflect the last row's running balance rather than the date-level aggregate)

#### /category-trend

1.6 WHEN multiple compliance uploads exist with the same `report_date` THEN the query grouped by `cu.id, cu.report_date, category` returns one row per (upload, category) pair, producing duplicated stacked bars per date when the chart is keyed on `report_date`

1.7 WHEN per-category counts are surfaced for a date with multiple uploads THEN counts are reported per-vertical instead of aggregated across all verticals sharing that `report_date`, so no row in the response represents the full date-level category total

#### /summary

1.8 WHEN multiple uploads exist for the latest `report_date` THEN the query `WHERE vertical IS NULL ORDER BY id DESC LIMIT 1` (with fallback to `vertical = 'NTS_AEO'`) selects a single upload for that date and discards the `summary_json` of all other verticals, silently dropping their data

1.9 WHEN the summary returned by `/summary` is compared against `/trends`, `/waterfall`, or `/category-trend` for the same latest date THEN the figures do not reconcile, because `/summary` reflects one vertical's upload while the other endpoints aggregate (or duplicate) across all verticals

#### `compliance_snapshots` creation in `persistUpload()`

1.10 WHEN `persistUpload()` computes per-vertical compliance stats THEN the query filters only `WHERE team IS NOT NULL` and groups by `team`, with no filter or grouping on `vertical`, so item counts pulled from `compliance_items` are aggregated across every vertical present in the table

1.11 WHEN the resulting per-team totals are written into `compliance_snapshots` for a single vertical's upload THEN the `total_devices`, `compliant`, and `non_compliant` columns reflect cross-vertical totals rather than the snapshotted vertical, corrupting the monthly snapshot record

### Expected Behavior (Correct)

#### /trends (originally reported)

2.1 WHEN multiple compliance uploads exist with the same `report_date` THEN the system SHALL aggregate their counts (new_count, recurring_count, resolved_count, total_active) into a single trend data point per unique date

2.2 WHEN the chart renders trend data THEN each unique `report_date` SHALL appear exactly once on the x-axis regardless of how many upload records exist for that date

2.3 WHEN per-team counts are computed for a date with multiple uploads THEN the system SHALL aggregate team item counts across all uploads sharing that `report_date`, producing a single per-team total per date

#### /waterfall (route handler `GET /top-recurring`)

2.4 WHEN multiple compliance uploads exist with the same `report_date` THEN the system SHALL aggregate `new_count`, `recurring_count`, and `resolved_count` across all uploads sharing that `report_date` into a single per-date row before passing rows to `computeWaterfall()`

2.5 WHEN `computeWaterfall()` consumes the aggregated rows THEN it SHALL emit exactly one waterfall entry per unique `report_date` and the running `start`/`end` totals SHALL advance using each date's date-level aggregate deltas (not per-upload deltas)

#### /category-trend

2.6 WHEN multiple compliance uploads exist with the same `report_date` THEN the query SHALL group by `cu.report_date, category` (without `cu.id` in the GROUP BY) and `SUM`/`COUNT` items across all uploads sharing the date, producing one row per (date, category) pair

2.7 WHEN per-category counts are returned for a date with multiple uploads THEN the `count` field SHALL be the sum of items in that category across every upload for that `report_date`

#### /summary

2.8 WHEN multiple uploads exist for the latest `report_date` THEN the system SHALL either (a) merge the `summary_json` of all uploads sharing that date into a single combined summary response, or (b) return a documented, well-defined selection (e.g., a named "primary" vertical) along with metadata indicating which uploads were considered, rather than silently picking one by `ORDER BY id DESC LIMIT 1`

2.9 WHEN the response is constructed for a date with multiple uploads THEN the `upload` field SHALL identify the set of uploads that contributed to the response (or, if a single representative is returned, the response SHALL include a flag/field indicating other uploads exist for the same date that were not merged)

#### `compliance_snapshots` creation in `persistUpload()`

2.10 WHEN `persistUpload()` computes per-vertical compliance stats THEN the query SHALL filter `compliance_items` by the `vertical` of the upload being persisted (in addition to `team IS NOT NULL`) and group by `vertical, team`, so each snapshot row reflects only the items belonging to that vertical

2.11 WHEN snapshots are written into `compliance_snapshots` THEN the `total_devices`, `compliant`, and `non_compliant` values SHALL match the items belonging to the snapshotted vertical only and SHALL NOT be inflated by items from other verticals

### Unchanged Behavior (Regression Prevention)

3.1 WHEN only one compliance upload exists per `report_date` (single-file upload workflow) THEN the system SHALL CONTINUE TO return that date's counts unchanged as a single trend data point

3.2 WHEN the chart displays trend data THEN the system SHALL CONTINUE TO show all existing data fields (new_count, recurring_count, resolved_count, total_active, per-team breakdowns) with correct values

3.3 WHEN no compliance uploads exist THEN the system SHALL CONTINUE TO return an empty trends array and the chart SHALL CONTINUE TO display the "no data" state

3.4 WHEN only one compliance upload exists per `report_date` THEN `GET /waterfall` SHALL CONTINUE TO emit one entry per date with the same `start`, `new_count`, `recurring_count`, `resolved_count`, and `end` fields and the same running-total semantics as before

3.5 WHEN only one compliance upload exists per `report_date` THEN `GET /category-trend` SHALL CONTINUE TO return one row per (date, category) pair with the same `report_date`, `category`, and `count` field shape as before

3.6 WHEN only one compliance upload exists for the latest `report_date` THEN `GET /summary` SHALL CONTINUE TO return the same `entries`, `overall_scores`, and `upload` shape as before, including the existing `vertical IS NULL` → `vertical = 'NTS_AEO'` fallback for selecting which upload's summary to surface

3.7 WHEN `/summary` is called with a `team` query parameter THEN the system SHALL CONTINUE TO filter `entries` by the requested team and SHALL CONTINUE TO reject teams not in `ALLOWED_TEAMS` with HTTP 400

3.8 WHEN `persistUpload()` writes a snapshot for a vertical that is the only vertical present in `compliance_items` for that month THEN the snapshot row's `total_devices`, `compliant`, `non_compliant`, and `compliance_pct` SHALL CONTINUE TO be identical to the pre-fix values (no behavioural change in the single-vertical case)

3.9 WHEN `persistUpload()` encounters an error during snapshot creation THEN the system SHALL CONTINUE TO log the error and complete the upload commit successfully (snapshot creation remains non-critical)

3.10 WHEN any of these endpoints are queried with no matching data (no uploads, no items for a vertical, no items in a category) THEN the system SHALL CONTINUE TO return the existing empty-state response shapes