cve-dashboard/.kiro/specs/compliance-schema-drift-check/tasks.md

Implementation Plan: Compliance Schema Drift Check

Overview

This plan implements schema drift detection in the compliance upload flow. The work proceeds in layers: first extract the shared config file, then build the Python schema extractor, then the Node.js drift checker, then wire it into the preview endpoint, and finally update the upload modal with the drift-review phase. Property-based tests validate the drift checker's correctness properties using fast-check.

Tasks

• 1. Create shared parser configuration file and update Python parser

  • 1.1 Create backend/scripts/compliance_config.json with metric_categories, core_cols, and skip_sheets

    • Extract the exact values from the inline dicts METRIC_CATEGORIES, CORE_COLS, and SKIP_SHEETS in parse_compliance_xlsx.py
    • metric_categories is an object mapping metric ID strings to category strings
    • core_cols is an array of column name strings
    • skip_sheets is an array of sheet name strings
    • Requirements: 1.1, 1.2

  • 1.2 Modify backend/scripts/parse_compliance_xlsx.py to read config from JSON file

    • Remove the inline METRIC_CATEGORIES, CORE_COLS, and SKIP_SHEETS definitions
    • Load them from compliance_config.json (resolved relative to the script's directory)
    • If the config file is missing or contains invalid JSON, print a descriptive error to stderr and exit with non-zero code
    • Ensure CORE_COLS is converted to a set after loading from the JSON array
    • Requirements: 1.3, 1.4

  • * 1.3 Write unit tests for Python parser config loading

    • Test that parser loads config correctly and produces same output as before
    • Test that missing config file causes non-zero exit with descriptive stderr
    • Test that invalid JSON in config file causes non-zero exit with descriptive stderr
    • Requirements: 1.3, 1.4

• 2. Create Python schema extractor script

  • 2.1 Create backend/scripts/extract_xlsx_schema.py

    • Accept file path as CLI argument
    • Use openpyxl in read-only mode to extract: sheet names, first-row column headers per sheet, and unique metric values from the Summary sheet (header at row 4, data from row 5 onward)
    • Output JSON to stdout with shape { "sheets": [{ "name", "columns", "metric_values?" }] }
    • On error, return { "error": "..." } on stdout and exit with non-zero code
    • Reuse the approach from dump_xlsx_schema.py for Summary sheet metric extraction
    • Requirements: 2.1, 2.2, 2.3, 2.4

  • * 2.2 Write unit tests for schema extractor

    • Test that valid xlsx produces correct schema JSON
    • Test that missing file returns error JSON and non-zero exit
    • Test that file with no sheets returns error JSON
    • Requirements: 2.1, 2.4

• 3. Implement Node.js drift checker module

  • 3.1 Create backend/helpers/driftChecker.js with compareSchemaToDrift(schema, config) function

    • Implement breaking rules: missing core column in detail sheets, missing detail sheet (in metric_categories but not skip_sheets and absent from xlsx)
    • Implement silent-miss rules: unknown metric value in Summary not in metric_categories, unknown sheet not in skip_sheets and not in metric_categories
    • Implement cosmetic rules: new column in detail sheet not in core_cols, stale metric in metric_categories not in Summary metric values
    • Each finding has shape { severity, message, value, sheet } (sheet is null when not applicable)
    • Return { breaking: [], silent_miss: [], cosmetic: [] }
    • Export compareSchemaToDrift and a loadConfig(configPath) function that reads and validates compliance_config.json
    • Config loader validates: file exists, parses as JSON, contains metric_categories (object), core_cols (array), skip_sheets (array)
    • Requirements: 3.1, 3.2, 3.3, 4.1, 4.2, 4.3, 5.1, 5.2, 5.3, 1.5, 1.6

  • 3.2 Write property test: Breaking drift completeness (Property 1)

    • Property 1: Breaking drift completeness
    • For any generated schema and config, the set of breaking findings equals exactly the union of missing-core-column findings and missing-detail-sheet findings — no more, no fewer
    • Use fast-check with arbitrary generators for schema and config objects
    • Minimum 100 iterations
    • Validates: Requirements 3.1, 3.2, 3.3

  • * 3.3 Write property test: Silent-miss drift completeness (Property 2)

    • Property 2: Silent-miss drift completeness
    • For any generated schema and config, the set of silent-miss findings equals exactly the union of unknown-metric findings and unknown-sheet findings
    • Use fast-check with arbitrary generators for schema and config objects
    • Minimum 100 iterations
    • Validates: Requirements 4.1, 4.2, 4.3

  • * 3.4 Write property test: Cosmetic drift completeness (Property 3)

    • Property 3: Cosmetic drift completeness
    • For any generated schema and config, the set of cosmetic findings equals exactly the union of new-column findings and stale-metric findings
    • Use fast-check with arbitrary generators for schema and config objects
    • Minimum 100 iterations
    • Validates: Requirements 5.1, 5.2, 5.3

  • * 3.5 Write property test: Drift severity ordering (Property 4)

    • Property 4: Drift severity ordering
    • For any drift report, the grouped output always returns all breaking findings first, then all silent-miss, then all cosmetic
    • Use fast-check to generate mixed drift reports and verify ordering
    • Minimum 100 iterations
    • Validates: Requirements 8.1

  • * 3.6 Write unit tests for drift checker and config loader

    • Test each drift rule individually with hand-crafted schema/config pairs
    • Test config loader with valid file, missing file, invalid JSON, and missing required keys
    • Test that perfectly aligned schema and config produce zero findings
    • Test edge cases: empty metric_categories, empty core_cols, empty skip_sheets
    • Requirements: 3.1, 3.2, 3.3, 4.1, 4.2, 4.3, 5.1, 5.2, 5.3, 1.5, 1.6

• 4. Checkpoint — Verify backend modules

  • Ensure all tests pass, ask the user if questions arise.

• 5. Integrate drift check into preview endpoint

  • 5.1 Modify backend/routes/compliance.js to add drift checking in POST /preview

    • After receiving the uploaded file, spawn extract_xlsx_schema.py as a Python subprocess to get the xlsx schema
    • Read compliance_config.json using the loadConfig() function from driftChecker.js
    • Call compareSchemaToDrift(schema, config) to produce the drift report
    • Proceed with the existing parseXlsx() call and computeDiff()
    • Include drift (DriftReport object) and drift_error (string or null) in the response
    • If schema extraction or drift check throws, set drift: null and drift_error: <message>, then continue with normal flow
    • If config file is missing or invalid, return 500 with descriptive message
    • Preserve all existing response fields: diff, tempFile, filename, report_date, total_items
    • Requirements: 6.1, 6.2, 6.3, 6.4, 9.2

  • * 5.2 Write integration tests for preview endpoint drift behavior

    • Test that preview response includes drift field alongside existing diff data
    • Test that breaking drift still returns 200 (not an error)
    • Test graceful degradation when drift check fails (drift: null, drift_error present)
    • Test 500 response when config file is missing
    • Test that commit endpoint is unchanged and does not reference drift
    • Requirements: 6.1, 6.2, 6.3, 6.4, 9.3

• 6. Update upload modal with drift-review phase

  • 6.1 Modify frontend/src/components/pages/ComplianceUploadModal.js to add drift-review phase

    • Add drift-review phase between uploading and preview in the phase flow
    • After upload response, check if drift is non-null and has findings — if so, enter drift-review; otherwise skip to preview
    • When drift is null (drift check failed), skip drift-review and go straight to preview
    • Display findings grouped by severity: breaking first, then silent-miss, then cosmetic
    • Each severity group has a header with label and count badge
    • Groups with more than 5 findings collapse with a "Show N more" toggle
    • Each finding shows the message and the triggering value
    • Breaking findings: red text (#EF4444), red left-border accent
    • Silent-miss findings: amber text (#F59E0B), amber left-border accent
    • Cosmetic findings: muted text (#94A3B8), subtle left-border accent
    • "Cancel" button returns to idle phase; "Continue to Preview" button advances to diff preview
    • "Continue to Preview" disabled when breaking findings exist, with a message explaining the block
    • When no breaking findings but silent-miss exist, show warning message and enable "Continue to Preview"
    • When only cosmetic findings, enable "Continue to Preview" without warning
    • Follow dashboard dark theme and monospace typography from DESIGN_SYSTEM.md
    • Preserve existing diff preview, commit flow, done, and error phases unchanged
    • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 9.1, 9.4

  • * 6.2 Write unit tests for upload modal drift-review phase

    • Test drift-review phase renders when findings exist
    • Test "Continue to Preview" button disabled when breaking findings present
    • Test "Continue to Preview" button enabled when no breaking findings
    • Test groups collapse at 5+ findings with correct "Show N more" count
    • Test cancel returns to idle phase
    • Test skips drift-review when drift is null or has no findings
    • Requirements: 7.1, 7.5, 7.6, 7.7, 7.8, 8.3

• 7. Final checkpoint — Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

Notes

• Tasks marked with * are optional and can be skipped for faster MVP
• Each task references specific requirements for traceability
• Checkpoints ensure incremental validation
• Property tests (3.2–3.5) validate the four correctness properties from the design using fast-check
• Unit tests validate specific examples and edge cases
• The Python parser modification (1.2) must produce identical output to the current inline-dict version — this is a refactor, not a behavior change
• The commit endpoint (POST /api/compliance/commit) is intentionally unchanged