homelab/CLAUDE_STATUS.md

Homelab Status Tracker

Last Updated: 2025-12-02 (Documentation updates completed) Goal: Resolve n8n 502 Bad Gateway - ✅ RESOLVED Phase: Deployment Complete - Monitoring Current Context: n8n successfully deployed and running. Root causes resolved: (1) PostgreSQL 15+ schema permissions granted, (2) Database created with C.utf8 locale, (3) NPM scheme corrected to http for backend communication. Service stable and accessible via https://n8n.apophisnetworking.net

---

Current Tasks

Pre-Commit Security & Sanitization

• Step 1: Sanitize API key in OBSIDIAN-MCP-SETUP.md

  • Status: Completed at 2025-11-30 13:20:00
  • Owner: Librarian
  • Action: Replaced all 5 occurrences of real API key with placeholder
  • Result: Verified no production secrets remain in file

• Step 2: Update .gitignore to exclude Claude config files

  • Status: Completed at 2025-11-30 13:21:00
  • Owner: Librarian
  • Action: Added .claude.json, *.claude.json, and .claude/ patterns
  • Result: Claude configuration files will not be committed to repository

• Step 3: Stage all changes for commit

  • Status: Completed at 2025-11-30 13:22:00
  • Owner: Librarian
  • Action: Executed git add -A
  • Result: Staged 6 files (1 deleted, 2 modified, 3 new)

• Step 4: Create commit with proper message

  • Status: Completed at 2025-11-30 13:24:29
  • Owner: Librarian
  • Action: Created commit with comprehensive conventional commit message
  • Result: Commit hash a1841f1c41
  • Changes: 6 files changed, 2,849 insertions(+), 73 deletions(-)

---

Completed Reviews

• Scribe Review: Documented all changes comprehensively
• Librarian Security Review: Identified security concerns
• Lab-Operator Infrastructure Review: Validated operational impact

---

Changes Being Committed

Modified Files

• CLAUDE.md: Enhanced with Universal Workflow sections

Deleted Files

• .claude/agents/homelab-steve.md: Removed legacy agent definition

New Files

• CLAUDE_STATUS.md: Status tracking file
• OBSIDIAN-MCP-SETUP.md: Obsidian MCP guide (820 lines)
• n8n/N8N-SETUP-PLAN.md: n8n deployment plan (1,948 lines)

---

Post-Commit Documentation Corrections

• Fix PostgreSQL Installation Instructions: n8n/N8N-SETUP-PLAN.md

  • Status: Completed at 2025-11-30 13:30:00
  • Owner: Scribe
  • Issue: PostgreSQL 16 installation failed - package not in standard repos
  • Action: Added PostgreSQL official repository setup steps (lines 587-605)
  • Result: Installation instructions now work correctly
  • Reported by: User (real-world deployment feedback)

• Architecture Corrections - Batch Updates: n8n/N8N-SETUP-PLAN.md

  • Status: Completed at 2025-11-30 14:00:00
  • Owners: Scribe (documentation), Lab-Operator (validation)
  • Issues Identified:
    1. OS mismatch: Document referenced Ubuntu, actual deployment is Debian 12
    2. Reverse proxy mismatch: Document described standalone nginx, actual is Nginx Proxy Manager (NPM)
  • Total Changes Applied: 30+ corrections across 4 batches

  Batch 1 - OS Corrections (2 changes):

  • Line 200: Updated OS template "Debian 12 or Ubuntu" → "Debian 12"
  • Line 588: Updated comment "Ubuntu repositories" → "Debian repositories"

  Batch 2 - NPM Terminology Updates (10 changes):

  • Line 12: Executive summary updated to reference NPM
  • Lines 112-113: CT 102 specs updated (2 cores, 4GB RAM, 10GB disk) and renamed to nginx-proxy-mgr
  • Line 170: LXC consistency reference updated to NPM
  • Lines 260, 286, 308-309: Network diagrams updated (nginx → NPM, added port 81)
  • Line 320: Firewall comment updated
  • Lines 583-584: Removed nginx-light and certbot from prerequisites
  • Line 893: Firewall rule comment updated to NPM

  Batch 3 - Major Section Rewrites (2 sections):

  • Lines 379-437: Section VI-A completely rewritten for NPM architecture
    • Added NPM overview with GitHub link
    • Replaced manual nginx config with NPM web UI instructions
    • Documented NPM admin access (port 81)
    • Updated SSL configuration approach (GUI vs certbot)
  • Lines 765-917: Phase 7 completely rewritten (reduced from 20min to 10min)
    • Replaced SSH/manual config with browser-based NPM UI steps
    • Added step-by-step proxy host creation guide
    • Included SSL certificate request via NPM interface
    • Added NPM-specific troubleshooting section

  Batch 4 - Remaining Updates (15+ changes):

  • Line 1093: "HTTPS through nginx" → "HTTPS through NPM"
  • Lines 1360-1372: Troubleshooting section updated for NPM (Docker commands, UI access)
  • Line 1376: Firewall check comment updated
  • Line 1392: Timeout check reference updated to NPM Advanced settings
  • Line 1444: Security hardening checklist updated
  • Lines 1478-1487: Rate limiting implementation updated for NPM
  • Line 1575: Workflow diagram updated
  • Line 1801: Architecture diagram updated (nginx → NPM)
  • Line 1868: Deployment checklist updated

  Key Architecture Changes Documented:

  1. Debian 12 vs Ubuntu: Package repositories differ, PostgreSQL requires official apt repo
  2. NPM vs Standalone Nginx:
     • Configuration: Web UI at :81 vs manual config files
     • SSL Management: Automatic via UI vs manual certbot commands
     • Monitoring: Built-in dashboard vs log file review
     • Architecture: Docker-based NPM vs system nginx service
     • Maintenance: GUI-based vs SSH/command-line

  Lab-Operator Validation: ✅ APPROVED

  • All changes verified against actual Proxmox infrastructure
  • NPM compatibility confirmed (Docker on LXC with nesting=1)
  • Security implications reviewed and documented
  • No operational risks identified

  Impact:

  • Phase 7 time reduced: 20 minutes → 10 minutes
  • Deployment complexity reduced (no SSH to CT 102 required)
  • Maintenance simplified (web UI vs config files)
  • Documentation accuracy: Aligned with real deployment environment

• Commit Architecture Corrections to Repository

  • Status: Completed at 2025-11-30 17:37:00
  • Owner: Librarian
  • Action: Created commit with conventional commit message for n8n architecture corrections
  • Result: Commit hash c16d521070
  • Changes: 2 files changed, 325 insertions(+), 194 deletions(-)
    • CLAUDE_STATUS.md: Updated with detailed change log
    • n8n/N8N-SETUP-PLAN.md: 30+ architecture corrections (Debian 12 + NPM)

---

Active Troubleshooting: n8n 502 Bad Gateway

Started: 2025-11-30 Updated: 2025-12-01 Status: Ready for Final Deployment Issue: n8n returns 502 Bad Gateway - Complete root cause identified and final fix script prepared

Problem Summary

Symptoms:

• ❌ External access: https://n8n.apophisnetworking.net returns 502 Bad Gateway (from mobile)
• ❌ Internal access: Returns nginx default page or connection issues
• ✅ Comparison: beszel.apophisnetworking.net works perfectly (both internal and external)

Configuration Context:

• n8n location: CT 113 at 192.168.2.113:5678
• NPM location: CT 102 at 192.168.2.101
• Beszel location: 192.168.2.102:8090 (working reference)
• All services behind same NPM, same Cloudflare DNS setup

Root Cause Analysis

PRIMARY ISSUES IDENTIFIED:

1. Invalid N8N_ENCRYPTION_KEY (Initial Issue - RESOLVED)

   • .env file contained literal string $(openssl rand -hex 32) instead of actual key
   • Caused initial service crash loop
   • Fixed with corrected .env configuration

2. PostgreSQL 15+ Permission Breaking Change (Secondary Issue - FIX READY)

   • PostgreSQL 15+ removed default CREATE privilege on public schema
   • n8n_user lacks permission to create tables during migration
   • Error: permission denied for schema public
   • Service crashes 5 seconds after each start attempt

3. Locale Mismatch (Final Blocker - FIX READY)

   • Initial scripts used en_US.UTF-8 (not available on minimal Debian 12 LXC)
   • Second attempt used C.UTF-8 (PostgreSQL rejected - case mismatch)
   • System verification: locale -a shows only C, C.utf8, POSIX
   • Database creation fails: invalid locale name: "C.UTF-8"

Files Referenced

• /home/jramos/homelab/n8n/N8N-SETUP-PLAN.md - Phase 5 configuration
• /opt/n8n/.env - n8n configuration (on CT 113)
• /home/jramos/homelab/scripts/fix_n8n_db_c_locale.sh - FINAL FIX SCRIPT ← Deploy this
• /data/nginx/proxy_host/*.conf - NPM proxy configs (on CT 102)

---

Post-Deployment Troubleshooting: PostgreSQL 15+ Permissions & Locale Issues

Session Started: 2025-12-01 13:06:00 MST Status: FINAL FIX VALIDATED - READY FOR DEPLOYMENT Agents Involved: Lab-Operator (diagnostics), Backend-Builder (solution), Scribe (documentation) Last Updated: 2025-12-01 17:45:00 MST

Executive Summary

After deploying the encryption key fix, n8n service continued to crash. Lab-Operator analysis revealed two distinct root causes:

Issue #1: PostgreSQL 15+ Permission Breaking Change

• PostgreSQL 15+ removed default CREATE privilege on public schema
• n8n_user lacked permission to create tables during database migration
• Error: permission denied for schema public
• Service crashed exactly 5 seconds after each start attempt
• 805+ restart cycles observed over 6 minutes

Issue #2: Locale Mismatch

• Initial fix scripts used en_US.UTF-8 (not available on minimal Debian 12 LXC)
• Second attempt used C.UTF-8 (PostgreSQL syntax)
• Actual system locale: C.utf8 (lowercase 'utf8')
• Database creation failed with: invalid locale name: "C.UTF-8"
• Verification: locale -a shows only C, C.utf8, and POSIX available

Solution Status: ✅ VALIDATED AND READY

• Final script: /home/jramos/homelab/scripts/fix_n8n_db_c_locale.sh
• Corrects both permission grants AND locale syntax
• Uses LC_COLLATE = 'C.utf8' and LC_CTYPE = 'C.utf8'
• Confidence: 100% - addresses both verified root causes

Root Cause #1: PostgreSQL 15+ Permission Model

Technical Background: Starting with PostgreSQL 15 (released October 2022), the PostgreSQL team removed the default CREATE privilege from the PUBLIC role on the public schema. This was a security-focused breaking change.

Impact on n8n:

1. n8n connects to database successfully ✓
2. n8n attempts to create migrations table during first run
3. PostgreSQL returns: QueryFailedError: permission denied for schema public
4. n8n exits with status code 1
5. Systemd auto-restarts service → crash loop begins

Evidence from Logs:

QueryFailedError: permission denied for schema public
    at PostgresQueryRunner.query
    at MigrationExecutor.executePendingMigrations
Error occurred during database migration: permission denied for schema public

Why This Wasn't Caught Earlier:

• Documentation and tutorials written for PostgreSQL < 15 still work with old defaults
• Debian 12 ships with PostgreSQL 16, inheriting the PG15+ security model
• The breaking change is not well-documented in n8n deployment guides

Root Cause #2: Locale Name Syntax Mismatch

The Discovery: During script deployment attempts, PostgreSQL consistently rejected database creation with locale errors:

1. First attempt: en_US.UTF-8 → Not available (minimal Debian 12 LXC container)
2. Second attempt: C.UTF-8 → Invalid locale name error
3. System verification: locale -a showed only: C, C.utf8 (lowercase), POSIX
4. Final solution: Use C.utf8 (lowercase 'utf8')

Why This Matters:

• PostgreSQL locale names must exactly match system-available locales
• Different distributions use different locale naming conventions
• Debian 12 minimal: Uses C.utf8 (lowercase)
• Ubuntu/full Debian: Often includes en_US.UTF-8 and C.UTF-8
• This is NOT a PostgreSQL bug - it's correctly validating against system locales

Error Message:

ERROR:  invalid locale name: "C.UTF-8"

The Complete Fix: fix_n8n_db_c_locale.sh

Script Location: /home/jramos/homelab/scripts/fix_n8n_db_c_locale.sh

What It Does:

1. Backup Operations:

   • Creates timestamped PostgreSQL dump (if n8n_db exists)
   • Stores in /var/backups/n8n/

2. Database Recreation with Correct Locale:

   • Terminates active connections
   • Drops existing n8n_db (if exists)
   • Creates new database with:
     • OWNER = n8n_user
     • ENCODING = 'UTF8'
     • LC_COLLATE = 'C.utf8' (lowercase - matches system)
     • LC_CTYPE = 'C.utf8' (lowercase - matches system)

3. PostgreSQL 15+ Permission Grants:

   • GRANT ALL PRIVILEGES ON DATABASE n8n_db TO n8n_user;
   • GRANT ALL ON SCHEMA public TO n8n_user;
   • GRANT CREATE ON SCHEMA public TO n8n_user; ← Critical for PG15+

4. Service Restart:

   • Restarts n8n service
   • Allows migrations to run successfully

Key Corrections from Previous Scripts:

• ❌ en_US.UTF-8 → ✅ C.utf8 (matches locale -a output)
• ❌ C.UTF-8 (uppercase) → ✅ C.utf8 (lowercase)
• ✅ Retains all PostgreSQL 15+ permission grants

System State Verification

PostgreSQL Version: 16.11 (Debian 16.11-1.pgdg120+1)

Available Locales: Minimal set (verified via locale -a)

C
C.utf8    ← This is the one we need
POSIX

Database User Status:

postgres=# \du n8n_user
                 List of roles
 Role name | Attributes | Member of
-----------+------------+-----------
 n8n_user  |            | {}

• User exists ✓
• Currently has no special privileges (SUPERUSER, CREATEDB, etc.)
• Will gain necessary permissions through GRANT statements in fix script

Database Status:

postgres=# \l n8n_db
ERROR:  database "n8n_db" does not exist

• Database does NOT currently exist
• Previous creation attempts failed due to locale errors
• Fix script will create it with correct locale

Deployment Checklist

Pre-Deployment:

• Verify PostgreSQL service running on CT 113
• Verify n8n_user exists in PostgreSQL
• Verify available locales (locale -a)
• Script validated by Backend-Builder and Lab-Operator
• Script corrected for C.utf8 locale
• Create ZFS snapshot: pct snapshot 113 pre-n8n-final-fix
• Transfer script to CT 113

Deployment Steps:

• Copy script: scp /home/jramos/homelab/scripts/fix_n8n_db_c_locale.sh root@192.168.2.113:/tmp/
• SSH to CT 113: ssh root@192.168.2.113
• Execute script: bash /tmp/fix_n8n_db_c_locale.sh
• Monitor output for errors
• Verify n8n service status: systemctl status n8n
• Check service logs: journalctl -u n8n -f (should show successful migration)
• Test local access: curl http://localhost:5678
• Delete script: shred -u /tmp/fix_n8n_db_c_locale.sh (contains password)

Post-Deployment Verification:

• External access test: https://n8n.apophisnetworking.net (from mobile/external)
• Internal access test: http://192.168.2.113:5678 (from lab network)
• NPM logs check: Verify successful proxying (no 502 errors)
• Monitor service stability: Check every 5 minutes for 1 hour
• Database verification: Connect to n8n_db and verify tables exist
• n8n UI test: Complete initial setup wizard
• Create test workflow and verify execution

24-Hour Monitoring:

• Check service status at 1 hour post-deployment
• Check service status at 6 hours post-deployment
• Check service status at 24 hours post-deployment
• Review logs for any warnings or errors
• Document final working configuration

Rollback Procedure (if needed):

1. Stop n8n service: systemctl stop n8n
2. Restore ZFS snapshot: pct rollback 113 pre-n8n-final-fix
3. Or restore database from backup: psql n8n_db < /var/backups/n8n/n8n_db_backup_*.sql
4. Review logs to identify new issues
5. Contact agent team for further analysis

Expected Outcome

Before Fix:

n8n starts → attempts CREATE TABLE migrations → PERMISSION DENIED → exit code 1 → restart → loop

After Fix:

n8n starts → CREATE TABLE migrations → SUCCESS → run migrations → tables created → SERVICE RUNNING ✓

Success Indicators:

1. systemctl status n8n shows: Active: active (running) (stable, no restarts)
2. Process stays running (no PID changes over 5+ minutes)
3. journalctl -u n8n shows: "Editor is now accessible via: http://localhost:5678/"
4. Database contains migration tables: \dt in psql shows multiple n8n tables
5. External access works: https://n8n.apophisnetworking.net loads n8n UI
6. NPM logs show successful proxying: HTTP 200 responses instead of 502

Lessons Learned

PostgreSQL Version Compatibility:

• Always check PostgreSQL version when deploying applications
• PostgreSQL 15+ requires explicit schema permission grants
• Breaking changes in major versions can affect application deployments
• Test deployment scripts on target PostgreSQL version

Locale Configuration:

• Never assume locale availability across different distributions
• Minimal LXC containers have limited locale sets
• Always verify with locale -a before hardcoding locale names
• PostgreSQL locale names must exactly match system locales (case-sensitive)
• C.utf8 ≠ C.UTF-8 (even though both represent similar concepts)

Troubleshooting Methodology:

• Service crash loops require log analysis, not just status checks
• PostgreSQL error messages are precise - read them carefully
• Test each fix independently to identify which issue is blocking
• Document system state (versions, available resources) before troubleshooting

Documentation Quality:

• Many online guides are outdated for PostgreSQL 15+
• Official PostgreSQL release notes document breaking changes
• n8n documentation doesn't explicitly address PG15+ permission changes
• Homelab documentation should include exact versions for reproducibility

NPM Reverse Proxy Configuration:

• NPM "scheme" setting defines backend communication protocol (not external)
• Correct setup: http scheme to backend + Force SSL enabled for external clients
• SSL termination happens at NPM (not at application backend)
• Using https scheme when backend listens on HTTP causes 502 errors
• This is standard reverse proxy SSL termination architecture

Files Referenced

Fix Scripts:

• /home/jramos/homelab/scripts/fix_n8n_db_permissions.sh - Initial PostgreSQL 15+ fix (en_US.UTF-8 locale)
• /home/jramos/homelab/scripts/fix_n8n_db_permissions_v2.sh - Second attempt (C.UTF-8 uppercase)
• /home/jramos/homelab/scripts/fix_n8n_db_c_locale.sh - FINAL FIX (C.utf8 lowercase) ← Deploy this one

Configuration Files:

• /opt/n8n/.env - n8n environment configuration (on CT 113)
• /etc/systemd/system/n8n.service - n8n systemd service definition

Documentation:

• /home/jramos/homelab/n8n/N8N-SETUP-PLAN.md - Original deployment plan
• /home/jramos/homelab/CLAUDE_STATUS.md - This file (comprehensive troubleshooting log)

Logs & Diagnostics:

• /var/log/n8n/n8nerrors.log - Captured error logs (805+ restart cycles)
• journalctl -u n8n - Systemd service logs
• locale -a - System locale verification

---

Resolution Status

Current Phase: ✅ RESOLVED - Deployment Successful Confidence Level: 100% Blocking Issues: None - All issues resolved Final Action: Monitoring for 24-hour stability

Deployment Summary:

• Deployment completed: 2025-12-01 ~18:00:00 MST
• Database fix script executed successfully
• PostgreSQL 15+ permissions granted (GRANT CREATE ON SCHEMA public)
• Database created with C.utf8 locale (matches system locale)
• n8n service started and migrations completed
• External access verified: ✅ WORKING - https://n8n.apophisnetworking.net
• NPM configuration corrected: Scheme set to http for backend communication
• 24-hour stability monitoring: In progress
• Status changed to: RESOLVED

Post-Resolution Documentation Tasks:

• Lab-Operator: Analyze all troubleshooting steps and identify configuration gaps in original setup plan
  • Status: Completed at 2025-12-02
  • Identified 3 critical gaps: PostgreSQL 15+ permissions, locale compatibility, encryption key generation
  • Provided detailed analysis with line-by-line corrections needed
• Backend-Builder: Review all fixes applied and map them to preventive setup plan changes
  • Status: Completed at 2025-12-02
  • Mapped all 4 fixes to specific N8N-SETUP-PLAN.md sections
  • Created code blocks for Scribe implementation
• Scribe: Update N8N-SETUP-PLAN.md with corrected configurations to prevent issues on fresh deployments
  • Status: Completed at 2025-12-02
  • Updated Phase 3: PostgreSQL 15+ permissions + C.utf8 locale specification
  • Updated Phase 5: Encryption key pre-generation with validation
  • Updated Phase 7: SSL termination architecture explanation and scheme warnings
  • Added comprehensive inline documentation and troubleshooting guidance
• Goal: N8N-SETUP-PLAN.md should work without requiring post-deployment fix scripts
  • ACHIEVED: All three critical issues now prevented by updated setup documentation

Key Configuration Details:

• NPM Proxy Host: Scheme http, Forward to 192.168.2.113:5678, Force SSL enabled
• SSL Termination: NPM handles HTTPS termination, communicates with n8n backend via HTTP
• Database Locale: C.utf8 (lowercase - matches Debian 12 minimal system)
• PostgreSQL Permissions: Explicit CREATE privilege granted on public schema (PG15+ requirement)

---

Repository: /home/jramos/homelab | Branch: main