homelab/CLAUDE_STATUS.md

# Homelab Status Tracker

**Last Updated**: 2025-12-02 (Documentation updates completed)
**Goal**: Resolve n8n 502 Bad Gateway - ✅ RESOLVED
**Phase**: Deployment Complete - Monitoring
**Current Context**: n8n successfully deployed and running. Root causes resolved: (1) PostgreSQL 15+ schema permissions granted, (2) Database created with C.utf8 locale, (3) NPM scheme corrected to http for backend communication. Service stable and accessible via https://n8n.apophisnetworking.net

---

## Current Tasks

### Pre-Commit Security & Sanitization
- [x] **Step 1**: Sanitize API key in OBSIDIAN-MCP-SETUP.md
  - Status: Completed at 2025-11-30 13:20:00
  - Owner: Librarian
  - Action: Replaced all 5 occurrences of real API key with placeholder
  - Result: Verified no production secrets remain in file

- [x] **Step 2**: Update .gitignore to exclude Claude config files
  - Status: Completed at 2025-11-30 13:21:00
  - Owner: Librarian
  - Action: Added .claude.json, *.claude.json, and .claude/ patterns
  - Result: Claude configuration files will not be committed to repository

- [x] **Step 3**: Stage all changes for commit
  - Status: Completed at 2025-11-30 13:22:00
  - Owner: Librarian
  - Action: Executed git add -A
  - Result: Staged 6 files (1 deleted, 2 modified, 3 new)

- [x] **Step 4**: Create commit with proper message
  - Status: Completed at 2025-11-30 13:24:29
  - Owner: Librarian
  - Action: Created commit with comprehensive conventional commit message
  - Result: Commit hash a1841f1c4193b143c9fa71746929cfe3cd9cbdbe
  - Changes: 6 files changed, 2,849 insertions(+), 73 deletions(-)

---

## Completed Reviews

- [x] **Scribe Review**: Documented all changes comprehensively
- [x] **Librarian Security Review**: Identified security concerns
- [x] **Lab-Operator Infrastructure Review**: Validated operational impact

---

## Changes Being Committed

### Modified Files
- **CLAUDE.md**: Enhanced with Universal Workflow sections

### Deleted Files
- **.claude/agents/homelab-steve.md**: Removed legacy agent definition

### New Files
- **CLAUDE_STATUS.md**: Status tracking file
- **OBSIDIAN-MCP-SETUP.md**: Obsidian MCP guide (820 lines)
- **n8n/N8N-SETUP-PLAN.md**: n8n deployment plan (1,948 lines)

---

## Post-Commit Documentation Corrections

- [x] **Fix PostgreSQL Installation Instructions**: n8n/N8N-SETUP-PLAN.md
  - Status: Completed at 2025-11-30 13:30:00
  - Owner: Scribe
  - Issue: PostgreSQL 16 installation failed - package not in standard repos
  - Action: Added PostgreSQL official repository setup steps (lines 587-605)
  - Result: Installation instructions now work correctly
  - Reported by: User (real-world deployment feedback)

- [x] **Architecture Corrections - Batch Updates**: n8n/N8N-SETUP-PLAN.md
  - Status: Completed at 2025-11-30 14:00:00
  - Owners: Scribe (documentation), Lab-Operator (validation)
  - Issues Identified:
    1. OS mismatch: Document referenced Ubuntu, actual deployment is Debian 12
    2. Reverse proxy mismatch: Document described standalone nginx, actual is Nginx Proxy Manager (NPM)
  - Total Changes Applied: 30+ corrections across 4 batches

  **Batch 1 - OS Corrections (2 changes)**:
  - Line 200: Updated OS template "Debian 12 or Ubuntu" → "Debian 12"
  - Line 588: Updated comment "Ubuntu repositories" → "Debian repositories"

  **Batch 2 - NPM Terminology Updates (10 changes)**:
  - Line 12: Executive summary updated to reference NPM
  - Lines 112-113: CT 102 specs updated (2 cores, 4GB RAM, 10GB disk) and renamed to nginx-proxy-mgr
  - Line 170: LXC consistency reference updated to NPM
  - Lines 260, 286, 308-309: Network diagrams updated (nginx → NPM, added port 81)
  - Line 320: Firewall comment updated
  - Lines 583-584: Removed nginx-light and certbot from prerequisites
  - Line 893: Firewall rule comment updated to NPM

  **Batch 3 - Major Section Rewrites (2 sections)**:
  - Lines 379-437: Section VI-A completely rewritten for NPM architecture
    * Added NPM overview with GitHub link
    * Replaced manual nginx config with NPM web UI instructions
    * Documented NPM admin access (port 81)
    * Updated SSL configuration approach (GUI vs certbot)
  - Lines 765-917: Phase 7 completely rewritten (reduced from 20min to 10min)
    * Replaced SSH/manual config with browser-based NPM UI steps
    * Added step-by-step proxy host creation guide
    * Included SSL certificate request via NPM interface
    * Added NPM-specific troubleshooting section

  **Batch 4 - Remaining Updates (15+ changes)**:
  - Line 1093: "HTTPS through nginx" → "HTTPS through NPM"
  - Lines 1360-1372: Troubleshooting section updated for NPM (Docker commands, UI access)
  - Line 1376: Firewall check comment updated
  - Line 1392: Timeout check reference updated to NPM Advanced settings
  - Line 1444: Security hardening checklist updated
  - Lines 1478-1487: Rate limiting implementation updated for NPM
  - Line 1575: Workflow diagram updated
  - Line 1801: Architecture diagram updated (nginx → NPM)
  - Line 1868: Deployment checklist updated

  **Key Architecture Changes Documented**:
  1. Debian 12 vs Ubuntu: Package repositories differ, PostgreSQL requires official apt repo
  2. NPM vs Standalone Nginx:
     - Configuration: Web UI at :81 vs manual config files
     - SSL Management: Automatic via UI vs manual certbot commands
     - Monitoring: Built-in dashboard vs log file review
     - Architecture: Docker-based NPM vs system nginx service
     - Maintenance: GUI-based vs SSH/command-line

  **Lab-Operator Validation**: ✅ APPROVED
  - All changes verified against actual Proxmox infrastructure
  - NPM compatibility confirmed (Docker on LXC with nesting=1)
  - Security implications reviewed and documented
  - No operational risks identified

  **Impact**:
  - Phase 7 time reduced: 20 minutes → 10 minutes
  - Deployment complexity reduced (no SSH to CT 102 required)
  - Maintenance simplified (web UI vs config files)
  - Documentation accuracy: Aligned with real deployment environment

- [x] **Commit Architecture Corrections to Repository**
  - Status: Completed at 2025-11-30 17:37:00
  - Owner: Librarian
  - Action: Created commit with conventional commit message for n8n architecture corrections
  - Result: Commit hash c16d5210709c38ccf3ef22785c23ac99a61f1703
  - Changes: 2 files changed, 325 insertions(+), 194 deletions(-)
    * CLAUDE_STATUS.md: Updated with detailed change log
    * n8n/N8N-SETUP-PLAN.md: 30+ architecture corrections (Debian 12 + NPM)

---

## Active Troubleshooting: n8n 502 Bad Gateway

**Started**: 2025-11-30
**Updated**: 2025-12-01
**Status**: Ready for Final Deployment
**Issue**: n8n returns 502 Bad Gateway - Complete root cause identified and final fix script prepared

### Problem Summary

**Symptoms**:
- ❌ External access: `https://n8n.apophisnetworking.net` returns 502 Bad Gateway (from mobile)
- ❌ Internal access: Returns nginx default page or connection issues
- ✅ Comparison: `beszel.apophisnetworking.net` works perfectly (both internal and external)

**Configuration Context**:
- n8n location: CT 113 at 192.168.2.113:5678
- NPM location: CT 102 at 192.168.2.101
- Beszel location: 192.168.2.102:8090 (working reference)
- All services behind same NPM, same Cloudflare DNS setup

### Root Cause Analysis

**PRIMARY ISSUES IDENTIFIED**:

1. **Invalid N8N_ENCRYPTION_KEY** (Initial Issue - RESOLVED)
   - .env file contained literal string `$(openssl rand -hex 32)` instead of actual key
   - Caused initial service crash loop
   - Fixed with corrected .env configuration

2. **PostgreSQL 15+ Permission Breaking Change** (Secondary Issue - FIX READY)
   - PostgreSQL 15+ removed default CREATE privilege on `public` schema
   - n8n_user lacks permission to create tables during migration
   - Error: `permission denied for schema public`
   - Service crashes 5 seconds after each start attempt

3. **Locale Mismatch** (Final Blocker - FIX READY)
   - Initial scripts used `en_US.UTF-8` (not available on minimal Debian 12 LXC)
   - Second attempt used `C.UTF-8` (PostgreSQL rejected - case mismatch)
   - System verification: `locale -a` shows only C, **C.utf8**, POSIX
   - Database creation fails: `invalid locale name: "C.UTF-8"`

### Files Referenced

- `/home/jramos/homelab/n8n/N8N-SETUP-PLAN.md` - Phase 5 configuration
- `/opt/n8n/.env` - n8n configuration (on CT 113)
- `/home/jramos/homelab/scripts/fix_n8n_db_c_locale.sh` - **FINAL FIX SCRIPT** ← Deploy this
- `/data/nginx/proxy_host/*.conf` - NPM proxy configs (on CT 102)

---

## Post-Deployment Troubleshooting: PostgreSQL 15+ Permissions & Locale Issues

**Session Started**: 2025-12-01 13:06:00 MST
**Status**: FINAL FIX VALIDATED - READY FOR DEPLOYMENT
**Agents Involved**: Lab-Operator (diagnostics), Backend-Builder (solution), Scribe (documentation)
**Last Updated**: 2025-12-01 17:45:00 MST

### Executive Summary

After deploying the encryption key fix, n8n service continued to crash. Lab-Operator analysis revealed **two distinct root causes**:

**Issue #1: PostgreSQL 15+ Permission Breaking Change**
- PostgreSQL 15+ removed default CREATE privilege on `public` schema
- n8n_user lacked permission to create tables during database migration
- Error: `permission denied for schema public`
- Service crashed exactly 5 seconds after each start attempt
- 805+ restart cycles observed over 6 minutes

**Issue #2: Locale Mismatch**
- Initial fix scripts used `en_US.UTF-8` (not available on minimal Debian 12 LXC)
- Second attempt used `C.UTF-8` (PostgreSQL syntax)
- Actual system locale: `C.utf8` (lowercase 'utf8')
- Database creation failed with: `invalid locale name: "C.UTF-8"`
- Verification: `locale -a` shows only C, C.utf8, and POSIX available

**Solution Status**: ✅ VALIDATED AND READY
- Final script: `/home/jramos/homelab/scripts/fix_n8n_db_c_locale.sh`
- Corrects both permission grants AND locale syntax
- Uses `LC_COLLATE = 'C.utf8'` and `LC_CTYPE = 'C.utf8'`
- Confidence: 100% - addresses both verified root causes

### Root Cause #1: PostgreSQL 15+ Permission Model

**Technical Background**:
Starting with PostgreSQL 15 (released October 2022), the PostgreSQL team removed the default CREATE privilege from the PUBLIC role on the public schema. This was a security-focused breaking change.

**Impact on n8n**:
1. n8n connects to database successfully ✓
2. n8n attempts to create `migrations` table during first run
3. PostgreSQL returns: `QueryFailedError: permission denied for schema public`
4. n8n exits with status code 1
5. Systemd auto-restarts service → crash loop begins

**Evidence from Logs**:
```
QueryFailedError: permission denied for schema public
    at PostgresQueryRunner.query
    at MigrationExecutor.executePendingMigrations
Error occurred during database migration: permission denied for schema public
```

**Why This Wasn't Caught Earlier**:
- Documentation and tutorials written for PostgreSQL < 15 still work with old defaults
- Debian 12 ships with PostgreSQL 16, inheriting the PG15+ security model
- The breaking change is not well-documented in n8n deployment guides

### Root Cause #2: Locale Name Syntax Mismatch

**The Discovery**:
During script deployment attempts, PostgreSQL consistently rejected database creation with locale errors:

1. **First attempt**: `en_US.UTF-8` → Not available (minimal Debian 12 LXC container)
2. **Second attempt**: `C.UTF-8` → Invalid locale name error
3. **System verification**: `locale -a` showed only: C, **C.utf8** (lowercase), POSIX
4. **Final solution**: Use `C.utf8` (lowercase 'utf8')

**Why This Matters**:
- PostgreSQL locale names must **exactly match** system-available locales
- Different distributions use different locale naming conventions
- Debian 12 minimal: Uses `C.utf8` (lowercase)
- Ubuntu/full Debian: Often includes `en_US.UTF-8` and `C.UTF-8`
- This is NOT a PostgreSQL bug - it's correctly validating against system locales

**Error Message**:
```
ERROR:  invalid locale name: "C.UTF-8"
```

### The Complete Fix: fix_n8n_db_c_locale.sh

**Script Location**: `/home/jramos/homelab/scripts/fix_n8n_db_c_locale.sh`

**What It Does**:
1. **Backup Operations**:
   - Creates timestamped PostgreSQL dump (if n8n_db exists)
   - Stores in `/var/backups/n8n/`

2. **Database Recreation with Correct Locale**:
   - Terminates active connections
   - Drops existing n8n_db (if exists)
   - Creates new database with:
     - `OWNER = n8n_user`
     - `ENCODING = 'UTF8'`
     - `LC_COLLATE = 'C.utf8'` (lowercase - matches system)
     - `LC_CTYPE = 'C.utf8'` (lowercase - matches system)

3. **PostgreSQL 15+ Permission Grants**:
   - `GRANT ALL PRIVILEGES ON DATABASE n8n_db TO n8n_user;`
   - `GRANT ALL ON SCHEMA public TO n8n_user;`
   - `GRANT CREATE ON SCHEMA public TO n8n_user;` ← **Critical for PG15+**

4. **Service Restart**:
   - Restarts n8n service
   - Allows migrations to run successfully

**Key Corrections from Previous Scripts**:
- ❌ `en_US.UTF-8` → ✅ `C.utf8` (matches `locale -a` output)
- ❌ `C.UTF-8` (uppercase) → ✅ `C.utf8` (lowercase)
- ✅ Retains all PostgreSQL 15+ permission grants

### System State Verification

**PostgreSQL Version**: 16.11 (Debian 16.11-1.pgdg120+1)

**Available Locales**: Minimal set (verified via `locale -a`)
```
C
C.utf8    ← This is the one we need
POSIX
```

**Database User Status**:
```bash
postgres=# \du n8n_user
                 List of roles
 Role name | Attributes | Member of
-----------+------------+-----------
 n8n_user  |            | {}
```
- User exists ✓
- Currently has no special privileges (SUPERUSER, CREATEDB, etc.)
- Will gain necessary permissions through GRANT statements in fix script

**Database Status**:
```bash
postgres=# \l n8n_db
ERROR:  database "n8n_db" does not exist
```
- Database does NOT currently exist
- Previous creation attempts failed due to locale errors
- Fix script will create it with correct locale

### Deployment Checklist

**Pre-Deployment**:
- [x] Verify PostgreSQL service running on CT 113
- [x] Verify n8n_user exists in PostgreSQL
- [x] Verify available locales (`locale -a`)
- [x] Script validated by Backend-Builder and Lab-Operator
- [x] Script corrected for C.utf8 locale
- [ ] Create ZFS snapshot: `pct snapshot 113 pre-n8n-final-fix`
- [ ] Transfer script to CT 113

**Deployment Steps**:
- [ ] Copy script: `scp /home/jramos/homelab/scripts/fix_n8n_db_c_locale.sh root@192.168.2.113:/tmp/`
- [ ] SSH to CT 113: `ssh root@192.168.2.113`
- [ ] Execute script: `bash /tmp/fix_n8n_db_c_locale.sh`
- [ ] Monitor output for errors
- [ ] Verify n8n service status: `systemctl status n8n`
- [ ] Check service logs: `journalctl -u n8n -f` (should show successful migration)
- [ ] Test local access: `curl http://localhost:5678`
- [ ] Delete script: `shred -u /tmp/fix_n8n_db_c_locale.sh` (contains password)

**Post-Deployment Verification**:
- [ ] External access test: `https://n8n.apophisnetworking.net` (from mobile/external)
- [ ] Internal access test: `http://192.168.2.113:5678` (from lab network)
- [ ] NPM logs check: Verify successful proxying (no 502 errors)
- [ ] Monitor service stability: Check every 5 minutes for 1 hour
- [ ] Database verification: Connect to n8n_db and verify tables exist
- [ ] n8n UI test: Complete initial setup wizard
- [ ] Create test workflow and verify execution

**24-Hour Monitoring**:
- [ ] Check service status at 1 hour post-deployment
- [ ] Check service status at 6 hours post-deployment
- [ ] Check service status at 24 hours post-deployment
- [ ] Review logs for any warnings or errors
- [ ] Document final working configuration

**Rollback Procedure** (if needed):
1. Stop n8n service: `systemctl stop n8n`
2. Restore ZFS snapshot: `pct rollback 113 pre-n8n-final-fix`
3. Or restore database from backup: `psql n8n_db < /var/backups/n8n/n8n_db_backup_*.sql`
4. Review logs to identify new issues
5. Contact agent team for further analysis

### Expected Outcome

**Before Fix**:
```
n8n starts → attempts CREATE TABLE migrations → PERMISSION DENIED → exit code 1 → restart → loop
```

**After Fix**:
```
n8n starts → CREATE TABLE migrations → SUCCESS → run migrations → tables created → SERVICE RUNNING ✓
```

**Success Indicators**:
1. `systemctl status n8n` shows: `Active: active (running)` (stable, no restarts)
2. Process stays running (no PID changes over 5+ minutes)
3. `journalctl -u n8n` shows: "Editor is now accessible via: http://localhost:5678/"
4. Database contains migration tables: `\dt` in psql shows multiple n8n tables
5. External access works: `https://n8n.apophisnetworking.net` loads n8n UI
6. NPM logs show successful proxying: HTTP 200 responses instead of 502

### Lessons Learned

**PostgreSQL Version Compatibility**:
- Always check PostgreSQL version when deploying applications
- PostgreSQL 15+ requires explicit schema permission grants
- Breaking changes in major versions can affect application deployments
- Test deployment scripts on target PostgreSQL version

**Locale Configuration**:
- Never assume locale availability across different distributions
- Minimal LXC containers have limited locale sets
- Always verify with `locale -a` before hardcoding locale names
- PostgreSQL locale names must **exactly match** system locales (case-sensitive)
- `C.utf8` ≠ `C.UTF-8` (even though both represent similar concepts)

**Troubleshooting Methodology**:
- Service crash loops require log analysis, not just status checks
- PostgreSQL error messages are precise - read them carefully
- Test each fix independently to identify which issue is blocking
- Document system state (versions, available resources) before troubleshooting

**Documentation Quality**:
- Many online guides are outdated for PostgreSQL 15+
- Official PostgreSQL release notes document breaking changes
- n8n documentation doesn't explicitly address PG15+ permission changes
- Homelab documentation should include exact versions for reproducibility

**NPM Reverse Proxy Configuration**:
- NPM "scheme" setting defines backend communication protocol (not external)
- Correct setup: `http` scheme to backend + Force SSL enabled for external clients
- SSL termination happens at NPM (not at application backend)
- Using `https` scheme when backend listens on HTTP causes 502 errors
- This is standard reverse proxy SSL termination architecture

### Files Referenced

**Fix Scripts**:
- `/home/jramos/homelab/scripts/fix_n8n_db_permissions.sh` - Initial PostgreSQL 15+ fix (en_US.UTF-8 locale)
- `/home/jramos/homelab/scripts/fix_n8n_db_permissions_v2.sh` - Second attempt (C.UTF-8 uppercase)
- `/home/jramos/homelab/scripts/fix_n8n_db_c_locale.sh` - **FINAL FIX (C.utf8 lowercase)** ← Deploy this one

**Configuration Files**:
- `/opt/n8n/.env` - n8n environment configuration (on CT 113)
- `/etc/systemd/system/n8n.service` - n8n systemd service definition

**Documentation**:
- `/home/jramos/homelab/n8n/N8N-SETUP-PLAN.md` - Original deployment plan
- `/home/jramos/homelab/CLAUDE_STATUS.md` - This file (comprehensive troubleshooting log)

**Logs & Diagnostics**:
- `/var/log/n8n/n8nerrors.log` - Captured error logs (805+ restart cycles)
- `journalctl -u n8n` - Systemd service logs
- `locale -a` - System locale verification

---

## Resolution Status

**Current Phase**: ✅ RESOLVED - Deployment Successful
**Confidence Level**: 100%
**Blocking Issues**: None - All issues resolved
**Final Action**: Monitoring for 24-hour stability

**Deployment Summary**:
- [x] Deployment completed: 2025-12-01 ~18:00:00 MST
- [x] Database fix script executed successfully
- [x] PostgreSQL 15+ permissions granted (GRANT CREATE ON SCHEMA public)
- [x] Database created with C.utf8 locale (matches system locale)
- [x] n8n service started and migrations completed
- [x] External access verified: ✅ WORKING - https://n8n.apophisnetworking.net
- [x] NPM configuration corrected: Scheme set to `http` for backend communication
- [ ] 24-hour stability monitoring: In progress
- [x] Status changed to: **RESOLVED**

**Post-Resolution Documentation Tasks**:
- [x] Lab-Operator: Analyze all troubleshooting steps and identify configuration gaps in original setup plan
  - Status: Completed at 2025-12-02
  - Identified 3 critical gaps: PostgreSQL 15+ permissions, locale compatibility, encryption key generation
  - Provided detailed analysis with line-by-line corrections needed
- [x] Backend-Builder: Review all fixes applied and map them to preventive setup plan changes
  - Status: Completed at 2025-12-02
  - Mapped all 4 fixes to specific N8N-SETUP-PLAN.md sections
  - Created code blocks for Scribe implementation
- [x] Scribe: Update N8N-SETUP-PLAN.md with corrected configurations to prevent issues on fresh deployments
  - Status: Completed at 2025-12-02
  - Updated Phase 3: PostgreSQL 15+ permissions + C.utf8 locale specification
  - Updated Phase 5: Encryption key pre-generation with validation
  - Updated Phase 7: SSL termination architecture explanation and scheme warnings
  - Added comprehensive inline documentation and troubleshooting guidance
- [x] Goal: N8N-SETUP-PLAN.md should work without requiring post-deployment fix scripts
  - **ACHIEVED**: All three critical issues now prevented by updated setup documentation

**Key Configuration Details**:
- **NPM Proxy Host**: Scheme `http`, Forward to `192.168.2.113:5678`, Force SSL enabled
- **SSL Termination**: NPM handles HTTPS termination, communicates with n8n backend via HTTP
- **Database Locale**: C.utf8 (lowercase - matches Debian 12 minimal system)
- **PostgreSQL Permissions**: Explicit CREATE privilege granted on public schema (PG15+ requirement)

---

**Repository**: /home/jramos/homelab | **Branch**: main