Commit Graph

10 Commits

Author SHA1 Message Date
0366c63d51 docs(status): update status after repository reorganization commit
Mark all three steps in repository reorganization task as completed:
- Step 1: Librarian review (90 files, identified sensitive file exclusion)
- Step 2: Commit creation (hash 4f69420, 935 insertions, 349 deletions)
- Step 3: Push to Gitea (successful, processed eec4c4b..4f69420)

Task completed at 2025-12-02 21:45 MST.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-02 21:40:24 -07:00
4f69420aaa refactor(repo): reorganize repository structure for improved navigation and maintainability
Implement comprehensive directory reorganization to improve discoverability,
logical grouping, and separation of concerns across documentation, scripts,
and infrastructure snapshots.

Major Changes:

1. Documentation Reorganization:
   - Created start-here-docs/ for onboarding documentation
     * Moved QUICK-START.md, START-HERE.md, GIT-SETUP-GUIDE.md
     * Moved GIT-QUICK-REFERENCE.md, SCRIPT-USAGE.md, SETUP-COMPLETE.md
   - Created troubleshooting/ directory
     * Moved BUGFIX-SUMMARY.md for centralized issue resolution
   - Created mcp/ directory for Model Context Protocol configurations
     * Moved OBSIDIAN-MCP-SETUP.md to mcp/obsidian/

2. Scripts Reorganization:
   - Created scripts/crawlers-exporters/ for infrastructure collection
     * Moved collect*.sh scripts and collection documentation
     * Consolidates Proxmox homelab export tooling
   - Created scripts/fixers/ for operational repair scripts
     * Moved fix_n8n_db_*.sh scripts
     * Isolated scripts with embedded credentials (templates tracked)
   - Created scripts/qol/ for quality-of-life utilities
     * Moved git-aliases.sh and git-first-commit.sh

3. Infrastructure Snapshots:
   - Created disaster-recovery/ for active infrastructure state
     * Moved latest homelab-export-20251202-204939/ snapshot
     * Contains current VM/CT configurations and system state
   - Created archive-homelab/ for historical snapshots
     * Moved homelab-export-*.tar.gz archives
     * Preserves point-in-time backups for reference

4. Agent Definitions:
   - Created sub-agents/ directory
     * Added backend-builder.md (development agent)
     * Added lab-operator.md (infrastructure operations agent)
     * Added librarian.md (git/version control agent)
     * Added scribe.md (documentation agent)

5. Updated INDEX.md:
   - Reflects new directory structure throughout
   - Updated all file path references
   - Enhanced navigation with new sections
   - Added agent roles documentation
   - Updated quick reference commands

6. Security Improvements:
   - Updated .gitignore to match reorganized file locations
   - Corrected path for scripts/fixers/fix_n8n_db_c_locale.sh exclusion
   - Maintained template-based credential management pattern

Infrastructure State Update:
   - Latest snapshot: 2025-12-02 20:49:54
   - Removed: VM 101 (gitlab), CT 112 (Anytype)
   - Added: CT 113 (n8n)
   - Total: 9 VMs, 3 Containers

Impact:
   - Improved repository navigation and discoverability
   - Logical separation of documentation, scripts, and snapshots
   - Clearer onboarding path for new users
   - Enhanced maintainability through organized structure
   - Foundation for multi-agent workflow support

Files changed: 90 files (+935/-349)
   - 3 modified, 14 new files, 73 renames/moves

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-02 21:39:33 -07:00
eec4c4b298 feat(security): implement template-based credential management for sensitive configurations
Introduce template-based approach to prevent credential exposure in version control.
This security enhancement establishes a standard pattern for managing sensitive data
across the homelab repository.

Changes:
- Create services/homepage/services.yaml.template with env var placeholders
  * Replace 7 hardcoded credentials with ${VARIABLE_NAME} format
  * Add OPNSense, Proxmox, Plex, Radarr, Sonarr, Deluge placeholders
- Create scripts/fix_n8n_db_c_locale.sh.template with env var validation
  * Remove hardcoded PostgreSQL password
  * Add N8N_DB_PASSWORD environment variable requirement
  * Include security reminder to shred script after use
- Update .gitignore with explicit exclusions for sensitive files
  * Add services/homepage/services.yaml exclusion
  * Add scripts/fix_n8n_db_c_locale.sh exclusion
- Create services/homepage/README.md with comprehensive setup guide
  * Document environment variable usage (recommended method)
  * Provide API key acquisition instructions for all services
  * Include troubleshooting and security best practices
- Update scripts/README.md with template pattern documentation
  * Add fix_n8n_db_c_locale.sh template usage instructions
  * Create "Template-Based Script Pattern" section
  * Enhance security guidelines with shred usage

Template Pattern Benefits:
- Repository remains credential-free
- Templates serve as documentation
- Easy to recreate configs on new systems
- Supports CI/CD pipelines with secret injection

Security Validation:
- No API keys in staged files (verified)
- No passwords in staged files (verified)
- .gitignore properly excludes sensitive files
- Templates contain clear usage instructions

Related: n8n troubleshooting (CLAUDE_STATUS.md), Docker Compose migration

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-02 19:49:28 -07:00
3eea6b1b4e feat(services): migrate Docker Compose configurations from GitLab
Migrate all docker-compose service configurations from legacy GitLab instance
to homelab repository for consolidation and version control.

Services migrated (6):
- bytestash: Code snippet management
- filebrowser: Web-based file browser
- gitlab: QoL scripts for NPM cert sync
- paperless-ngx: Document management with OCR
- portainer: Docker management UI
- speedtest-tracker: Internet speed test tracker

Changes:
- Add services/ directory with complete configurations
- Update .gitignore with Docker Compose exclusions
- Create comprehensive services/README.md documentation
- Document migration process in CLAUDE_STATUS.md

Migration details:
- Source: https://vulcan.apophisnetworking.net/jramos/homelab.git
- Files migrated: 10 files (6 compose + 3 utilities + 1 README)
- Total size: 84 KB
- Lines added: 836

Security notes:
- .env files excluded from git
- Hardcoded secrets identified in documentation
- Review and update secrets before deployment

Related: GitLab VM 101 decommissioning preparation

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-02 14:34:32 -07:00
779ae2fb24 docs(n8n): enhance setup guide with PostgreSQL 15+ fixes and encryption key validation
Update n8n deployment documentation to prevent three critical issues discovered during troubleshooting:

1. PostgreSQL 15+ Compatibility (Phase 3):
   - Add explicit schema permission grants for public schema
   - Include C.utf8 locale specification for Debian 12 minimal LXC
   - Add permission validation test before proceeding

2. Encryption Key Generation (Phase 5):
   - Add pre-generation validation to prevent literal command strings in .env
   - Include verification steps for 64-character hex key format
   - Document common misconfiguration and remediation steps

3. SSL Termination Architecture (Phase 7):
   - Clarify NPM scheme setting (http backend vs https external)
   - Explain reverse proxy SSL termination pattern
   - Document why https scheme causes 502 Bad Gateway errors

Update CLAUDE_STATUS.md to mark troubleshooting session complete and document deployment success.

These preventive measures ensure clean deployments on PostgreSQL 16 and avoid the 805+ restart crash loops encountered during initial deployment.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-02 08:55:41 -07:00
a626c48e7b docs(n8n): complete PostgreSQL 15+ troubleshooting and add operational scripts
This commit documents the comprehensive troubleshooting session that identified
and resolved the n8n 502 Bad Gateway issue, along with production-ready fix scripts.

Root Cause Identified:
- PostgreSQL 15+ removed default CREATE privilege on public schema
- n8n_user unable to create tables during database migration
- Service trapped in crash loop (805+ restart cycles over 6 minutes)
- Error: "permission denied for schema public"

CLAUDE_STATUS.md Updates:
- Executive summary with key findings and 95% deployment confidence
- Complete error log evidence (exact error messages from 805+ restart cycles)
- Detailed root cause analysis of PostgreSQL 15+ breaking change
- Fix script validation by backend-builder (92/100 rating)
- Quick deployment guide with pre/post-deployment procedures
- Communication log documenting all three agent contributions
- Lessons learned for future Debian 12 + PostgreSQL 16 deployments

Scripts Added (All Sanitized):
1. fix_n8n_db_permissions.sh
   - Fixes PostgreSQL 15+ permission issue for n8n database
   - Creates backups before changes (pg_dump to /var/backups/n8n/)
   - Recreates database with proper ownership and explicit schema grants
   - Tests permissions before restarting service
   - Parameterized password (via N8N_DB_PASSWORD env var)
   - Comprehensive logging to /var/log/n8n_db_fix_*.log
   - Production-ready with error handling and validation

2. export_cf_dns.py (Cloudflare DNS Export Tool)
   - Exports Cloudflare DNS records and zone settings
   - Supports pagination for large zone configurations
   - Parameterized credentials (CF_ZONE_ID, CF_API_TOKEN)
   - Useful for backup/disaster recovery workflows
   - Includes validation function to prevent misconfiguration

3. scripts/README.md
   - Comprehensive documentation for all scripts
   - Usage examples with environment variable approach
   - Security notes and best practices
   - Directory structure and use cases

Security Measures:
- All scripts parameterized (no hardcoded credentials)
- Updated .gitignore to exclude script variants with embedded credentials
- Added patterns for *_with_creds.*, *.local.*, *_prod.* variants
- Documentation emphasizes environment variable usage

Agent Contributions:
- Lab-Operator: Analyzed error logs, identified PostgreSQL 15+ permission issue (100% confidence)
- Backend-Builder: Created fix script, validated against errors (92/100 rating, 95% deployment confidence)
- Scribe: Documented complete troubleshooting session with evidence and deployment guides
- Librarian: Sanitized scripts, managed git operations, ensured no credential exposure

Files Changed:
- Modified: CLAUDE_STATUS.md (+313 lines comprehensive troubleshooting documentation)
- Modified: .gitignore (+9 lines for script credential protection)
- New: scripts/fix_n8n_db_permissions.sh (349 lines, production-ready)
- New: scripts/crawlers-exporters/export_cf_dns.py (144 lines, sanitized)
- New: scripts/README.md (138 lines documentation)
- New: scripts/crawlers-exporters/*.json (DNS export examples)

Ready for Deployment: User can now execute fix script with 95% confidence
Expected Result: n8n service will successfully complete database migrations and start

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-01 17:16:20 -07:00
fe75402738 docs(n8n): document troubleshooting session for 502 Bad Gateway issue
Root Cause:
- N8N_ENCRYPTION_KEY in /opt/n8n/.env contained literal shell command
  string $(openssl rand -hex 32) instead of executed value
- .env files do not execute shell commands, only parse literal strings
- Caused n8n service crash loop preventing startup

Troubleshooting Process:
- Identified service crash loop via journalctl logs
- Backend-Builder diagnosed invalid encryption key issue
- Multiple heredoc script attempts failed due to Windows/Linux line
  ending issues in WSL environment
- Created simple fix script using echo statements (no heredoc)

Solution:
- Fix script created at /tmp/fix_n8n_simple.sh
- Generates proper encryption key using openssl rand -hex 32
- Recreates .env with corrected configuration including missing
  N8N_LISTEN_ADDRESS=0.0.0.0 and NODE_ENV=production
- Backs up existing .env before changes
- Sets proper permissions (600, n8n:n8n)

Reviews:
- Backend-Builder: APPROVED (95% confidence, technically sound)
- Lab-Operator: APPROVED with safeguards (ZFS snapshot, DB backup)

Status: Ready for deployment by user on CT 113 tomorrow

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-01 00:17:55 -07:00
c16d521070 docs(n8n): correct architecture for Debian 12 and Nginx Proxy Manager
Real-world deployment feedback revealed documentation mismatches:
- OS: Ubuntu references → Debian 12 (actual deployment)
- Reverse Proxy: Standalone nginx → Nginx Proxy Manager (NPM)

Changes Applied (30+ corrections in 4 batches):

Batch 1 - OS Corrections:
- Update OS template and PostgreSQL repo references to Debian 12

Batch 2 - NPM Terminology (10 updates):
- Update CT 102 specs (2 cores, 4GB RAM, 10GB disk)
- Rename nginx → nginx-proxy-mgr throughout
- Add NPM admin UI port 81 to diagrams
- Remove nginx-light/certbot from prerequisites

Batch 3 - Major Rewrites:
- Section VI-A: Complete NPM architecture overview
- Phase 7: Rewrite for NPM web UI (20min → 10min)
  * Replace SSH/manual config with browser-based setup
  * Add step-by-step proxy host creation guide
  * Include NPM-specific troubleshooting

Batch 4 - Minor Updates (15+ changes):
- Update troubleshooting sections for NPM
- Update architecture diagrams
- Update deployment workflows

Impact:
- Deployment time reduced (Phase 7: 20min → 10min)
- Complexity reduced (GUI vs manual nginx config)
- Accuracy improved (matches actual Debian 12 + NPM deployment)

Validated-by: Lab-Operator
Real-world-tested: PostgreSQL installation, NPM configuration

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 17:37:00 -07:00
a1841f1c41 docs(infrastructure): add MCP setup and n8n deployment documentation
- Add Obsidian MCP server setup guide for WSL2 integration (820 lines)
- Add comprehensive n8n workflow automation deployment plan (1,948 lines)
- Add agent workflow coordination via CLAUDE_STATUS.md
- Update CLAUDE.md with universal agent workflow protocol
- Remove deprecated homelab-steve agent definition
- Enhance .gitignore with Claude config exclusions

Security: API key sanitized, no production secrets exposed
Infrastructure Impact: None (documentation only)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 13:24:29 -07:00
4b62fb0a27 Initial commit: Homelab infrastructure repository with automated collection system
- Added Proxmox VE configuration collection scripts
- Included documentation and quick-start guides
- First infrastructure snapshot from serviceslab (2025-11-29)
- All VM configs (10 VMs) and LXC configs (3 containers)
- Git setup complete with .gitignore protecting sensitive data
2025-11-29 15:55:56 -07:00