homelab/INDEX.md

Homelab Infrastructure Collection - File Index

Welcome to your homelab infrastructure collection toolkit! This index will help you navigate the various files and understand what each one does.

Quick Navigation

New to this? Start here: start-here-docs/QUICK-START.md

Ready to run? Execute: bash scripts/crawlers-exporters/collect.sh

Need help? Check: scripts/crawlers-exporters/COLLECTION-GUIDE.md

Repository Structure

homelab/
├── services/                           # Docker Compose service configurations
│   ├── n8n/                           # n8n workflow automation
│   ├── netbox/                        # Network documentation & IPAM
│   └── README.md                      # Services overview
├── scripts/
│   ├── crawlers-exporters/            # Infrastructure collection scripts
│   │   ├── collect-homelab-config.sh  # Main collection engine (runs on Proxmox)
│   │   ├── collect-remote.sh          # SSH wrapper (orchestrates from WSL)
│   │   ├── collect.sh                 # Convenience wrapper (uses .env)
│   │   ├── COLLECTION-GUIDE.md        # Comprehensive collection reference
│   │   └── README-COLLECTION.md       # Collection overview
│   ├── fixers/                        # Problem-solving scripts
│   │   ├── fix_n8n_db_c_locale.sh     # PostgreSQL locale fix for n8n
│   │   └── fix_n8n_db_permissions.sh  # Database permissions repair
│   └── qol/                           # Quality of life improvements
│       ├── git-aliases.sh             # Git command shortcuts
│       └── git-first-commit.sh        # Repository initialization helper
├── start-here-docs/                   # Getting started documentation
│   ├── START-HERE.md                  # Primary entry point
│   ├── QUICK-START.md                 # 5-minute quick start guide
│   ├── GIT-SETUP-GUIDE.md            # Git configuration guide
│   ├── GIT-QUICK-REFERENCE.md        # Git command reference
│   ├── SCRIPT-USAGE.md               # Script usage examples
│   └── SETUP-COMPLETE.md             # Post-setup checklist
├── troubleshooting/                   # Problem resolution documentation
│   └── BUGFIX-SUMMARY.md             # Known issues and fixes
├── n8n/                              # n8n-specific documentation
│   └── N8N-SETUP-PLAN.md             # n8n deployment guide
├── mcp/                              # Model Context Protocol integrations
│   └── obsidian/
│       └── OBSIDIAN-MCP-SETUP.md     # Obsidian MCP configuration
├── sub-agents/                       # Agent role definitions
│   ├── scribe.md                     # Documentation & architecture agent
│   ├── backend-builder.md            # Development agent
│   ├── lab-operator.md               # Infrastructure operations agent
│   └── librarian.md                  # Knowledge management agent
├── homelab-export-20251202-204939/   # Latest infrastructure snapshot
│   ├── README.md                     # Export documentation
│   ├── SUMMARY.md                    # Collection statistics
│   ├── collection.log                # Detailed collection log
│   ├── configs/                      # Configuration files
│   │   ├── proxmox/                 # Proxmox core configs
│   │   ├── vms/                     # VM configurations
│   │   ├── lxc/                     # Container configurations
│   │   ├── storage/                 # Storage pool configs
│   │   ├── network/                 # Network configs
│   │   ├── backup/                  # Backup job configs
│   │   └── services/                # Systemd services
│   └── exports/                      # Structured data exports
│       ├── system/                   # System information
│       ├── cluster/                  # Cluster status (JSON)
│       └── guests/                   # VM/CT details
├── archive-homelab/                  # Historical exports and backups
├── CLAUDE.md                         # AI assistant project guidance
├── INDEX.md                          # This file - navigation index
└── README.md                         # Repository overview

File Inventory

Core Scripts

File	Location	Size	Purpose
collect-homelab-config.sh	scripts/crawlers-exporters/	32K	Main collection engine - runs on Proxmox host
collect-remote.sh	scripts/crawlers-exporters/	13K	SSH wrapper - orchestrates remote execution from WSL
collect.sh	scripts/crawlers-exporters/	4.2K	Convenience wrapper - uses .env configuration

Which script should I use?

• Easiest: scripts/crawlers-exporters/collect.sh (if you have .env configured)
• Direct control: scripts/crawlers-exporters/collect-remote.sh <proxmox-ip>
• On Proxmox: scripts/crawlers-exporters/collect-homelab-config.sh (if SSHed into Proxmox)

Configuration

File	Size	Purpose
.env.example	1.5K	Configuration template with all options
.env	-	YOUR configuration (create from .env.example)

Setup: cp .env.example .env then edit .env with your Proxmox IP

Documentation

File	Location	Size	Purpose	Audience
QUICK-START.md	start-here-docs/	9.6K	Get started in 5 minutes	First-time users
START-HERE.md	start-here-docs/	-	Primary entry point	All users
README-COLLECTION.md	scripts/crawlers-exporters/	8.5K	Overview and common patterns	General users
COLLECTION-GUIDE.md	scripts/crawlers-exporters/	15K	Comprehensive reference	Power users
GIT-SETUP-GUIDE.md	start-here-docs/	-	Git configuration	New Git users
GIT-QUICK-REFERENCE.md	start-here-docs/	-	Git command cheat sheet	All users
SCRIPT-USAGE.md	start-here-docs/	-	Script examples	Script users
INDEX.md	Root	This file	Navigation and file index	Everyone
CLAUDE.md	Root	5.7K	Project context for Claude	AI assistant
README.md	Root	-	Repository overview	All users

Which Documentation Should I Read?

┌─────────────────────────────────────────────────────┐
│  Just want to get started?                          │
│  → Read: start-here-docs/QUICK-START.md            │
│  → Time: 5 minutes                                  │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│  Want to understand common usage patterns?          │
│  → Read: scripts/crawlers-exporters/               │
│           README-COLLECTION.md                      │
│  → Time: 15 minutes                                 │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│  Need complete reference with all options?          │
│  → Read: scripts/crawlers-exporters/               │
│           COLLECTION-GUIDE.md                       │
│  → Time: 30-45 minutes                              │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│  Setting up services (n8n, netbox, etc)?           │
│  → Read: services/README.md                        │
│         n8n/N8N-SETUP-PLAN.md                      │
│  → Time: Variable                                   │
└─────────────────────────────────────────────────────┘

Typical Workflow

First-Time Setup (5 minutes)

1. Configure environment

   cp .env.example .env
   nano .env  # Set PROXMOX_HOST to your Proxmox IP
   

2. Test SSH access

   ssh root@your-proxmox-ip
   

3. Run first collection

   bash scripts/crawlers-exporters/collect.sh
   

4. Review results

   cat homelab-export-*/SUMMARY.md
   

Regular Use (1 minute)

# Just run this whenever you want to snapshot your infrastructure
bash scripts/crawlers-exporters/collect.sh

Advanced Use

# Full collection with verbose output
bash scripts/crawlers-exporters/collect.sh --level full --verbose

# Sanitized export for sharing
bash scripts/crawlers-exporters/collect.sh --sanitize all

# Different Proxmox host
bash scripts/crawlers-exporters/collect.sh --host 192.168.1.200

What Gets Created When You Run The Scripts?

After running bash scripts/crawlers-exporters/collect.sh, you'll have:

/home/jramos/homelab/
└── homelab-export-20251202-204939/
    ├── README.md                    # Documentation of this specific export
    ├── SUMMARY.md                   # Statistics and overview
    ├── collection.log               # Detailed collection log
    ├── homelab-export-*.tar.gz      # Compressed archive
    ├── configs/
    │   ├── proxmox/                # Proxmox core configs
    │   ├── vms/                    # All VM configs (9 VMs)
    │   ├── lxc/                    # All container configs (3 containers)
    │   ├── storage/                # Storage pool configs
    │   ├── network/                # Network configs
    │   ├── backup/                 # Backup job configs
    │   └── services/               # Systemd services
    ├── exports/
    │   ├── system/                 # System information
    │   ├── cluster/                # Cluster status
    │   └── guests/                 # VM/CT details
    ├── docs/                        # For your manual documentation
    ├── scripts/                     # For your automation scripts
    └── diagrams/                    # For network diagrams

Script Capabilities Matrix

Feature	collect.sh	collect-remote.sh	collect-homelab-config.sh
Uses .env config	✓	✗	✗
Runs from WSL	✓	✓	✗
Runs on Proxmox	✗	✗	✓
SSH automation	✓	✓	N/A
File transfer	✓	✓	N/A
Easiest to use	✓	-	-
Most flexible	-	✓	✓

Command Quick Reference

Setup Commands

# First-time configuration
cp .env.example .env
nano .env

# Set up SSH keys (recommended)
ssh-keygen -t ed25519
ssh-copy-id root@<proxmox-ip>

Collection Commands

# Standard collection (default)
bash scripts/crawlers-exporters/collect.sh

# Basic collection (minimal)
bash scripts/crawlers-exporters/collect.sh --level basic

# Full collection (comprehensive)
bash scripts/crawlers-exporters/collect.sh --level full

# With verbose output
bash scripts/crawlers-exporters/collect.sh --verbose

# Sanitized (safe for sharing)
bash scripts/crawlers-exporters/collect.sh --sanitize all

# Keep copy on Proxmox
bash scripts/crawlers-exporters/collect.sh --keep-remote

# Custom output location
bash scripts/crawlers-exporters/collect.sh --output ~/backups

Review Commands

# View summary
cat homelab-export-20251202-204939/SUMMARY.md

# View collection log
less homelab-export-20251202-204939/collection.log

# List collected files
ls -R homelab-export-20251202-204939/

# Browse VM configs
cat homelab-export-20251202-204939/configs/vms/*.conf

# Browse container configs
cat homelab-export-20251202-204939/configs/lxc/*.conf

# Check storage setup
cat homelab-export-20251202-204939/configs/proxmox/storage.cfg

Help Commands

# Script help
bash scripts/crawlers-exporters/collect.sh --help
bash scripts/crawlers-exporters/collect-remote.sh --help
bash scripts/crawlers-exporters/collect-homelab-config.sh --help

# Documentation
cat start-here-docs/QUICK-START.md
cat scripts/crawlers-exporters/README-COLLECTION.md
cat scripts/crawlers-exporters/COLLECTION-GUIDE.md

Collection Levels Explained

Level	Time	Size	Contents	Use Case
basic	~30s	1-3 MB	Configs only	Quick snapshots
standard	~1min	2-6 MB	Configs + status	Regular documentation
full	~2min	5-20 MB	Everything	Pre-maintenance backup
paranoid	~5min	10-50 MB	Maximum detail	Disaster recovery planning

Your Infrastructure

Based on the latest export (2025-12-02 20:49:54), your environment includes:

Virtual Machines (QEMU/KVM) - 9 VMs

VM ID	Name	Status	Purpose
100	docker-hub	Running	Container registry/Docker hub mirror
104	ubuntu-dev	Stopped	Ubuntu development environment
105	dev	Stopped	General-purpose development workstation
106	Ansible-Control	Running	IaC orchestration, configuration management
107	ubuntu-docker	Stopped	Ubuntu Docker host
108	CML	Stopped	Cisco Modeling Labs - network simulation
109	web-server-01	Running	Web application server (clustered)
110	web-server-02	Running	Load-balanced pair with web-server-01
111	db-server-01	Running	Backend database server

Note: VM 101 (gitlab) has been removed from the infrastructure.

Containers (LXC) - 3 Containers

CT ID	Name	Status	Purpose
102	nginx	Running	Reverse proxy/load balancer
103	netbox	Stopped	Network documentation/IPAM
113	n8n	Running	Workflow automation platform

Note: CT 112 (Anytype) has been replaced by CT 113 (n8n).

Storage Pools

• local (Directory) - 14.8% used - System files, ISOs, templates
• local-lvm (LVM-Thin) - 0.0% used - VM disk images (thin provisioned)
• Vault (NFS/Directory) - 11.9% used - Secure storage for sensitive data
• PBS-Backups (Proxmox Backup Server) - 21.6% used - Automated backup repository
• iso-share (NFS/CIFS) - 1.4% used - Installation media library
• localnetwork (Network share) - Shared resources across infrastructure

All of these are documented in your collection exports!

Latest Export Information

• Export Directory: /home/jramos/homelab/homelab-export-20251202-204939/
• Collection Date: 2025-12-02 20:49:54
• Hostname: serviceslab
• Collection Level: full
• Script Version: 1.0.0

Common Questions

Q: Which file do I run?

A: Start with bash scripts/crawlers-exporters/collect.sh - it's the easiest.

Q: Do I need to set up .env?

A: Yes! Copy .env.example to .env and set your PROXMOX_HOST.

Q: Will this modify my Proxmox setup?

A: No! All operations are read-only. Zero risk.

Q: How much disk space do I need?

A: On Proxmox: ~100-500 MB. On WSL: ~50-250 MB for archives.

Q: Can I run this regularly?

A: Absolutely! That's what it's designed for. Run weekly or monthly.

Q: What if I get errors?

A: Check SUMMARY.md and collection.log in the export. Most "errors" are just skipped features you don't use (like ZFS if you don't have ZFS).

Q: Is my data secure?

A: By default, passwords and tokens are sanitized. Use --sanitize all for complete sanitization including IPs.

Q: Can I use this for disaster recovery?

A: Yes! The configs show you exactly how your infrastructure is set up. Restore disk images from PBS-Backups and recreate VMs using these configs.

Q: How do I fix n8n database issues?

A: Use the scripts in scripts/fixers/:

bash scripts/fixers/fix_n8n_db_permissions.sh
bash scripts/fixers/fix_n8n_db_c_locale.sh

Integration Examples

Git Version Control

cd homelab-export-20251202-204939/
git init
git add .
git commit -m "Infrastructure snapshot 2025-12-02"
git remote add origin <your-private-repo>
git push -u origin main

Scheduled Collections (on Proxmox)

# Add to Proxmox crontab
0 3 * * 0 /root/collect-homelab-config.sh -l standard -o /backup/weekly-$(date +\%Y\%U)

Automated Retention

# Keep only last 4 weeks
find ./homelab-export-* -maxdepth 0 -type d -mtime +28 -exec rm -rf {} \;

Troubleshooting Quick Guide

Problem	Solution
"Cannot connect to Proxmox"	Check IP in .env, test with ssh root@<ip>
"Permission denied"	Use bash collect.sh not ./collect.sh on Windows FS
"PROXMOX_HOST not set"	Edit .env file and set your Proxmox IP
"Some items skipped"	Normal! Check SUMMARY.md - usually ZFS/cluster features you don't use
"Script won't execute"	Use: bash scripts/crawlers-exporters/collect.sh (explicitly call bash)
SSH asks for password	Set up SSH keys: ssh-copy-id root@<proxmox-ip>
n8n 502 Bad Gateway	Check troubleshooting/BUGFIX-SUMMARY.md and n8n/N8N-SETUP-PLAN.md
PostgreSQL locale errors	Run bash scripts/fixers/fix_n8n_db_c_locale.sh

For detailed troubleshooting, see: troubleshooting/BUGFIX-SUMMARY.md

File Sizes Reference

Component	Size	Notes
Scripts	~50K total	All three collection scripts
Documentation	~65K total	All markdown and text files
Configuration	~2K	.env.example template
Total Package	~117K	Everything you need
Output (basic)	1-3 MB	Per collection run
Output (standard)	2-6 MB	Per collection run
Output (full)	5-20 MB	Per collection run

Service Management

n8n Workflow Automation

• Container: CT 113
• Service Config: services/n8n/
• Setup Guide: n8n/N8N-SETUP-PLAN.md
• Common Issues: See troubleshooting/BUGFIX-SUMMARY.md

NetBox IPAM

• Container: CT 103
• Service Config: services/netbox/
• Purpose: Network documentation and IP address management

Nginx Reverse Proxy

• Container: CT 102
• Purpose: Front-end traffic management, SSL termination

Agent Roles

This repository works with multiple AI agent personas:

Agent	File	Purpose
Scribe	sub-agents/scribe.md	Documentation & architecture
Backend Builder	sub-agents/backend-builder.md	Development & coding
Lab Operator	sub-agents/lab-operator.md	Infrastructure operations
Librarian	sub-agents/librarian.md	Knowledge management

Next Steps

1. Immediate: Read start-here-docs/QUICK-START.md (5 minutes)

2. Setup: Create your .env file

   cp .env.example .env
   nano .env  # Set PROXMOX_HOST
   

3. Execute: Run your first collection

   bash scripts/crawlers-exporters/collect.sh
   

4. Review: Check the results

   cat homelab-export-20251202-204939/SUMMARY.md
   

5. Explore: Browse the collected configs

   ls -R homelab-export-20251202-204939/
   

6. Learn More: Read scripts/crawlers-exporters/README-COLLECTION.md for common patterns

7. Master It: Study scripts/crawlers-exporters/COLLECTION-GUIDE.md for advanced usage

Support and Resources

• Quick help: bash scripts/crawlers-exporters/collect.sh --help
• Detailed help: See scripts/crawlers-exporters/COLLECTION-GUIDE.md
• Getting started: See start-here-docs/QUICK-START.md
• Examples: See scripts/crawlers-exporters/README-COLLECTION.md
• Troubleshooting: See troubleshooting/BUGFIX-SUMMARY.md

Summary

You now have a complete, production-ready infrastructure collection system that:

✓ Automatically collects all Proxmox configurations ✓ Works seamlessly from WSL2 via SSH ✓ Sanitizes sensitive information ✓ Creates organized, documented exports ✓ Supports automation and scheduling ✓ Provides comprehensive documentation ✓ Is completely safe (read-only operations) ✓ Includes service management for n8n, netbox, nginx ✓ Has troubleshooting resources and fix scripts

Ready to begin?

# Set up configuration
cp .env.example .env
nano .env

# Run your first collection
bash scripts/crawlers-exporters/collect.sh

---

Repository Version: 2.0.0 Last Updated: 2025-12-02 Latest Export: homelab-export-20251202-204939 Infrastructure: 9 VMs, 3 Containers, Proxmox VE 8.3.3 Maintained by: Your homelab automation system