Implement comprehensive directory reorganization to improve discoverability,
logical grouping, and separation of concerns across documentation, scripts,
and infrastructure snapshots.
Major Changes:
1. Documentation Reorganization:
- Created start-here-docs/ for onboarding documentation
* Moved QUICK-START.md, START-HERE.md, GIT-SETUP-GUIDE.md
* Moved GIT-QUICK-REFERENCE.md, SCRIPT-USAGE.md, SETUP-COMPLETE.md
- Created troubleshooting/ directory
* Moved BUGFIX-SUMMARY.md for centralized issue resolution
- Created mcp/ directory for Model Context Protocol configurations
* Moved OBSIDIAN-MCP-SETUP.md to mcp/obsidian/
2. Scripts Reorganization:
- Created scripts/crawlers-exporters/ for infrastructure collection
* Moved collect*.sh scripts and collection documentation
* Consolidates Proxmox homelab export tooling
- Created scripts/fixers/ for operational repair scripts
* Moved fix_n8n_db_*.sh scripts
* Isolated scripts with embedded credentials (templates tracked)
- Created scripts/qol/ for quality-of-life utilities
* Moved git-aliases.sh and git-first-commit.sh
3. Infrastructure Snapshots:
- Created disaster-recovery/ for active infrastructure state
* Moved latest homelab-export-20251202-204939/ snapshot
* Contains current VM/CT configurations and system state
- Created archive-homelab/ for historical snapshots
* Moved homelab-export-*.tar.gz archives
* Preserves point-in-time backups for reference
4. Agent Definitions:
- Created sub-agents/ directory
* Added backend-builder.md (development agent)
* Added lab-operator.md (infrastructure operations agent)
* Added librarian.md (git/version control agent)
* Added scribe.md (documentation agent)
5. Updated INDEX.md:
- Reflects new directory structure throughout
- Updated all file path references
- Enhanced navigation with new sections
- Added agent roles documentation
- Updated quick reference commands
6. Security Improvements:
- Updated .gitignore to match reorganized file locations
- Corrected path for scripts/fixers/fix_n8n_db_c_locale.sh exclusion
- Maintained template-based credential management pattern
Infrastructure State Update:
- Latest snapshot: 2025-12-02 20:49:54
- Removed: VM 101 (gitlab), CT 112 (Anytype)
- Added: CT 113 (n8n)
- Total: 9 VMs, 3 Containers
Impact:
- Improved repository navigation and discoverability
- Logical separation of documentation, scripts, and snapshots
- Clearer onboarding path for new users
- Enhanced maintainability through organized structure
- Foundation for multi-agent workflow support
Files changed: 90 files (+935/-349)
- 3 modified, 14 new files, 73 renames/moves
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
9.5 KiB
Homelab Collection - Quick Start Guide
Welcome! This guide will get you up and running with automated homelab infrastructure collection in under 5 minutes.
What This Does
Automatically collects your entire Proxmox homelab configuration including:
- ✓ All VM configurations
- ✓ All LXC container configurations
- ✓ Storage pool definitions
- ✓ Network configurations
- ✓ Backup job schedules
- ✓ System information
- ✓ Cluster resources
Completely safe - read-only operations, no modifications to your system.
Prerequisites Checklist
- Proxmox VE server running (yours:
serviceslab) - SSH access to Proxmox from WSL
- Proxmox IP address or hostname
5-Minute Setup
Step 1: Test SSH Access
# Replace with your Proxmox IP
ssh root@192.168.1.100
If you get a password prompt, set up SSH keys (recommended):
ssh-keygen -t ed25519 -f ~/.ssh/homelab
ssh-copy-id -i ~/.ssh/homelab root@192.168.1.100
Step 2: Configure Your Environment
cd /mnt/c/Users/fam1n/Documents/homelab
# Copy the example configuration
cp .env.example .env
# Edit with your Proxmox details
nano .env # or use 'code .env' for VS Code
Minimum required change in .env:
PROXMOX_HOST="192.168.1.100" # Change to your Proxmox IP
Step 3: Run Your First Collection
# Simple! Just run:
bash collect.sh
That's it! The script will:
- SSH to your Proxmox server
- Collect all configurations
- Download results to
./exports/ - Show you a summary
Step 4: Review Your Export
# View the summary
cat exports/homelab-export-*/SUMMARY.md
# Browse collected configs
ls -la exports/homelab-export-*/configs/
What Just Happened?
Your infrastructure is now documented! Here's what you have:
exports/homelab-export-20241128-143000/
├── configs/
│ ├── proxmox/ ← Core Proxmox settings
│ ├── vms/ ← All your VMs (100-docker-hub, 101-gitlab, etc.)
│ ├── lxc/ ← All containers (102-nginx, 103-netbox, etc.)
│ ├── storage/ ← Storage pools (Vault, PBS-Backups, etc.)
│ ├── network/ ← Network configuration
│ └── backup/ ← Backup jobs
├── exports/
│ ├── system/ ← System info (CPU, RAM, disk, etc.)
│ └── cluster/ ← Resource usage
├── README.md ← Complete documentation
└── SUMMARY.md ← What was collected
Next Steps
Immediate Actions
-
Review your infrastructure
# See your VM configs cat exports/homelab-export-*/configs/vms/*.conf # Check storage setup cat exports/homelab-export-*/configs/proxmox/storage.cfg -
Store safely
# Initialize git repository cd exports/homelab-export-*/ git init git add . git commit -m "Initial homelab snapshot" -
Schedule automatic collections (see Automation section below)
Common Commands
# Standard collection (default)
bash collect.sh
# Full collection with everything
bash collect.sh --level full --verbose
# Quick basic snapshot
bash collect.sh --level basic
# Sanitized export (safe for sharing)
bash collect.sh --sanitize all
# Override Proxmox host
bash collect.sh --host 192.168.1.200
Automation
Weekly Snapshots
On your Proxmox server, run:
# SSH to Proxmox
ssh root@your-proxmox-ip
# Copy the collection script
cat > /root/collect-homelab-config.sh
# Paste the script content, then Ctrl+D
# Make executable
chmod +x /root/collect-homelab-config.sh
# Add to crontab for weekly Sunday 3 AM runs
crontab -e
Add this line:
0 3 * * 0 /root/collect-homelab-config.sh -l standard -o /backup/homelab/weekly-$(date +\%Y\%U) 2>&1 | logger -t homelab-collection
Understanding Your Infrastructure
Now that you have the export, here's what you can learn:
Your Virtual Machines
Based on your environment, you should see configs for:
100-docker-hub.conf- Your container registry101-gitlab.conf- GitLab CE/EE instance105-dev.conf- Development environment106-Ansible-Control.conf- Automation control node108-CML.conf- Cisco Modeling Labs109-web-server-01.conf- Web server (clustered)110-web-server-02.conf- Web server (clustered)111-db-server-01.conf- Database server
Your Containers
102-nginx.conf- Reverse proxy/load balancer103-netbox.conf- Network documentation/IPAM112-Anytype.conf- Knowledge management
Your Storage
Check configs/proxmox/storage.cfg to see:
- local (Directory) - System storage
- local-lvm (LVM-Thin) - VM disk images
- Vault (NFS/Directory) - Secure storage
- PBS-Backups (Proxmox Backup Server) - Backup repository
- iso-share (NFS/CIFS) - Installation media
Files in This Directory
| File | Purpose |
|---|---|
collect.sh |
Start here - Easy wrapper that uses .env |
collect-remote.sh |
Advanced - Direct remote execution |
collect-homelab-config.sh |
Core script (runs on Proxmox) |
.env.example |
Configuration template |
.env |
Your configuration (create from .env.example) |
QUICK-START.md |
This file |
README-COLLECTION.md |
Detailed overview |
COLLECTION-GUIDE.md |
Complete reference manual |
Troubleshooting
"Cannot connect to Proxmox"
# Test connection
ping 192.168.1.100
ssh root@192.168.1.100
# If password prompt, set up keys:
ssh-keygen -t ed25519
ssh-copy-id root@192.168.1.100
"PROXMOX_HOST not set"
Edit your .env file:
nano .env
# Set PROXMOX_HOST="your-ip"
"Permission denied"
The scripts might not be executable on Windows filesystems. That's OK! When you run:
bash collect.sh
This explicitly uses bash to execute the script.
Need More Help?
- Quick reference:
cat README-COLLECTION.md - Complete guide:
cat COLLECTION-GUIDE.md - Script help:
bash collect-remote.sh --help
Security Notes
What's Sanitized by Default
- ✓ Passwords (replaced with
<REDACTED>) - ✓ API tokens (replaced with
<REDACTED>) - ✗ IP addresses (kept for documentation)
Full Sanitization
For exports leaving your network:
bash collect.sh --sanitize all
This will also replace IP addresses with 10.X.X.X.
Keep It Secure
# Store exports securely
chmod 700 exports/
# Use private Git repositories
# Never commit unsanitized exports to public repos
# Encrypt sensitive exports
gpg --symmetric exports/homelab-export-*.tar.gz
Use Cases
1. Documentation
"What's running on VM 109?" → Check configs/vms/109-web-server-01.conf
2. Disaster Recovery
If Proxmox crashes, you have all configurations to rebuild
3. Change Tracking
# Compare this month vs last month
diff -u exports/january/ exports/february/
4. Infrastructure as Code
Use the collected configs as templates for Terraform/Ansible
5. Compliance/Audit
"What was the configuration on date X?" → Check that export
Advanced Usage
Different Collection Levels
# Minimal - just configs
bash collect.sh --level basic
# Standard - configs + status (default)
bash collect.sh --level standard
# Everything - including service configs
bash collect.sh --level full
# Maximum detail
bash collect.sh --level paranoid
Custom Output Location
# Save to specific directory
bash collect.sh --output ~/backups/homelab
Keep Remote Copy
# Don't delete from Proxmox after download
bash collect.sh --keep-remote
Verbose Output
# See detailed progress
bash collect.sh --verbose
Success Indicators
After running, you should see:
[✓] SSH connection successful
[✓] Confirmed Proxmox VE installation
[✓] Script uploaded successfully
[✓] Collection completed successfully
[✓] Archive downloaded successfully
[✓] Archive extracted
And in your summary:
Total items collected: 45+
Total items skipped: 0-5 (normal)
Total errors: 0
What's Next?
- Explore your export - Browse the configs directory
- Set up Git - Version control your infrastructure
- Schedule automation - Weekly/monthly snapshots
- Build IaC - Convert to Terraform/Ansible
- Document - Add notes to the docs/ folder in each export
- Share - Use sanitized exports to get help or show off your setup
Complete Example Workflow
# 1. Setup (one-time)
cd /mnt/c/Users/fam1n/Documents/homelab
cp .env.example .env
nano .env # Set PROXMOX_HOST
# 2. Collect
bash collect.sh
# 3. Review
cat exports/homelab-export-*/SUMMARY.md
ls -R exports/homelab-export-*/
# 4. Store
cd exports/homelab-export-*/
git init
git add .
git commit -m "Homelab snapshot $(date +%Y-%m-%d)"
# 5. Automate (on Proxmox)
ssh root@your-proxmox-ip
# Set up cron job (see Automation section)
# 6. Repeat monthly
bash collect.sh --level full
You're All Set!
You now have:
- ✓ Complete infrastructure documentation
- ✓ Automated collection capability
- ✓ Disaster recovery reference
- ✓ Foundation for Infrastructure as Code
- ✓ Change tracking capability
Run bash collect.sh anytime to refresh your documentation.
Quick Reference Card:
# Basic collection with defaults
bash collect.sh
# Full collection, verbose, sanitized
bash collect.sh --level full --sanitize all --verbose
# Quick snapshot
bash collect.sh --level basic
# Help
bash collect.sh --help
bash collect-remote.sh --help
Need more details? See COLLECTION-GUIDE.md for comprehensive documentation.
Ready to proceed? Just run: bash collect.sh