homelab/scripts/crawlers-exporters/README-COLLECTION.md

Homelab Infrastructure Collection Scripts

Automated collection tools for documenting and backing up your Proxmox VE homelab infrastructure.

Quick Start

From WSL2 (Your Current Environment)

The easiest way to collect your Proxmox configuration from WSL2:

# 1. Make scripts executable
chmod +x collect-homelab-config.sh collect-remote.sh

# 2. Set your Proxmox IP address
PROXMOX_IP="192.168.1.100"  # Replace with your actual IP

# 3. Run the remote collection
./collect-remote.sh $PROXMOX_IP

# 4. Review results
ls -la homelab-export-*/
cat homelab-export-*/SUMMARY.md

That's it! The script will SSH to your Proxmox host, collect all configurations, and download them to your WSL environment.

Directly on Proxmox Host

If you prefer to run directly on the Proxmox server:

# 1. Copy script to Proxmox
scp collect-homelab-config.sh root@<proxmox-ip>:/root/

# 2. SSH to Proxmox
ssh root@<proxmox-ip>

# 3. Run collection
chmod +x /root/collect-homelab-config.sh
./collect-homelab-config.sh

# 4. Copy results back to WSL
# From WSL:
scp -r root@<proxmox-ip>:/root/homelab-export-* ./

What You Get

After running the collection script, you'll have:

homelab-export-20241128-143022/
├── README.md              # Detailed documentation
├── SUMMARY.md            # Collection statistics
├── collection.log        # Detailed log
├── configs/
│   ├── proxmox/         # Core Proxmox settings
│   ├── vms/             # All VM configurations
│   ├── lxc/             # All container configurations
│   ├── storage/         # Storage pool configs
│   ├── network/         # Network interface configs
│   └── backup/          # Backup job definitions
└── exports/
    ├── system/          # System information
    ├── cluster/         # Cluster status
    └── guests/          # VM/container lists

Files in This Directory

File	Purpose
collect-homelab-config.sh	Main collection script (runs on Proxmox host)
collect-remote.sh	Wrapper for remote execution from WSL/Linux
COLLECTION-GUIDE.md	Comprehensive usage guide and reference
README-COLLECTION.md	This file - quick start guide

Common Use Cases

Daily Documentation Snapshot

# Run from WSL - executes on Proxmox and brings results back
./collect-remote.sh 192.168.1.100 --level standard

Pre-Maintenance Full Backup

# Full collection before making changes
./collect-remote.sh 192.168.1.100 --level full --keep-remote

Sanitized Export for Sharing

# Remove all sensitive data (IPs, passwords, tokens)
./collect-remote.sh 192.168.1.100 --sanitize all

Weekly Automated Collection

On your Proxmox host, add to crontab:

# Edit crontab
sudo crontab -e

# Add weekly Sunday 3 AM collection
0 3 * * 0 /root/collect-homelab-config.sh -l standard -o /backup/homelab/weekly-$(date +\%Y\%W)

Collection Levels

Level	What It Includes	Best For
basic	System info, Proxmox configs, VM/LXC configs, network	Quick snapshots
standard	Basic + storage, backups, cluster info	Regular documentation
full	Standard + service configs, detailed state	Pre-maintenance
paranoid	Everything possible	Maximum detail

Prerequisites

SSH Access (Required for Remote Collection)

Set up passwordless SSH authentication:

# Generate SSH key (if you don't have one)
ssh-keygen -t ed25519

# Copy to Proxmox host
ssh-copy-id root@<proxmox-ip>

# Test connection
ssh root@<proxmox-ip> 'hostname'

Disk Space

Ensure adequate space:

• On Proxmox: ~100-500 MB (depending on collection level)
• On WSL: ~50-250 MB for compressed archives

Permissions

The collection script needs root access on the Proxmox host to read all configurations.

Security Considerations

Default Sanitization

By default, the script sanitizes:

• ✓ Passwords
• ✓ API tokens
• ✗ IP addresses (preserved for documentation)

Full Sanitization

For exports that leave your network:

./collect-remote.sh <proxmox-ip> --sanitize all

This will redact:

• Passwords
• API tokens
• IP addresses (replaced with 10.x.x.x)

Storage Recommendations

1. Keep unsanitized exports secure - store in encrypted locations
2. Use private Git repositories - never commit to public repos without sanitization
3. Encrypt sensitive exports:

   gpg --symmetric --cipher-algo AES256 homelab-export-*.tar.gz
   

Next Steps

1. Run your first collection using the Quick Start steps above

2. Review the comprehensive guide:

   cat COLLECTION-GUIDE.md
   

3. Examine your infrastructure:

   # View VM configurations
   cat homelab-export-*/configs/vms/*.conf
   
   # Check storage setup
   cat homelab-export-*/configs/proxmox/storage.cfg
   
   # Review network configuration
   cat homelab-export-*/configs/network/interfaces
   

4. Set up version control:

   cd homelab-export-<timestamp>/
   git init
   git add .
   git commit -m "Initial infrastructure snapshot"
   git remote add origin <your-private-repo>
   git push -u origin main
   

5. Schedule regular collections (see COLLECTION-GUIDE.md for automation)

6. Use exports to build Infrastructure as Code (Terraform, Ansible)

Troubleshooting

"Cannot connect to Proxmox host"

Check:

• Is the IP address correct?
• Is SSH running on Proxmox? ssh root@<proxmox-ip>
• Can you reach the host? ping <proxmox-ip>
• Is firewall blocking port 22?

"Permission denied"

Solutions:

• Ensure scripts are executable: chmod +x *.sh
• Use root user: ./collect-remote.sh 192.168.1.100 --user root
• Set up SSH keys: ssh-copy-id root@<proxmox-ip>

"Not a Proxmox VE host"

This script requires a Proxmox VE installation. Verify:

ssh root@<proxmox-ip> 'cat /etc/pve/.version'

"Some items skipped"

This is normal! Review SUMMARY.md to see what was skipped. Common skipped items:

• ZFS tools (if not using ZFS)
• Cluster configs (on single-node setups)
• HA configs (if High Availability not configured)

Need more help?

1. Run with verbose flag: ./collect-remote.sh <proxmox-ip> -v
2. Check the log: cat homelab-export-*/collection.log
3. Review the full guide: cat COLLECTION-GUIDE.md

Examples

Standard Collection from WSL

./collect-remote.sh 192.168.1.100

Full Collection with Verbose Output

./collect-remote.sh 192.168.1.100 --level full --verbose

Custom SSH Port and User

./collect-remote.sh proxmox.local --port 2222 --user admin

Keep Remote Copy

./collect-remote.sh 192.168.1.100 --keep-remote

Sanitized Export to Custom Location

./collect-remote.sh 192.168.1.100 --sanitize all --output ~/Documents/homelab-backups

File Sizes

Approximate sizes for reference:

Collection Level	Uncompressed	Compressed
Basic	5-15 MB	1-3 MB
Standard	10-30 MB	2-6 MB
Full	20-100 MB	5-20 MB
Paranoid	50-500 MB	10-50 MB

Actual sizes vary based on number of VMs/containers and configurations

Support

For detailed information on any topic:

• Complete Usage Guide: See COLLECTION-GUIDE.md
• Script Help: Run ./collect-homelab-config.sh --help
• Remote Wrapper Help: Run ./collect-remote.sh --help

What's Collected vs. What's NOT

✓ Collected (READ-ONLY)

• Configuration files
• System information
• Resource lists
• Status information
• Network settings
• Storage definitions

✗ NOT Collected

• Actual VM/container disk images
• ISO files
• Backup archives
• User data within VMs/containers
• Authentication credentials (sanitized)

Note: This script collects configuration and metadata, not actual data. For complete backups, use Proxmox Backup Server or vzdump.

Integration Ideas

Once you have collected your infrastructure:

1. Documentation: Use as source of truth for runbooks
2. Change Tracking: Diff exports to see what changed over time
3. Disaster Recovery: Reference during rebuilds
4. IaC Development: Convert to Terraform/Ansible
5. Compliance: Evidence of configuration state
6. Training: Sandbox for learning without risk
7. Migration Planning: Understand dependencies before moving

---

Version: 1.0.0 Last Updated: 2024-11-28 Maintained By: Your homelab automation assistant, Steve