# Homelab Collection Script - Quick Reference ## Overview The `collect-homelab-config.sh` script performs comprehensive, read-only collection of your Proxmox VE infrastructure configuration and exports it in an organized, documented format. **Status**: ✅ **Fully Operational** (bugs fixed as of 2025-11-29) --- ## Quick Start ### On Proxmox Host (Direct Execution) ```bash # Copy script to Proxmox host scp collect-homelab-config.sh root@192.168.2.100:/tmp/ # SSH to Proxmox host ssh root@192.168.2.100 # Run with default settings cd /tmp bash collect-homelab-config.sh # Or with verbose output bash collect-homelab-config.sh --verbose ``` ### From Your Workstation (Remote Execution) ```bash # Upload and execute in one command scp collect-homelab-config.sh root@192.168.2.100:/tmp/ && \ ssh root@192.168.2.100 'cd /tmp && bash /tmp/collect-homelab-config.sh' # Download the results scp root@192.168.2.100:/tmp/homelab-export-*.tar.gz ./ ``` --- ## Command-Line Options ### Collection Levels ```bash # Basic: System info + Proxmox/VM/CT configs only ./collect-homelab-config.sh --level basic # Standard: Basic + storage, network, backup, cluster info (DEFAULT) ./collect-homelab-config.sh --level standard # Full: Standard + service configs, detailed system state ./collect-homelab-config.sh --level full # Paranoid: Everything possible (experimental) ./collect-homelab-config.sh --level paranoid ``` ### Sanitization Options ```bash # Sanitize everything (IPs, passwords, tokens) ./collect-homelab-config.sh --sanitize all # Sanitize only IP addresses ./collect-homelab-config.sh --sanitize ips # No sanitization (CAUTION: sensitive data will be in clear text) ./collect-homelab-config.sh --sanitize none # Default: Passwords and tokens sanitized, IPs preserved ``` ### Output Control ```bash # Custom output directory ./collect-homelab-config.sh --output /backup/my-homelab-export # Disable compression ./collect-homelab-config.sh --no-compress # Enable compression (default) ./collect-homelab-config.sh --compress ``` ### Debugging ```bash # Verbose output (shows DEBUG messages) ./collect-homelab-config.sh --verbose # Standard output (INFO/SUCCESS/WARN/ERROR only) ./collect-homelab-config.sh ``` ### Combined Examples ```bash # Full collection with complete sanitization and verbose output ./collect-homelab-config.sh --level full --sanitize all --verbose # Basic collection to specific directory, no compression ./collect-homelab-config.sh --level basic --output /root/configs --no-compress # Standard collection with IP sanitization ./collect-homelab-config.sh --sanitize all --verbose ``` --- ## What Gets Collected ### ✅ Always Collected (All Levels) **System Information**: - Proxmox VE version - Hostname, kernel, uptime - CPU, memory, disk information - LVM configuration (PV/VG/LV) - Network interfaces and routing - Listening services - Installed packages **Proxmox Configurations**: - Datacenter settings (`datacenter.cfg`) - Storage pools (`storage.cfg`) - Users and permissions (`user.cfg`) - Authentication keys (`authkey.pub`) - Firewall rules (if configured) - Corosync cluster config (if in cluster) - HA configuration (if configured) **VM Configurations**: - All QEMU/KVM VM configs (`/etc/pve/nodes/*/qemu-server/*.conf`) - VM firewall rules - Format: `{VMID}-{VM-name}.conf` **LXC Container Configurations**: - All LXC container configs (`/etc/pve/nodes/*/lxc/*.conf`) - Container firewall rules - Format: `{CTID}-{container-name}.conf` **Network Configurations**: - Interface definitions (`/etc/network/interfaces`) - Additional interface configs (`/etc/network/interfaces.d/`) - SDN configuration (if configured) - Hosts file - DNS resolver config **Storage Information**: - Storage status (`pvesm status`) - ZFS pools and datasets (if ZFS is used) - NFS exports (if configured) - Samba configuration (if configured) - iSCSI initiator config (if configured) ### ➕ Standard Level and Above **Backup Configurations**: - Vzdump configuration - Scheduled backup jobs - Proxmox Backup Server integration **Cluster Information**: - Cluster status and membership - Resource allocation - Recent tasks history **Guest Information**: - VM and container lists - Resource usage statistics - JSON exports for programmatic access ### ➕ Full Level and Above **Service Configurations**: - Systemd service status - Proxmox-specific service states - pve-cluster - pvedaemon - pveproxy - pvestatd - pve-firewall - pvescheduler --- ## Output Structure ``` homelab-export-20251129-141328/ ├── README.md # Comprehensive documentation ├── SUMMARY.md # Collection statistics and overview ├── collection.log # Detailed execution log ├── configs/ # Configuration files │ ├── proxmox/ # Proxmox VE configurations │ │ ├── datacenter.cfg │ │ ├── storage.cfg │ │ ├── user.cfg │ │ └── authkey.pub │ ├── vms/ # Virtual machine configs │ │ ├── 100-docker-hub.conf │ │ ├── 101-gitlab.conf │ │ └── ... │ ├── lxc/ # LXC container configs │ │ ├── 102-nginx.conf │ │ ├── 103-netbox.conf │ │ └── 112-Anytype.conf │ ├── storage/ # Storage configurations │ ├── network/ # Network configurations │ ├── backup/ # Backup job configurations │ └── services/ # System service configs (full level) ├── exports/ # System state exports │ ├── system/ # System information │ │ ├── pve-version.txt │ │ ├── cpuinfo.txt │ │ ├── meminfo.txt │ │ └── ... │ ├── cluster/ # Cluster status and resources │ └── guests/ # Guest VM/CT information ├── docs/ # For manual documentation additions ├── scripts/ # For automation scripts └── diagrams/ # For network diagrams ``` --- ## Sanitization Behavior ### Default Sanitization (Passwords + Tokens) **Patterns Removed**: - `password=*` → `password=` - `passwd=*` → `passwd=` - `"password": "*"` → `"password": ""` - `token=*` → `token=` - `api_key=*` → `api_key=` - `secret=*` → `secret=` **IP Addresses**: Preserved (not sanitized) ### All Sanitization (--sanitize all) Everything above PLUS: - All IPv4 addresses → `10.X.X.X` ### No Sanitization (--sanitize none) ⚠️ **DANGER**: All sensitive data remains in clear text. Use only for: - Air-gapped environments - Encrypted storage destinations - Troubleshooting purposes --- ## Expected Output ### Success Indicators ``` ================================================================================ Collection Complete ================================================================================ [✓] Total items collected: 50 [INFO] Total items skipped: 1 [WARN] Total errors: 5 Export Location: ./homelab-export-20251129-141328 Summary Report: ./homelab-export-20251129-141328/SUMMARY.md Collection Log: ./homelab-export-20251129-141328/collection.log ``` ### Typical Statistics - **Items Collected**: 45-60 (depending on your infrastructure) - **Items Skipped**: 0-5 (files that don't exist, like domains.cfg) - **Errors**: 0-10 (warnings for optional configs that don't exist) - **Archive Size**: 30-100KB (compressed) - **Execution Time**: 5-15 seconds ### Common Warnings (Non-Critical) ``` [WARN] Failed to copy directory HA configuration from /etc/pve/ha [WARN] Failed to copy directory Additional interface configs from /etc/network/interfaces.d [WARN] Failed to copy directory SDN configuration from /etc/pve/sdn [WARN] Failed to execute: pvecm status (Cluster status) [WARN] Failed to execute: pvecm nodes (Cluster nodes) ``` These are **expected** if you don't have: - High Availability (HA) configured - Additional network interface configs - Software Defined Networking (SDN) - Multi-node cluster --- ## Post-Collection Workflow ### 1. Review the Collection ```bash # Extract the archive tar -xzf homelab-export-20251129-141328.tar.gz # Review the summary cat homelab-export-20251129-141328/SUMMARY.md # Check for errors grep -E "\[ERROR\]|\[WARN\]" homelab-export-20251129-141328/collection.log ``` ### 2. Verify Sensitive Data ```bash # Verify passwords are sanitized grep -r "password=" homelab-export-20251129-141328/configs/ # Should show: password= ``` ### 3. Version Control ```bash # Initialize git (if not already done) cd /mnt/c/Users/fam1n/Documents/homelab git init # Add the export git add homelab-export-20251129-141328/ git commit -m "Infrastructure snapshot: 2025-11-29" # Or just commit the archive git add homelab-export-20251129-141328.tar.gz git commit -m "Homelab export archive: 2025-11-29" ``` ### 4. Documentation Enhancement Add custom documentation to the collected export: ```bash cd homelab-export-20251129-141328 # Add network diagrams cp ~/network-diagram.png diagrams/ # Add runbooks cat > docs/disaster-recovery.md < /etc/cron.d/homelab-export <<'EOF' # Monthly infrastructure export 0 2 1 * * root cd /backup && /opt/scripts/collect-homelab-config.sh --level standard --output /backup/homelab-export-$(date +\%Y\%m) --compress EOF ``` --- ## Troubleshooting ### Script Stops Prematurely ✅ **FIXED** as of 2025-11-29. If you're still experiencing issues: 1. Ensure you're using the latest version of the script 2. Run with `--verbose` to see detailed progress 3. Check `collection.log` for specific errors 4. Verify you have root permissions: `whoami` should return `root` ### Permission Denied Errors ```bash # Ensure script is executable chmod +x collect-homelab-config.sh # Run as root sudo bash collect-homelab-config.sh # or ssh root@proxmox-host 'bash /tmp/collect-homelab-config.sh' ``` ### Missing Commands The script requires standard Proxmox utilities: - `pveversion`, `pvesh`, `pvesm`, `pvecm` - `qm`, `pct` - `lscpu`, `df`, `lsblk`, `ip`, `ss` These are all included in a standard Proxmox VE installation. ### Archive Too Large ```bash # Exclude large log files (manual step after collection) cd homelab-export-20251129-141328 find . -name "*.log" -size +10M -delete # Re-compress cd .. tar -czf homelab-export-20251129-141328-slim.tar.gz homelab-export-20251129-141328/ ``` --- ## Security Considerations ### ⚠️ Warning: Sensitive Data Even with sanitization enabled, the export contains: - ✅ Network topology and IP addressing - ✅ Usernames (passwords redacted) - ✅ Storage paths and mount points - ✅ VM/CT configurations - ✅ Installed package lists ### Best Practices 1. **Never commit to public repositories** without `--sanitize all` 2. **Encrypt archives** before cloud storage: ```bash gpg -c homelab-export-20251129-141328.tar.gz ``` 3. **Use private Git repos** (GitLab private, GitHub private) 4. **Restrict file permissions**: ```bash chmod 600 homelab-export-20251129-141328.tar.gz ``` 5. **Audit before sharing**: Always review contents before sharing with others --- ## Integration Examples ### Ansible Playbook ```yaml --- - name: Collect Proxmox infrastructure state hosts: proxmox tasks: - name: Upload collection script copy: src: collect-homelab-config.sh dest: /tmp/collect-homelab-config.sh mode: '0755' - name: Run collection shell: cd /tmp && bash collect-homelab-config.sh --level full - name: Download export fetch: src: /tmp/homelab-export-{{ ansible_date_time.date }}.tar.gz dest: ./exports/ flat: yes ``` ### Backup Script ```bash #!/bin/bash # backup-homelab-config.sh BACKUP_DIR="/mnt/backup/homelab-configs" PROXMOX_HOST="192.168.2.100" # Run collection on Proxmox ssh root@${PROXMOX_HOST} 'cd /tmp && bash /tmp/collect-homelab-config.sh --level full --sanitize all' # Download archive scp root@${PROXMOX_HOST}:/tmp/homelab-export-*.tar.gz ${BACKUP_DIR}/ # Clean up old exports (keep last 12) cd ${BACKUP_DIR} ls -t homelab-export-*.tar.gz | tail -n +13 | xargs rm -f # Cleanup remote ssh root@${PROXMOX_HOST} 'rm -rf /tmp/homelab-export-*' ``` --- ## Files in This Directory - **collect-homelab-config.sh**: The main collection script (FIXED) - **BUGFIX-SUMMARY.md**: Detailed technical analysis of bugs and fixes - **SCRIPT-USAGE.md**: This file - quick reference guide - **homelab-export-*.tar.gz**: Collected infrastructure exports - **CLAUDE.md**: Repository context and infrastructure overview --- ## Support For issues, improvements, or questions: 1. Review BUGFIX-SUMMARY.md for technical details 2. Check collection.log in the export for error details 3. Run with `--verbose` for debugging output 4. Verify you're running the latest fixed version --- *Script version: 1.0.0 (Bugs fixed: 2025-11-29)* *Documentation maintained in: /mnt/c/Users/fam1n/Documents/homelab/*