refactor(repo): reorganize repository structure for improved navigation and maintainability

Implement comprehensive directory reorganization to improve discoverability, logical grouping, and separation of concerns across documentation, scripts, and infrastructure snapshots. Major Changes: 1. Documentation Reorganization: - Created start-here-docs/ for onboarding documentation * Moved QUICK-START.md, START-HERE.md, GIT-SETUP-GUIDE.md * Moved GIT-QUICK-REFERENCE.md, SCRIPT-USAGE.md, SETUP-COMPLETE.md - Created troubleshooting/ directory * Moved BUGFIX-SUMMARY.md for centralized issue resolution - Created mcp/ directory for Model Context Protocol configurations * Moved OBSIDIAN-MCP-SETUP.md to mcp/obsidian/ 2. Scripts Reorganization: - Created scripts/crawlers-exporters/ for infrastructure collection * Moved collect*.sh scripts and collection documentation * Consolidates Proxmox homelab export tooling - Created scripts/fixers/ for operational repair scripts * Moved fix_n8n_db_*.sh scripts * Isolated scripts with embedded credentials (templates tracked) - Created scripts/qol/ for quality-of-life utilities * Moved git-aliases.sh and git-first-commit.sh 3. Infrastructure Snapshots: - Created disaster-recovery/ for active infrastructure state * Moved latest homelab-export-20251202-204939/ snapshot * Contains current VM/CT configurations and system state - Created archive-homelab/ for historical snapshots * Moved homelab-export-*.tar.gz archives * Preserves point-in-time backups for reference 4. Agent Definitions: - Created sub-agents/ directory * Added backend-builder.md (development agent) * Added lab-operator.md (infrastructure operations agent) * Added librarian.md (git/version control agent) * Added scribe.md (documentation agent) 5. Updated INDEX.md: - Reflects new directory structure throughout - Updated all file path references - Enhanced navigation with new sections - Added agent roles documentation - Updated quick reference commands 6. Security Improvements: - Updated .gitignore to match reorganized file locations - Corrected path for scripts/fixers/fix_n8n_db_c_locale.sh exclusion - Maintained template-based credential management pattern Infrastructure State Update: - Latest snapshot: 2025-12-02 20:49:54 - Removed: VM 101 (gitlab), CT 112 (Anytype) - Added: CT 113 (n8n) - Total: 9 VMs, 3 Containers Impact: - Improved repository navigation and discoverability - Logical separation of documentation, scripts, and snapshots - Clearer onboarding path for new users - Enhanced maintainability through organized structure - Foundation for multi-agent workflow support Files changed: 90 files (+935/-349) - 3 modified, 14 new files, 73 renames/moves 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-02 21:39:33 -07:00
parent eec4c4b298
commit 4f69420aaa
90 changed files with 935 additions and 349 deletions
--- a/scripts/crawlers-exporters/COLLECTION-GUIDE.md
+++ b/scripts/crawlers-exporters/COLLECTION-GUIDE.md
@@ -0,0 +1,596 @@
+# Homelab Infrastructure Collection Guide
+
+This guide explains how to use the `collect-homelab-config.sh` script to automatically gather your Proxmox infrastructure configuration for documentation, backup, and Infrastructure as Code purposes.
+
+## Overview
+
+The collection script is a sophisticated, read-only tool that:
+- Gathers Proxmox VE configurations (VMs, containers, storage, network)
+- Exports system information and cluster state
+- Organizes everything in a well-structured directory
+- Sanitizes sensitive information (passwords, tokens, optionally IPs)
+- Generates summary reports and documentation
+- Creates compressed archives for easy storage
+
+**Important**: This script performs NO modifications to your system. It is entirely safe to run.
+
+## Prerequisites
+
+### Running from WSL2 (Your Current Environment)
+
+You'll need SSH access to your Proxmox host. The script must run **on the Proxmox node itself**, not from WSL.
+
+### Access Methods
+
+**Option 1: SSH to Proxmox and run directly**
+```bash
+# From WSL, SSH to your Proxmox host
+ssh root@<proxmox-ip>
+
+# Transfer the script
+scp collect-homelab-config.sh root@<proxmox-ip>:/root/
+
+# SSH into Proxmox
+ssh root@<proxmox-ip>
+
+# Make executable and run
+chmod +x /root/collect-homelab-config.sh
+/root/collect-homelab-config.sh
+```
+
+**Option 2: Run via SSH from WSL**
+```bash
+# Copy script to Proxmox
+scp collect-homelab-config.sh root@<proxmox-ip>:/tmp/
+
+# Execute remotely
+ssh root@<proxmox-ip> 'bash /tmp/collect-homelab-config.sh'
+
+# Copy results back to WSL
+scp -r root@<proxmox-ip>:/root/homelab-export-* ./
+```
+
+**Option 3: Create a wrapper script** (recommended)
+
+We'll create a convenient wrapper that handles the SSH transfer and execution automatically.
+
+## Quick Start
+
+### 1. Make the Script Executable
+
+```bash
+chmod +x collect-homelab-config.sh
+```
+
+### 2. Run with Default Settings
+
+On your Proxmox host:
+```bash
+sudo ./collect-homelab-config.sh
+```
+
+This will:
+- Use **standard** collection level
+- Sanitize passwords and tokens (but preserve IPs)
+- Create output in `./homelab-export-<timestamp>/`
+- Generate a compressed `.tar.gz` archive
+
+### 3. Review the Results
+
+```bash
+# View the summary
+cat homelab-export-*/SUMMARY.md
+
+# Check for any errors
+cat homelab-export-*/collection.log
+
+# Browse the collected configs
+ls -R homelab-export-*/
+```
+
+## Collection Levels
+
+The script supports four collection levels:
+
+### Basic
+**What it collects:**
+- System information (CPU, memory, disk, network)
+- Proxmox core configurations (datacenter, storage, users)
+- VM and container configurations
+- Network configurations
+
+**Use case:** Quick snapshot for documentation
+
+**Example:**
+```bash
+./collect-homelab-config.sh --level basic
+```
+
+### Standard (Default)
+**What it collects:**
+- Everything in Basic
+- Storage pool configurations and status
+- Backup job definitions
+- Cluster information and resources
+- Guest VM/container details
+
+**Use case:** Regular infrastructure snapshots
+
+**Example:**
+```bash
+./collect-homelab-config.sh --level standard
+```
+
+### Full
+**What it collects:**
+- Everything in Standard
+- System service configurations
+- Detailed service status
+- Extended system state
+
+**Use case:** Comprehensive documentation before major changes
+
+**Example:**
+```bash
+./collect-homelab-config.sh --level full
+```
+
+### Paranoid
+**What it collects:**
+- Everything possible (experimental features included)
+
+**Use case:** Maximum detail for disaster recovery planning
+
+**Example:**
+```bash
+./collect-homelab-config.sh --level paranoid
+```
+
+## Sanitization Options
+
+Protect sensitive information in your exports:
+
+### Default Sanitization
+Sanitizes passwords and tokens, preserves IP addresses:
+```bash
+./collect-homelab-config.sh
+```
+
+### Full Sanitization
+Sanitizes everything (IPs, passwords, tokens):
+```bash
+./collect-homelab-config.sh --sanitize all
+```
+
+### IP Sanitization Only
+```bash
+./collect-homelab-config.sh --sanitize ips
+```
+
+### No Sanitization
+**Warning:** Only use if storing in a secure location
+```bash
+./collect-homelab-config.sh --sanitize none
+```
+
+## Command Reference
+
+### Common Usage Patterns
+
+**Daily snapshot with compression:**
+```bash
+./collect-homelab-config.sh -l standard -o /backup/homelab/daily
+```
+
+**Pre-maintenance full backup:**
+```bash
+./collect-homelab-config.sh -l full -o /backup/pre-maintenance-$(date +%Y%m%d)
+```
+
+**Documentation export (sanitized for sharing):**
+```bash
+./collect-homelab-config.sh -l standard --sanitize all -o ./docs-export
+```
+
+**Verbose output for troubleshooting:**
+```bash
+./collect-homelab-config.sh -v
+```
+
+**Quick collection without compression:**
+```bash
+./collect-homelab-config.sh --no-compress
+```
+
+### All Options
+
+```
+-l, --level LEVEL       Collection level: basic, standard, full, paranoid
+-o, --output DIR        Output directory
+-s, --sanitize WHAT     Sanitization: all, ips, none
+-c, --compress          Compress output (default: true)
+--no-compress           Skip compression
+-v, --verbose           Verbose logging
+-h, --help              Show help
+```
+
+## Output Structure
+
+After running the script, you'll have this structure:
+
+```
+homelab-export-<timestamp>/
+├── README.md                    # Complete documentation of the export
+├── SUMMARY.md                   # Collection statistics and overview
+├── collection.log               # Detailed collection log
+├── configs/
+│   ├── proxmox/                # Proxmox VE configurations
+│   │   ├── datacenter.cfg
+│   │   ├── storage.cfg
+│   │   ├── user.cfg
+│   │   └── firewall-cluster.fw
+│   ├── vms/                    # VM configs (VMID-name.conf)
+│   │   ├── 100-docker-hub.conf
+│   │   ├── 101-gitlab.conf
+│   │   └── ...
+│   ├── lxc/                    # Container configs (CTID-name.conf)
+│   │   ├── 102-nginx.conf
+│   │   ├── 103-netbox.conf
+│   │   └── ...
+│   ├── storage/                # Storage configurations and status
+│   ├── network/                # Network interface configs
+│   ├── backup/                 # Backup job definitions
+│   └── services/               # System service configs (full/paranoid)
+├── exports/
+│   ├── system/                 # System information
+│   │   ├── pve-version.txt
+│   │   ├── hostname.txt
+│   │   ├── cpuinfo.txt
+│   │   ├── meminfo.txt
+│   │   └── ...
+│   ├── cluster/                # Cluster status and resources
+│   │   ├── cluster-status.txt
+│   │   └── cluster-resources.json
+│   └── guests/                 # VM/CT lists and details
+│       ├── vm-list.txt
+│       ├── container-list.txt
+│       └── all-guests.json
+├── docs/                        # (Empty - for manual documentation)
+├── scripts/                     # (Empty - for automation scripts)
+└── diagrams/                    # (Empty - for network diagrams)
+```
+
+## Automated Collection
+
+### Setting Up Regular Snapshots
+
+Create a cron job on your Proxmox host to collect weekly snapshots:
+
+```bash
+# Edit root's crontab
+sudo crontab -e
+
+# Add weekly collection every Sunday at 3 AM
+0 3 * * 0 /root/collect-homelab-config.sh -l standard -o /backup/homelab/weekly-$(date +\%Y\%U) >> /var/log/homelab-collection.log 2>&1
+```
+
+### Retention Policy Example
+
+Keep 4 weekly backups and clean up old ones:
+
+```bash
+#!/bin/bash
+# /root/homelab-collection-cleanup.sh
+
+BACKUP_DIR="/backup/homelab"
+KEEP_WEEKS=4
+
+# Remove archives older than KEEP_WEEKS
+find "${BACKUP_DIR}" -name "homelab-export-*.tar.gz" -mtime +$((KEEP_WEEKS * 7)) -delete
+find "${BACKUP_DIR}" -name "weekly-*" -type d -mtime +$((KEEP_WEEKS * 7)) -exec rm -rf {} \;
+```
+
+Add to crontab:
+```bash
+0 4 * * 0 /root/homelab-collection-cleanup.sh
+```
+
+## Integration with Git
+
+### Initial Repository Setup
+
+```bash
+# On your Proxmox host or after copying to WSL
+cd homelab-export-<timestamp>/
+
+# Initialize repository
+git init
+
+# Create .gitignore for sensitive files
+cat > .gitignore <<'EOF'
+# Exclude logs
+*.log
+
+# Exclude any unsanitized exports if keeping both
+*-unsanitized/
+
+# Exclude compressed archives (large files)
+*.tar.gz
+*.zip
+EOF
+
+# Initial commit
+git add .
+git commit -m "Initial homelab infrastructure snapshot $(date +%Y-%m-%d)"
+
+# Add remote (replace with your repository)
+git remote add origin git@your-git-server:homelab/infrastructure.git
+
+# Push
+git push -u origin main
+```
+
+### Tracking Changes Over Time
+
+```bash
+# After subsequent collections
+cd /path/to/homelab-export-<new-timestamp>/
+
+# Copy to your git repo
+cp -r configs/ /path/to/git-repo/
+cp -r exports/ /path/to/git-repo/
+
+cd /path/to/git-repo/
+git add .
+git diff --staged  # Review what changed
+git commit -m "Infrastructure snapshot $(date +%Y-%m-%d)"
+git push
+```
+
+## Using Collected Data
+
+### 1. Documentation
+
+The collected configs serve as your infrastructure's source of truth:
+
+```bash
+# Review your VM configurations
+cat configs/vms/*.conf
+
+# Check storage setup
+cat configs/proxmox/storage.cfg
+
+# Understand network topology
+cat configs/network/interfaces
+```
+
+### 2. Disaster Recovery
+
+In case of catastrophic failure:
+
+1. Reinstall Proxmox VE on new hardware
+2. Reference `configs/network/interfaces` to recreate network configuration
+3. Reference `configs/proxmox/storage.cfg` to recreate storage pools
+4. Use VM/container configs to understand what to restore
+5. Restore actual disk images from your Proxmox Backup Server
+
+### 3. Infrastructure as Code Development
+
+Use the collected configs to build Terraform/Ansible:
+
+**Example: Convert to Terraform**
+```bash
+# Review a VM config
+cat configs/vms/100-docker-hub.conf
+
+# Create Terraform resource based on it
+# (Use the config as reference for memory, CPU, disks, etc.)
+```
+
+**Example: Create Ansible Playbook**
+```bash
+# Use the exports to understand current state
+cat exports/guests/vm-list.txt
+
+# Build Ansible inventory
+# Create playbooks to manage these VMs
+```
+
+### 4. Change Tracking
+
+```bash
+# Compare two exports
+diff -u homelab-export-20240101-120000/configs/vms/ \
+        homelab-export-20240201-120000/configs/vms/
+
+# What changed in storage?
+diff homelab-export-20240101-120000/configs/proxmox/storage.cfg \
+     homelab-export-20240201-120000/configs/proxmox/storage.cfg
+```
+
+## Troubleshooting
+
+### Permission Denied Errors
+
+**Problem:** Script fails with permission errors
+
+**Solution:** Run as root or with sudo
+```bash
+sudo ./collect-homelab-config.sh
+```
+
+### Some Items Skipped
+
+**Problem:** SUMMARY.md shows skipped items
+
+**Cause:** Likely normal - some features may not be enabled on your system
+
+**Action:** Review `collection.log` to see what was skipped. Common skipped items:
+- ZFS tools (if you're not using ZFS)
+- Cluster configs (if running single node)
+- HA configs (if HA is not configured)
+
+This is typically not an error.
+
+### SSH Connection Issues (Remote Execution)
+
+**Problem:** Cannot connect to Proxmox from WSL
+
+**Solution:**
+```bash
+# Test SSH connection
+ssh root@<proxmox-ip> 'hostname'
+
+# If prompted for password, set up key-based auth
+ssh-keygen -t ed25519
+ssh-copy-id root@<proxmox-ip>
+```
+
+### Disk Space
+
+**Problem:** Out of disk space
+
+**Solution:**
+- Use `--no-compress` to skip compression
+- Use `-l basic` for smaller exports
+- Specify output directory on a larger filesystem: `-o /path/to/larger/disk/`
+
+### Script Doesn't Execute
+
+**Problem:** `bash: ./collect-homelab-config.sh: Permission denied`
+
+**Solution:**
+```bash
+chmod +x collect-homelab-config.sh
+```
+
+## Security Best Practices
+
+1. **Storage:** Keep exports in secure locations
+   ```bash
+   # Set restrictive permissions
+   chmod 700 homelab-export-*/
+   ```
+
+2. **Git:** Use private repositories for unsanitized exports
+   ```bash
+   # Never push unsanitized configs to public repos
+   # Always review before pushing:
+   git diff --staged
+   ```
+
+3. **Sanitization:** Use `--sanitize all` for any exports that leave your network
+
+4. **Encryption:** Encrypt sensitive exports
+   ```bash
+   # Encrypt the archive
+   gpg --symmetric --cipher-algo AES256 homelab-export-*.tar.gz
+
+   # Decrypt when needed
+   gpg --decrypt homelab-export-*.tar.gz.gpg > homelab-export.tar.gz
+   ```
+
+5. **Cleanup:** Remove old exports
+   ```bash
+   # After archiving to secure backup storage
+   rm -rf homelab-export-20240101-*/
+   ```
+
+## Advanced Usage
+
+### Custom Collection Script
+
+Create a wrapper for your specific needs:
+
+```bash
+#!/bin/bash
+# /root/my-homelab-backup.sh
+
+BACKUP_ROOT="/backup/homelab"
+DATE=$(date +%Y%m%d)
+
+# Run collection
+/root/collect-homelab-config.sh \
+    --level full \
+    --output "${BACKUP_ROOT}/export-${DATE}" \
+    --sanitize all \
+    --verbose
+
+# Copy to NFS share
+cp "${BACKUP_ROOT}/export-${DATE}.tar.gz" /mnt/backups/
+
+# Upload to off-site (example)
+# rclone copy "${BACKUP_ROOT}/export-${DATE}.tar.gz" remote:homelab-backups/
+
+# Cleanup old exports (keep 30 days)
+find "${BACKUP_ROOT}" -name "export-*.tar.gz" -mtime +30 -delete
+
+echo "Backup completed: ${DATE}"
+```
+
+### Selective Collection
+
+To collect only specific components, modify the script or extract them manually:
+
+```bash
+# Just VM configs
+mkdir -p /tmp/just-vms
+cp homelab-export-*/configs/vms/* /tmp/just-vms/
+
+# Just network info
+cp -r homelab-export-*/configs/network /tmp/network-backup/
+```
+
+### Integration with Monitoring
+
+Alert if collection fails:
+
+```bash
+#!/bin/bash
+# Wrapper with monitoring
+
+if /root/collect-homelab-config.sh -l standard -o /backup/daily; then
+    echo "Homelab collection successful" | mail -s "Backup OK" admin@example.com
+else
+    echo "Homelab collection FAILED - check logs" | mail -s "Backup FAILED" admin@example.com
+fi
+```
+
+## Next Steps
+
+1. **Run your first collection:**
+   ```bash
+   ./collect-homelab-config.sh --level standard --verbose
+   ```
+
+2. **Review the output:**
+   ```bash
+   cat homelab-export-*/SUMMARY.md
+   less homelab-export-*/collection.log
+   ```
+
+3. **Set up automated collections** (see Automated Collection section)
+
+4. **Initialize a Git repository** for your exports (see Integration with Git section)
+
+5. **Create documentation** in the `docs/` folder within your exports
+
+6. **Build network diagrams** and save them in `diagrams/`
+
+7. **Use the configs** to start building Infrastructure as Code (Terraform, Ansible)
+
+## Support and Issues
+
+If you encounter issues:
+
+1. Run with `--verbose` flag for detailed output
+2. Review `collection.log` for error messages
+3. Check that you're running as root/sudo
+4. Verify you're on a Proxmox VE host (`cat /etc/pve/.version`)
+5. Ensure adequate disk space (`df -h`)
+
+---
+
+**Prepared by:** Your homelab automation assistant, Steve
+**Script Version:** 1.0.0
+**Last Updated:** 2024-11-28
--- a/scripts/crawlers-exporters/README-COLLECTION.md
+++ b/scripts/crawlers-exporters/README-COLLECTION.md
@@ -0,0 +1,341 @@
+# Homelab Infrastructure Collection Scripts
+
+Automated collection tools for documenting and backing up your Proxmox VE homelab infrastructure.
+
+## Quick Start
+
+### From WSL2 (Your Current Environment)
+
+The easiest way to collect your Proxmox configuration from WSL2:
+
+```bash
+# 1. Make scripts executable
+chmod +x collect-homelab-config.sh collect-remote.sh
+
+# 2. Set your Proxmox IP address
+PROXMOX_IP="192.168.1.100"  # Replace with your actual IP
+
+# 3. Run the remote collection
+./collect-remote.sh $PROXMOX_IP
+
+# 4. Review results
+ls -la homelab-export-*/
+cat homelab-export-*/SUMMARY.md
+```
+
+That's it! The script will SSH to your Proxmox host, collect all configurations, and download them to your WSL environment.
+
+### Directly on Proxmox Host
+
+If you prefer to run directly on the Proxmox server:
+
+```bash
+# 1. Copy script to Proxmox
+scp collect-homelab-config.sh root@<proxmox-ip>:/root/
+
+# 2. SSH to Proxmox
+ssh root@<proxmox-ip>
+
+# 3. Run collection
+chmod +x /root/collect-homelab-config.sh
+./collect-homelab-config.sh
+
+# 4. Copy results back to WSL
+# From WSL:
+scp -r root@<proxmox-ip>:/root/homelab-export-* ./
+```
+
+## What You Get
+
+After running the collection script, you'll have:
+
+```
+homelab-export-20241128-143022/
+├── README.md              # Detailed documentation
+├── SUMMARY.md            # Collection statistics
+├── collection.log        # Detailed log
+├── configs/
+│   ├── proxmox/         # Core Proxmox settings
+│   ├── vms/             # All VM configurations
+│   ├── lxc/             # All container configurations
+│   ├── storage/         # Storage pool configs
+│   ├── network/         # Network interface configs
+│   └── backup/          # Backup job definitions
+└── exports/
+    ├── system/          # System information
+    ├── cluster/         # Cluster status
+    └── guests/          # VM/container lists
+```
+
+## Files in This Directory
+
+| File | Purpose |
+|------|---------|
+| `collect-homelab-config.sh` | Main collection script (runs on Proxmox host) |
+| `collect-remote.sh` | Wrapper for remote execution from WSL/Linux |
+| `COLLECTION-GUIDE.md` | Comprehensive usage guide and reference |
+| `README-COLLECTION.md` | This file - quick start guide |
+
+## Common Use Cases
+
+### Daily Documentation Snapshot
+
+```bash
+# Run from WSL - executes on Proxmox and brings results back
+./collect-remote.sh 192.168.1.100 --level standard
+```
+
+### Pre-Maintenance Full Backup
+
+```bash
+# Full collection before making changes
+./collect-remote.sh 192.168.1.100 --level full --keep-remote
+```
+
+### Sanitized Export for Sharing
+
+```bash
+# Remove all sensitive data (IPs, passwords, tokens)
+./collect-remote.sh 192.168.1.100 --sanitize all
+```
+
+### Weekly Automated Collection
+
+On your Proxmox host, add to crontab:
+
+```bash
+# Edit crontab
+sudo crontab -e
+
+# Add weekly Sunday 3 AM collection
+0 3 * * 0 /root/collect-homelab-config.sh -l standard -o /backup/homelab/weekly-$(date +\%Y\%W)
+```
+
+## Collection Levels
+
+| Level | What It Includes | Best For |
+|-------|------------------|----------|
+| **basic** | System info, Proxmox configs, VM/LXC configs, network | Quick snapshots |
+| **standard** | Basic + storage, backups, cluster info | Regular documentation |
+| **full** | Standard + service configs, detailed state | Pre-maintenance |
+| **paranoid** | Everything possible | Maximum detail |
+
+## Prerequisites
+
+### SSH Access (Required for Remote Collection)
+
+Set up passwordless SSH authentication:
+
+```bash
+# Generate SSH key (if you don't have one)
+ssh-keygen -t ed25519
+
+# Copy to Proxmox host
+ssh-copy-id root@<proxmox-ip>
+
+# Test connection
+ssh root@<proxmox-ip> 'hostname'
+```
+
+### Disk Space
+
+Ensure adequate space:
+- On Proxmox: ~100-500 MB (depending on collection level)
+- On WSL: ~50-250 MB for compressed archives
+
+### Permissions
+
+The collection script needs root access on the Proxmox host to read all configurations.
+
+## Security Considerations
+
+### Default Sanitization
+
+By default, the script sanitizes:
+- ✓ Passwords
+- ✓ API tokens
+- ✗ IP addresses (preserved for documentation)
+
+### Full Sanitization
+
+For exports that leave your network:
+
+```bash
+./collect-remote.sh <proxmox-ip> --sanitize all
+```
+
+This will redact:
+- Passwords
+- API tokens
+- IP addresses (replaced with 10.x.x.x)
+
+### Storage Recommendations
+
+1. **Keep unsanitized exports secure** - store in encrypted locations
+2. **Use private Git repositories** - never commit to public repos without sanitization
+3. **Encrypt sensitive exports**:
+   ```bash
+   gpg --symmetric --cipher-algo AES256 homelab-export-*.tar.gz
+   ```
+
+## Next Steps
+
+1. **Run your first collection** using the Quick Start steps above
+
+2. **Review the comprehensive guide**:
+   ```bash
+   cat COLLECTION-GUIDE.md
+   ```
+
+3. **Examine your infrastructure**:
+   ```bash
+   # View VM configurations
+   cat homelab-export-*/configs/vms/*.conf
+
+   # Check storage setup
+   cat homelab-export-*/configs/proxmox/storage.cfg
+
+   # Review network configuration
+   cat homelab-export-*/configs/network/interfaces
+   ```
+
+4. **Set up version control**:
+   ```bash
+   cd homelab-export-<timestamp>/
+   git init
+   git add .
+   git commit -m "Initial infrastructure snapshot"
+   git remote add origin <your-private-repo>
+   git push -u origin main
+   ```
+
+5. **Schedule regular collections** (see COLLECTION-GUIDE.md for automation)
+
+6. **Use exports to build Infrastructure as Code** (Terraform, Ansible)
+
+## Troubleshooting
+
+### "Cannot connect to Proxmox host"
+
+Check:
+- Is the IP address correct?
+- Is SSH running on Proxmox? `ssh root@<proxmox-ip>`
+- Can you reach the host? `ping <proxmox-ip>`
+- Is firewall blocking port 22?
+
+### "Permission denied"
+
+Solutions:
+- Ensure scripts are executable: `chmod +x *.sh`
+- Use root user: `./collect-remote.sh 192.168.1.100 --user root`
+- Set up SSH keys: `ssh-copy-id root@<proxmox-ip>`
+
+### "Not a Proxmox VE host"
+
+This script requires a Proxmox VE installation. Verify:
+```bash
+ssh root@<proxmox-ip> 'cat /etc/pve/.version'
+```
+
+### "Some items skipped"
+
+This is normal! Review `SUMMARY.md` to see what was skipped. Common skipped items:
+- ZFS tools (if not using ZFS)
+- Cluster configs (on single-node setups)
+- HA configs (if High Availability not configured)
+
+### Need more help?
+
+1. Run with verbose flag: `./collect-remote.sh <proxmox-ip> -v`
+2. Check the log: `cat homelab-export-*/collection.log`
+3. Review the full guide: `cat COLLECTION-GUIDE.md`
+
+## Examples
+
+### Standard Collection from WSL
+
+```bash
+./collect-remote.sh 192.168.1.100
+```
+
+### Full Collection with Verbose Output
+
+```bash
+./collect-remote.sh 192.168.1.100 --level full --verbose
+```
+
+### Custom SSH Port and User
+
+```bash
+./collect-remote.sh proxmox.local --port 2222 --user admin
+```
+
+### Keep Remote Copy
+
+```bash
+./collect-remote.sh 192.168.1.100 --keep-remote
+```
+
+### Sanitized Export to Custom Location
+
+```bash
+./collect-remote.sh 192.168.1.100 --sanitize all --output ~/Documents/homelab-backups
+```
+
+## File Sizes
+
+Approximate sizes for reference:
+
+| Collection Level | Uncompressed | Compressed |
+|------------------|--------------|------------|
+| Basic | 5-15 MB | 1-3 MB |
+| Standard | 10-30 MB | 2-6 MB |
+| Full | 20-100 MB | 5-20 MB |
+| Paranoid | 50-500 MB | 10-50 MB |
+
+*Actual sizes vary based on number of VMs/containers and configurations*
+
+## Support
+
+For detailed information on any topic:
+
+- **Complete Usage Guide**: See `COLLECTION-GUIDE.md`
+- **Script Help**: Run `./collect-homelab-config.sh --help`
+- **Remote Wrapper Help**: Run `./collect-remote.sh --help`
+
+## What's Collected vs. What's NOT
+
+### ✓ Collected (READ-ONLY)
+- Configuration files
+- System information
+- Resource lists
+- Status information
+- Network settings
+- Storage definitions
+
+### ✗ NOT Collected
+- Actual VM/container disk images
+- ISO files
+- Backup archives
+- User data within VMs/containers
+- Authentication credentials (sanitized)
+
+**Note:** This script collects *configuration* and *metadata*, not actual data. For complete backups, use Proxmox Backup Server or vzdump.
+
+## Integration Ideas
+
+Once you have collected your infrastructure:
+
+1. **Documentation**: Use as source of truth for runbooks
+2. **Change Tracking**: Diff exports to see what changed over time
+3. **Disaster Recovery**: Reference during rebuilds
+4. **IaC Development**: Convert to Terraform/Ansible
+5. **Compliance**: Evidence of configuration state
+6. **Training**: Sandbox for learning without risk
+7. **Migration Planning**: Understand dependencies before moving
+
+---
+
+**Version:** 1.0.0
+**Last Updated:** 2024-11-28
+**Maintained By:** Your homelab automation assistant, Steve
--- a/scripts/crawlers-exporters/collect-homelab-config.sh
+++ b/scripts/crawlers-exporters/collect-homelab-config.sh
--- a/scripts/crawlers-exporters/collect-remote.sh
+++ b/scripts/crawlers-exporters/collect-remote.sh
@@ -0,0 +1,416 @@
+#!/usr/bin/env bash
+
+################################################################################
+# Remote Homelab Collection Wrapper
+# Purpose: Executes the collection script on a remote Proxmox host via SSH
+#          and retrieves the results back to your local machine (WSL/Linux)
+#
+# Usage: ./collect-remote.sh [PROXMOX_HOST] [OPTIONS]
+################################################################################
+
+set -euo pipefail
+
+# Color codes
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+CYAN='\033[0;36m'
+BOLD='\033[1m'
+NC='\033[0m'
+
+# Script configuration
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+COLLECTION_SCRIPT="${SCRIPT_DIR}/collect-homelab-config.sh"
+REMOTE_SCRIPT_PATH="/tmp/collect-homelab-config.sh"
+LOCAL_OUTPUT_DIR="${SCRIPT_DIR}"
+
+# SSH configuration
+SSH_USER="${SSH_USER:-root}"
+SSH_PORT="${SSH_PORT:-22}"
+SSH_OPTS="-o ConnectTimeout=10 -o StrictHostKeyChecking=no"
+
+################################################################################
+# Functions
+################################################################################
+
+log() {
+    local level="$1"
+    shift
+    local message="$*"
+
+    case "${level}" in
+        INFO)
+            echo -e "${BLUE}[INFO]${NC} ${message}"
+            ;;
+        SUCCESS)
+            echo -e "${GREEN}[✓]${NC} ${message}"
+            ;;
+        WARN)
+            echo -e "${YELLOW}[WARN]${NC} ${message}"
+            ;;
+        ERROR)
+            echo -e "${RED}[ERROR]${NC} ${message}" >&2
+            ;;
+    esac
+}
+
+banner() {
+    echo ""
+    echo -e "${BOLD}${CYAN}======================================================================${NC}"
+    echo -e "${BOLD}${CYAN}  $1${NC}"
+    echo -e "${BOLD}${CYAN}======================================================================${NC}"
+    echo ""
+}
+
+usage() {
+    cat <<EOF
+${BOLD}Remote Homelab Collection Wrapper${NC}
+
+${BOLD}USAGE:${NC}
+    $0 PROXMOX_HOST [OPTIONS]
+
+${BOLD}DESCRIPTION:${NC}
+    Executes the homelab collection script on a remote Proxmox host via SSH,
+    then retrieves the results back to your local machine.
+
+${BOLD}ARGUMENTS:${NC}
+    PROXMOX_HOST        IP address or hostname of your Proxmox server
+
+${BOLD}OPTIONS:${NC}
+    -u, --user USER     SSH username (default: root)
+    -p, --port PORT     SSH port (default: 22)
+    -l, --level LEVEL   Collection level: basic, standard, full, paranoid
+                        (default: standard)
+    -s, --sanitize OPT  Sanitization: all, ips, none (default: passwords/tokens only)
+    -o, --output DIR    Local directory to store results (default: current directory)
+    -k, --keep-remote   Keep the export on the remote host (default: remove after download)
+    -v, --verbose       Verbose output
+    -h, --help          Show this help message
+
+${BOLD}ENVIRONMENT VARIABLES:${NC}
+    SSH_USER            Default SSH username (default: root)
+    SSH_PORT            Default SSH port (default: 22)
+
+${BOLD}EXAMPLES:${NC}
+    # Basic usage
+    $0 192.168.1.100
+
+    # Full collection with complete sanitization
+    $0 192.168.1.100 --level full --sanitize all
+
+    # Custom SSH user and port
+    $0 proxmox.local --user admin --port 2222
+
+    # Keep results on remote host
+    $0 192.168.1.100 --keep-remote
+
+    # Verbose output with custom output directory
+    $0 192.168.1.100 -v -o ~/backups/homelab
+
+${BOLD}PREREQUISITES:${NC}
+    1. SSH access to the Proxmox host
+    2. collect-homelab-config.sh in the same directory as this script
+    3. Sufficient disk space on both remote and local machines
+
+${BOLD}WORKFLOW:${NC}
+    1. Copies collection script to remote Proxmox host
+    2. Executes the script remotely
+    3. Downloads the compressed archive to local machine
+    4. Optionally removes the remote copy
+    5. Extracts the archive locally
+
+${BOLD}NOTES:${NC}
+    - Requires passwordless SSH or SSH key authentication (recommended)
+    - The script will be run as the specified SSH user (typically root)
+    - Remote execution output is displayed in real-time
+
+EOF
+}
+
+check_prerequisites() {
+    # Check if collection script exists
+    if [[ ! -f "${COLLECTION_SCRIPT}" ]]; then
+        log ERROR "Collection script not found: ${COLLECTION_SCRIPT}"
+        log ERROR "Ensure collect-homelab-config.sh is in the same directory as this script"
+        exit 1
+    fi
+
+    # Check if ssh is available
+    if ! command -v ssh &> /dev/null; then
+        log ERROR "SSH client not found. Please install openssh-client"
+        exit 1
+    fi
+
+    # Check if scp is available
+    if ! command -v scp &> /dev/null; then
+        log ERROR "SCP not found. Please install openssh-client"
+        exit 1
+    fi
+}
+
+test_ssh_connection() {
+    local host="$1"
+
+    log INFO "Testing SSH connection to ${SSH_USER}@${host}:${SSH_PORT}..."
+
+    if ssh ${SSH_OPTS} -p "${SSH_PORT}" "${SSH_USER}@${host}" "exit 0" 2>/dev/null; then
+        log SUCCESS "SSH connection successful"
+        return 0
+    else
+        log ERROR "Cannot connect to ${SSH_USER}@${host}:${SSH_PORT}"
+        log ERROR "Possible issues:"
+        log ERROR "  - Host is unreachable"
+        log ERROR "  - SSH service is not running"
+        log ERROR "  - Incorrect credentials"
+        log ERROR "  - Firewall blocking connection"
+        log ERROR ""
+        log ERROR "Try manually: ssh -p ${SSH_PORT} ${SSH_USER}@${host}"
+        return 1
+    fi
+}
+
+verify_proxmox_host() {
+    local host="$1"
+
+    log INFO "Verifying Proxmox installation on remote host..."
+
+    if ssh ${SSH_OPTS} -p "${SSH_PORT}" "${SSH_USER}@${host}" "test -f /etc/pve/.version" 2>/dev/null; then
+        local pve_version=$(ssh ${SSH_OPTS} -p "${SSH_PORT}" "${SSH_USER}@${host}" "cat /etc/pve/.version" 2>/dev/null)
+        log SUCCESS "Confirmed Proxmox VE installation (version: ${pve_version})"
+        return 0
+    else
+        log WARN "Remote host does not appear to be a Proxmox VE server"
+        log WARN "Proceeding anyway, but collection may fail..."
+        return 0
+    fi
+}
+
+upload_script() {
+    local host="$1"
+
+    banner "Uploading Collection Script"
+
+    log INFO "Copying collection script to ${host}..."
+
+    if scp ${SSH_OPTS} -P "${SSH_PORT}" "${COLLECTION_SCRIPT}" "${SSH_USER}@${host}:${REMOTE_SCRIPT_PATH}"; then
+        log SUCCESS "Script uploaded successfully"
+
+        # Make executable
+        ssh ${SSH_OPTS} -p "${SSH_PORT}" "${SSH_USER}@${host}" "chmod +x ${REMOTE_SCRIPT_PATH}"
+        log SUCCESS "Script permissions set"
+        return 0
+    else
+        log ERROR "Failed to upload script"
+        return 1
+    fi
+}
+
+execute_remote_collection() {
+    local host="$1"
+    shift
+    local collection_args=("$@")
+
+    banner "Executing Collection on Remote Host"
+
+    log INFO "Running collection script on ${host}..."
+    log INFO "Arguments: ${collection_args[*]}"
+
+    # Build the remote command
+    local remote_cmd="${REMOTE_SCRIPT_PATH} ${collection_args[*]}"
+
+    # Execute remotely and stream output
+    if ssh ${SSH_OPTS} -p "${SSH_PORT}" "${SSH_USER}@${host}" "${remote_cmd}"; then
+        log SUCCESS "Collection completed successfully on remote host"
+        return 0
+    else
+        log ERROR "Collection failed on remote host"
+        return 1
+    fi
+}
+
+download_results() {
+    local host="$1"
+    local output_dir="$2"
+
+    banner "Downloading Results"
+
+    log INFO "Finding remote export archive..."
+
+    # Find the most recent export archive
+    local remote_archive=$(ssh ${SSH_OPTS} -p "${SSH_PORT}" "${SSH_USER}@${host}" \
+        "ls -t /root/homelab-export-*.tar.gz 2>/dev/null | head -1" 2>/dev/null)
+
+    if [[ -z "${remote_archive}" ]]; then
+        log ERROR "No export archive found on remote host"
+        log ERROR "Collection may have failed or compression was disabled"
+        return 1
+    fi
+
+    log INFO "Found archive: ${remote_archive}"
+
+    # Create output directory
+    mkdir -p "${output_dir}"
+
+    # Download the archive
+    local local_archive="${output_dir}/$(basename "${remote_archive}")"
+
+    log INFO "Downloading to: ${local_archive}"
+
+    if scp ${SSH_OPTS} -P "${SSH_PORT}" "${SSH_USER}@${host}:${remote_archive}" "${local_archive}"; then
+        log SUCCESS "Archive downloaded successfully"
+
+        # Extract the archive
+        log INFO "Extracting archive..."
+        if tar -xzf "${local_archive}" -C "${output_dir}"; then
+            log SUCCESS "Archive extracted to: ${output_dir}/$(basename "${local_archive}" .tar.gz)"
+
+            # Show summary
+            local extracted_dir="${output_dir}/$(basename "${local_archive}" .tar.gz)"
+            if [[ -f "${extracted_dir}/SUMMARY.md" ]]; then
+                echo ""
+                log INFO "Collection Summary:"
+                echo ""
+                head -30 "${extracted_dir}/SUMMARY.md"
+                echo ""
+                log INFO "Full summary: ${extracted_dir}/SUMMARY.md"
+            fi
+
+            return 0
+        else
+            log ERROR "Failed to extract archive"
+            return 1
+        fi
+    else
+        log ERROR "Failed to download archive"
+        return 1
+    fi
+}
+
+cleanup_remote() {
+    local host="$1"
+    local keep_remote="$2"
+
+    if [[ "${keep_remote}" == "true" ]]; then
+        log INFO "Keeping export on remote host (--keep-remote specified)"
+        return 0
+    fi
+
+    banner "Cleaning Up Remote Host"
+
+    log INFO "Removing export files from remote host..."
+
+    # Remove the script
+    ssh ${SSH_OPTS} -p "${SSH_PORT}" "${SSH_USER}@${host}" "rm -f ${REMOTE_SCRIPT_PATH}" 2>/dev/null || true
+
+    # Remove export directories and archives
+    ssh ${SSH_OPTS} -p "${SSH_PORT}" "${SSH_USER}@${host}" \
+        "rm -rf /root/homelab-export-* 2>/dev/null" 2>/dev/null || true
+
+    log SUCCESS "Remote cleanup completed"
+}
+
+################################################################################
+# Main Execution
+################################################################################
+
+main() {
+    # Parse arguments
+    if [[ $# -eq 0 ]]; then
+        usage
+        exit 1
+    fi
+
+    local proxmox_host=""
+    local collection_level="standard"
+    local sanitize_option=""
+    local keep_remote="false"
+    local verbose="false"
+
+    # First argument is the host
+    proxmox_host="$1"
+    shift
+
+    # Parse remaining options
+    local collection_args=()
+
+    while [[ $# -gt 0 ]]; do
+        case "$1" in
+            -u|--user)
+                SSH_USER="$2"
+                shift 2
+                ;;
+            -p|--port)
+                SSH_PORT="$2"
+                shift 2
+                ;;
+            -l|--level)
+                collection_level="$2"
+                collection_args+=("--level" "$2")
+                shift 2
+                ;;
+            -s|--sanitize)
+                sanitize_option="$2"
+                collection_args+=("--sanitize" "$2")
+                shift 2
+                ;;
+            -o|--output)
+                LOCAL_OUTPUT_DIR="$2"
+                shift 2
+                ;;
+            -k|--keep-remote)
+                keep_remote="true"
+                shift
+                ;;
+            -v|--verbose)
+                verbose="true"
+                collection_args+=("--verbose")
+                shift
+                ;;
+            -h|--help)
+                usage
+                exit 0
+                ;;
+            *)
+                log ERROR "Unknown option: $1"
+                usage
+                exit 1
+                ;;
+        esac
+    done
+
+    # Validate host
+    if [[ -z "${proxmox_host}" ]]; then
+        log ERROR "Proxmox host not specified"
+        usage
+        exit 1
+    fi
+
+    # Display configuration
+    banner "Remote Homelab Collection"
+    echo -e "${BOLD}Target Host:${NC}     ${proxmox_host}"
+    echo -e "${BOLD}SSH User:${NC}        ${SSH_USER}"
+    echo -e "${BOLD}SSH Port:${NC}        ${SSH_PORT}"
+    echo -e "${BOLD}Collection Level:${NC} ${collection_level}"
+    echo -e "${BOLD}Output Directory:${NC} ${LOCAL_OUTPUT_DIR}"
+    echo -e "${BOLD}Keep Remote:${NC}     ${keep_remote}"
+    echo ""
+
+    # Execute workflow
+    check_prerequisites
+    test_ssh_connection "${proxmox_host}" || exit 1
+    verify_proxmox_host "${proxmox_host}"
+    upload_script "${proxmox_host}" || exit 1
+    execute_remote_collection "${proxmox_host}" "${collection_args[@]}" || exit 1
+    download_results "${proxmox_host}" "${LOCAL_OUTPUT_DIR}" || exit 1
+    cleanup_remote "${proxmox_host}" "${keep_remote}"
+
+    banner "Collection Complete"
+
+    log SUCCESS "Homelab infrastructure export completed successfully"
+    log INFO "Results are available in: ${LOCAL_OUTPUT_DIR}"
+    echo ""
+}
+
+# Run main function
+main "$@"
--- a/scripts/crawlers-exporters/collect.sh
+++ b/scripts/crawlers-exporters/collect.sh
@@ -0,0 +1,158 @@
+#!/usr/bin/env bash
+
+################################################################################
+# Homelab Collection Convenience Wrapper
+# Purpose: Easy-to-use wrapper that loads settings from .env file
+################################################################################
+
+set -euo pipefail
+
+# Color codes
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+RED='\033[0;31m'
+BOLD='\033[1m'
+NC='\033[0m'
+
+# Script directory
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# Load .env file if it exists
+if [[ -f "${SCRIPT_DIR}/.env" ]]; then
+    echo -e "${BLUE}[INFO]${NC} Loading configuration from .env file..."
+    # shellcheck disable=SC1091
+    source "${SCRIPT_DIR}/.env"
+else
+    echo -e "${YELLOW}[WARN]${NC} No .env file found. Using defaults and command-line arguments."
+    echo -e "${YELLOW}[WARN]${NC} Copy .env.example to .env and customize for your environment."
+    echo ""
+fi
+
+# Set defaults if not in .env
+PROXMOX_HOST="${PROXMOX_HOST:-}"
+PROXMOX_SSH_USER="${PROXMOX_SSH_USER:-root}"
+PROXMOX_SSH_PORT="${PROXMOX_SSH_PORT:-22}"
+COLLECTION_LEVEL="${COLLECTION_LEVEL:-standard}"
+OUTPUT_DIR="${OUTPUT_DIR:-./exports}"
+KEEP_REMOTE="${KEEP_REMOTE:-false}"
+VERBOSE="${VERBOSE:-false}"
+
+# Allow command-line override
+SANITIZE_ARG=""
+
+usage() {
+    cat <<EOF
+${BOLD}Homelab Collection Convenience Wrapper${NC}
+
+${BOLD}USAGE:${NC}
+    $0 [OPTIONS]
+
+${BOLD}DESCRIPTION:${NC}
+    Convenience wrapper that loads settings from .env file and executes
+    the remote collection script.
+
+${BOLD}OPTIONS:${NC}
+    -h, --host HOST     Override Proxmox host from .env
+    -l, --level LEVEL   Override collection level (basic, standard, full, paranoid)
+    -s, --sanitize OPT  Sanitization option (all, ips, none)
+    -v, --verbose       Enable verbose output
+    -k, --keep-remote   Keep export on remote host
+    --help              Show this help
+
+${BOLD}CONFIGURATION:${NC}
+    Settings are loaded from .env file in the same directory.
+    Create .env from .env.example and customize for your environment.
+
+${BOLD}EXAMPLES:${NC}
+    # Use settings from .env file
+    $0
+
+    # Override specific settings
+    $0 --level full --verbose
+
+    # Override host
+    $0 --host 192.168.1.200
+
+    # Full sanitization
+    $0 --sanitize all
+
+${BOLD}SETUP:${NC}
+    1. Copy .env.example to .env
+    2. Edit .env with your Proxmox details
+    3. Run this script: $0
+
+EOF
+}
+
+# Parse command-line arguments (they override .env)
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        -h|--host)
+            PROXMOX_HOST="$2"
+            shift 2
+            ;;
+        -l|--level)
+            COLLECTION_LEVEL="$2"
+            shift 2
+            ;;
+        -s|--sanitize)
+            SANITIZE_ARG="--sanitize $2"
+            shift 2
+            ;;
+        -v|--verbose)
+            VERBOSE="true"
+            shift
+            ;;
+        -k|--keep-remote)
+            KEEP_REMOTE="true"
+            shift
+            ;;
+        --help)
+            usage
+            exit 0
+            ;;
+        *)
+            echo -e "${RED}[ERROR]${NC} Unknown option: $1"
+            usage
+            exit 1
+            ;;
+    esac
+done
+
+# Validate required settings
+if [[ -z "${PROXMOX_HOST}" ]]; then
+    echo -e "${RED}[ERROR]${NC} PROXMOX_HOST not set"
+    echo -e "${RED}[ERROR]${NC} Either set it in .env or use --host option"
+    echo ""
+    usage
+    exit 1
+fi
+
+# Build command
+CMD=("${SCRIPT_DIR}/collect-remote.sh" "${PROXMOX_HOST}")
+CMD+=("--user" "${PROXMOX_SSH_USER}")
+CMD+=("--port" "${PROXMOX_SSH_PORT}")
+CMD+=("--level" "${COLLECTION_LEVEL}")
+CMD+=("--output" "${OUTPUT_DIR}")
+
+[[ -n "${SANITIZE_ARG}" ]] && CMD+=("${SANITIZE_ARG}")
+[[ "${VERBOSE}" == "true" ]] && CMD+=("--verbose")
+[[ "${KEEP_REMOTE}" == "true" ]] && CMD+=("--keep-remote")
+
+# Display configuration
+echo -e "${BOLD}${BLUE}Configuration:${NC}"
+echo -e "  Host:       ${PROXMOX_HOST}"
+echo -e "  SSH User:   ${PROXMOX_SSH_USER}"
+echo -e "  SSH Port:   ${PROXMOX_SSH_PORT}"
+echo -e "  Level:      ${COLLECTION_LEVEL}"
+echo -e "  Output:     ${OUTPUT_DIR}"
+echo -e "  Verbose:    ${VERBOSE}"
+echo -e "  Keep Remote: ${KEEP_REMOTE}"
+echo ""
+
+# Execute
+echo -e "${GREEN}[INFO]${NC} Executing: ${CMD[*]}"
+echo ""
+
+exec "${CMD[@]}"