homelab/scripts/crawlers-exporters/README-COLLECTION.md

# Homelab Infrastructure Collection Scripts

Automated collection tools for documenting and backing up your Proxmox VE homelab infrastructure.

## Quick Start

### From WSL2 (Your Current Environment)

The easiest way to collect your Proxmox configuration from WSL2:

```bash
# 1. Make scripts executable
chmod +x collect-homelab-config.sh collect-remote.sh

# 2. Set your Proxmox IP address
PROXMOX_IP="192.168.1.100"  # Replace with your actual IP

# 3. Run the remote collection
./collect-remote.sh $PROXMOX_IP

# 4. Review results
ls -la homelab-export-*/
cat homelab-export-*/SUMMARY.md
```

That's it! The script will SSH to your Proxmox host, collect all configurations, and download them to your WSL environment.

### Directly on Proxmox Host

If you prefer to run directly on the Proxmox server:

```bash
# 1. Copy script to Proxmox
scp collect-homelab-config.sh root@<proxmox-ip>:/root/

# 2. SSH to Proxmox
ssh root@<proxmox-ip>

# 3. Run collection
chmod +x /root/collect-homelab-config.sh
./collect-homelab-config.sh

# 4. Copy results back to WSL
# From WSL:
scp -r root@<proxmox-ip>:/root/homelab-export-* ./
```

## What You Get

After running the collection script, you'll have:

```
homelab-export-20241128-143022/
├── README.md              # Detailed documentation
├── SUMMARY.md            # Collection statistics
├── collection.log        # Detailed log
├── configs/
│   ├── proxmox/         # Core Proxmox settings
│   ├── vms/             # All VM configurations
│   ├── lxc/             # All container configurations
│   ├── storage/         # Storage pool configs
│   ├── network/         # Network interface configs
│   └── backup/          # Backup job definitions
└── exports/
    ├── system/          # System information
    ├── cluster/         # Cluster status
    └── guests/          # VM/container lists
```

## Files in This Directory

| File | Purpose |
|------|---------|
| `collect-homelab-config.sh` | Main collection script (runs on Proxmox host) |
| `collect-remote.sh` | Wrapper for remote execution from WSL/Linux |
| `COLLECTION-GUIDE.md` | Comprehensive usage guide and reference |
| `README-COLLECTION.md` | This file - quick start guide |

## Common Use Cases

### Daily Documentation Snapshot

```bash
# Run from WSL - executes on Proxmox and brings results back
./collect-remote.sh 192.168.1.100 --level standard
```

### Pre-Maintenance Full Backup

```bash
# Full collection before making changes
./collect-remote.sh 192.168.1.100 --level full --keep-remote
```

### Sanitized Export for Sharing

```bash
# Remove all sensitive data (IPs, passwords, tokens)
./collect-remote.sh 192.168.1.100 --sanitize all
```

### Weekly Automated Collection

On your Proxmox host, add to crontab:

```bash
# Edit crontab
sudo crontab -e

# Add weekly Sunday 3 AM collection
0 3 * * 0 /root/collect-homelab-config.sh -l standard -o /backup/homelab/weekly-$(date +\%Y\%W)
```

## Collection Levels

| Level | What It Includes | Best For |
|-------|------------------|----------|
| **basic** | System info, Proxmox configs, VM/LXC configs, network | Quick snapshots |
| **standard** | Basic + storage, backups, cluster info | Regular documentation |
| **full** | Standard + service configs, detailed state | Pre-maintenance |
| **paranoid** | Everything possible | Maximum detail |

## Prerequisites

### SSH Access (Required for Remote Collection)

Set up passwordless SSH authentication:

```bash
# Generate SSH key (if you don't have one)
ssh-keygen -t ed25519

# Copy to Proxmox host
ssh-copy-id root@<proxmox-ip>

# Test connection
ssh root@<proxmox-ip> 'hostname'
```

### Disk Space

Ensure adequate space:
- On Proxmox: ~100-500 MB (depending on collection level)
- On WSL: ~50-250 MB for compressed archives

### Permissions

The collection script needs root access on the Proxmox host to read all configurations.

## Security Considerations

### Default Sanitization

By default, the script sanitizes:
- ✓ Passwords
- ✓ API tokens
- ✗ IP addresses (preserved for documentation)

### Full Sanitization

For exports that leave your network:

```bash
./collect-remote.sh <proxmox-ip> --sanitize all
```

This will redact:
- Passwords
- API tokens
- IP addresses (replaced with 10.x.x.x)

### Storage Recommendations

1. **Keep unsanitized exports secure** - store in encrypted locations
2. **Use private Git repositories** - never commit to public repos without sanitization
3. **Encrypt sensitive exports**:
   ```bash
   gpg --symmetric --cipher-algo AES256 homelab-export-*.tar.gz
   ```

## Next Steps

1. **Run your first collection** using the Quick Start steps above

2. **Review the comprehensive guide**:
   ```bash
   cat COLLECTION-GUIDE.md
   ```

3. **Examine your infrastructure**:
   ```bash
   # View VM configurations
   cat homelab-export-*/configs/vms/*.conf

   # Check storage setup
   cat homelab-export-*/configs/proxmox/storage.cfg

   # Review network configuration
   cat homelab-export-*/configs/network/interfaces
   ```

4. **Set up version control**:
   ```bash
   cd homelab-export-<timestamp>/
   git init
   git add .
   git commit -m "Initial infrastructure snapshot"
   git remote add origin <your-private-repo>
   git push -u origin main
   ```

5. **Schedule regular collections** (see COLLECTION-GUIDE.md for automation)

6. **Use exports to build Infrastructure as Code** (Terraform, Ansible)

## Troubleshooting

### "Cannot connect to Proxmox host"

Check:
- Is the IP address correct?
- Is SSH running on Proxmox? `ssh root@<proxmox-ip>`
- Can you reach the host? `ping <proxmox-ip>`
- Is firewall blocking port 22?

### "Permission denied"

Solutions:
- Ensure scripts are executable: `chmod +x *.sh`
- Use root user: `./collect-remote.sh 192.168.1.100 --user root`
- Set up SSH keys: `ssh-copy-id root@<proxmox-ip>`

### "Not a Proxmox VE host"

This script requires a Proxmox VE installation. Verify:
```bash
ssh root@<proxmox-ip> 'cat /etc/pve/.version'
```

### "Some items skipped"

This is normal! Review `SUMMARY.md` to see what was skipped. Common skipped items:
- ZFS tools (if not using ZFS)
- Cluster configs (on single-node setups)
- HA configs (if High Availability not configured)

### Need more help?

1. Run with verbose flag: `./collect-remote.sh <proxmox-ip> -v`
2. Check the log: `cat homelab-export-*/collection.log`
3. Review the full guide: `cat COLLECTION-GUIDE.md`

## Examples

### Standard Collection from WSL

```bash
./collect-remote.sh 192.168.1.100
```

### Full Collection with Verbose Output

```bash
./collect-remote.sh 192.168.1.100 --level full --verbose
```

### Custom SSH Port and User

```bash
./collect-remote.sh proxmox.local --port 2222 --user admin
```

### Keep Remote Copy

```bash
./collect-remote.sh 192.168.1.100 --keep-remote
```

### Sanitized Export to Custom Location

```bash
./collect-remote.sh 192.168.1.100 --sanitize all --output ~/Documents/homelab-backups
```

## File Sizes

Approximate sizes for reference:

| Collection Level | Uncompressed | Compressed |
|------------------|--------------|------------|
| Basic | 5-15 MB | 1-3 MB |
| Standard | 10-30 MB | 2-6 MB |
| Full | 20-100 MB | 5-20 MB |
| Paranoid | 50-500 MB | 10-50 MB |

*Actual sizes vary based on number of VMs/containers and configurations*

## Support

For detailed information on any topic:

- **Complete Usage Guide**: See `COLLECTION-GUIDE.md`
- **Script Help**: Run `./collect-homelab-config.sh --help`
- **Remote Wrapper Help**: Run `./collect-remote.sh --help`

## What's Collected vs. What's NOT

### ✓ Collected (READ-ONLY)
- Configuration files
- System information
- Resource lists
- Status information
- Network settings
- Storage definitions

### ✗ NOT Collected
- Actual VM/container disk images
- ISO files
- Backup archives
- User data within VMs/containers
- Authentication credentials (sanitized)

**Note:** This script collects *configuration* and *metadata*, not actual data. For complete backups, use Proxmox Backup Server or vzdump.

## Integration Ideas

Once you have collected your infrastructure:

1. **Documentation**: Use as source of truth for runbooks
2. **Change Tracking**: Diff exports to see what changed over time
3. **Disaster Recovery**: Reference during rebuilds
4. **IaC Development**: Convert to Terraform/Ansible
5. **Compliance**: Evidence of configuration state
6. **Training**: Sandbox for learning without risk
7. **Migration Planning**: Understand dependencies before moving

---

**Version:** 1.0.0
**Last Updated:** 2024-11-28
**Maintained By:** Your homelab automation assistant, Steve