homelab/start-here-docs/SCRIPT-USAGE.md

# Homelab Collection Script - Quick Reference

## Overview

The `collect-homelab-config.sh` script performs comprehensive, read-only collection of your Proxmox VE infrastructure configuration and exports it in an organized, documented format.

**Status**: ✅ **Fully Operational** (bugs fixed as of 2025-11-29)

---

## Quick Start

### On Proxmox Host (Direct Execution)

```bash
# Copy script to Proxmox host
scp collect-homelab-config.sh root@192.168.2.100:/tmp/

# SSH to Proxmox host
ssh root@192.168.2.100

# Run with default settings
cd /tmp
bash collect-homelab-config.sh

# Or with verbose output
bash collect-homelab-config.sh --verbose
```

### From Your Workstation (Remote Execution)

```bash
# Upload and execute in one command
scp collect-homelab-config.sh root@192.168.2.100:/tmp/ && \
ssh root@192.168.2.100 'cd /tmp && bash /tmp/collect-homelab-config.sh'

# Download the results
scp root@192.168.2.100:/tmp/homelab-export-*.tar.gz ./
```

---

## Command-Line Options

### Collection Levels

```bash
# Basic: System info + Proxmox/VM/CT configs only
./collect-homelab-config.sh --level basic

# Standard: Basic + storage, network, backup, cluster info (DEFAULT)
./collect-homelab-config.sh --level standard

# Full: Standard + service configs, detailed system state
./collect-homelab-config.sh --level full

# Paranoid: Everything possible (experimental)
./collect-homelab-config.sh --level paranoid
```

### Sanitization Options

```bash
# Sanitize everything (IPs, passwords, tokens)
./collect-homelab-config.sh --sanitize all

# Sanitize only IP addresses
./collect-homelab-config.sh --sanitize ips

# No sanitization (CAUTION: sensitive data will be in clear text)
./collect-homelab-config.sh --sanitize none

# Default: Passwords and tokens sanitized, IPs preserved
```

### Output Control

```bash
# Custom output directory
./collect-homelab-config.sh --output /backup/my-homelab-export

# Disable compression
./collect-homelab-config.sh --no-compress

# Enable compression (default)
./collect-homelab-config.sh --compress
```

### Debugging

```bash
# Verbose output (shows DEBUG messages)
./collect-homelab-config.sh --verbose

# Standard output (INFO/SUCCESS/WARN/ERROR only)
./collect-homelab-config.sh
```

### Combined Examples

```bash
# Full collection with complete sanitization and verbose output
./collect-homelab-config.sh --level full --sanitize all --verbose

# Basic collection to specific directory, no compression
./collect-homelab-config.sh --level basic --output /root/configs --no-compress

# Standard collection with IP sanitization
./collect-homelab-config.sh --sanitize all --verbose
```

---

## What Gets Collected

### ✅ Always Collected (All Levels)

**System Information**:
- Proxmox VE version
- Hostname, kernel, uptime
- CPU, memory, disk information
- LVM configuration (PV/VG/LV)
- Network interfaces and routing
- Listening services
- Installed packages

**Proxmox Configurations**:
- Datacenter settings (`datacenter.cfg`)
- Storage pools (`storage.cfg`)
- Users and permissions (`user.cfg`)
- Authentication keys (`authkey.pub`)
- Firewall rules (if configured)
- Corosync cluster config (if in cluster)
- HA configuration (if configured)

**VM Configurations**:
- All QEMU/KVM VM configs (`/etc/pve/nodes/*/qemu-server/*.conf`)
- VM firewall rules
- Format: `{VMID}-{VM-name}.conf`

**LXC Container Configurations**:
- All LXC container configs (`/etc/pve/nodes/*/lxc/*.conf`)
- Container firewall rules
- Format: `{CTID}-{container-name}.conf`

**Network Configurations**:
- Interface definitions (`/etc/network/interfaces`)
- Additional interface configs (`/etc/network/interfaces.d/`)
- SDN configuration (if configured)
- Hosts file
- DNS resolver config

**Storage Information**:
- Storage status (`pvesm status`)
- ZFS pools and datasets (if ZFS is used)
- NFS exports (if configured)
- Samba configuration (if configured)
- iSCSI initiator config (if configured)

### ➕ Standard Level and Above

**Backup Configurations**:
- Vzdump configuration
- Scheduled backup jobs
- Proxmox Backup Server integration

**Cluster Information**:
- Cluster status and membership
- Resource allocation
- Recent tasks history

**Guest Information**:
- VM and container lists
- Resource usage statistics
- JSON exports for programmatic access

### ➕ Full Level and Above

**Service Configurations**:
- Systemd service status
- Proxmox-specific service states
  - pve-cluster
  - pvedaemon
  - pveproxy
  - pvestatd
  - pve-firewall
  - pvescheduler

---

## Output Structure

```
homelab-export-20251129-141328/
├── README.md                    # Comprehensive documentation
├── SUMMARY.md                   # Collection statistics and overview
├── collection.log               # Detailed execution log
├── configs/                     # Configuration files
│   ├── proxmox/                # Proxmox VE configurations
│   │   ├── datacenter.cfg
│   │   ├── storage.cfg
│   │   ├── user.cfg
│   │   └── authkey.pub
│   ├── vms/                    # Virtual machine configs
│   │   ├── 100-docker-hub.conf
│   │   ├── 101-gitlab.conf
│   │   └── ...
│   ├── lxc/                    # LXC container configs
│   │   ├── 102-nginx.conf
│   │   ├── 103-netbox.conf
│   │   └── 112-Anytype.conf
│   ├── storage/                # Storage configurations
│   ├── network/                # Network configurations
│   ├── backup/                 # Backup job configurations
│   └── services/               # System service configs (full level)
├── exports/                     # System state exports
│   ├── system/                 # System information
│   │   ├── pve-version.txt
│   │   ├── cpuinfo.txt
│   │   ├── meminfo.txt
│   │   └── ...
│   ├── cluster/                # Cluster status and resources
│   └── guests/                 # Guest VM/CT information
├── docs/                        # For manual documentation additions
├── scripts/                     # For automation scripts
└── diagrams/                    # For network diagrams
```

---

## Sanitization Behavior

### Default Sanitization (Passwords + Tokens)

**Patterns Removed**:
- `password=*` → `password=<REDACTED>`
- `passwd=*` → `passwd=<REDACTED>`
- `"password": "*"` → `"password": "<REDACTED>"`
- `token=*` → `token=<REDACTED>`
- `api_key=*` → `api_key=<REDACTED>`
- `secret=*` → `secret=<REDACTED>`

**IP Addresses**: Preserved (not sanitized)

### All Sanitization (--sanitize all)

Everything above PLUS:
- All IPv4 addresses → `10.X.X.X`

### No Sanitization (--sanitize none)

⚠️ **DANGER**: All sensitive data remains in clear text. Use only for:
- Air-gapped environments
- Encrypted storage destinations
- Troubleshooting purposes

---

## Expected Output

### Success Indicators

```
================================================================================
  Collection Complete
================================================================================

[✓] Total items collected: 50
[INFO] Total items skipped: 1
[WARN] Total errors: 5

Export Location: ./homelab-export-20251129-141328
Summary Report: ./homelab-export-20251129-141328/SUMMARY.md
Collection Log: ./homelab-export-20251129-141328/collection.log
```

### Typical Statistics

- **Items Collected**: 45-60 (depending on your infrastructure)
- **Items Skipped**: 0-5 (files that don't exist, like domains.cfg)
- **Errors**: 0-10 (warnings for optional configs that don't exist)
- **Archive Size**: 30-100KB (compressed)
- **Execution Time**: 5-15 seconds

### Common Warnings (Non-Critical)

```
[WARN] Failed to copy directory HA configuration from /etc/pve/ha
[WARN] Failed to copy directory Additional interface configs from /etc/network/interfaces.d
[WARN] Failed to copy directory SDN configuration from /etc/pve/sdn
[WARN] Failed to execute: pvecm status (Cluster status)
[WARN] Failed to execute: pvecm nodes (Cluster nodes)
```

These are **expected** if you don't have:
- High Availability (HA) configured
- Additional network interface configs
- Software Defined Networking (SDN)
- Multi-node cluster

---

## Post-Collection Workflow

### 1. Review the Collection

```bash
# Extract the archive
tar -xzf homelab-export-20251129-141328.tar.gz

# Review the summary
cat homelab-export-20251129-141328/SUMMARY.md

# Check for errors
grep -E "\[ERROR\]|\[WARN\]" homelab-export-20251129-141328/collection.log
```

### 2. Verify Sensitive Data

```bash
# Verify passwords are sanitized
grep -r "password=" homelab-export-20251129-141328/configs/

# Should show: password=<REDACTED>
```

### 3. Version Control

```bash
# Initialize git (if not already done)
cd /mnt/c/Users/fam1n/Documents/homelab
git init

# Add the export
git add homelab-export-20251129-141328/
git commit -m "Infrastructure snapshot: 2025-11-29"

# Or just commit the archive
git add homelab-export-20251129-141328.tar.gz
git commit -m "Homelab export archive: 2025-11-29"
```

### 4. Documentation Enhancement

Add custom documentation to the collected export:

```bash
cd homelab-export-20251129-141328

# Add network diagrams
cp ~/network-diagram.png diagrams/

# Add runbooks
cat > docs/disaster-recovery.md <<EOF
# Disaster Recovery Procedures
...
EOF

# Add automation scripts
cp ~/ansible-playbooks/*.yml scripts/
```

### 5. Scheduled Collection

Create a cron job on your Proxmox host:

```bash
# Run collection monthly and keep archives
cat > /etc/cron.d/homelab-export <<'EOF'
# Monthly infrastructure export
0 2 1 * * root cd /backup && /opt/scripts/collect-homelab-config.sh --level standard --output /backup/homelab-export-$(date +\%Y\%m) --compress
EOF
```

---

## Troubleshooting

### Script Stops Prematurely

✅ **FIXED** as of 2025-11-29. If you're still experiencing issues:

1. Ensure you're using the latest version of the script
2. Run with `--verbose` to see detailed progress
3. Check `collection.log` for specific errors
4. Verify you have root permissions: `whoami` should return `root`

### Permission Denied Errors

```bash
# Ensure script is executable
chmod +x collect-homelab-config.sh

# Run as root
sudo bash collect-homelab-config.sh
# or
ssh root@proxmox-host 'bash /tmp/collect-homelab-config.sh'
```

### Missing Commands

The script requires standard Proxmox utilities:
- `pveversion`, `pvesh`, `pvesm`, `pvecm`
- `qm`, `pct`
- `lscpu`, `df`, `lsblk`, `ip`, `ss`

These are all included in a standard Proxmox VE installation.

### Archive Too Large

```bash
# Exclude large log files (manual step after collection)
cd homelab-export-20251129-141328
find . -name "*.log" -size +10M -delete

# Re-compress
cd ..
tar -czf homelab-export-20251129-141328-slim.tar.gz homelab-export-20251129-141328/
```

---

## Security Considerations

### ⚠️ Warning: Sensitive Data

Even with sanitization enabled, the export contains:
- ✅ Network topology and IP addressing
- ✅ Usernames (passwords redacted)
- ✅ Storage paths and mount points
- ✅ VM/CT configurations
- ✅ Installed package lists

### Best Practices

1. **Never commit to public repositories** without `--sanitize all`
2. **Encrypt archives** before cloud storage:
   ```bash
   gpg -c homelab-export-20251129-141328.tar.gz
   ```
3. **Use private Git repos** (GitLab private, GitHub private)
4. **Restrict file permissions**:
   ```bash
   chmod 600 homelab-export-20251129-141328.tar.gz
   ```
5. **Audit before sharing**: Always review contents before sharing with others

---

## Integration Examples

### Ansible Playbook

```yaml
---
- name: Collect Proxmox infrastructure state
  hosts: proxmox
  tasks:
    - name: Upload collection script
      copy:
        src: collect-homelab-config.sh
        dest: /tmp/collect-homelab-config.sh
        mode: '0755'

    - name: Run collection
      shell: cd /tmp && bash collect-homelab-config.sh --level full

    - name: Download export
      fetch:
        src: /tmp/homelab-export-{{ ansible_date_time.date }}.tar.gz
        dest: ./exports/
        flat: yes
```

### Backup Script

```bash
#!/bin/bash
# backup-homelab-config.sh

BACKUP_DIR="/mnt/backup/homelab-configs"
PROXMOX_HOST="192.168.2.100"

# Run collection on Proxmox
ssh root@${PROXMOX_HOST} 'cd /tmp && bash /tmp/collect-homelab-config.sh --level full --sanitize all'

# Download archive
scp root@${PROXMOX_HOST}:/tmp/homelab-export-*.tar.gz ${BACKUP_DIR}/

# Clean up old exports (keep last 12)
cd ${BACKUP_DIR}
ls -t homelab-export-*.tar.gz | tail -n +13 | xargs rm -f

# Cleanup remote
ssh root@${PROXMOX_HOST} 'rm -rf /tmp/homelab-export-*'
```

---

## Files in This Directory

- **collect-homelab-config.sh**: The main collection script (FIXED)
- **BUGFIX-SUMMARY.md**: Detailed technical analysis of bugs and fixes
- **SCRIPT-USAGE.md**: This file - quick reference guide
- **homelab-export-*.tar.gz**: Collected infrastructure exports
- **CLAUDE.md**: Repository context and infrastructure overview

---

## Support

For issues, improvements, or questions:
1. Review BUGFIX-SUMMARY.md for technical details
2. Check collection.log in the export for error details
3. Run with `--verbose` for debugging output
4. Verify you're running the latest fixed version

---

*Script version: 1.0.0 (Bugs fixed: 2025-11-29)*
*Documentation maintained in: /mnt/c/Users/fam1n/Documents/homelab/*