# Homelab Infrastructure Collection - File Index

Welcome to your homelab infrastructure collection toolkit! This index will help you navigate the various files and understand what each one does.

## Quick Navigation

**New to this?** Start here: **[start-here-docs/QUICK-START.md](start-here-docs/QUICK-START.md)**

**Ready to run?** Execute: `bash scripts/crawlers-exporters/collect.sh`

**Need help?** Check: **[scripts/crawlers-exporters/COLLECTION-GUIDE.md](scripts/crawlers-exporters/COLLECTION-GUIDE.md)**

## Repository Structure

```
homelab/
├── services/                           # Docker Compose service configurations
│   ├── n8n/                           # n8n workflow automation
│   ├── netbox/                        # Network documentation & IPAM
│   └── README.md                      # Services overview
├── scripts/
│   ├── crawlers-exporters/            # Infrastructure collection scripts
│   │   ├── collect-homelab-config.sh  # Main collection engine (runs on Proxmox)
│   │   ├── collect-remote.sh          # SSH wrapper (orchestrates from WSL)
│   │   ├── collect.sh                 # Convenience wrapper (uses .env)
│   │   ├── COLLECTION-GUIDE.md        # Comprehensive collection reference
│   │   └── README-COLLECTION.md       # Collection overview
│   ├── fixers/                        # Problem-solving scripts
│   │   ├── fix_n8n_db_c_locale.sh     # PostgreSQL locale fix for n8n
│   │   └── fix_n8n_db_permissions.sh  # Database permissions repair
│   └── qol/                           # Quality of life improvements
│       ├── git-aliases.sh             # Git command shortcuts
│       └── git-first-commit.sh        # Repository initialization helper
├── start-here-docs/                   # Getting started documentation
│   ├── START-HERE.md                  # Primary entry point
│   ├── QUICK-START.md                 # 5-minute quick start guide
│   ├── GIT-SETUP-GUIDE.md            # Git configuration guide
│   ├── GIT-QUICK-REFERENCE.md        # Git command reference
│   ├── SCRIPT-USAGE.md               # Script usage examples
│   └── SETUP-COMPLETE.md             # Post-setup checklist
├── troubleshooting/                   # Problem resolution documentation
│   └── BUGFIX-SUMMARY.md             # Known issues and fixes
├── n8n/                              # n8n-specific documentation
│   └── N8N-SETUP-PLAN.md             # n8n deployment guide
├── mcp/                              # Model Context Protocol integrations
│   └── obsidian/
│       └── OBSIDIAN-MCP-SETUP.md     # Obsidian MCP configuration
├── sub-agents/                       # Agent role definitions
│   ├── scribe.md                     # Documentation & architecture agent
│   ├── backend-builder.md            # Development agent
│   ├── lab-operator.md               # Infrastructure operations agent
│   └── librarian.md                  # Knowledge management agent
├── homelab-export-20251202-204939/   # Latest infrastructure snapshot
│   ├── README.md                     # Export documentation
│   ├── SUMMARY.md                    # Collection statistics
│   ├── collection.log                # Detailed collection log
│   ├── configs/                      # Configuration files
│   │   ├── proxmox/                 # Proxmox core configs
│   │   ├── vms/                     # VM configurations
│   │   ├── lxc/                     # Container configurations
│   │   ├── storage/                 # Storage pool configs
│   │   ├── network/                 # Network configs
│   │   ├── backup/                  # Backup job configs
│   │   └── services/                # Systemd services
│   └── exports/                      # Structured data exports
│       ├── system/                   # System information
│       ├── cluster/                  # Cluster status (JSON)
│       └── guests/                   # VM/CT details
├── archive-homelab/                  # Historical exports and backups
├── CLAUDE.md                         # AI assistant project guidance
├── INDEX.md                          # This file - navigation index
└── README.md                         # Repository overview
```

## File Inventory

### Core Scripts

| File | Location | Size | Purpose |
|------|----------|------|---------|
| `collect-homelab-config.sh` | `scripts/crawlers-exporters/` | 32K | Main collection engine - runs on Proxmox host |
| `collect-remote.sh` | `scripts/crawlers-exporters/` | 13K | SSH wrapper - orchestrates remote execution from WSL |
| `collect.sh` | `scripts/crawlers-exporters/` | 4.2K | Convenience wrapper - uses .env configuration |

**Which script should I use?**
- **Easiest**: `scripts/crawlers-exporters/collect.sh` (if you have `.env` configured)
- **Direct control**: `scripts/crawlers-exporters/collect-remote.sh <proxmox-ip>`
- **On Proxmox**: `scripts/crawlers-exporters/collect-homelab-config.sh` (if SSHed into Proxmox)

### Configuration

| File | Size | Purpose |
|------|------|---------|
| `.env.example` | 1.5K | Configuration template with all options |
| `.env` | - | YOUR configuration (create from .env.example) |

**Setup**: `cp .env.example .env` then edit `.env` with your Proxmox IP

### Documentation

| File | Location | Size | Purpose | Audience |
|------|----------|------|---------|----------|
| `QUICK-START.md` | `start-here-docs/` | 9.6K | Get started in 5 minutes | First-time users |
| `START-HERE.md` | `start-here-docs/` | - | Primary entry point | All users |
| `README-COLLECTION.md` | `scripts/crawlers-exporters/` | 8.5K | Overview and common patterns | General users |
| `COLLECTION-GUIDE.md` | `scripts/crawlers-exporters/` | 15K | Comprehensive reference | Power users |
| `GIT-SETUP-GUIDE.md` | `start-here-docs/` | - | Git configuration | New Git users |
| `GIT-QUICK-REFERENCE.md` | `start-here-docs/` | - | Git command cheat sheet | All users |
| `SCRIPT-USAGE.md` | `start-here-docs/` | - | Script examples | Script users |
| `INDEX.md` | Root | This file | Navigation and file index | Everyone |
| `CLAUDE.md` | Root | 5.7K | Project context for Claude | AI assistant |
| `README.md` | Root | - | Repository overview | All users |

### Which Documentation Should I Read?

```
┌─────────────────────────────────────────────────────┐
│  Just want to get started?                          │
│  → Read: start-here-docs/QUICK-START.md            │
│  → Time: 5 minutes                                  │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│  Want to understand common usage patterns?          │
│  → Read: scripts/crawlers-exporters/               │
│           README-COLLECTION.md                      │
│  → Time: 15 minutes                                 │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│  Need complete reference with all options?          │
│  → Read: scripts/crawlers-exporters/               │
│           COLLECTION-GUIDE.md                       │
│  → Time: 30-45 minutes                              │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│  Setting up services (n8n, netbox, etc)?           │
│  → Read: services/README.md                        │
│         n8n/N8N-SETUP-PLAN.md                      │
│  → Time: Variable                                   │
└─────────────────────────────────────────────────────┘
```

## Typical Workflow

### First-Time Setup (5 minutes)

1. **Configure environment**
   ```bash
   cp .env.example .env
   nano .env  # Set PROXMOX_HOST to your Proxmox IP
   ```

2. **Test SSH access**
   ```bash
   ssh root@your-proxmox-ip
   ```

3. **Run first collection**
   ```bash
   bash scripts/crawlers-exporters/collect.sh
   ```

4. **Review results**
   ```bash
   cat homelab-export-*/SUMMARY.md
   ```

### Regular Use (1 minute)

```bash
# Just run this whenever you want to snapshot your infrastructure
bash scripts/crawlers-exporters/collect.sh
```

### Advanced Use

```bash
# Full collection with verbose output
bash scripts/crawlers-exporters/collect.sh --level full --verbose

# Sanitized export for sharing
bash scripts/crawlers-exporters/collect.sh --sanitize all

# Different Proxmox host
bash scripts/crawlers-exporters/collect.sh --host 192.168.1.200
```

## What Gets Created When You Run The Scripts?

After running `bash scripts/crawlers-exporters/collect.sh`, you'll have:

```
/home/jramos/homelab/
└── homelab-export-20251202-204939/
    ├── README.md                    # Documentation of this specific export
    ├── SUMMARY.md                   # Statistics and overview
    ├── collection.log               # Detailed collection log
    ├── homelab-export-*.tar.gz      # Compressed archive
    ├── configs/
    │   ├── proxmox/                # Proxmox core configs
    │   ├── vms/                    # All VM configs (9 VMs)
    │   ├── lxc/                    # All container configs (3 containers)
    │   ├── storage/                # Storage pool configs
    │   ├── network/                # Network configs
    │   ├── backup/                 # Backup job configs
    │   └── services/               # Systemd services
    ├── exports/
    │   ├── system/                 # System information
    │   ├── cluster/                # Cluster status
    │   └── guests/                 # VM/CT details
    ├── docs/                        # For your manual documentation
    ├── scripts/                     # For your automation scripts
    └── diagrams/                    # For network diagrams
```

## Script Capabilities Matrix

| Feature | collect.sh | collect-remote.sh | collect-homelab-config.sh |
|---------|-----------|------------------|---------------------------|
| Uses .env config | ✓ | ✗ | ✗ |
| Runs from WSL | ✓ | ✓ | ✗ |
| Runs on Proxmox | ✗ | ✗ | ✓ |
| SSH automation | ✓ | ✓ | N/A |
| File transfer | ✓ | ✓ | N/A |
| Easiest to use | ✓ | - | - |
| Most flexible | - | ✓ | ✓ |

## Command Quick Reference

### Setup Commands
```bash
# First-time configuration
cp .env.example .env
nano .env

# Set up SSH keys (recommended)
ssh-keygen -t ed25519
ssh-copy-id root@<proxmox-ip>
```

### Collection Commands
```bash
# Standard collection (default)
bash scripts/crawlers-exporters/collect.sh

# Basic collection (minimal)
bash scripts/crawlers-exporters/collect.sh --level basic

# Full collection (comprehensive)
bash scripts/crawlers-exporters/collect.sh --level full

# With verbose output
bash scripts/crawlers-exporters/collect.sh --verbose

# Sanitized (safe for sharing)
bash scripts/crawlers-exporters/collect.sh --sanitize all

# Keep copy on Proxmox
bash scripts/crawlers-exporters/collect.sh --keep-remote

# Custom output location
bash scripts/crawlers-exporters/collect.sh --output ~/backups
```

### Review Commands
```bash
# View summary
cat homelab-export-20251202-204939/SUMMARY.md

# View collection log
less homelab-export-20251202-204939/collection.log

# List collected files
ls -R homelab-export-20251202-204939/

# Browse VM configs
cat homelab-export-20251202-204939/configs/vms/*.conf

# Browse container configs
cat homelab-export-20251202-204939/configs/lxc/*.conf

# Check storage setup
cat homelab-export-20251202-204939/configs/proxmox/storage.cfg
```

### Help Commands
```bash
# Script help
bash scripts/crawlers-exporters/collect.sh --help
bash scripts/crawlers-exporters/collect-remote.sh --help
bash scripts/crawlers-exporters/collect-homelab-config.sh --help

# Documentation
cat start-here-docs/QUICK-START.md
cat scripts/crawlers-exporters/README-COLLECTION.md
cat scripts/crawlers-exporters/COLLECTION-GUIDE.md
```

## Collection Levels Explained

| Level | Time | Size | Contents | Use Case |
|-------|------|------|----------|----------|
| `basic` | ~30s | 1-3 MB | Configs only | Quick snapshots |
| `standard` | ~1min | 2-6 MB | Configs + status | Regular documentation |
| `full` | ~2min | 5-20 MB | Everything | Pre-maintenance backup |
| `paranoid` | ~5min | 10-50 MB | Maximum detail | Disaster recovery planning |

## Your Infrastructure

Based on the latest export (2025-12-07 12:00:40), your environment includes:

### Virtual Machines (QEMU/KVM) - 10 VMs

| VM ID | Name | Status | Purpose |
|-------|------|--------|---------|
| 100 | docker-hub | Running | Container registry/Docker hub mirror |
| 101 | monitoring-docker | Running | Monitoring stack (Grafana/Prometheus/PVE Exporter) at 192.168.2.114 |
| 104 | ubuntu-dev | Stopped | Ubuntu development environment |
| 105 | dev | Stopped | General-purpose development workstation |
| 106 | Ansible-Control | Running | IaC orchestration, configuration management |
| 107 | ubuntu-docker | Stopped | Ubuntu Docker host |
| 108 | CML | Stopped | Cisco Modeling Labs - network simulation |
| 109 | web-server-01 | Running | Web application server (clustered) |
| 110 | web-server-02 | Running | Load-balanced pair with web-server-01 |
| 111 | db-server-01 | Running | Backend database server |

**Recent Changes**: Added VM 101 (monitoring-docker) for dedicated observability infrastructure.

### Containers (LXC) - 4 Containers

| CT ID | Name | Status | Purpose |
|-------|------|--------|---------|
| 102 | nginx | Running | Reverse proxy/load balancer |
| 103 | netbox | Stopped | Network documentation/IPAM |
| 112 | twingate-connector | Running | Zero-trust network access connector |
| 113 | n8n | Running | Workflow automation platform at 192.168.2.107 |

**Recent Changes**: Added CT 112 (twingate-connector) for zero-trust security, CT 113 (n8n) for workflow automation.

### Storage Pools
- **local** (Directory) - 15.13% used - System files, ISOs, templates
- **local-lvm** (LVM-Thin) - 0.0% used - VM disk images (thin provisioned)
- **Vault** (NFS/Directory) - 10.88% used - Secure storage for sensitive data
- **PBS-Backups** (Proxmox Backup Server) - 27.43% used - Automated backup repository
- **iso-share** (NFS/CIFS) - 1.4% used - Installation media library
- **localnetwork** (Network share) - Shared resources across infrastructure

All of these are documented in your collection exports!

## Latest Export Information

- **Export Directory**: `/home/jramos/homelab/disaster-recovery/homelab-export-20251207-120040/`
- **Collection Date**: 2025-12-07 12:00:40
- **Hostname**: serviceslab
- **Collection Level**: full
- **Script Version**: 1.0.0

## Common Questions

### Q: Which file do I run?
**A:** Start with `bash scripts/crawlers-exporters/collect.sh` - it's the easiest.

### Q: Do I need to set up .env?
**A:** Yes! Copy `.env.example` to `.env` and set your `PROXMOX_HOST`.

### Q: Will this modify my Proxmox setup?
**A:** No! All operations are read-only. Zero risk.

### Q: How much disk space do I need?
**A:** On Proxmox: ~100-500 MB. On WSL: ~50-250 MB for archives.

### Q: Can I run this regularly?
**A:** Absolutely! That's what it's designed for. Run weekly or monthly.

### Q: What if I get errors?
**A:** Check `SUMMARY.md` and `collection.log` in the export. Most "errors" are just skipped features you don't use (like ZFS if you don't have ZFS).

### Q: Is my data secure?
**A:** By default, passwords and tokens are sanitized. Use `--sanitize all` for complete sanitization including IPs.

### Q: Can I use this for disaster recovery?
**A:** Yes! The configs show you exactly how your infrastructure is set up. Restore disk images from PBS-Backups and recreate VMs using these configs.

### Q: How do I fix n8n database issues?
**A:** Use the scripts in `scripts/fixers/`:
```bash
bash scripts/fixers/fix_n8n_db_permissions.sh
bash scripts/fixers/fix_n8n_db_c_locale.sh
```

## Integration Examples

### Git Version Control
```bash
cd homelab-export-20251202-204939/
git init
git add .
git commit -m "Infrastructure snapshot 2025-12-02"
git remote add origin <your-private-repo>
git push -u origin main
```

### Scheduled Collections (on Proxmox)
```bash
# Add to Proxmox crontab
0 3 * * 0 /root/collect-homelab-config.sh -l standard -o /backup/weekly-$(date +\%Y\%U)
```

### Automated Retention
```bash
# Keep only last 4 weeks
find ./homelab-export-* -maxdepth 0 -type d -mtime +28 -exec rm -rf {} \;
```

## Troubleshooting Quick Guide

| Problem | Solution |
|---------|----------|
| "Cannot connect to Proxmox" | Check IP in `.env`, test with `ssh root@<ip>` |
| "Permission denied" | Use `bash collect.sh` not `./collect.sh` on Windows FS |
| "PROXMOX_HOST not set" | Edit `.env` file and set your Proxmox IP |
| "Some items skipped" | Normal! Check `SUMMARY.md` - usually ZFS/cluster features you don't use |
| "Script won't execute" | Use: `bash scripts/crawlers-exporters/collect.sh` (explicitly call bash) |
| SSH asks for password | Set up SSH keys: `ssh-copy-id root@<proxmox-ip>` |
| n8n 502 Bad Gateway | Check `troubleshooting/BUGFIX-SUMMARY.md` and `n8n/N8N-SETUP-PLAN.md` |
| PostgreSQL locale errors | Run `bash scripts/fixers/fix_n8n_db_c_locale.sh` |

For detailed troubleshooting, see: **[troubleshooting/BUGFIX-SUMMARY.md](troubleshooting/BUGFIX-SUMMARY.md)**

## File Sizes Reference

| Component | Size | Notes |
|-----------|------|-------|
| Scripts | ~50K total | All three collection scripts |
| Documentation | ~65K total | All markdown and text files |
| Configuration | ~2K | .env.example template |
| **Total Package** | **~117K** | Everything you need |
| **Output (basic)** | 1-3 MB | Per collection run |
| **Output (standard)** | 2-6 MB | Per collection run |
| **Output (full)** | 5-20 MB | Per collection run |

## Monitoring Stack

The infrastructure now includes a comprehensive monitoring and observability stack deployed on VM 101 (monitoring-docker) at 192.168.2.114:

### Components
- **Grafana** (Port 3000): Visualization and dashboards
- **Prometheus** (Port 9090): Metrics collection and time-series database
- **PVE Exporter** (Port 9221): Proxmox VE metrics exporter

### Features
- Real-time Proxmox infrastructure monitoring
- VM and container resource utilization tracking
- Storage pool metrics and capacity planning
- Network traffic analysis
- Pre-configured dashboards for Proxmox VE
- Alerting capabilities (configurable)

### Access
- **Grafana UI**: http://192.168.2.114:3000
- **Prometheus UI**: http://192.168.2.114:9090
- **Metrics Endpoint**: http://192.168.2.114:9221/pve

### Documentation
For comprehensive setup, configuration, and troubleshooting:
- **Monitoring Guide**: `monitoring/README.md`
- **Docker Compose Configs**: `monitoring/grafana/`, `monitoring/prometheus/`, `monitoring/pve-exporter/`

### Key Metrics
- Node CPU, memory, and disk usage
- VM/CT resource consumption
- Storage pool utilization trends
- Backup job success rates
- Network interface statistics

## Service Management

### n8n Workflow Automation
- **Container**: CT 113
- **Service Config**: `services/n8n/`
- **Setup Guide**: `n8n/N8N-SETUP-PLAN.md`
- **Common Issues**: See `troubleshooting/BUGFIX-SUMMARY.md`

### NetBox IPAM
- **Container**: CT 103
- **Service Config**: `services/netbox/`
- **Purpose**: Network documentation and IP address management

### Nginx Reverse Proxy
- **Container**: CT 102
- **Purpose**: Front-end traffic management, SSL termination

## Agent Roles

This repository works with multiple AI agent personas:

| Agent | File | Purpose |
|-------|------|---------|
| **Scribe** | `sub-agents/scribe.md` | Documentation & architecture |
| **Backend Builder** | `sub-agents/backend-builder.md` | Development & coding |
| **Lab Operator** | `sub-agents/lab-operator.md` | Infrastructure operations |
| **Librarian** | `sub-agents/librarian.md` | Knowledge management |

## Next Steps

1. **Immediate**: Read [start-here-docs/QUICK-START.md](start-here-docs/QUICK-START.md) (5 minutes)

2. **Setup**: Create your `.env` file
   ```bash
   cp .env.example .env
   nano .env  # Set PROXMOX_HOST
   ```

3. **Execute**: Run your first collection
   ```bash
   bash scripts/crawlers-exporters/collect.sh
   ```

4. **Review**: Check the results
   ```bash
   cat homelab-export-20251202-204939/SUMMARY.md
   ```

5. **Explore**: Browse the collected configs
   ```bash
   ls -R homelab-export-20251202-204939/
   ```

6. **Learn More**: Read [scripts/crawlers-exporters/README-COLLECTION.md](scripts/crawlers-exporters/README-COLLECTION.md) for common patterns

7. **Master It**: Study [scripts/crawlers-exporters/COLLECTION-GUIDE.md](scripts/crawlers-exporters/COLLECTION-GUIDE.md) for advanced usage

## Support and Resources

- **Quick help**: `bash scripts/crawlers-exporters/collect.sh --help`
- **Detailed help**: See [scripts/crawlers-exporters/COLLECTION-GUIDE.md](scripts/crawlers-exporters/COLLECTION-GUIDE.md)
- **Getting started**: See [start-here-docs/QUICK-START.md](start-here-docs/QUICK-START.md)
- **Examples**: See [scripts/crawlers-exporters/README-COLLECTION.md](scripts/crawlers-exporters/README-COLLECTION.md)
- **Troubleshooting**: See [troubleshooting/BUGFIX-SUMMARY.md](troubleshooting/BUGFIX-SUMMARY.md)

## Summary

You now have a complete, production-ready infrastructure collection system that:

✓ Automatically collects all Proxmox configurations
✓ Works seamlessly from WSL2 via SSH
✓ Sanitizes sensitive information
✓ Creates organized, documented exports
✓ Supports automation and scheduling
✓ Provides comprehensive documentation
✓ Is completely safe (read-only operations)
✓ Includes service management for n8n, netbox, nginx
✓ Has troubleshooting resources and fix scripts

**Ready to begin?**

```bash
# Set up configuration
cp .env.example .env
nano .env

# Run your first collection
bash scripts/crawlers-exporters/collect.sh
```

---

**Repository Version:** 2.1.0
**Last Updated**: 2025-12-07
**Latest Export**: disaster-recovery/homelab-export-20251207-120040
**Infrastructure**: 10 VMs, 4 Containers, Proxmox VE 8.3.3
**Maintained by**: Your homelab automation system