Files

Jordan Ramos 07f9638d8b docs(infrastructure): update documentation for PVE 8.4.0 upgrade and VM 114 deployment

Infrastructure updates reflected in latest collection (homelab-export-20251211-144345):

Platform Upgrade:
- Proxmox VE upgraded from 8.3.3 to 8.4.0
- Kernel 6.8.12-8-pve, pve-manager 8.4.14
- Enhanced performance, security updates, and bug fixes

New Deployment:
- VM 114 (haos): Home Assistant OS for smart home automation
  * Specs: 4GB RAM (87% utilized), 2 vCPUs, 50GB boot disk
  * Status: Running with ~3 days uptime
  * Purpose: Centralized IoT device management and home automation hub

Service Activation:
- CT 103 (netbox): Network documentation/IPAM now Running
  * Previously Stopped, now active with ~3.1 days uptime
  * Resource usage: 1.28GB/2GB memory (64%)
  * Required for ongoing infrastructure expansion planning

Storage Utilization Updates:
- PBS-Backups: 27.43% → 28.27% (normal backup retention growth)
- Vault (ZFS): 10.88% → 12.13% (data accumulation monitored)
- local: 15.13% → 19.11% (new VM deployment + system updates)
- local-lvm: 0.0% → 0.01% (thin provisioned baseline)
- iso-share: 1.4% → 1.45% (minimal change)

Infrastructure Scale:
- Total VMs: 8 → 9
- Templates: 2 (unchanged)
- LXC Containers: 4 (unchanged)

Files Updated:
- CLAUDE_STATUS.md: Added comprehensive change log section, updated all metrics
- INDEX.md: Updated infrastructure counts, VM/CT tables, storage utilization

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

2025-12-11 17:08:10 -07:00

23 KiB

Raw Blame History

Homelab Infrastructure Collection - File Index

Welcome to your homelab infrastructure collection toolkit! This index will help you navigate the various files and understand what each one does.

New to this? Start here: start-here-docs/QUICK-START.md

Ready to run? Execute: bash scripts/crawlers-exporters/collect.sh

Need help? Check: scripts/crawlers-exporters/COLLECTION-GUIDE.md

Repository Structure

homelab/
├── services/                           # Docker Compose service configurations
│   ├── n8n/                           # n8n workflow automation
│   ├── netbox/                        # Network documentation & IPAM
│   └── README.md                      # Services overview
├── scripts/
│   ├── crawlers-exporters/            # Infrastructure collection scripts
│   │   ├── collect-homelab-config.sh  # Main collection engine (runs on Proxmox)
│   │   ├── collect-remote.sh          # SSH wrapper (orchestrates from WSL)
│   │   ├── collect.sh                 # Convenience wrapper (uses .env)
│   │   ├── COLLECTION-GUIDE.md        # Comprehensive collection reference
│   │   └── README-COLLECTION.md       # Collection overview
│   ├── fixers/                        # Problem-solving scripts
│   │   ├── fix_n8n_db_c_locale.sh     # PostgreSQL locale fix for n8n
│   │   └── fix_n8n_db_permissions.sh  # Database permissions repair
│   └── qol/                           # Quality of life improvements
│       ├── git-aliases.sh             # Git command shortcuts
│       └── git-first-commit.sh        # Repository initialization helper
├── start-here-docs/                   # Getting started documentation
│   ├── START-HERE.md                  # Primary entry point
│   ├── QUICK-START.md                 # 5-minute quick start guide
│   ├── GIT-SETUP-GUIDE.md            # Git configuration guide
│   ├── GIT-QUICK-REFERENCE.md        # Git command reference
│   ├── SCRIPT-USAGE.md               # Script usage examples
│   └── SETUP-COMPLETE.md             # Post-setup checklist
├── troubleshooting/                   # Problem resolution documentation
│   └── BUGFIX-SUMMARY.md             # Known issues and fixes
├── n8n/                              # n8n-specific documentation
│   └── N8N-SETUP-PLAN.md             # n8n deployment guide
├── mcp/                              # Model Context Protocol integrations
│   └── obsidian/
│       └── OBSIDIAN-MCP-SETUP.md     # Obsidian MCP configuration
├── sub-agents/                       # Agent role definitions
│   ├── scribe.md                     # Documentation & architecture agent
│   ├── backend-builder.md            # Development agent
│   ├── lab-operator.md               # Infrastructure operations agent
│   └── librarian.md                  # Knowledge management agent
├── homelab-export-20251202-204939/   # Latest infrastructure snapshot
│   ├── README.md                     # Export documentation
│   ├── SUMMARY.md                    # Collection statistics
│   ├── collection.log                # Detailed collection log
│   ├── configs/                      # Configuration files
│   │   ├── proxmox/                 # Proxmox core configs
│   │   ├── vms/                     # VM configurations
│   │   ├── lxc/                     # Container configurations
│   │   ├── storage/                 # Storage pool configs
│   │   ├── network/                 # Network configs
│   │   ├── backup/                  # Backup job configs
│   │   └── services/                # Systemd services
│   └── exports/                      # Structured data exports
│       ├── system/                   # System information
│       ├── cluster/                  # Cluster status (JSON)
│       └── guests/                   # VM/CT details
├── archive-homelab/                  # Historical exports and backups
├── CLAUDE.md                         # AI assistant project guidance
├── INDEX.md                          # This file - navigation index
└── README.md                         # Repository overview

File Inventory

Core Scripts

File	Location	Size	Purpose
`collect-homelab-config.sh`	`scripts/crawlers-exporters/`	32K	Main collection engine - runs on Proxmox host
`collect-remote.sh`	`scripts/crawlers-exporters/`	13K	SSH wrapper - orchestrates remote execution from WSL
`collect.sh`	`scripts/crawlers-exporters/`	4.2K	Convenience wrapper - uses .env configuration

Which script should I use?

Easiest: scripts/crawlers-exporters/collect.sh (if you have .env configured)
Direct control: scripts/crawlers-exporters/collect-remote.sh <proxmox-ip>
On Proxmox: scripts/crawlers-exporters/collect-homelab-config.sh (if SSHed into Proxmox)

Configuration

File	Size	Purpose
`.env.example`	1.5K	Configuration template with all options
`.env`	-	YOUR configuration (create from .env.example)

Setup: cp .env.example .env then edit .env with your Proxmox IP

Documentation

File	Location	Size	Purpose	Audience
`QUICK-START.md`	`start-here-docs/`	9.6K	Get started in 5 minutes	First-time users
`START-HERE.md`	`start-here-docs/`	-	Primary entry point	All users
`README-COLLECTION.md`	`scripts/crawlers-exporters/`	8.5K	Overview and common patterns	General users
`COLLECTION-GUIDE.md`	`scripts/crawlers-exporters/`	15K	Comprehensive reference	Power users
`GIT-SETUP-GUIDE.md`	`start-here-docs/`	-	Git configuration	New Git users
`GIT-QUICK-REFERENCE.md`	`start-here-docs/`	-	Git command cheat sheet	All users
`SCRIPT-USAGE.md`	`start-here-docs/`	-	Script examples	Script users
`INDEX.md`	Root	This file	Navigation and file index	Everyone
`CLAUDE.md`	Root	5.7K	Project context for Claude	AI assistant
`README.md`	Root	-	Repository overview	All users

Which Documentation Should I Read?

┌─────────────────────────────────────────────────────┐
│  Just want to get started?                          │
│  → Read: start-here-docs/QUICK-START.md            │
│  → Time: 5 minutes                                  │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│  Want to understand common usage patterns?          │
│  → Read: scripts/crawlers-exporters/               │
│           README-COLLECTION.md                      │
│  → Time: 15 minutes                                 │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│  Need complete reference with all options?          │
│  → Read: scripts/crawlers-exporters/               │
│           COLLECTION-GUIDE.md                       │
│  → Time: 30-45 minutes                              │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│  Setting up services (n8n, netbox, etc)?           │
│  → Read: services/README.md                        │
│         n8n/N8N-SETUP-PLAN.md                      │
│  → Time: Variable                                   │
└─────────────────────────────────────────────────────┘

Typical Workflow

First-Time Setup (5 minutes)

Configure environment

cp .env.example .env
nano .env  # Set PROXMOX_HOST to your Proxmox IP

Test SSH access
```
ssh root@your-proxmox-ip
```

Run first collection

bash scripts/crawlers-exporters/collect.sh

Review results
```
cat homelab-export-*/SUMMARY.md
```

Regular Use (1 minute)

# Just run this whenever you want to snapshot your infrastructure
bash scripts/crawlers-exporters/collect.sh

Advanced Use

# Full collection with verbose output
bash scripts/crawlers-exporters/collect.sh --level full --verbose

# Sanitized export for sharing
bash scripts/crawlers-exporters/collect.sh --sanitize all

# Different Proxmox host
bash scripts/crawlers-exporters/collect.sh --host 192.168.1.200

What Gets Created When You Run The Scripts?

After running bash scripts/crawlers-exporters/collect.sh, you'll have:

/home/jramos/homelab/
└── homelab-export-20251202-204939/
    ├── README.md                    # Documentation of this specific export
    ├── SUMMARY.md                   # Statistics and overview
    ├── collection.log               # Detailed collection log
    ├── homelab-export-*.tar.gz      # Compressed archive
    ├── configs/
    │   ├── proxmox/                # Proxmox core configs
    │   ├── vms/                    # All VM configs (9 VMs)
    │   ├── lxc/                    # All container configs (3 containers)
    │   ├── storage/                # Storage pool configs
    │   ├── network/                # Network configs
    │   ├── backup/                 # Backup job configs
    │   └── services/               # Systemd services
    ├── exports/
    │   ├── system/                 # System information
    │   ├── cluster/                # Cluster status
    │   └── guests/                 # VM/CT details
    ├── docs/                        # For your manual documentation
    ├── scripts/                     # For your automation scripts
    └── diagrams/                    # For network diagrams

Script Capabilities Matrix

Feature	collect.sh	collect-remote.sh	collect-homelab-config.sh
Uses .env config	✓	✗	✗
Runs from WSL	✓	✓	✗
Runs on Proxmox	✗	✗	✓
SSH automation	✓	✓	N/A
File transfer	✓	✓	N/A
Easiest to use	✓	-	-
Most flexible	-	✓	✓

Command Quick Reference

Setup Commands

# First-time configuration
cp .env.example .env
nano .env

# Set up SSH keys (recommended)
ssh-keygen -t ed25519
ssh-copy-id root@<proxmox-ip>

Collection Commands

# Standard collection (default)
bash scripts/crawlers-exporters/collect.sh

# Basic collection (minimal)
bash scripts/crawlers-exporters/collect.sh --level basic

# Full collection (comprehensive)
bash scripts/crawlers-exporters/collect.sh --level full

# With verbose output
bash scripts/crawlers-exporters/collect.sh --verbose

# Sanitized (safe for sharing)
bash scripts/crawlers-exporters/collect.sh --sanitize all

# Keep copy on Proxmox
bash scripts/crawlers-exporters/collect.sh --keep-remote

# Custom output location
bash scripts/crawlers-exporters/collect.sh --output ~/backups

Review Commands

# View summary
cat homelab-export-20251202-204939/SUMMARY.md

# View collection log
less homelab-export-20251202-204939/collection.log

# List collected files
ls -R homelab-export-20251202-204939/

# Browse VM configs
cat homelab-export-20251202-204939/configs/vms/*.conf

# Browse container configs
cat homelab-export-20251202-204939/configs/lxc/*.conf

# Check storage setup
cat homelab-export-20251202-204939/configs/proxmox/storage.cfg

Help Commands

# Script help
bash scripts/crawlers-exporters/collect.sh --help
bash scripts/crawlers-exporters/collect-remote.sh --help
bash scripts/crawlers-exporters/collect-homelab-config.sh --help

# Documentation
cat start-here-docs/QUICK-START.md
cat scripts/crawlers-exporters/README-COLLECTION.md
cat scripts/crawlers-exporters/COLLECTION-GUIDE.md

Collection Levels Explained

Level	Time	Size	Contents	Use Case
`basic`	~30s	1-3 MB	Configs only	Quick snapshots
`standard`	~1min	2-6 MB	Configs + status	Regular documentation
`full`	~2min	5-20 MB	Everything	Pre-maintenance backup
`paranoid`	~5min	10-50 MB	Maximum detail	Disaster recovery planning

Your Infrastructure

Based on the latest export (2025-12-11 14:43:55), your environment includes:

Virtual Machines (QEMU/KVM) - 9 VMs

VM ID	Name	Status	Purpose
100	docker-hub	Running	Container registry/Docker hub mirror
101	monitoring-docker	Running	Monitoring stack (Grafana/Prometheus/PVE Exporter) at 192.168.2.114
105	dev	Stopped	General-purpose development workstation
106	Ansible-Control	Running	IaC orchestration, configuration management
108	CML	Stopped	Cisco Modeling Labs - network simulation
109	web-server-01	Running	Web application server (clustered)
110	web-server-02	Running	Load-balanced pair with web-server-01
111	db-server-01	Running	Backend database server
114	haos	Running	Home Assistant OS - smart home automation platform

Recent Changes: Added VM 101 (monitoring-docker) for observability, VM 114 (haos) for home automation (2025-12-11).

VM Templates - 2 Templates

Template ID	Name	Purpose
104	ubuntu-dev	Ubuntu development environment template for cloning
107	ubuntu-docker	Ubuntu Docker host template for rapid deployment

Containers (LXC) - 4 Containers

CT ID	Name	Status	Purpose
102	nginx	Running	Reverse proxy/load balancer
103	netbox	Running	Network documentation/IPAM
112	twingate-connector	Running	Zero-trust network access connector
113	n8n	Running	Workflow automation platform at 192.168.2.107

Recent Changes: Added CT 112 (twingate-connector) for zero-trust security, CT 113 (n8n) for workflow automation. CT 103 (netbox) activated 2025-12-11.

Storage Pools

local (Directory) - 19.11% used - System files, ISOs, templates
local-lvm (LVM-Thin) - 0.01% used - VM disk images (thin provisioned)
Vault (NFS/Directory) - 12.13% used - Secure storage for sensitive data
PBS-Backups (Proxmox Backup Server) - 28.27% used - Automated backup repository
iso-share (NFS/CIFS) - 1.45% used - Installation media library
localnetwork (Network share) - Shared resources across infrastructure

All of these are documented in your collection exports!

Latest Export Information

Export Directory: /home/jramos/homelab/disaster-recovery/homelab-export-20251211-144345/
Collection Date: 2025-12-11 14:43:55
Hostname: serviceslab
Collection Level: full
Script Version: 1.0.0

Common Questions

Q: Which file do I run?

A: Start with bash scripts/crawlers-exporters/collect.sh - it's the easiest.

Q: Do I need to set up .env?

A: Yes! Copy .env.example to .env and set your PROXMOX_HOST.

Q: Will this modify my Proxmox setup?

A: No! All operations are read-only. Zero risk.

Q: How much disk space do I need?

A: On Proxmox: ~100-500 MB. On WSL: ~50-250 MB for archives.

Q: Can I run this regularly?

A: Absolutely! That's what it's designed for. Run weekly or monthly.

Q: What if I get errors?

A: Check SUMMARY.md and collection.log in the export. Most "errors" are just skipped features you don't use (like ZFS if you don't have ZFS).

Q: Is my data secure?

A: By default, passwords and tokens are sanitized. Use --sanitize all for complete sanitization including IPs.

Q: Can I use this for disaster recovery?

A: Yes! The configs show you exactly how your infrastructure is set up. Restore disk images from PBS-Backups and recreate VMs using these configs.

Q: How do I fix n8n database issues?

A: Use the scripts in scripts/fixers/:

bash scripts/fixers/fix_n8n_db_permissions.sh
bash scripts/fixers/fix_n8n_db_c_locale.sh

Integration Examples

Git Version Control

cd homelab-export-20251202-204939/
git init
git add .
git commit -m "Infrastructure snapshot 2025-12-02"
git remote add origin <your-private-repo>
git push -u origin main

Scheduled Collections (on Proxmox)

# Add to Proxmox crontab
0 3 * * 0 /root/collect-homelab-config.sh -l standard -o /backup/weekly-$(date +\%Y\%U)

Automated Retention

# Keep only last 4 weeks
find ./homelab-export-* -maxdepth 0 -type d -mtime +28 -exec rm -rf {} \;

Troubleshooting Quick Guide

Problem	Solution
"Cannot connect to Proxmox"	Check IP in `.env`, test with `ssh root@<ip>`
"Permission denied"	Use `bash collect.sh` not `./collect.sh` on Windows FS
"PROXMOX_HOST not set"	Edit `.env` file and set your Proxmox IP
"Some items skipped"	Normal! Check `SUMMARY.md` - usually ZFS/cluster features you don't use
"Script won't execute"	Use: `bash scripts/crawlers-exporters/collect.sh` (explicitly call bash)
SSH asks for password	Set up SSH keys: `ssh-copy-id root@<proxmox-ip>`
n8n 502 Bad Gateway	Check `troubleshooting/BUGFIX-SUMMARY.md` and `n8n/N8N-SETUP-PLAN.md`
PostgreSQL locale errors	Run `bash scripts/fixers/fix_n8n_db_c_locale.sh`

For detailed troubleshooting, see: troubleshooting/BUGFIX-SUMMARY.md

File Sizes Reference

Component	Size	Notes
Scripts	~50K total	All three collection scripts
Documentation	~65K total	All markdown and text files
Configuration	~2K	.env.example template
Total Package	~117K	Everything you need
Output (basic)	1-3 MB	Per collection run
Output (standard)	2-6 MB	Per collection run
Output (full)	5-20 MB	Per collection run

Monitoring Stack

The infrastructure now includes a comprehensive monitoring and observability stack deployed on VM 101 (monitoring-docker) at 192.168.2.114:

Components

Grafana (Port 3000): Visualization and dashboards
Prometheus (Port 9090): Metrics collection and time-series database
PVE Exporter (Port 9221): Proxmox VE metrics exporter

Features

Real-time Proxmox infrastructure monitoring
VM and container resource utilization tracking
Storage pool metrics and capacity planning
Network traffic analysis
Pre-configured dashboards for Proxmox VE
Alerting capabilities (configurable)

Access

Grafana UI: http://192.168.2.114:3000
Prometheus UI: http://192.168.2.114:9090
Metrics Endpoint: http://192.168.2.114:9221/pve

Documentation

For comprehensive setup, configuration, and troubleshooting:

Monitoring Guide: monitoring/README.md
Docker Compose Configs: monitoring/grafana/, monitoring/prometheus/, monitoring/pve-exporter/

Key Metrics

Node CPU, memory, and disk usage
VM/CT resource consumption
Storage pool utilization trends
Backup job success rates
Network interface statistics

Service Management

n8n Workflow Automation

Container: CT 113
Service Config: services/n8n/
Setup Guide: n8n/N8N-SETUP-PLAN.md
Common Issues: See troubleshooting/BUGFIX-SUMMARY.md

NetBox IPAM

Container: CT 103
Service Config: services/netbox/
Purpose: Network documentation and IP address management

Nginx Reverse Proxy

Container: CT 102
Purpose: Front-end traffic management, SSL termination

Agent Roles

This repository works with multiple AI agent personas:

Agent	File	Purpose
Scribe	`sub-agents/scribe.md`	Documentation & architecture
Backend Builder	`sub-agents/backend-builder.md`	Development & coding
Lab Operator	`sub-agents/lab-operator.md`	Infrastructure operations
Librarian	`sub-agents/librarian.md`	Knowledge management

Next Steps

Immediate: Read start-here-docs/QUICK-START.md (5 minutes)

Setup: Create your .env file

cp .env.example .env
nano .env  # Set PROXMOX_HOST

Execute: Run your first collection

bash scripts/crawlers-exporters/collect.sh

Review: Check the results

cat homelab-export-20251202-204939/SUMMARY.md

Explore: Browse the collected configs
```
ls -R homelab-export-20251202-204939/
```
Learn More: Read scripts/crawlers-exporters/README-COLLECTION.md for common patterns
Master It: Study scripts/crawlers-exporters/COLLECTION-GUIDE.md for advanced usage

Support and Resources

Quick help: bash scripts/crawlers-exporters/collect.sh --help
Detailed help: See scripts/crawlers-exporters/COLLECTION-GUIDE.md
Getting started: See start-here-docs/QUICK-START.md
Examples: See scripts/crawlers-exporters/README-COLLECTION.md
Troubleshooting: See troubleshooting/BUGFIX-SUMMARY.md

Summary

You now have a complete, production-ready infrastructure collection system that:

✓ Automatically collects all Proxmox configurations ✓ Works seamlessly from WSL2 via SSH ✓ Sanitizes sensitive information ✓ Creates organized, documented exports ✓ Supports automation and scheduling ✓ Provides comprehensive documentation ✓ Is completely safe (read-only operations) ✓ Includes service management for n8n, netbox, nginx ✓ Has troubleshooting resources and fix scripts

Ready to begin?

# Set up configuration
cp .env.example .env
nano .env

# Run your first collection
bash scripts/crawlers-exporters/collect.sh

Repository Version: 2.1.0 Last Updated: 2025-12-07 Latest Export: disaster-recovery/homelab-export-20251207-120040 Infrastructure: 8 VMs, 2 Templates, 4 Containers, Proxmox VE 8.3.3 Maintained by: Your homelab automation system

23 KiB Raw Blame History

Homelab Infrastructure Collection - File Index

Quick Navigation

Repository Structure

File Inventory

Core Scripts

Configuration

Documentation

Which Documentation Should I Read?

Typical Workflow

First-Time Setup (5 minutes)

Regular Use (1 minute)

Advanced Use

What Gets Created When You Run The Scripts?

Script Capabilities Matrix

Command Quick Reference

Setup Commands

Collection Commands

Review Commands

Help Commands

Collection Levels Explained

Your Infrastructure

Virtual Machines (QEMU/KVM) - 9 VMs

VM Templates - 2 Templates

Containers (LXC) - 4 Containers

Storage Pools

Latest Export Information

Common Questions

Q: Which file do I run?

Q: Do I need to set up .env?

Q: Will this modify my Proxmox setup?

Q: How much disk space do I need?

Q: Can I run this regularly?

Q: What if I get errors?

Q: Is my data secure?

Q: Can I use this for disaster recovery?

Q: How do I fix n8n database issues?

Integration Examples

Git Version Control

Scheduled Collections (on Proxmox)

Automated Retention

Troubleshooting Quick Guide

File Sizes Reference

Monitoring Stack

Components

Features

Access

Documentation

Key Metrics

Service Management

n8n Workflow Automation

NetBox IPAM

Nginx Reverse Proxy

Agent Roles

Next Steps

Support and Resources

Summary

23 KiB

Raw Blame History