diff --git a/README.md b/README.md index e219097..e841e60 100644 --- a/README.md +++ b/README.md @@ -4,178 +4,165 @@ Version-controlled infrastructure configuration for my Proxmox-based homelab env ## Overview -This repository contains configuration files, scripts, and documentation for managing a Proxmox VE 8.3.3 homelab environment. The infrastructure follows a hybrid architecture combining traditional virtualization (KVM/QEMU) with containerization (LXC) for optimal resource utilization. +This repository contains configuration files, scripts, and documentation for managing a Proxmox VE 8.4.0 homelab environment. The infrastructure follows a hybrid architecture combining traditional virtualization (KVM/QEMU) with containerization (LXC) for optimal resource utilization. ## Infrastructure Components ### Proxmox Host - **Node**: serviceslab -- **Version**: Proxmox VE 8.3.3 +- **IP**: 192.168.2.100 +- **Version**: Proxmox VE 8.4.0 (kernel 6.8.12-17-pve) - **Architecture**: Single-node cluster - **Primary Use**: Services and development laboratory -### Virtual Machines (QEMU/KVM) -- **100** - docker-hub: Container registry and Docker hub mirror -- **101** - monitoring-docker: Monitoring stack (Grafana/Prometheus/PVE Exporter) at 192.168.2.114 -- **105** - dev: General-purpose development environment -- **106** - Ansible-Control: Infrastructure automation control node -- **108** - CML: Cisco Modeling Labs for network simulation -- **109** - web-server-01: Web application server (clustered) -- **110** - web-server-02: Web application server (load-balanced) -- **111** - db-server-01: Database server +### Virtual Machines — Running -### VM Templates -- **104** - ubuntu-dev: Ubuntu development environment template for cloning -- **107** - ubuntu-docker: Ubuntu Docker host template for rapid deployment +| VMID | Name | vCPU | RAM | Disk | Purpose | +|------|------|------|-----|------|---------| +| 100 | docker-hub | 4 | 10GB | 100GB | Container registry and Docker hub mirror | +| 101 | monitoring-docker | 2 | 8GB | 50GB | Monitoring stack (Grafana/Prometheus/PVE Exporter) | +| 102 | CML | 8 | 32GB | 200GB | Cisco Modeling Labs — network simulation lab | -### Containers (LXC) -- **102** - nginx: Reverse proxy and load balancer (Nginx Proxy Manager) -- **103** - netbox: Network documentation and IPAM -- **112** - twingate-connector: Zero-trust network access connector -- **113** - n8n: Workflow automation platform at 192.168.2.107 +### Virtual Machines — Stopped / Templates + +| VMID | Name | vCPU | RAM | Notes | +|------|------|------|-----|-------| +| 104 | ubuntu-dev | 2 | 5GB | Template — Ubuntu dev environment | +| 105 | pfSense-Firewall | 2 | 2GB | Stopped — firewall lab VM | +| 106 | Ansible-Control | 2 | 4GB | Stopped — IaC control node | +| 107 | ubuntu-docker | 2 | 4GB | Template — Ubuntu Docker host | +| 114 | haos | 2 | 4GB | Stopped — Home Assistant OS | + +### Containers (LXC) — Running + +| CTID | Name | vCPU | RAM | IP | Purpose | +|------|------|------|-----|----|---------| +| 109 | caddy | 2 | 2GB | 192.168.2.129 | Reverse proxy and SSL termination (replaced NPM) | +| 112 | twingate-connector | 1 | 1GB | DHCP | Zero-trust network access connector | +| 113 | n8n | 2 | 4GB | 192.168.2.113 | Workflow automation (PostgreSQL 16 + pgvector) | + +### Containers (LXC) — Stopped + +| CTID | Name | vCPU | RAM | Notes | +|------|------|------|-----|-------| +| 117 | test-cve-database | 4 | 8GB | Stopped — CVE database test environment | ### Storage Pools -- **local**: System files, ISOs, and templates -- **local-lvm**: VM disk images (thin provisioned) -- **Vault**: Secure storage for sensitive data -- **PBS-Backups**: Proxmox Backup Server repository -- **iso-share**: Installation media library + +| Name | Type | Used | Total | Purpose | +|------|------|------|-------|---------| +| Vault | ZFS Pool | ~2% (110GB) | 4.36TB | Primary VM/CT disk storage | +| PBS-Backups | Proxmox Backup Server | ~29.78% | ~1TB | Automated backup repository | +| iso-share | NFS | ~1.61% | ~3TB | Installation media library | +| local | Directory | ~22.57% | 45GB | System files, ISOs, templates | +| local-lvm | LVM-Thin | ~0.01% | 69GB | VM disk images (thin provisioned) | + +### Network + +| Bridge | IP | Purpose | +|--------|-----|---------| +| vmbr0 | 192.168.2.100/24 | Primary LAN bridge (eno1) | +| vmbr1 | 192.168.3.0/24 | Internal/isolated bridge | + +--- ## Repository Structure ``` homelab/ -├── .env.example # Template for environment configuration -├── collect*.sh # Scripts for collecting Proxmox configuration -├── homelab-export-*/ # Exported configuration snapshots -├── CLAUDE.md # AI assistant guidance for this repository -├── INDEX.md # Comprehensive documentation index -├── START-HERE.md # Getting started guide -└── README.md # This file +├── services/ # Docker Compose service configurations +│ ├── n8n/ # n8n workflow automation +│ └── README.md # Services overview +├── monitoring/ # Observability stack configs +│ ├── grafana/ +│ ├── prometheus/ +│ └── pve-exporter/ +├── scripts/ +│ ├── crawlers-exporters/ # Infrastructure collection scripts +│ │ ├── collect.sh # Convenience wrapper (uses .env) +│ │ ├── collect-remote.sh # SSH wrapper for WSL2 +│ │ └── collect-homelab-config.sh # Main collection engine +│ ├── fixers/ # Problem-solving scripts +│ └── qol/ # Git utilities +├── start-here-docs/ # Getting started guides +├── sub-agents/ # AI agent role definitions +├── troubleshooting/ # Bug fixes and audit findings +├── disaster-recovery/ # Infrastructure export snapshots +├── .env.example # Configuration template +├── CLAUDE.md # AI assistant project context +├── INDEX.md # Comprehensive documentation index +└── README.md # This file ``` -## Monitoring & Observability - -The infrastructure includes a comprehensive monitoring stack deployed on VM 101 (monitoring-docker) at 192.168.2.114: - -### Components -- **Grafana** (Port 3000): Visualization and dashboards -- **Prometheus** (Port 9090): Metrics collection and time-series database -- **PVE Exporter** (Port 9221): Proxmox VE metrics exporter - -### Features -- Real-time infrastructure monitoring -- Resource utilization tracking for VMs and containers -- Storage pool metrics and trends -- Network traffic analysis -- Pre-configured Proxmox VE dashboards -- Alerting capabilities - -**Documentation**: See `monitoring/README.md` for complete setup and configuration guide. - -## Network Security - -### Zero-Trust Access -- **CT 112** - twingate-connector: Provides secure remote access without traditional VPN -- **Technology**: Twingate zero-trust network access -- **Benefits**: Simplified secure access, no complex VPN configurations - -## Automation & Integration - -### Workflow Automation -- **CT 113** - n8n at 192.168.2.107 -- **Database**: PostgreSQL 15+ -- **Features**: API integrations, scheduled workflows, webhook triggers -- **Documentation**: See `services/README.md` for n8n setup and troubleshooting - -## Quick Start - -### Prerequisites -- SSH access to your Proxmox server -- Basic familiarity with Linux command line -- WSL2 (if running on Windows) or native Linux environment - -### Initial Setup - -1. **Configure Environment**: - ```bash - cp .env.example .env - # Edit .env with your Proxmox host details - ``` - -2. **Collect Current Configuration**: - ```bash - ./collect-homelab-config.sh - ``` - -3. **Review Exported Data**: - ```bash - # The script creates timestamped exports in homelab-export-YYYYMMDD-HHMMSS/ - ``` - -## Scripts - -### collect-homelab-config.sh -Comprehensive configuration collection script with multiple collection levels: -- **basic**: Essential configurations only -- **standard**: Recommended for most users -- **full**: Detailed system information -- **paranoid**: Complete infrastructure snapshot - -### collect-remote.sh -Remote collection wrapper for running collection on Proxmox host via SSH. - -### collect.sh -Simplified collection interface. - -## Usage Guides - -- **START-HERE.md**: Complete beginner's guide -- **COLLECTION-GUIDE.md**: Detailed collection script documentation -- **SCRIPT-USAGE.md**: Advanced script usage and customization -- **INDEX.md**: Comprehensive documentation index - -## Security Notes - -- The `.env` file is ignored by git (contains sensitive information) -- Exported configurations may contain IP addresses and hostnames -- Review sanitization options in `.env` before sharing exports -- Always use `.env.example` as template and never commit actual `.env` - -## Backup Strategy - -- Automated backups via Proxmox Backup Server (PBS-Backups pool) -- Configuration exports stored in timestamped directories -- Version control for Infrastructure as Code changes - -## Contributing - -This is a personal homelab repository. If you're using this as a template: - -1. Fork the repository -2. Customize `.env` for your environment -3. Update VM/CT IDs and names to match your infrastructure -4. Modify scripts as needed for your use case - -## Documentation - -See the following files for detailed information: -- **CLAUDE.md**: AI assistant context and repository guidelines -- **WORKFLOW-DIAGRAM.txt**: Visual workflow representation -- **QUICK-START.md**: Fast-track setup guide - -## License - -This is a personal infrastructure repository. Use at your own risk. - -## Support - -For questions about: -- **Proxmox**: https://pve.proxmox.com/wiki/Main_Page -- **This Repository**: See documentation files or create an issue - --- -*Last Updated: 2025-12-07* -*Proxmox Version: 8.3.3* -*Infrastructure: 8 VMs, 2 Templates, 4 LXC Containers* +## Monitoring & Observability + +Deployed on VM 101 (monitoring-docker): + +| Component | Port | Purpose | +|-----------|------|---------| +| Grafana | 3000 | Dashboards and visualization | +| Prometheus | 9090 | Metrics collection | +| PVE Exporter | 9221 | Proxmox metrics scraper | + +See `monitoring/README.md` for setup and configuration details. + +--- + +## Reverse Proxy + +**Caddy** (CT 109, 192.168.2.129) handles reverse proxying and automatic TLS for all services. Replaced Nginx Proxy Manager in early 2026. + +--- + +## Remote Access + +**Twingate** (CT 112) provides zero-trust remote access without a traditional VPN. No open inbound firewall rules required. + +--- + +## Workflow Automation + +**n8n** (CT 113) runs on PostgreSQL 16 with the pgvector extension for RAG/vector search workflows. See `services/n8n/` for configuration and `scripts/fixers/` for common database repair scripts. + +--- + +## Collecting Your Infrastructure State + +```bash +# 1. Configure your environment +cp .env.example .env +nano .env # Set PROXMOX_HOST=192.168.2.100 + +# 2. Run the collector +bash scripts/crawlers-exporters/collect.sh + +# 3. Review the output +cat homelab-export-*/SUMMARY.md +``` + +See `start-here-docs/QUICK-START.md` for the full 5-minute setup guide. + +--- + +## Security Notes + +- `.env` is git-ignored — never commit it +- Exported configs sanitize passwords and tokens by default +- Review `troubleshooting/` for the December 2025 security audit findings and remediation roadmap +- See `20260331 - Homelab GitOps Optimization Plan` in Obsidian for the full GitOps and security hardening roadmap + +--- + +## Backup Strategy + +- **Automated**: Proxmox Backup Server (PBS-Backups pool) handles VM/CT snapshots +- **Config snapshots**: Run `collect.sh` periodically; exports stored in `disaster-recovery/` +- **Repository**: All config changes version-controlled here + +--- + +*Last Updated: 2026-03-31* +*Proxmox Version: 8.4.0* +*Infrastructure: 3 VMs running, 5 VMs stopped/templates, 3 LXC running, 1 LXC stopped* diff --git "a/disaster-recovery/homelab-export-20251211-144345/configs/lxc/113-n8n\nn8n\nn8n.conf" "b/disaster-recovery/homelab-export-20251211-144345/configs/lxc/113-n8n\nn8n\nn8n.conf" deleted file mode 100644 index b14dd32..0000000 --- "a/disaster-recovery/homelab-export-20251211-144345/configs/lxc/113-n8n\nn8n\nn8n.conf" +++ /dev/null @@ -1,46 +0,0 @@ -arch: amd64 -cores: 2 -features: nesting=1 -hostname: n8n -memory: 4096 -nameserver: 8.8.8.8 8.8.4.4 1.1.1.1 -net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.2.1,hwaddr=BC:24:11:BD:35:B7,ip=192.168.2.113/24,type=veth -ostype: debian -parent: pre-db-permission-fix -rootfs: Vault:subvol-113-disk-0,size=20G -searchdomain: apophisnetworking.net -swap: 2048 -unprivileged: 1 - -[pre-db-permission-fix] -#Before PostgreSQL schema permission fix -arch: amd64 -cores: 2 -features: nesting=1 -hostname: n8n -memory: 4096 -nameserver: 8.8.8.8 8.8.4.4 1.1.1.1 -net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.2.1,hwaddr=BC:24:11:BD:35:B7,ip=192.168.2.113/24,type=veth -ostype: debian -parent: pre-n8n-fix -rootfs: Vault:subvol-113-disk-0,size=20G -searchdomain: apophisnetworking.net -snaptime: 1764644598 -swap: 2048 -unprivileged: 1 - -[pre-n8n-fix] -#Before encryption key fix 2025-12-01_12%3A58 -arch: amd64 -cores: 2 -features: nesting=1 -hostname: n8n -memory: 4096 -nameserver: 8.8.8.8 8.8.4.4 1.1.1.1 -net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.2.1,hwaddr=BC:24:11:BD:35:B7,ip=192.168.2.113/24,type=veth -ostype: debian -rootfs: Vault:subvol-113-disk-0,size=20G -searchdomain: apophisnetworking.net -snaptime: 1764619109 -swap: 2048 -unprivileged: 1