# Homelab Infrastructure Status **Last Updated**: 2025-12-11 14:43:55 **Export Reference**: disaster-recovery/homelab-export-20251211-144345 ## Current Infrastructure Snapshot ### Proxmox Environment - **Node**: serviceslab - **Version**: Proxmox VE 8.4.0 - **Management IP**: 192.168.2.200 - **Architecture**: Single-node cluster - **Total Resources**: 9 VMs, 2 Templates, 4 LXC Containers --- ## Virtual Machines (QEMU/KVM) - 9 VMs | VM ID | Name | IP Address | Status | Purpose | |-------|------|------------|--------|---------| | 100 | docker-hub | 192.168.2.XXX | Running | Container registry/Docker hub mirror | | 101 | monitoring-docker | 192.168.2.114 | Running | Monitoring stack (Grafana/Prometheus/PVE Exporter) | | 105 | dev | - | Stopped | General-purpose development workstation | | 106 | Ansible-Control | 192.168.2.XXX | Running | IaC orchestration, configuration management | | 108 | CML | - | Stopped | Cisco Modeling Labs - network simulation | | 109 | web-server-01 | 192.168.2.XXX | Running | Web application server (clustered) | | 110 | web-server-02 | 192.168.2.XXX | Running | Load-balanced pair with web-server-01 | | 111 | db-server-01 | 192.168.2.XXX | Running | Backend database server | | 114 | haos | 192.168.2.XXX | Running | Home Assistant OS - smart home automation platform | **Recent Changes**: - Added VM 101 (monitoring-docker) for dedicated monitoring infrastructure - Removed VM 101 (gitlab) - service decommissioned --- ## VM Templates - 2 Templates | Template ID | Name | Purpose | |-------------|------|---------| | 104 | ubuntu-dev | Ubuntu development environment template for cloning | | 107 | ubuntu-docker | Ubuntu Docker host template for rapid deployment | **Note**: Templates are immutable base images used for cloning new VMs, not running workloads. They provide standardized configurations for consistent infrastructure provisioning. --- ## Containers (LXC) - 4 Containers | CT ID | Name | IP Address | Status | Purpose | |-------|------|------------|--------|---------| | 102 | nginx | 192.168.2.101 | Running | Reverse proxy/load balancer & NPM | | 103 | netbox | 192.168.2.XXX | Running | Network documentation/IPAM | | 112 | twingate-connector | 192.168.2.XXX | Running | Zero-trust network access connector | | 113 | n8n | 192.168.2.107 | Running | Workflow automation platform | **Recent Changes**: - Added CT 112 (twingate-connector) for zero-trust network security - Added CT 113 (n8n) for workflow automation - Removed CT 112 (Anytype) - replaced by n8n --- ## Storage Architecture | Storage Pool | Type | Total | Used | % Used | Purpose | |--------------|------|-------|------|--------|---------| | local | Directory | - | - | 19.11% | System files, ISOs, templates | | local-lvm | LVM-Thin | - | - | 0.01% | VM disk images (thin provisioned) | | Vault | NFS/Directory | - | - | 12.13% | Secure storage for sensitive data | | PBS-Backups | PBS | - | - | 28.27% | Automated backup repository | | iso-share | NFS/CIFS | - | - | 1.45% | Installation media library | | localnetwork | Network Share | - | - | N/A | Shared resources across infrastructure | **Capacity Notes**: - PBS-Backups utilization increased to 28.27% (healthy retention) - Vault utilization increased to 12.13% (data growth monitored) - local storage at 19.11% (system overhead within normal range) --- ## Key Services & Stacks ### Monitoring & Observability (NEW) **VM 101** - monitoring-docker (192.168.2.114) - **Grafana**: Port 3000 - Visualization and dashboards - **Prometheus**: Port 9090 - Metrics collection and time-series database - **PVE Exporter**: Port 9221 - Proxmox VE metrics exporter - **Documentation**: `/home/jramos/homelab/monitoring/README.md` - **Status**: Fully operational ### Network Security (NEW) **CT 112** - twingate-connector - **Purpose**: Zero-trust network access - **Type**: Lightweight connector - **Status**: Running - **Integration**: Connects homelab to Twingate network ### Automation & Integration **CT 113** - n8n (192.168.2.107) - **Purpose**: Workflow automation platform - **Technology**: n8n.io - **Database**: PostgreSQL 15+ - **Features**: API integration, scheduled workflows, webhook triggers - **Documentation**: `/home/jramos/homelab/services/README.md#n8n-workflow-automation` - **Status**: Operational (resolved database locale issues) ### Infrastructure Documentation **CT 103** - netbox - **Purpose**: Network documentation and IPAM - **Status**: Stopped (on-demand use) - **Function**: Infrastructure source of truth ### Reverse Proxy & Load Balancing **CT 102** - nginx (192.168.2.101) - **Purpose**: Nginx Proxy Manager - **Ports**: 80, 81, 443 - **Function**: SSL termination, reverse proxy, certificate management - **Upstream Services**: All web-facing applications ### Three-Tier Application Stack **Web Tier**: - VM 109 (web-server-01) - Primary web server - VM 110 (web-server-02) - Load-balanced pair **Database Tier**: - VM 111 (db-server-01) - Backend database **Proxy Tier**: - CT 102 (nginx) - Load balancer and SSL termination ### Development & Automation **VM 106** - Ansible-Control - **Purpose**: Infrastructure as Code orchestration - **Tools**: Ansible, Terraform/OpenTofu (potential) - **Status**: Running ### Container Registry **VM 100** - docker-hub - **Purpose**: Local Docker registry and hub mirror - **Function**: Caching container images for faster deployments - **Status**: Running ### Network Simulation **VM 108** - CML - **Purpose**: Cisco Modeling Labs - **Function**: Network topology testing and simulation - **Status**: Stopped (resource-intensive, on-demand use) --- ## Architecture Patterns ### Monitoring & Observability (NEW) The infrastructure now implements a comprehensive monitoring stack following industry best practices: - **Metrics Collection**: Prometheus scraping Proxmox metrics via PVE Exporter - **Visualization**: Grafana providing real-time dashboards and alerting - **Isolation**: Dedicated VM for monitoring services (fault isolation) - **Integration**: Ready for AlertManager, additional exporters, and integrations **Design Decision**: VM-based deployment provides kernel-level isolation and prevents resource contention with critical infrastructure services. ### Zero-Trust Security (NEW) Implementation of zero-trust network access principles: - **Twingate Connector**: Lightweight connector providing secure access without VPNs - **Container Deployment**: LXC container for minimal resource overhead - **Network Segmentation**: Secure access to homelab from external networks **Design Decision**: LXC container chosen for quick provisioning and low resource consumption. ### Automation-First Approach Workflow automation and infrastructure orchestration: - **n8n Platform**: Visual workflow builder for API integrations - **Scheduled Tasks**: Automated backup checks, monitoring alerts, reports - **Integration Hub**: Connects monitoring, documentation, and operational tools **Design Decision**: PostgreSQL backend ensures data persistence and supports complex workflows. ### Tiered Application Architecture Classic three-tier design for production-like environments: - **Presentation Tier**: Paired web servers (109, 110) behind load balancer - **Business Logic**: Application processing on web tier - **Data Tier**: Dedicated database server (111) with backup strategy **Design Decision**: Separation of concerns, scalability testing, high availability patterns. ### Selective Containerization Strategy Hybrid approach balancing performance and resource efficiency: - **LXC Containers**: Stateless services (nginx, netbox, twingate, n8n) - **Full VMs**: Complex applications, kernel dependencies, heavy workloads - **Rationale**: LXC for ~10x lower overhead, VMs for isolation and compatibility --- ## Recent Infrastructure Changes ### 2025-12-11: Loki-Stack Monitoring Fully Operational **Issue Resolved:** Centralized logging pipeline now receiving syslog from UniFi router **Root Cause:** rsyslog filter in `/etc/rsyslog.d/unifi-router.conf` was configured for wrong source IP (192.168.1.1 instead of 192.168.2.1) **Fix Applied:** Updated rsyslog filter to match VLAN 2 gateway IP (192.168.2.1) **Status:** ✅ Complete - Logs flowing UniFi → rsyslog → Promtail → Loki → Grafana **Services Affected:** - VM 101 (monitoring-docker): rsyslog configuration updated - Loki-stack: All components operational - Grafana: Dashboards receiving real-time syslog data **Technical Details:** See `troubleshooting/loki-stack-bugfix.md` for complete 5-phase troubleshooting history --- ### 2025-12-11: Infrastructure Expansion & System Updates #### Proxmox VE Platform Upgrade - **Upgraded**: Proxmox VE 8.3.3 → 8.4.0 - **Kernel**: 6.8.12-8-pve - **pve-manager**: 8.4.14 - **Impact**: Enhanced performance, security updates, bug fixes - **Status**: ✅ Complete - All VMs and containers operating normally #### New VM 114: Home Assistant OS Deployment - **Service**: haos (Home Assistant Operating System) - **Purpose**: Smart home automation and integration platform - **Specifications**: - Memory: 4 GB (87% utilized) - CPU: 2 vCPUs - Boot Disk: 50 GB - Status: Running (~3 days uptime) - **Rationale**: Centralized home automation hub for IoT device management - **Integration**: Will integrate with monitoring stack for infrastructure metrics #### CT 103: NetBox IPAM Activated - **Service**: netbox (Network Documentation & IPAM) - **Status Change**: Stopped → Running - **Uptime**: ~3.1 days - **Resource Usage**: 1.28 GB / 2 GB memory (64%) - **Purpose**: Active network documentation and IP address management - **Rationale**: Required for ongoing infrastructure expansion planning #### Storage Utilization Trends - **PBS-Backups**: 27.43% → 28.27% (+0.84%) - Normal backup retention growth - **Vault (ZFS)**: 10.88% → 12.13% (+1.25%) - Data accumulation monitored - **local**: 15.13% → 19.11% (+3.98%) - New VM deployment and system updates - **iso-share**: 1.4% → 1.45% (+0.05%) - Minimal change - **local-lvm**: 0.0% → 0.01% (+0.01%) - Thin provisioned storage baseline --- ### 2025-12-07: Infrastructure Documentation & Monitoring Stack #### Additions 1. **VM 101 (monitoring-docker)**: New dedicated monitoring infrastructure - Grafana for visualization - Prometheus for metrics collection - PVE Exporter for Proxmox integration - IP: 192.168.2.114 2. **CT 112 (twingate-connector)**: Zero-trust network security - Lightweight connector - Secure remote access without VPN 3. **CT 113 (n8n)**: Workflow automation platform - PostgreSQL 15+ backend - IP: 192.168.2.107 - Resolved database locale issues ### Modifications - Storage utilization updated across all pools - PBS-Backups now at 27.43% (increased retention) - Vault optimized to 10.88% (reduced usage) ### Removals - **VM 101 (gitlab)**: Decommissioned (previously at this ID) - **CT 112 (Anytype)**: Replaced by n8n for better integration ### Documentation Updates - Created comprehensive monitoring stack documentation - Updated all infrastructure tables with current VMs/CTs - Added architecture patterns for observability and zero-trust - Updated storage statistics - Referenced latest export: disaster-recovery/homelab-export-20251207-120040 --- ## Repository Structure ``` homelab/ monitoring/ # NEW: Monitoring stack configurations README.md # Comprehensive monitoring documentation grafana/ docker-compose.yml prometheus/ docker-compose.yml prometheus.yml pve-exporter/ docker-compose.yml pve.yml .env services/ # Docker Compose service configurations n8n/ # n8n workflow automation netbox/ # Network documentation & IPAM README.md # Services overview (updated) disaster-recovery/ homelab-export-20251207-120040/ # Latest infrastructure export scripts/ crawlers-exporters/ # Infrastructure collection scripts fixers/ # Problem-solving scripts qol/ # Quality of life improvements CLAUDE.md # AI assistant guidance (updated) INDEX.md # Navigation index (updated) README.md # Repository overview (updated) CLAUDE_STATUS.md # This file - current infrastructure status ``` --- ## Current Initiative: Sub-Agent Architecture Optimization (2025-12-07) ### Goal Improve the quality and effectiveness of all sub-agent prompt definitions to match best practices identified through comprehensive Opus-powered prompt engineering analysis. Target: bring all sub-agents to the quality standard established by librarian.md (~120-340 lines with comprehensive examples, safety protocols, and decision frameworks). ### Phase COMPLETED - All sub-agent improvements and validations finished ### Progress Checklist - [x] Prompt engineering analysis completed (Opus model) - Analyzed CLAUDE.md and all 4 sub-agent files - Identified 5 critical issues, 12 high-impact improvements - Generated comprehensive improvement recommendations - [x] scribe.md improved (29 340 lines) - Added 6 usage examples (4 positive, 2 negative redirects) - Implemented comprehensive responsibilities section - Added 3 complete ASCII diagram templates - Included safety protocols and decision frameworks - Quality now matches librarian.md standard - [x] backend-builder.md improved (40 291 lines) - Added 6 usage examples with clear boundaries - Expanded core responsibilities with Ansible, Terraform, Docker Compose, Python, Shell - Added technology stack table and validation rules table - Included safety protocols for secrets and destructive operations - Added handoff protocol for lab-operator deployment - Defined clear boundaries (CREATES code, does NOT deploy) - [x] lab-operator.md improved (37 193 lines) - Added 6 usage examples with role clarity - Expanded domain expertise with specific commands - Added command style guide (5-step pattern) - Included safety protocols and decision-making framework - Added error handling and escalation guidelines - Defined clear boundaries (DEPLOYS/OPERATES, does NOT create IaC) - [x] CLAUDE.md structural fixes - Moved YAML frontmatter to line 1 (was at line 89) - Fixed trailing pipe character on line 87 - Completed incomplete sentence about backup strategy - Completed incomplete sentence about storage growth - Removed redundant "Key Services" reference - Expanded status file template with actual structure and recovery instructions - [x] Final validation and testing - librarian: Git status check successful, clear output format - scribe: File reading functional (note: reported encoding issue, likely false positive) - backend-builder: YAML validation successful, proper syntax checking - lab-operator: Directory listing successful, proper command execution - All agents demonstrate improved structure and clarity ### Context **Why It Matters**: Well-designed sub-agent prompts improve task routing accuracy, execution quality, error reduction, and maintainability. The librarian.md agent (143 lines) sets the quality standard; scribe was severely underdeveloped at 29 lines before improvement. **Next Steps**: Improve backend-builder.md and lab-operator.md using scribe.md as quality template. --- ## Previous Phase: Infrastructure Documentation Complete ### Goal Comprehensive documentation of monitoring stack and updated infrastructure inventory. ### Phase Documentation & Maintenance ### Completed Tasks - [x] Created `/home/jramos/homelab/monitoring/README.md` with comprehensive monitoring documentation - [x] Updated `CLAUDE_STATUS.md` with current infrastructure state - [x] Documented 8 VMs, 2 Templates, and 4 LXC containers - [x] Updated storage statistics (PBS 27.43%, Vault 10.88%, local 15.13%) - [x] Added monitoring stack architecture and deployment procedures - [x] Documented new services: monitoring-docker, twingate-connector, n8n - [x] Referenced latest export: disaster-recovery/homelab-export-20251207-120040 ### Remaining Documentation Tasks - [x] Update INDEX.md with monitoring section and current VM/CT counts - [x] Update README.md with infrastructure (8 VMs, 2 Templates, 4 LXC) - [x] Update CLAUDE.md with architecture tables for monitoring and zero-trust - [x] Update services/README.md with monitoring stack and twingate sections - [x] Verify all documentation cross-references are accurate - [ ] Test monitoring stack deployment procedures --- ## Access Information ### Management Interfaces - **Proxmox UI**: https://192.168.2.200:8006 - **Grafana**: http://192.168.2.114:3000 - **Prometheus**: http://192.168.2.114:9090 - **Nginx Proxy Manager**: http://192.168.2.101:81 - **n8n**: http://192.168.2.107:5678 ### Key Network Segments - **Management Network**: 192.168.2.0/24 - **Proxmox Host**: 192.168.2.200 - **Reverse Proxy**: 192.168.2.101 (CT 102) - **n8n**: 192.168.2.107 (CT 113) - **Monitoring**: 192.168.2.114 (VM 101) --- ## Maintenance Schedule ### Automated Tasks - **Backups**: Proxmox Backup Server - Daily incremental, Weekly full - **Monitoring Scrapes**: Prometheus - Every 30 seconds - **Certificate Renewal**: Nginx Proxy Manager - Automatic via Let's Encrypt ### Recommended Manual Tasks - **Weekly**: Review Grafana dashboards for anomalies - **Monthly**: Update monitoring stack Docker images - **Quarterly**: Review backup retention policies - **Semi-Annual**: Kernel updates on Proxmox host and VMs --- ## Known Issues & Resolutions ### Resolved - n8n PostgreSQL locale errors (fixed with `fix_n8n_db_c_locale.sh`) - n8n database permissions (fixed with `fix_n8n_db_permissions.sh`) ### Active Monitoring - PVE Exporter SSL verification (set to false for self-signed certificates) - Prometheus retention policies (currently 15 days, may need adjustment) ### Deferred - NetBox container offline (on-demand service) - Development VMs stopped (resource conservation) --- ## Version History - **v2.1.0** (2025-12-07): Added monitoring stack, twingate connector, updated infrastructure counts - **v2.0.0** (2025-12-02): Repository reorganization, services migration from GitLab - **v1.0.0** (2025-11-29): Initial infrastructure documentation --- **Maintained by**: jramos **Repository**: Homelab Infrastructure Configuration **Platform**: Proxmox VE 8.4.0 **Infrastructure Scale**: 9 VMs, 2 Templates, 4 Containers **Current Status**: Operational - Home Automation Integration Deployed