403 lines
25 KiB
Plaintext
403 lines
25 KiB
Plaintext
|
================================================================================
|
||
|
|
HOMELAB INFRASTRUCTURE COLLECTION - WORKFLOW DIAGRAM
|
||
|
|
================================================================================
|
||
|
|
|
||
|
|
OVERVIEW
|
||
|
|
--------
|
||
|
|
This diagram shows how the collection scripts work together to gather your
|
||
|
|
Proxmox infrastructure configuration from WSL2.
|
||
|
|
|
||
|
|
|
||
|
|
ARCHITECTURE
|
||
|
|
------------
|
||
|
|
|
||
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||
|
|
│ WSL2 Environment (/mnt/c/Users/fam1n/Documents/homelab) │
|
||
|
|
│ │
|
||
|
|
│ ┌────────────────┐ ┌──────────────────┐ │
|
||
|
|
│ │ collect.sh │────────>│ .env file │ │
|
||
|
|
│ │ (convenience) │ reads │ (your config) │ │
|
||
|
|
│ └────────┬───────┘ └──────────────────┘ │
|
||
|
|
│ │ │
|
||
|
|
│ │ calls │
|
||
|
|
│ v │
|
||
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
||
|
|
│ │ collect-remote.sh │ │
|
||
|
|
│ │ (SSH orchestration & file transfer) │ │
|
||
|
|
│ └────────┬─────────────────────────────────────────┘ │
|
||
|
|
│ │ │
|
||
|
|
└───────────┼──────────────────────────────────────────────────────────────┘
|
||
|
|
│
|
||
|
|
│ SSH connection
|
||
|
|
│ (transfers & executes script)
|
||
|
|
v
|
||
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||
|
|
│ Proxmox VE Host (serviceslab) │
|
||
|
|
│ │
|
||
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
||
|
|
│ │ collect-homelab-config.sh │ │
|
||
|
|
│ │ (main collection engine - runs on Proxmox) │ │
|
||
|
|
│ └────────┬─────────────────────────────────────────┘ │
|
||
|
|
│ │ │
|
||
|
|
│ │ reads (READ-ONLY operations) │
|
||
|
|
│ │ │
|
||
|
|
│ ├──> /etc/pve/ (Proxmox configs) │
|
||
|
|
│ ├──> /etc/pve/qemu-server/ (VM configs) │
|
||
|
|
│ ├──> /etc/pve/lxc/ (Container configs) │
|
||
|
|
│ ├──> /etc/network/interfaces (Network config) │
|
||
|
|
│ ├──> Storage configs │
|
||
|
|
│ ├──> Backup configs │
|
||
|
|
│ └──> System information │
|
||
|
|
│ │
|
||
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
||
|
|
│ │ Generated Output: │ │
|
||
|
|
│ │ /root/homelab-export-<timestamp>/ │ │
|
||
|
|
│ │ /root/homelab-export-<timestamp>.tar.gz │ │
|
||
|
|
│ └────────┬─────────────────────────────────────────┘ │
|
||
|
|
│ │ │
|
||
|
|
└───────────┼──────────────────────────────────────────────────────────────┘
|
||
|
|
│
|
||
|
|
│ SCP download
|
||
|
|
│ (transfers archive back)
|
||
|
|
v
|
||
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||
|
|
│ WSL2 Environment (./exports/) │
|
||
|
|
│ │
|
||
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
||
|
|
│ │ homelab-export-<timestamp>/ │ │
|
||
|
|
│ │ ├── README.md │ │
|
||
|
|
│ │ ├── SUMMARY.md │ │
|
||
|
|
│ │ ├── configs/ │ │
|
||
|
|
│ │ │ ├── proxmox/ │ │
|
||
|
|
│ │ │ ├── vms/ │ │
|
||
|
|
│ │ │ ├── lxc/ │ │
|
||
|
|
│ │ │ ├── storage/ │ │
|
||
|
|
│ │ │ ├── network/ │ │
|
||
|
|
│ │ │ └── backup/ │ │
|
||
|
|
│ │ └── exports/ │ │
|
||
|
|
│ │ ├── system/ │ │
|
||
|
|
│ │ ├── cluster/ │ │
|
||
|
|
│ │ └── guests/ │ │
|
||
|
|
│ └──────────────────────────────────────────────────┘ │
|
||
|
|
│ │
|
||
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
||
|
|
|
||
|
|
|
||
|
|
EXECUTION FLOW
|
||
|
|
--------------
|
||
|
|
|
||
|
|
Step 1: User Executes on WSL
|
||
|
|
┌──────────────────────┐
|
||
|
|
│ $ bash collect.sh │
|
||
|
|
└──────────┬───────────┘
|
||
|
|
│
|
||
|
|
v
|
||
|
|
Step 2: Load Configuration
|
||
|
|
┌─────────────────────────────────┐
|
||
|
|
│ Read .env file │
|
||
|
|
│ Set PROXMOX_HOST, SSH user, etc│
|
||
|
|
└──────────┬──────────────────────┘
|
||
|
|
│
|
||
|
|
v
|
||
|
|
Step 3: Call Remote Wrapper
|
||
|
|
┌─────────────────────────────────────────┐
|
||
|
|
│ collect-remote.sh 192.168.1.100 │
|
||
|
|
│ --level standard --output ./exports │
|
||
|
|
└──────────┬──────────────────────────────┘
|
||
|
|
│
|
||
|
|
v
|
||
|
|
Step 4: Test SSH Connection
|
||
|
|
┌─────────────────────────────────┐
|
||
|
|
│ SSH to Proxmox │
|
||
|
|
│ Verify /etc/pve/.version exists │
|
||
|
|
└──────────┬──────────────────────┘
|
||
|
|
│
|
||
|
|
v
|
||
|
|
Step 5: Upload Collection Script
|
||
|
|
┌──────────────────────────────────────────┐
|
||
|
|
│ SCP collect-homelab-config.sh to │
|
||
|
|
│ /tmp/collect-homelab-config.sh │
|
||
|
|
└──────────┬───────────────────────────────┘
|
||
|
|
│
|
||
|
|
v
|
||
|
|
Step 6: Execute on Proxmox
|
||
|
|
┌──────────────────────────────────────────────┐
|
||
|
|
│ SSH: /tmp/collect-homelab-config.sh │
|
||
|
|
│ --level standard --compress │
|
||
|
|
│ │
|
||
|
|
│ ┌──────────────────────────────────────────┐ │
|
||
|
|
│ │ Create directory structure │ │
|
||
|
|
│ │ Collect system information │ │
|
||
|
|
│ │ Collect Proxmox configs │ │
|
||
|
|
│ │ Collect VM configs (100-111) │ │
|
||
|
|
│ │ Collect LXC configs (102, 103, 112) │ │
|
||
|
|
│ │ Collect network configs │ │
|
||
|
|
│ │ Collect storage configs │ │
|
||
|
|
│ │ Collect backup configs │ │
|
||
|
|
│ │ Generate README and SUMMARY │ │
|
||
|
|
│ │ Compress to .tar.gz │ │
|
||
|
|
│ └──────────────────────────────────────────┘ │
|
||
|
|
└──────────┬───────────────────────────────────┘
|
||
|
|
│
|
||
|
|
v
|
||
|
|
Step 7: Download Results
|
||
|
|
┌──────────────────────────────────────────┐
|
||
|
|
│ Find newest .tar.gz on Proxmox │
|
||
|
|
│ SCP to WSL: ./exports/ │
|
||
|
|
│ Extract archive locally │
|
||
|
|
└──────────┬───────────────────────────────┘
|
||
|
|
│
|
||
|
|
v
|
||
|
|
Step 8: Cleanup (optional)
|
||
|
|
┌──────────────────────────────────────────┐
|
||
|
|
│ Remove export from Proxmox │
|
||
|
|
│ (unless --keep-remote specified) │
|
||
|
|
└──────────┬───────────────────────────────┘
|
||
|
|
│
|
||
|
|
v
|
||
|
|
Step 9: Display Summary
|
||
|
|
┌──────────────────────────────────────────┐
|
||
|
|
│ Show collection statistics │
|
||
|
|
│ Display path to results │
|
||
|
|
│ Show SUMMARY.md excerpt │
|
||
|
|
└──────────────────────────────────────────┘
|
||
|
|
|
||
|
|
|
||
|
|
DATA FLOW
|
||
|
|
---------
|
||
|
|
|
||
|
|
Your Proxmox VMs/CTs: What Gets Collected:
|
||
|
|
┌─────────────────────┐ ┌──────────────────────────┐
|
||
|
|
│ 100: docker-hub │──────────────>│ 100-docker-hub.conf │
|
||
|
|
│ 101: gitlab │──────────────>│ 101-gitlab.conf │
|
||
|
|
│ 105: dev │──────────────>│ 105-dev.conf │
|
||
|
|
│ 106: Ansible-Control│──────────────>│ 106-Ansible-Control.conf │
|
||
|
|
│ 108: CML │──────────────>│ 108-CML.conf │
|
||
|
|
│ 109: web-server-01 │──────────────>│ 109-web-server-01.conf │
|
||
|
|
│ 110: web-server-02 │──────────────>│ 110-web-server-02.conf │
|
||
|
|
│ 111: db-server-01 │──────────────>│ 111-db-server-01.conf │
|
||
|
|
│ 102: nginx (CT) │──────────────>│ 102-nginx.conf │
|
||
|
|
│ 103: netbox (CT) │──────────────>│ 103-netbox.conf │
|
||
|
|
│ 112: Anytype (CT) │──────────────>│ 112-Anytype.conf │
|
||
|
|
└─────────────────────┘ └──────────────────────────┘
|
||
|
|
|
||
|
|
Storage Pools: What Gets Collected:
|
||
|
|
┌─────────────────────┐ ┌──────────────────────────┐
|
||
|
|
│ local (Directory) │──────────────>│ storage.cfg │
|
||
|
|
│ local-lvm (LVM) │──────────────>│ pvesm status │
|
||
|
|
│ Vault (NFS) │──────────────>│ ZFS/LVM details │
|
||
|
|
│ PBS-Backups │──────────────>│ │
|
||
|
|
│ iso-share │──────────────>│ │
|
||
|
|
└─────────────────────┘ └──────────────────────────┘
|
||
|
|
|
||
|
|
|
||
|
|
COLLECTION LEVELS
|
||
|
|
-----------------
|
||
|
|
|
||
|
|
┌──────────┬─────────────────────────────────────────────────────────┐
|
||
|
|
│ Level │ What's Collected │
|
||
|
|
├──────────┼─────────────────────────────────────────────────────────┤
|
||
|
|
│ basic │ • System info (CPU, RAM, disk) │
|
||
|
|
│ │ • Proxmox configs (datacenter, storage, users) │
|
||
|
|
│ │ • VM/CT configs │
|
||
|
|
│ │ • Network configs │
|
||
|
|
├──────────┼─────────────────────────────────────────────────────────┤
|
||
|
|
│ standard │ • Everything in basic │
|
||
|
|
│ (default)│ • Storage pool status │
|
||
|
|
│ │ • Backup job definitions │
|
||
|
|
│ │ • Cluster information │
|
||
|
|
│ │ • Guest details (resource usage) │
|
||
|
|
├──────────┼─────────────────────────────────────────────────────────┤
|
||
|
|
│ full │ • Everything in standard │
|
||
|
|
│ │ • System service configs │
|
||
|
|
│ │ • Detailed service status │
|
||
|
|
│ │ • Extended system state │
|
||
|
|
├──────────┼─────────────────────────────────────────────────────────┤
|
||
|
|
│ paranoid │ • Everything possible │
|
||
|
|
│ │ • Experimental features │
|
||
|
|
│ │ • Maximum detail │
|
||
|
|
└──────────┴─────────────────────────────────────────────────────────┘
|
||
|
|
|
||
|
|
|
||
|
|
SANITIZATION OPTIONS
|
||
|
|
--------------------
|
||
|
|
|
||
|
|
┌─────────────┬──────────┬───────────┬──────────┐
|
||
|
|
│ Option │ Passwords│ Tokens │ IPs │
|
||
|
|
├─────────────┼──────────┼───────────┼──────────┤
|
||
|
|
│ default │ REDACTED │ REDACTED │ Preserved│
|
||
|
|
│ --sanitize │ REDACTED │ REDACTED │ REDACTED │
|
||
|
|
│ all │ │ │ (10.X.X.X)│
|
||
|
|
│ --sanitize │ Preserved│ Preserved │ REDACTED │
|
||
|
|
│ ips │ │ │ │
|
||
|
|
│ --sanitize │ Preserved│ Preserved │ Preserved│
|
||
|
|
│ none │ │ │ │
|
||
|
|
└─────────────┴──────────┴───────────┴──────────┘
|
||
|
|
|
||
|
|
|
||
|
|
SECURITY MODEL
|
||
|
|
--------------
|
||
|
|
|
||
|
|
┌─────────────────────────────────────────────────────────────────┐
|
||
|
|
│ SAFE (Read-Only Operations) │
|
||
|
|
├─────────────────────────────────────────────────────────────────┤
|
||
|
|
│ ✓ Reading configuration files │
|
||
|
|
│ ✓ Listing VMs and containers │
|
||
|
|
│ ✓ Querying system information │
|
||
|
|
│ ✓ Checking storage status │
|
||
|
|
│ ✓ Viewing network configuration │
|
||
|
|
│ ✓ Copying files to temporary locations │
|
||
|
|
└─────────────────────────────────────────────────────────────────┘
|
||
|
|
|
||
|
|
┌─────────────────────────────────────────────────────────────────┐
|
||
|
|
│ NOT COLLECTED (For Security/Size) │
|
||
|
|
├─────────────────────────────────────────────────────────────────┤
|
||
|
|
│ ✗ VM/CT disk images │
|
||
|
|
│ ✗ ISO files │
|
||
|
|
│ ✗ Backup archives │
|
||
|
|
│ ✗ User data inside VMs/CTs │
|
||
|
|
│ ✗ Private SSH keys │
|
||
|
|
│ ✗ Actual passwords (only structure/config) │
|
||
|
|
└─────────────────────────────────────────────────────────────────┘
|
||
|
|
|
||
|
|
|
||
|
|
FILES CREATED
|
||
|
|
-------------
|
||
|
|
|
||
|
|
After execution, you'll have these files:
|
||
|
|
|
||
|
|
WSL Directory Structure:
|
||
|
|
/mnt/c/Users/fam1n/Documents/homelab/
|
||
|
|
├── collect-homelab-config.sh (Main script - runs on Proxmox)
|
||
|
|
├── collect-remote.sh (SSH orchestration wrapper)
|
||
|
|
├── collect.sh (Convenience wrapper with .env)
|
||
|
|
├── .env.example (Configuration template)
|
||
|
|
├── .env (Your configuration - create this!)
|
||
|
|
├── QUICK-START.md (This guide)
|
||
|
|
├── README-COLLECTION.md (Overview and common usage)
|
||
|
|
├── COLLECTION-GUIDE.md (Comprehensive reference)
|
||
|
|
├── WORKFLOW-DIAGRAM.txt (This file)
|
||
|
|
└── exports/ (Output directory)
|
||
|
|
└── homelab-export-<timestamp>/
|
||
|
|
├── README.md
|
||
|
|
├── SUMMARY.md
|
||
|
|
├── collection.log
|
||
|
|
├── configs/
|
||
|
|
└── exports/
|
||
|
|
|
||
|
|
|
||
|
|
TIMING ESTIMATES
|
||
|
|
----------------
|
||
|
|
|
||
|
|
Collection Level | Proxmox Execution | Transfer Time | Total
|
||
|
|
-------------------|--------------------|-----------------|-----------
|
||
|
|
basic | 10-30 seconds | 5-10 seconds | ~30s
|
||
|
|
standard (default)| 20-60 seconds | 10-20 seconds | ~1min
|
||
|
|
full | 30-120 seconds | 15-30 seconds | ~2min
|
||
|
|
paranoid | 60-300 seconds | 20-60 seconds | ~5min
|
||
|
|
|
||
|
|
* Times vary based on number of VMs/CTs and network speed
|
||
|
|
|
||
|
|
|
||
|
|
TROUBLESHOOTING FLOW
|
||
|
|
--------------------
|
||
|
|
|
||
|
|
Problem: Script won't run
|
||
|
|
│
|
||
|
|
├─> Check: Is script in correct directory?
|
||
|
|
│ └─> cd /mnt/c/Users/fam1n/Documents/homelab
|
||
|
|
│
|
||
|
|
├─> Check: Using bash to execute?
|
||
|
|
│ └─> bash collect.sh (not ./collect.sh on Windows FS)
|
||
|
|
│
|
||
|
|
└─> Check: Is .env configured?
|
||
|
|
└─> cat .env | grep PROXMOX_HOST
|
||
|
|
|
||
|
|
Problem: Cannot connect to Proxmox
|
||
|
|
│
|
||
|
|
├─> Check: Can you ping Proxmox?
|
||
|
|
│ └─> ping 192.168.1.100
|
||
|
|
│
|
||
|
|
├─> Check: Can you SSH manually?
|
||
|
|
│ └─> ssh root@192.168.1.100
|
||
|
|
│
|
||
|
|
└─> Setup: SSH key authentication
|
||
|
|
└─> ssh-keygen -t ed25519
|
||
|
|
ssh-copy-id root@192.168.1.100
|
||
|
|
|
||
|
|
Problem: Some items skipped
|
||
|
|
│
|
||
|
|
└─> Normal! Check SUMMARY.md
|
||
|
|
Common skips:
|
||
|
|
• ZFS (if not using ZFS)
|
||
|
|
• Cluster (if single node)
|
||
|
|
• HA (if not configured)
|
||
|
|
|
||
|
|
|
||
|
|
INTEGRATION WORKFLOWS
|
||
|
|
---------------------
|
||
|
|
|
||
|
|
1. Version Control Integration:
|
||
|
|
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||
|
|
│ Collection │───>│ Git Init │───>│ Git Push │
|
||
|
|
│ Script │ │ & Commit │ │ to Remote │
|
||
|
|
└──────────────┘ └──────────────┘ └──────────────┘
|
||
|
|
|
||
|
|
2. Scheduled Automation:
|
||
|
|
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||
|
|
│ Cron Job │───>│ Collection │───>│ Retention │
|
||
|
|
│ (Weekly) │ │ Script │ │ Cleanup │
|
||
|
|
└──────────────┘ └──────────────┘ └──────────────┘
|
||
|
|
|
||
|
|
3. Infrastructure as Code:
|
||
|
|
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||
|
|
│ Collection │───>│ Analyze │───>│ Generate │
|
||
|
|
│ Export │ │ Configs │ │ Terraform │
|
||
|
|
└──────────────┘ └──────────────┘ └──────────────┘
|
||
|
|
|
||
|
|
|
||
|
|
QUICK COMMAND REFERENCE
|
||
|
|
-----------------------
|
||
|
|
|
||
|
|
# First time setup
|
||
|
|
cp .env.example .env
|
||
|
|
nano .env # Set PROXMOX_HOST
|
||
|
|
|
||
|
|
# Basic collection
|
||
|
|
bash collect.sh
|
||
|
|
|
||
|
|
# Full collection with verbose output
|
||
|
|
bash collect.sh --level full --verbose
|
||
|
|
|
||
|
|
# Sanitized for sharing
|
||
|
|
bash collect.sh --sanitize all
|
||
|
|
|
||
|
|
# Override Proxmox host
|
||
|
|
bash collect.sh --host 192.168.1.200
|
||
|
|
|
||
|
|
# Keep copy on Proxmox
|
||
|
|
bash collect.sh --keep-remote
|
||
|
|
|
||
|
|
# View results
|
||
|
|
cat exports/homelab-export-*/SUMMARY.md
|
||
|
|
ls -R exports/homelab-export-*/configs/
|
||
|
|
|
||
|
|
# Initialize git
|
||
|
|
cd exports/homelab-export-*/
|
||
|
|
git init && git add . && git commit -m "Initial snapshot"
|
||
|
|
|
||
|
|
|
||
|
|
SUPPORT RESOURCES
|
||
|
|
-----------------
|
||
|
|
|
||
|
|
1. Quick Start: QUICK-START.md (Getting started in 5 minutes)
|
||
|
|
2. Overview: README-COLLECTION.md (Common patterns and examples)
|
||
|
|
3. Full Guide: COLLECTION-GUIDE.md (Comprehensive reference)
|
||
|
|
4. This Diagram: WORKFLOW-DIAGRAM.txt (Architecture and flow)
|
||
|
|
5. Script Help: bash collect.sh --help
|
||
|
|
|
||
|
|
|
||
|
|
================================================================================
|
||
|
|
Ready to begin? Run: bash collect.sh
|
||
|
|
================================================================================
|