jramos/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0366c63d5156d2bdbaacf800c64e58919564d67c
homelab/INDEX.md
Jordan Ramos 4f69420aaa refactor(repo): reorganize repository structure for improved navigation and maintainability 
Implement comprehensive directory reorganization to improve discoverability,
logical grouping, and separation of concerns across documentation, scripts,
and infrastructure snapshots.

Major Changes:

1. Documentation Reorganization:
   - Created start-here-docs/ for onboarding documentation
     * Moved QUICK-START.md, START-HERE.md, GIT-SETUP-GUIDE.md
     * Moved GIT-QUICK-REFERENCE.md, SCRIPT-USAGE.md, SETUP-COMPLETE.md
   - Created troubleshooting/ directory
     * Moved BUGFIX-SUMMARY.md for centralized issue resolution
   - Created mcp/ directory for Model Context Protocol configurations
     * Moved OBSIDIAN-MCP-SETUP.md to mcp/obsidian/

2. Scripts Reorganization:
   - Created scripts/crawlers-exporters/ for infrastructure collection
     * Moved collect*.sh scripts and collection documentation
     * Consolidates Proxmox homelab export tooling
   - Created scripts/fixers/ for operational repair scripts
     * Moved fix_n8n_db_*.sh scripts
     * Isolated scripts with embedded credentials (templates tracked)
   - Created scripts/qol/ for quality-of-life utilities
     * Moved git-aliases.sh and git-first-commit.sh

3. Infrastructure Snapshots:
   - Created disaster-recovery/ for active infrastructure state
     * Moved latest homelab-export-20251202-204939/ snapshot
     * Contains current VM/CT configurations and system state
   - Created archive-homelab/ for historical snapshots
     * Moved homelab-export-*.tar.gz archives
     * Preserves point-in-time backups for reference

4. Agent Definitions:
   - Created sub-agents/ directory
     * Added backend-builder.md (development agent)
     * Added lab-operator.md (infrastructure operations agent)
     * Added librarian.md (git/version control agent)
     * Added scribe.md (documentation agent)

5. Updated INDEX.md:
   - Reflects new directory structure throughout
   - Updated all file path references
   - Enhanced navigation with new sections
   - Added agent roles documentation
   - Updated quick reference commands

6. Security Improvements:
   - Updated .gitignore to match reorganized file locations
   - Corrected path for scripts/fixers/fix_n8n_db_c_locale.sh exclusion
   - Maintained template-based credential management pattern

Infrastructure State Update:
   - Latest snapshot: 2025-12-02 20:49:54
   - Removed: VM 101 (gitlab), CT 112 (Anytype)
   - Added: CT 113 (n8n)
   - Total: 9 VMs, 3 Containers

Impact:
   - Improved repository navigation and discoverability
   - Logical separation of documentation, scripts, and snapshots
   - Clearer onboarding path for new users
   - Enhanced maintainability through organized structure
   - Foundation for multi-agent workflow support

Files changed: 90 files (+935/-349)
   - 3 modified, 14 new files, 73 renames/moves

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-02 21:39:33 -07:00

539 lines
21 KiB
Markdown
Raw Blame History

# Homelab Infrastructure Collection - File Index
Welcome to your homelab infrastructure collection toolkit! This index will help you navigate the various files and understand what each one does.
## Quick Navigation
**New to this?** Start here: **[start-here-docs/QUICK-START.md](start-here-docs/QUICK-START.md)**
**Ready to run?** Execute: `bash scripts/crawlers-exporters/collect.sh`
**Need help?** Check: **[scripts/crawlers-exporters/COLLECTION-GUIDE.md](scripts/crawlers-exporters/COLLECTION-GUIDE.md)**
## Repository Structure
```
homelab/
├── services/ # Docker Compose service configurations
│ ├── n8n/ # n8n workflow automation
│ ├── netbox/ # Network documentation & IPAM
│ └── README.md # Services overview
├── scripts/
│ ├── crawlers-exporters/ # Infrastructure collection scripts
│ │ ├── collect-homelab-config.sh # Main collection engine (runs on Proxmox)
│ │ ├── collect-remote.sh # SSH wrapper (orchestrates from WSL)
│ │ ├── collect.sh # Convenience wrapper (uses .env)
│ │ ├── COLLECTION-GUIDE.md # Comprehensive collection reference
│ │ └── README-COLLECTION.md # Collection overview
│ ├── fixers/ # Problem-solving scripts
│ │ ├── fix_n8n_db_c_locale.sh # PostgreSQL locale fix for n8n
│ │ └── fix_n8n_db_permissions.sh # Database permissions repair
│ └── qol/ # Quality of life improvements
│ ├── git-aliases.sh # Git command shortcuts
│ └── git-first-commit.sh # Repository initialization helper
├── start-here-docs/ # Getting started documentation
│ ├── START-HERE.md # Primary entry point
│ ├── QUICK-START.md # 5-minute quick start guide
│ ├── GIT-SETUP-GUIDE.md # Git configuration guide
│ ├── GIT-QUICK-REFERENCE.md # Git command reference
│ ├── SCRIPT-USAGE.md # Script usage examples
│ └── SETUP-COMPLETE.md # Post-setup checklist
├── troubleshooting/ # Problem resolution documentation
│ └── BUGFIX-SUMMARY.md # Known issues and fixes
├── n8n/ # n8n-specific documentation
│ └── N8N-SETUP-PLAN.md # n8n deployment guide
├── mcp/ # Model Context Protocol integrations
│ └── obsidian/
│ └── OBSIDIAN-MCP-SETUP.md # Obsidian MCP configuration
├── sub-agents/ # Agent role definitions
│ ├── scribe.md # Documentation & architecture agent
│ ├── backend-builder.md # Development agent
│ ├── lab-operator.md # Infrastructure operations agent
│ └── librarian.md # Knowledge management agent
├── homelab-export-20251202-204939/ # Latest infrastructure snapshot
│ ├── README.md # Export documentation
│ ├── SUMMARY.md # Collection statistics
│ ├── collection.log # Detailed collection log
│ ├── configs/ # Configuration files
│ │ ├── proxmox/ # Proxmox core configs
│ │ ├── vms/ # VM configurations
│ │ ├── lxc/ # Container configurations
│ │ ├── storage/ # Storage pool configs
│ │ ├── network/ # Network configs
│ │ ├── backup/ # Backup job configs
│ │ └── services/ # Systemd services
│ └── exports/ # Structured data exports
│ ├── system/ # System information
│ ├── cluster/ # Cluster status (JSON)
│ └── guests/ # VM/CT details
├── archive-homelab/ # Historical exports and backups
├── CLAUDE.md # AI assistant project guidance
├── INDEX.md # This file - navigation index
└── README.md # Repository overview
```
## File Inventory
### Core Scripts
| File | Location | Size | Purpose |
|------|----------|------|---------|
| `collect-homelab-config.sh` | `scripts/crawlers-exporters/` | 32K | Main collection engine - runs on Proxmox host |
| `collect-remote.sh` | `scripts/crawlers-exporters/` | 13K | SSH wrapper - orchestrates remote execution from WSL |
| `collect.sh` | `scripts/crawlers-exporters/` | 4.2K | Convenience wrapper - uses .env configuration |
**Which script should I use?**
- **Easiest**: `scripts/crawlers-exporters/collect.sh` (if you have `.env` configured)
- **Direct control**: `scripts/crawlers-exporters/collect-remote.sh <proxmox-ip>`
- **On Proxmox**: `scripts/crawlers-exporters/collect-homelab-config.sh` (if SSHed into Proxmox)
### Configuration
| File | Size | Purpose |
|------|------|---------|
| `.env.example` | 1.5K | Configuration template with all options |
| `.env` | - | YOUR configuration (create from .env.example) |
**Setup**: `cp .env.example .env` then edit `.env` with your Proxmox IP
### Documentation
| File | Location | Size | Purpose | Audience |
|------|----------|------|---------|----------|
| `QUICK-START.md` | `start-here-docs/` | 9.6K | Get started in 5 minutes | First-time users |
| `START-HERE.md` | `start-here-docs/` | - | Primary entry point | All users |
| `README-COLLECTION.md` | `scripts/crawlers-exporters/` | 8.5K | Overview and common patterns | General users |
| `COLLECTION-GUIDE.md` | `scripts/crawlers-exporters/` | 15K | Comprehensive reference | Power users |
| `GIT-SETUP-GUIDE.md` | `start-here-docs/` | - | Git configuration | New Git users |
| `GIT-QUICK-REFERENCE.md` | `start-here-docs/` | - | Git command cheat sheet | All users |
| `SCRIPT-USAGE.md` | `start-here-docs/` | - | Script examples | Script users |
| `INDEX.md` | Root | This file | Navigation and file index | Everyone |
| `CLAUDE.md` | Root | 5.7K | Project context for Claude | AI assistant |
| `README.md` | Root | - | Repository overview | All users |
### Which Documentation Should I Read?
```
┌─────────────────────────────────────────────────────┐
│ Just want to get started? │
│ → Read: start-here-docs/QUICK-START.md │
│ → Time: 5 minutes │
└─────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────┐
│ Want to understand common usage patterns? │
│ → Read: scripts/crawlers-exporters/ │
│ README-COLLECTION.md │
│ → Time: 15 minutes │
└─────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────┐
│ Need complete reference with all options? │
│ → Read: scripts/crawlers-exporters/ │
│ COLLECTION-GUIDE.md │
│ → Time: 30-45 minutes │
└─────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────┐
│ Setting up services (n8n, netbox, etc)? │
│ → Read: services/README.md │
│ n8n/N8N-SETUP-PLAN.md │
│ → Time: Variable │
└─────────────────────────────────────────────────────┘
```
## Typical Workflow
### First-Time Setup (5 minutes)
1. **Configure environment**
```bash
cp .env.example .env
nano .env # Set PROXMOX_HOST to your Proxmox IP
```
2. **Test SSH access**
```bash
ssh root@your-proxmox-ip
```
3. **Run first collection**
```bash
bash scripts/crawlers-exporters/collect.sh
```
4. **Review results**
```bash
cat homelab-export-*/SUMMARY.md
```
### Regular Use (1 minute)
```bash
# Just run this whenever you want to snapshot your infrastructure
bash scripts/crawlers-exporters/collect.sh
```
### Advanced Use
```bash
# Full collection with verbose output
bash scripts/crawlers-exporters/collect.sh --level full --verbose
# Sanitized export for sharing
bash scripts/crawlers-exporters/collect.sh --sanitize all
# Different Proxmox host
bash scripts/crawlers-exporters/collect.sh --host 192.168.1.200
```
## What Gets Created When You Run The Scripts?
After running `bash scripts/crawlers-exporters/collect.sh`, you'll have:
```
/home/jramos/homelab/
└── homelab-export-20251202-204939/
├── README.md # Documentation of this specific export
├── SUMMARY.md # Statistics and overview
├── collection.log # Detailed collection log
├── homelab-export-*.tar.gz # Compressed archive
├── configs/
│ ├── proxmox/ # Proxmox core configs
│ ├── vms/ # All VM configs (9 VMs)
│ ├── lxc/ # All container configs (3 containers)
│ ├── storage/ # Storage pool configs
│ ├── network/ # Network configs
│ ├── backup/ # Backup job configs
│ └── services/ # Systemd services
├── exports/
│ ├── system/ # System information
│ ├── cluster/ # Cluster status
│ └── guests/ # VM/CT details
├── docs/ # For your manual documentation
├── scripts/ # For your automation scripts
└── diagrams/ # For network diagrams
```
## Script Capabilities Matrix
| Feature | collect.sh | collect-remote.sh | collect-homelab-config.sh |
|---------|-----------|------------------|---------------------------|
| Uses .env config | ✓ | ✗ | ✗ |
| Runs from WSL | ✓ | ✓ | ✗ |
| Runs on Proxmox | ✗ | ✗ | ✓ |
| SSH automation | ✓ | ✓ | N/A |
| File transfer | ✓ | ✓ | N/A |
| Easiest to use | ✓ | - | - |
| Most flexible | - | ✓ | ✓ |
## Command Quick Reference
### Setup Commands
```bash
# First-time configuration
cp .env.example .env
nano .env
# Set up SSH keys (recommended)
ssh-keygen -t ed25519
ssh-copy-id root@<proxmox-ip>
```
### Collection Commands
```bash
# Standard collection (default)
bash scripts/crawlers-exporters/collect.sh
# Basic collection (minimal)
bash scripts/crawlers-exporters/collect.sh --level basic
# Full collection (comprehensive)
bash scripts/crawlers-exporters/collect.sh --level full
# With verbose output
bash scripts/crawlers-exporters/collect.sh --verbose
# Sanitized (safe for sharing)
bash scripts/crawlers-exporters/collect.sh --sanitize all
# Keep copy on Proxmox
bash scripts/crawlers-exporters/collect.sh --keep-remote
# Custom output location
bash scripts/crawlers-exporters/collect.sh --output ~/backups
```
### Review Commands
```bash
# View summary
cat homelab-export-20251202-204939/SUMMARY.md
# View collection log
less homelab-export-20251202-204939/collection.log
# List collected files
ls -R homelab-export-20251202-204939/
# Browse VM configs
cat homelab-export-20251202-204939/configs/vms/*.conf
# Browse container configs
cat homelab-export-20251202-204939/configs/lxc/*.conf
# Check storage setup
cat homelab-export-20251202-204939/configs/proxmox/storage.cfg
```
### Help Commands
```bash
# Script help
bash scripts/crawlers-exporters/collect.sh --help
bash scripts/crawlers-exporters/collect-remote.sh --help
bash scripts/crawlers-exporters/collect-homelab-config.sh --help
# Documentation
cat start-here-docs/QUICK-START.md
cat scripts/crawlers-exporters/README-COLLECTION.md
cat scripts/crawlers-exporters/COLLECTION-GUIDE.md
```
## Collection Levels Explained
| Level | Time | Size | Contents | Use Case |
|-------|------|------|----------|----------|
| `basic` | ~30s | 1-3 MB | Configs only | Quick snapshots |
| `standard` | ~1min | 2-6 MB | Configs + status | Regular documentation |
| `full` | ~2min | 5-20 MB | Everything | Pre-maintenance backup |
| `paranoid` | ~5min | 10-50 MB | Maximum detail | Disaster recovery planning |
## Your Infrastructure
Based on the latest export (2025-12-02 20:49:54), your environment includes:
### Virtual Machines (QEMU/KVM) - 9 VMs
| VM ID | Name | Status | Purpose |
|-------|------|--------|---------|
| 100 | docker-hub | Running | Container registry/Docker hub mirror |
| 104 | ubuntu-dev | Stopped | Ubuntu development environment |
| 105 | dev | Stopped | General-purpose development workstation |
| 106 | Ansible-Control | Running | IaC orchestration, configuration management |
| 107 | ubuntu-docker | Stopped | Ubuntu Docker host |
| 108 | CML | Stopped | Cisco Modeling Labs - network simulation |
| 109 | web-server-01 | Running | Web application server (clustered) |
| 110 | web-server-02 | Running | Load-balanced pair with web-server-01 |
| 111 | db-server-01 | Running | Backend database server |
**Note**: VM 101 (gitlab) has been removed from the infrastructure.
### Containers (LXC) - 3 Containers
| CT ID | Name | Status | Purpose |
|-------|------|--------|---------|
| 102 | nginx | Running | Reverse proxy/load balancer |
| 103 | netbox | Stopped | Network documentation/IPAM |
| 113 | n8n | Running | Workflow automation platform |
**Note**: CT 112 (Anytype) has been replaced by CT 113 (n8n).
### Storage Pools
- **local** (Directory) - 14.8% used - System files, ISOs, templates
- **local-lvm** (LVM-Thin) - 0.0% used - VM disk images (thin provisioned)
- **Vault** (NFS/Directory) - 11.9% used - Secure storage for sensitive data
- **PBS-Backups** (Proxmox Backup Server) - 21.6% used - Automated backup repository
- **iso-share** (NFS/CIFS) - 1.4% used - Installation media library
- **localnetwork** (Network share) - Shared resources across infrastructure
All of these are documented in your collection exports!
## Latest Export Information
- **Export Directory**: `/home/jramos/homelab/homelab-export-20251202-204939/`
- **Collection Date**: 2025-12-02 20:49:54
- **Hostname**: serviceslab
- **Collection Level**: full
- **Script Version**: 1.0.0
## Common Questions
### Q: Which file do I run?
**A:** Start with `bash scripts/crawlers-exporters/collect.sh` - it's the easiest.
### Q: Do I need to set up .env?
**A:** Yes! Copy `.env.example` to `.env` and set your `PROXMOX_HOST`.
### Q: Will this modify my Proxmox setup?
**A:** No! All operations are read-only. Zero risk.
### Q: How much disk space do I need?
**A:** On Proxmox: ~100-500 MB. On WSL: ~50-250 MB for archives.
### Q: Can I run this regularly?
**A:** Absolutely! That's what it's designed for. Run weekly or monthly.
### Q: What if I get errors?
**A:** Check `SUMMARY.md` and `collection.log` in the export. Most "errors" are just skipped features you don't use (like ZFS if you don't have ZFS).
### Q: Is my data secure?
**A:** By default, passwords and tokens are sanitized. Use `--sanitize all` for complete sanitization including IPs.
### Q: Can I use this for disaster recovery?
**A:** Yes! The configs show you exactly how your infrastructure is set up. Restore disk images from PBS-Backups and recreate VMs using these configs.
### Q: How do I fix n8n database issues?
**A:** Use the scripts in `scripts/fixers/`:
```bash
bash scripts/fixers/fix_n8n_db_permissions.sh
bash scripts/fixers/fix_n8n_db_c_locale.sh
```
## Integration Examples
### Git Version Control
```bash
cd homelab-export-20251202-204939/
git init
git add .
git commit -m "Infrastructure snapshot 2025-12-02"
git remote add origin <your-private-repo>
git push -u origin main
```
### Scheduled Collections (on Proxmox)
```bash
# Add to Proxmox crontab
0 3 * * 0 /root/collect-homelab-config.sh -l standard -o /backup/weekly-$(date +\%Y\%U)
```
### Automated Retention
```bash
# Keep only last 4 weeks
find ./homelab-export-* -maxdepth 0 -type d -mtime +28 -exec rm -rf {} \;
```
## Troubleshooting Quick Guide
| Problem | Solution |
|---------|----------|
| "Cannot connect to Proxmox" | Check IP in `.env`, test with `ssh root@<ip>` |
| "Permission denied" | Use `bash collect.sh` not `./collect.sh` on Windows FS |
| "PROXMOX_HOST not set" | Edit `.env` file and set your Proxmox IP |
| "Some items skipped" | Normal! Check `SUMMARY.md` - usually ZFS/cluster features you don't use |
| "Script won't execute" | Use: `bash scripts/crawlers-exporters/collect.sh` (explicitly call bash) |
| SSH asks for password | Set up SSH keys: `ssh-copy-id root@<proxmox-ip>` |
| n8n 502 Bad Gateway | Check `troubleshooting/BUGFIX-SUMMARY.md` and `n8n/N8N-SETUP-PLAN.md` |
| PostgreSQL locale errors | Run `bash scripts/fixers/fix_n8n_db_c_locale.sh` |
For detailed troubleshooting, see: **[troubleshooting/BUGFIX-SUMMARY.md](troubleshooting/BUGFIX-SUMMARY.md)**
## File Sizes Reference
| Component | Size | Notes |
|-----------|------|-------|
| Scripts | ~50K total | All three collection scripts |
| Documentation | ~65K total | All markdown and text files |
| Configuration | ~2K | .env.example template |
| **Total Package** | **~117K** | Everything you need |
| **Output (basic)** | 1-3 MB | Per collection run |
| **Output (standard)** | 2-6 MB | Per collection run |
| **Output (full)** | 5-20 MB | Per collection run |
## Service Management
### n8n Workflow Automation
- **Container**: CT 113
- **Service Config**: `services/n8n/`
- **Setup Guide**: `n8n/N8N-SETUP-PLAN.md`
- **Common Issues**: See `troubleshooting/BUGFIX-SUMMARY.md`
### NetBox IPAM
- **Container**: CT 103
- **Service Config**: `services/netbox/`
- **Purpose**: Network documentation and IP address management
### Nginx Reverse Proxy
- **Container**: CT 102
- **Purpose**: Front-end traffic management, SSL termination
## Agent Roles
This repository works with multiple AI agent personas:
| Agent | File | Purpose |
|-------|------|---------|
| **Scribe** | `sub-agents/scribe.md` | Documentation & architecture |
| **Backend Builder** | `sub-agents/backend-builder.md` | Development & coding |
| **Lab Operator** | `sub-agents/lab-operator.md` | Infrastructure operations |
| **Librarian** | `sub-agents/librarian.md` | Knowledge management |
## Next Steps
1. **Immediate**: Read [start-here-docs/QUICK-START.md](start-here-docs/QUICK-START.md) (5 minutes)
2. **Setup**: Create your `.env` file
```bash
cp .env.example .env
nano .env # Set PROXMOX_HOST
```
3. **Execute**: Run your first collection
```bash
bash scripts/crawlers-exporters/collect.sh
```
4. **Review**: Check the results
```bash
cat homelab-export-20251202-204939/SUMMARY.md
```
5. **Explore**: Browse the collected configs
```bash
ls -R homelab-export-20251202-204939/
```
6. **Learn More**: Read [scripts/crawlers-exporters/README-COLLECTION.md](scripts/crawlers-exporters/README-COLLECTION.md) for common patterns
7. **Master It**: Study [scripts/crawlers-exporters/COLLECTION-GUIDE.md](scripts/crawlers-exporters/COLLECTION-GUIDE.md) for advanced usage
## Support and Resources
- **Quick help**: `bash scripts/crawlers-exporters/collect.sh --help`
- **Detailed help**: See [scripts/crawlers-exporters/COLLECTION-GUIDE.md](scripts/crawlers-exporters/COLLECTION-GUIDE.md)
- **Getting started**: See [start-here-docs/QUICK-START.md](start-here-docs/QUICK-START.md)
- **Examples**: See [scripts/crawlers-exporters/README-COLLECTION.md](scripts/crawlers-exporters/README-COLLECTION.md)
- **Troubleshooting**: See [troubleshooting/BUGFIX-SUMMARY.md](troubleshooting/BUGFIX-SUMMARY.md)
## Summary
You now have a complete, production-ready infrastructure collection system that:
✓ Automatically collects all Proxmox configurations
✓ Works seamlessly from WSL2 via SSH
✓ Sanitizes sensitive information
✓ Creates organized, documented exports
✓ Supports automation and scheduling
✓ Provides comprehensive documentation
✓ Is completely safe (read-only operations)
✓ Includes service management for n8n, netbox, nginx
✓ Has troubleshooting resources and fix scripts
**Ready to begin?**
```bash
# Set up configuration
cp .env.example .env
nano .env
# Run your first collection
bash scripts/crawlers-exporters/collect.sh
```
---
**Repository Version:** 2.0.0
**Last Updated**: 2025-12-02
**Latest Export**: homelab-export-20251202-204939
**Infrastructure**: 9 VMs, 3 Containers, Proxmox VE 8.3.3
**Maintained by**: Your homelab automation system
Reference in New Issue View Git Blame Copy Permalink