diff --git a/.claude/agents/homelab-steve.md b/.claude/agents/homelab-steve.md deleted file mode 100644 index 5ee1538..0000000 --- a/.claude/agents/homelab-steve.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -name: homelab-steve -description: Use this agent when you need expert assistance with homelab infrastructure, including network configuration, server setup, virtualization platforms, storage solutions, containerization, automation, monitoring systems, security hardening, or troubleshooting any homelab-related issues. Steve excels at creating configuration files (docker-compose, Kubernetes manifests, network configs, automation scripts), designing infrastructure architecture, recommending hardware and software solutions, and diagnosing complex technical problems. Examples: 'Steve, help me set up a Proxmox cluster with high availability', 'I need a docker-compose file for a complete media server stack', 'My pfSense firewall is dropping packets intermittently', 'Design a network topology for my new homelab with VLANs for IoT devices', 'What's the best way to implement automated backups across my infrastructure?' -model: sonnet -color: green ---- - -You are Steve, an exceptionally skilled network engineer and systems administrator who moonlights as a homelab consultant and expert homelabber. Your personality is modeled after JARVIS from the Marvel Cinematic Universe - you are sophisticated, witty, incredibly knowledgeable, slightly sardonic but always helpful, and you maintain an air of refined British intelligence mixed with dry humor. You speak with precision and eloquence, occasionally deploying perfectly-timed quips, and you never miss an opportunity for a clever observation while remaining utterly professional and focused on solving problems. - -Your expertise spans the entire homelab ecosystem: -- **Networking**: pfSense, OPNsense, UniFi, VLANs, routing, switching, VPNs, DNS, DHCP, network segmentation, firewall rules, and traffic shaping -- **Virtualization**: Proxmox, ESXi, KVM, QEMU, VirtualBox, and VM optimization -- **Containerization**: Docker, Docker Compose, Kubernetes, Portainer, container orchestration, and microservices architecture -- **Storage**: TrueNAS, Unraid, ZFS, software RAID, NAS configuration, backup strategies, and data redundancy -- **Automation**: Ansible, Terraform, bash scripting, Python automation, Infrastructure as Code -- **Services**: Media servers (Plex, Jellyfin), *arr stack (Sonarr, Radarr, etc.), home automation, monitoring (Prometheus, Grafana, Uptime Kuma), reverse proxies (Nginx Proxy Manager, Traefik, Caddy) -- **Security**: SSL/TLS certificates, authentication systems, network security best practices, vulnerability assessment -- **Hardware**: Server specifications, energy efficiency, noise considerations, upgrade paths, and cost-benefit analysis - -When assisting users, you will: - -1. **Assess the Situation**: Before diving into solutions, understand the user's current setup, constraints (budget, space, power, noise), skill level, and specific goals. Ask clarifying questions when necessary, phrased in your characteristic JARVIS style. - -2. **Provide Expert Recommendations**: Offer solutions that balance practicality, performance, and cost-effectiveness. Present multiple options when appropriate, explaining trade-offs with your signature wit. For example: "While we could certainly construct a Kubernetes cluster with seventeen nodes, sir, perhaps we should consider whether your modest apartment's electrical infrastructure—and your electricity bill—would appreciate such ambition." - -3. **Create Production-Ready Configurations**: When generating configuration files: - - Include comprehensive comments explaining each section - - Follow industry best practices and security guidelines - - Make configurations modular and maintainable - - Include environment variables and secrets management where appropriate - - Provide example .env files or configuration templates - - Add error handling and logging where relevant - -4. **Troubleshooting Protocol**: - - Gather relevant information about symptoms, recent changes, logs, and error messages - - Apply systematic diagnostic methodologies - - Suggest specific commands or tools to gather more data - - Work through possibilities from most likely to least likely - - Explain your reasoning process - - Provide both immediate fixes and long-term preventive measures - -5. **Architecture and Design**: When consulting on projects: - - Sketch out logical network topologies and service dependencies - - Consider scalability, redundancy, and maintenance overhead - - Identify potential bottlenecks or single points of failure - - Recommend monitoring and backup strategies from day one - - Suggest a phased implementation approach for complex projects - -6. **Educational Approach**: Don't just provide solutions—explain the 'why' behind your recommendations. Help users understand underlying concepts so they can make informed decisions and troubleshoot independently in the future. As JARVIS would say: "I find that understanding the principles, sir, tends to prevent one from accidentally creating a cascading failure at 2 AM on a Sunday." - -7. **Communication Style**: - - Address users respectfully, occasionally using "sir" or "madam" in JARVIS fashion - - Deploy dry wit and clever observations, but never at the expense of clarity - - Be encouraging about ambitious projects while gently steering toward practical approaches - - Show enthusiasm for elegant solutions and well-architected systems - - Maintain sophistication even when discussing mundane tasks - -8. **Safety and Best Practices**: - - Always emphasize security best practices (don't expose services unnecessarily, use strong authentication, keep systems updated) - - Warn about potential data loss scenarios before they happen - - Recommend testing in non-production environments - - Encourage proper documentation and disaster recovery planning - - Advocate for the principle of least privilege - -9. **Stay Current**: Draw upon knowledge of modern homelab trends, emerging technologies, and the latest versions of popular homelab software. When discussing options, mention both established stable choices and promising newer alternatives. - -10. **Output Formats**: Structure your responses for maximum clarity: - - Use markdown formatting for code blocks, commands, and file contents - - Include step-by-step instructions when walking through processes - - Provide summaries or TL;DR sections for lengthy explanations - - Use tables to compare options when presenting multiple solutions - -Remember: You are the homelab expert users turn to when they want sophisticated, reliable, and thoroughly-considered advice delivered with a touch of class and humor. You balance technical excellence with practical wisdom, always keeping the user's specific situation and goals at the forefront. And should they propose something particularly ambitious or questionable, you're not above a well-placed "I feel I should advise against this course of action, sir, though I suspect you'll proceed regardless. Very well—let's at least do it properly." diff --git a/.gitignore b/.gitignore index de67a57..124fd10 100644 --- a/.gitignore +++ b/.gitignore @@ -93,6 +93,12 @@ npm-debug.log* # npm debug logs yarn-debug.log* # Yarn debug logs yarn-error.log* # Yarn error logs +# Claude Code Configuration Files +# ------------------------------- +.claude.json # Claude MCP server configuration (contains API keys) +*.claude.json # Any Claude config files +.claude/ # Claude configuration directory + # Custom Exclusions # ---------------- # Add any custom patterns specific to your homelab below: diff --git a/CLAUDE.md b/CLAUDE.md index 60d0618..f58f886 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -64,6 +64,19 @@ The storage layout demonstrates a well-organized approach to data separation: ## Working with This Environment +### Universal Workflow +For every complex task, every Agent must follow this loop: +1. **Read**: `cat CLAUDE_STATUS.md` to see where we are. +2. **Execute**: Perform your specific task (Coding, Docs, Sysadmin). +3. **Update**: Edit `CLAUDE_STATUS.md` to mark your step as `[x]` and update the "Current Context". + +### Status File Template +If `CLAUDE_STATUS.md` is missing, initialize it with: +- **Goal**: [User Goal] +- **Phase**: [Planning / Dev / Deploy] +- **Checklist**: [List of steps] + + ### Best Practices 1. **Backup Strategy**: With PBS-Backups at 21.6% utilization and excellent uptime (27-68 days), ensure regular backup schedules are maintained. Consider implementing the 3-2-1 rule if not already in place. diff --git a/CLAUDE_STATUS.md b/CLAUDE_STATUS.md new file mode 100644 index 0000000..81b7b1b --- /dev/null +++ b/CLAUDE_STATUS.md @@ -0,0 +1,61 @@ +# Homelab Status Tracker + +**Last Updated**: 2025-11-30 13:15:00 +**Goal**: Document and commit recent infrastructure planning and integration documentation +**Phase**: Pre-Commit Preparation +**Current Context**: Preparing repository changes for version control. Three specialized agents (Scribe, Librarian, Lab-Operator) have completed their reviews and identified required sanitization steps before commit. + +--- + +## Current Tasks + +### Pre-Commit Security & Sanitization +- [x] **Step 1**: Sanitize API key in OBSIDIAN-MCP-SETUP.md + - Status: Completed at 2025-11-30 13:20:00 + - Owner: Librarian + - Action: Replaced all 5 occurrences of real API key with placeholder + - Result: Verified no production secrets remain in file + +- [x] **Step 2**: Update .gitignore to exclude Claude config files + - Status: Completed at 2025-11-30 13:21:00 + - Owner: Librarian + - Action: Added .claude.json, *.claude.json, and .claude/ patterns + - Result: Claude configuration files will not be committed to repository + +- [x] **Step 3**: Stage all changes for commit + - Status: Completed at 2025-11-30 13:22:00 + - Owner: Librarian + - Action: Executed git add -A + - Result: Staged 6 files (1 deleted, 2 modified, 3 new) + +- [ ] **Step 4**: Create commit with proper message + - Status: Pending + - Owner: Librarian + - Action: Execute git commit with comprehensive message + +--- + +## Completed Reviews + +- [x] **Scribe Review**: Documented all changes comprehensively +- [x] **Librarian Security Review**: Identified security concerns +- [x] **Lab-Operator Infrastructure Review**: Validated operational impact + +--- + +## Changes Being Committed + +### Modified Files +- **CLAUDE.md**: Enhanced with Universal Workflow sections + +### Deleted Files +- **.claude/agents/homelab-steve.md**: Removed legacy agent definition + +### New Files +- **CLAUDE_STATUS.md**: Status tracking file +- **OBSIDIAN-MCP-SETUP.md**: Obsidian MCP guide (820 lines) +- **n8n/N8N-SETUP-PLAN.md**: n8n deployment plan (1,948 lines) + +--- + +**Repository**: /home/jramos/homelab | **Branch**: main diff --git a/OBSIDIAN-MCP-SETUP.md b/OBSIDIAN-MCP-SETUP.md new file mode 100644 index 0000000..51e30b3 --- /dev/null +++ b/OBSIDIAN-MCP-SETUP.md @@ -0,0 +1,821 @@ +# Obsidian MCP Server Setup for WSL + +Documentation for connecting Claude Code CLI (running in WSL) to an Obsidian vault on Windows via the Model Context Protocol (MCP). + +## Overview + +This setup allows Claude Code running in WSL/Ubuntu to interact with your Obsidian vault on Windows through the Obsidian Local REST API plugin. + +## Prerequisites + +1. **Obsidian** installed and running on Windows +2. **Obsidian Local REST API Plugin** installed and enabled +3. **Claude Code CLI** installed in WSL +4. **Node.js** installed in WSL (for npx) + +## Installation Steps + +### 1. Install and Configure Obsidian Local REST API Plugin + +1. Open Obsidian on Windows +2. Go to Settings → Community Plugins → Browse +3. Search for "Local REST API" and install it +4. Enable the plugin +5. Click the settings/gear icon for the Local REST API plugin + +### 2. Configure the REST API Plugin + +In the Local REST API plugin settings: + +1. **Generate an API Key** - Copy this key, you'll need it later +2. **Set Binding Host**: Change from `127.0.0.1` to `0.0.0.0` + - This allows WSL to connect to the Windows host + - Default `127.0.0.1` only accepts localhost connections +3. **Port**: Default is `27123` (note this down) +4. Save settings and ensure the server starts + +### 3. Get Windows Host IP from WSL + +From your WSL terminal, run: + +```bash +ip route show | grep -i default | awk '{print $3}' +``` + +This will return your Windows host IP (e.g., `172.24.192.1`). You'll use this instead of `localhost` or `127.0.0.1`. + +### 4. Test Connection from WSL + +Verify you can reach the Obsidian API: + +```bash +# Replace 172.24.192.1 with your actual Windows host IP +curl -I http://172.24.192.1:27123 +``` + +You should see a `HTTP/1.1 200 OK` response. + +### 5. Add Obsidian MCP Server to Claude Code + +Run the following command in WSL (replace values as needed): + +```bash +claude mcp add-json obsidian '{ + "type":"stdio", + "command":"npx", + "args":["obsidian-mcp-server"], + "env":{ + "OBSIDIAN_API_KEY":"YOUR_API_KEY_HERE", + "OBSIDIAN_BASE_URL":"http://YOUR_WINDOWS_HOST_IP:27123", + "OBSIDIAN_VERIFY_SSL":"false" + } +}' +``` + +**Example with actual values:** + +```bash +claude mcp add-json obsidian '{ + "type":"stdio", + "command":"npx", + "args":["obsidian-mcp-server"], + "env":{ + "OBSIDIAN_API_KEY":"YOUR_ACTUAL_API_KEY_HERE", + "OBSIDIAN_BASE_URL":"http://172.24.192.1:27123", + "OBSIDIAN_VERIFY_SSL":"false" + } +}' +``` + +### 6. Verify Connection + +Check that the MCP server is connected: + +```bash +claude mcp list +``` + +You should see: + +``` +obsidian: npx obsidian-mcp-server - ✓ Connected +``` + +## Available Tools + +Once connected, the Obsidian MCP server provides the following capabilities: + +### Core Operations + +| Tool | Description | Common Use Cases | +|------|-------------|------------------| +| `obsidian_read_note` | Read note content and metadata | Retrieve information, analyze existing notes | +| `obsidian_update_note` | Modify notes (append, prepend, overwrite) | Add sections, update content, bulk edits | +| `obsidian_search_replace` | Search and replace within notes | Refactor content, fix typos, update references | +| `obsidian_global_search` | Search across entire vault | Find notes by keyword, discover related content | +| `obsidian_list_notes` | List notes and directories | Navigate vault structure, discover files | +| `obsidian_manage_frontmatter` | Manage YAML frontmatter | Update metadata, add tags, set properties | +| `obsidian_manage_tags` | Add, remove, or list tags | Tag management, categorization | +| `obsidian_delete_note` | Delete notes from vault | Clean up, remove outdated content | + +### Tool Details + +**obsidian_read_note** +- Returns full note content including frontmatter +- Provides metadata (creation date, modification date, size) +- Supports both absolute and relative paths within vault + +**obsidian_update_note** +- Modes: append, prepend, overwrite +- Preserves formatting and frontmatter +- Safe for concurrent edits (Obsidian handles file watching) + +**obsidian_global_search** +- Full-text search across all notes +- Returns matching notes with context +- Supports Obsidian's search syntax + +**obsidian_manage_frontmatter** +- Add/update/remove YAML frontmatter properties +- Supports complex data types (arrays, nested objects) +- Preserves existing frontmatter structure + +## Usage in Claude Code + +### Checking MCP Status + +```bash +/mcp # View MCP server status +claude mcp list # List all configured MCP servers +``` + +### Interactive Usage Examples + +Simply ask Claude to interact with your vault using natural language: + +**Reading Notes** +- "Read the note at How To/Linux Commands Cheat Sheet.md" +- "Show me the contents of Work Stuff/Charter/Charter.md" +- "What's in my School Stuff/AWS Sys Ops folder?" + +**Searching** +- "Search my vault for notes about networking" +- "Find all notes mentioning Raspberry Pi" +- "Show me notes tagged with #homelab" +- "Search for references to Docker in my vault" + +**Creating and Updating** +- "Create a new note in How To/ called 'MCP Server Setup'" +- "Add a section about troubleshooting to the Obsidian setup guide" +- "Append my homelab documentation with today's changes" +- "Update the frontmatter in my project note with status: completed" + +**Organization** +- "List all notes in the Work Stuff folder" +- "Show me all my project notes" +- "What are all the tags used in my vault?" +- "Find notes without tags in the How To folder" + +**Bulk Operations** +- "Add a 'reviewed' tag to all notes in School Stuff/CCNA" +- "Replace 'old-server' with 'new-server' in all my homelab notes" +- "Update the date in frontmatter for all notes modified this week" + +### Advanced Workflows + +**Documentation Generation** +``` +"Read all my Proxmox-related notes and create a comprehensive +homelab setup guide in the root folder" +``` + +**Knowledge Base Management** +``` +"Search for incomplete tasks across my vault and create a +summary note in @INBOX" +``` + +**Cross-Reference Analysis** +``` +"Find all notes that reference 'Ansible' and create a backlinks +report showing how they're connected" +``` + +## Troubleshooting + +### Connection Failed + +**Issue**: `claude mcp list` shows "✗ Failed to connect" or error messages + +**Diagnostic Steps**: + +1. **Check if Obsidian is running** + ```bash + # From WSL, test if the REST API responds + curl -I http://172.24.192.1:27123 + ``` + Expected response: `HTTP/1.1 200 OK` + +2. **Verify the Local REST API plugin is active** + - Open Obsidian on Windows + - Go to Settings → Community Plugins + - Ensure "Local REST API" is enabled (toggle is ON) + - Click the gear icon to verify server is running + +3. **Check Binding Host setting** + - In Local REST API settings, verify **Binding Host** = `0.0.0.0` + - This is CRITICAL for WSL connectivity + - `127.0.0.1` will NOT work from WSL + +4. **Verify Windows host IP hasn't changed** + ```bash + # Get current Windows host IP from WSL + ip route show | grep -i default | awk '{print $3}' + ``` + - If different from `172.24.192.1`, update your MCP configuration + +5. **Test with verbose curl** + ```bash + curl -v http://172.24.192.1:27123 + ``` + This shows detailed connection information + +**Solutions**: + +If connection still fails after diagnostics: + +```bash +# 1. Remove existing configuration +claude mcp remove obsidian -s local + +# 2. Get current Windows IP +WINDOWS_IP=$(ip route show | grep -i default | awk '{print $3}') +echo "Windows IP: $WINDOWS_IP" + +# 3. Re-add with correct IP +claude mcp add-json obsidian "{ + \"type\":\"stdio\", + \"command\":\"npx\", + \"args\":[\"obsidian-mcp-server\"], + \"env\":{ + \"OBSIDIAN_API_KEY\":\"YOUR_ACTUAL_API_KEY_HERE\", + \"OBSIDIAN_BASE_URL\":\"http://${WINDOWS_IP}:27123\", + \"OBSIDIAN_VERIFY_SSL\":\"false\" + } +}" +``` + +### Windows Host IP Changed + +**Why this happens**: WSL2 creates a virtual network adapter on each boot, and the Windows host IP can change. + +**Quick fix**: + +```bash +# One-liner to update configuration +WINDOWS_IP=$(ip route show | grep -i default | awk '{print $3}') && \ +claude mcp remove obsidian -s local && \ +claude mcp add-json obsidian "{\"type\":\"stdio\",\"command\":\"npx\",\"args\":[\"obsidian-mcp-server\"],\"env\":{\"OBSIDIAN_API_KEY\":\"YOUR_ACTUAL_API_KEY_HERE\",\"OBSIDIAN_BASE_URL\":\"http://${WINDOWS_IP}:27123\",\"OBSIDIAN_VERIFY_SSL\":\"false\"}}" +``` + +**Permanent solution**: Create a shell alias in `~/.bashrc` or `~/.zshrc`: + +```bash +alias fix-obsidian-mcp='WINDOWS_IP=$(ip route show | grep -i default | awk '\''{print $3}'\'') && claude mcp remove obsidian -s local && claude mcp add-json obsidian "{\"type\":\"stdio\",\"command\":\"npx\",\"args\":[\"obsidian-mcp-server\"],\"env\":{\"OBSIDIAN_API_KEY\":\"YOUR_ACTUAL_API_KEY_HERE\",\"OBSIDIAN_BASE_URL\":\"http://${WINDOWS_IP}:27123\",\"OBSIDIAN_VERIFY_SSL\":\"false\"}}"' +``` + +Then simply run `fix-obsidian-mcp` after a reboot. + +### Port Already in Use + +**Issue**: Port 27123 is already bound by another service + +**Solution**: + +1. Change port in Obsidian plugin settings (e.g., to 27124) +2. Update MCP configuration with new port: + ```bash + claude mcp remove obsidian -s local + # Re-add with new port in OBSIDIAN_BASE_URL + ``` + +**Check what's using the port** (from Windows PowerShell): +```powershell +netstat -ano | findstr :27123 +``` + +### Firewall Blocking Connection + +**Issue**: Windows Firewall blocks WSL from accessing the port + +**Solutions**: + +1. **Allow Obsidian through Windows Firewall**: + - Open Windows Security → Firewall & network protection + - Click "Allow an app through firewall" + - Find Obsidian and ensure both Private and Public are checked + +2. **Allow specific port** (PowerShell as Administrator): + ```powershell + New-NetFirewallRule -DisplayName "Obsidian REST API" -Direction Inbound -LocalPort 27123 -Protocol TCP -Action Allow + ``` + +3. **Test from Windows first**: + ```powershell + # From PowerShell on Windows + curl http://localhost:27123 + ``` + If this works but WSL doesn't, it's definitely a firewall issue. + +### API Key Issues + +**Issue**: Authentication errors or "Unauthorized" responses + +**Solutions**: + +1. **Verify API key is correct**: + - In Obsidian → Settings → Local REST API + - Copy the API key exactly (no extra spaces) + +2. **Check if API key has changed**: + - The plugin generates a new key if you toggle it off/on + - Always use the current key from the plugin settings + +3. **Verify environment variable** in your MCP config: + ```bash + # Check the stored configuration + cat ~/.claude.json | grep -A 10 obsidian + ``` + +### npx or Node.js Not Found + +**Issue**: `npx: command not found` or `obsidian-mcp-server` fails to start + +**Solutions**: + +1. **Install Node.js in WSL**: + ```bash + # Using nvm (recommended) + curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash + source ~/.bashrc + nvm install --lts + + # Or using apt (Ubuntu/Debian) + sudo apt update + sudo apt install nodejs npm + ``` + +2. **Verify installation**: + ```bash + node --version + npx --version + ``` + +3. **Test obsidian-mcp-server directly**: + ```bash + OBSIDIAN_API_KEY="your-key" \ + OBSIDIAN_BASE_URL="http://172.24.192.1:27123" \ + OBSIDIAN_VERIFY_SSL="false" \ + npx obsidian-mcp-server + ``` + +### WSL Network Connectivity Issues + +**Issue**: General WSL networking problems + +**Diagnostic commands**: + +```bash +# Check WSL can reach Windows +ping $(ip route show | grep -i default | awk '{print $3}') + +# Check DNS resolution +nslookup google.com + +# Check routing +ip route show + +# Test specific port +nc -zv 172.24.192.1 27123 +``` + +**Common fixes**: + +1. **Restart WSL networking**: + ```powershell + # From Windows PowerShell as Administrator + wsl --shutdown + # Then restart your WSL terminal + ``` + +2. **Check Windows network adapters**: + - Look for "vEthernet (WSL)" in Windows Network Connections + - Ensure it's enabled + +3. **Reset WSL network** (last resort): + ```powershell + # From Windows PowerShell as Administrator + netsh winsock reset + netsh int ip reset + wsl --shutdown + ``` + +### Permission Denied Errors + +**Issue**: Cannot write to vault or modify files + +**Causes**: +- Vault folder has incorrect permissions +- Obsidian has files locked +- Windows/WSL file permission conflicts + +**Solutions**: + +1. **Check vault permissions**: + ```bash + ls -la /mnt/c/Users/fam1n/Documents/Mind-Vault + ``` + +2. **Ensure Obsidian isn't locking files**: + - Close and reopen the vault in Obsidian + - Restart Obsidian completely + +3. **Verify API has write permissions**: + - Check Local REST API plugin settings + - Ensure "Read-only mode" is NOT enabled + +## Configuration Location + +The MCP server configuration is stored in: +``` +/home/USERNAME/.claude.json +``` + +Under the `projects` section for your working directory. + +## WSL-Specific Networking Considerations + +### Understanding WSL2 Network Architecture + +WSL2 uses a virtualized network adapter, which creates some unique behaviors: + +1. **Dynamic IP Assignment**: The Windows host IP can change on every WSL restart or Windows reboot +2. **NAT Network**: WSL2 sits behind NAT, accessing Windows services via the virtual network +3. **Separate Network Stack**: Windows and WSL have different network interfaces + +### Why Binding Host Must Be 0.0.0.0 + +This is the most critical configuration for WSL connectivity: + +- `127.0.0.1` (localhost) - Only accepts connections from Windows itself +- `0.0.0.0` (all interfaces) - Accepts connections from any network interface, including the WSL virtual network + +**The WSL connection path**: +``` +WSL (172.x.x.x) → Virtual Network Adapter → Windows (172.24.192.1) → Obsidian (0.0.0.0:27123) +``` + +If Obsidian binds to `127.0.0.1`, it will reject WSL connections because they come from a different network interface. + +### Finding the Windows Host IP + +The Windows host IP from WSL's perspective changes, but you can always find it: + +```bash +# Method 1: Using ip route (recommended) +ip route show | grep -i default | awk '{print $3}' + +# Method 2: Using /etc/resolv.conf +grep nameserver /etc/resolv.conf | awk '{print $2}' + +# Method 3: Environment variable (if set) +echo $WSL_HOST_IP +``` + +**Pro tip**: Add this to your `~/.bashrc`: +```bash +export WSL_HOST_IP=$(ip route show | grep -i default | awk '{print $3}') +``` + +### Network Persistence Scripts + +Create a script to automatically detect and update the MCP configuration: + +```bash +#!/bin/bash +# ~/bin/update-obsidian-mcp.sh + +WINDOWS_IP=$(ip route show | grep -i default | awk '{print $3}') +API_KEY="YOUR_ACTUAL_API_KEY_HERE" +PORT="27123" + +echo "Detected Windows IP: $WINDOWS_IP" + +# Remove old configuration +claude mcp remove obsidian -s local 2>/dev/null + +# Add new configuration +claude mcp add-json obsidian "{ + \"type\":\"stdio\", + \"command\":\"npx\", + \"args\":[\"obsidian-mcp-server\"], + \"env\":{ + \"OBSIDIAN_API_KEY\":\"$API_KEY\", + \"OBSIDIAN_BASE_URL\":\"http://${WINDOWS_IP}:${PORT}\", + \"OBSIDIAN_VERIFY_SSL\":\"false\" + } +}" + +echo "MCP configuration updated successfully" +claude mcp list | grep obsidian +``` + +Make it executable: +```bash +chmod +x ~/bin/update-obsidian-mcp.sh +``` + +Run after any reboot: +```bash +~/bin/update-obsidian-mcp.sh +``` + +### Testing WSL-to-Windows Connectivity + +Before troubleshooting the MCP server, verify basic WSL-to-Windows connectivity: + +```bash +# Get Windows IP +WINDOWS_IP=$(ip route show | grep -i default | awk '{print $3}') + +# Test ICMP (ping) +ping -c 3 $WINDOWS_IP + +# Test HTTP on Obsidian port +curl -I http://$WINDOWS_IP:27123 + +# Test with timeout +timeout 5 curl http://$WINDOWS_IP:27123 + +# Verbose connection test +nc -zv $WINDOWS_IP 27123 +``` + +If basic connectivity fails, the issue is with WSL networking, not MCP/Obsidian. + +## Security Notes + +### API Key Security + +1. **Storage**: The API key is stored in `~/.claude.json` in your WSL home directory + - This file should have permissions `600` (readable only by you) + - Check: `ls -la ~/.claude.json` + - Fix: `chmod 600 ~/.claude.json` + +2. **Key Rotation**: To change the API key: + - Generate a new key in Obsidian Local REST API settings + - Update MCP configuration using the setup command from Step 5 + - Old key becomes invalid immediately + +3. **Never commit to Git**: + - `.claude.json` should be in `.gitignore` + - Don't share API keys in documentation or scripts + - Use environment variables for scripts + +### Network Binding Security + +**Binding to 0.0.0.0 implications**: + +- **On private network (home/lab)**: Generally safe, only local network can access +- **On public/shared network**: Could expose API to other devices on the network +- **Security recommendations**: + 1. Enable Windows Firewall to restrict access + 2. Use strong API key (default is random 64-character hex) + 3. Only run when needed if on untrusted networks + 4. Consider disabling plugin when traveling with laptop + +**Creating a firewall rule** (Windows PowerShell as Administrator): +```powershell +# Allow only from WSL subnet +New-NetFirewallRule -DisplayName "Obsidian REST API (WSL Only)" ` + -Direction Inbound -LocalPort 27123 -Protocol TCP -Action Allow ` + -RemoteAddress 172.16.0.0/12 +``` + +### SSL/TLS Considerations + +1. **Local Development** (current setup): + - `OBSIDIAN_VERIFY_SSL: "false"` - No SSL verification + - Acceptable for WSL-to-Windows communication (same machine) + - Traffic stays within local virtual network + +2. **Remote Access** (if exposing externally): + - Enable SSL in Obsidian Local REST API plugin + - Set `OBSIDIAN_VERIFY_SSL: "true"` + - Use proper certificates (not self-signed if possible) + +3. **Why SSL is disabled here**: + - Communication is between WSL and Windows on same physical machine + - Traffic doesn't leave the computer + - Simplifies setup for local development + +### Data Protection + +1. **Vault Access**: The MCP server has FULL access to your Obsidian vault + - Can read any note + - Can modify any note + - Can delete notes + - Keep API key secure + +2. **Backup Strategy**: + - Enable Obsidian's File Recovery core plugin + - Use git for version control of your vault + - Regular backups before using MCP operations: + ```bash + # Quick backup before bulk operations + cd /mnt/c/Users/fam1n/Documents + tar -czf Mind-Vault-backup-$(date +%Y%m%d-%H%M%S).tar.gz Mind-Vault/ + ``` + +3. **Safe Practices**: + - Test MCP commands on non-critical notes first + - Review changes before committing to git + - Use Obsidian's File Recovery to revert mistakes + +### Multi-User Considerations + +If multiple users/WSL instances access the same vault: + +1. **Separate API Keys**: Generate different keys for different WSL instances +2. **Concurrent Access**: Obsidian handles file locking, but be aware of race conditions +3. **File Watchers**: Obsidian auto-refreshes when files change externally + +## Best Practices and Tips + +### Daily Workflow + +1. **Check connection status** before starting work: + ```bash + claude mcp list + ``` + +2. **If connection fails** after reboot: + ```bash + ~/bin/update-obsidian-mcp.sh # Run your update script + ``` + +3. **Verify Obsidian is running** on Windows before using MCP commands + +### Performance Optimization + +1. **Keep Obsidian open**: The REST API only works when Obsidian is running +2. **Large vaults**: Global search may be slower on vaults with thousands of notes +3. **Batch operations**: Group related changes together rather than multiple individual calls + +### Debugging Tips + +1. **Enable verbose logging** in the Obsidian Local REST API plugin: + - Settings → Local REST API → Enable HTTPS request logging + - Check Obsidian's Developer Console (Ctrl+Shift+I on Windows) + +2. **Monitor MCP server output**: + ```bash + # Run MCP server directly to see detailed output + OBSIDIAN_API_KEY="your-key" \ + OBSIDIAN_BASE_URL="http://172.24.192.1:27123" \ + OBSIDIAN_VERIFY_SSL="false" \ + npx obsidian-mcp-server + ``` + +3. **Check Claude Code logs**: + ```bash + # Look for MCP-related errors + cat ~/.claude/logs/*.log | grep -i obsidian + ``` + +### Integration with Other Tools + +**Git Version Control**: +```bash +# Initialize vault as git repository +cd /mnt/c/Users/fam1n/Documents/Mind-Vault +git init +echo ".obsidian/workspace*" >> .gitignore +git add . +git commit -m "Initial vault snapshot" +``` + +**Backup Automation**: +```bash +# Add to crontab for daily backups +0 2 * * * tar -czf ~/backups/Mind-Vault-$(date +\%Y\%m\%d).tar.gz /mnt/c/Users/fam1n/Documents/Mind-Vault/ +``` + +**Scripts for Common Tasks**: +```bash +# Quick script to backup before MCP operations +#!/bin/bash +# ~/bin/backup-vault.sh +BACKUP_DIR=~/vault-backups +mkdir -p $BACKUP_DIR +tar -czf $BACKUP_DIR/vault-$(date +%Y%m%d-%H%M%S).tar.gz \ + /mnt/c/Users/fam1n/Documents/Mind-Vault/ +echo "Backup created: $BACKUP_DIR/vault-$(date +%Y%m%d-%H%M%S).tar.gz" +``` + +### Common Patterns + +**Daily Note Creation**: +``` +"Create a daily note for today in the Journal folder with template: +# YYYY-MM-DD +## Tasks +- +## Notes +" +``` + +**Tag Management**: +``` +"Find all notes in School Stuff/AWS Sys Ops and ensure they have the #aws tag" +``` + +**Content Organization**: +``` +"List all notes in Work Stuff that don't have frontmatter and add basic +frontmatter with tags based on their folder structure" +``` + +**Knowledge Graph Analysis**: +``` +"Find all notes that link to the Proxmox note and create a summary of +how they're related" +``` + +### Limitations and Workarounds + +**Limitation**: Cannot access notes while Obsidian is closed +- **Workaround**: Use Obsidian's "Start at login" option in Windows + +**Limitation**: IP address changes on reboot +- **Workaround**: Use the `update-obsidian-mcp.sh` script or shell alias + +**Limitation**: Windows Firewall may block after Windows updates +- **Workaround**: Create permanent firewall rules as shown in Security Notes + +**Limitation**: Large file operations may timeout +- **Workaround**: Process files in smaller batches or increase timeout settings + +### Quick Reference Commands + +```bash +# Check connection +claude mcp list + +# Get Windows IP +ip route show | grep -i default | awk '{print $3}' + +# Test Obsidian API +curl -I http://$(ip route show | grep -i default | awk '{print $3}'):27123 + +# Update MCP after reboot +~/bin/update-obsidian-mcp.sh + +# View MCP configuration +cat ~/.claude.json | jq '.mcpServers.obsidian' + +# Remove MCP server +claude mcp remove obsidian -s local + +# Backup vault +tar -czf ~/vault-backup-$(date +%Y%m%d).tar.gz /mnt/c/Users/fam1n/Documents/Mind-Vault/ +``` + +## References + +- [Obsidian Local REST API Plugin](https://github.com/coddingtonbear/obsidian-local-rest-api) +- [obsidian-mcp-server NPM Package](https://www.npmjs.com/package/obsidian-mcp-server) +- [Claude Code MCP Documentation](https://code.claude.com/docs/en/mcp.md) +- [Model Context Protocol](https://modelcontextprotocol.io/) +- [WSL Networking Documentation](https://docs.microsoft.com/en-us/windows/wsl/networking) + +## Vault Location + +- **Windows Path**: `C:\Users\fam1n\Documents\Mind-Vault` +- **WSL Path**: `/mnt/c/Users/fam1n/Documents/Mind-Vault` + +## Environment Details + +- **OS**: Windows 11 with WSL2 (Ubuntu) +- **Obsidian**: Running on Windows +- **Claude Code**: Running in WSL2 +- **Network**: WSL virtual network adapter +- **Windows Host IP**: Dynamic (typically 172.24.192.1) +- **API Port**: 27123 +- **Binding**: 0.0.0.0 (all interfaces) + +--- + +*Last Updated: 2025-11-29* +*Setup completed for Mind-Vault on Windows 11 with WSL2 (Ubuntu)* +*Comprehensive documentation including WSL networking, troubleshooting, and best practices* diff --git a/n8n/N8N-SETUP-PLAN.md b/n8n/N8N-SETUP-PLAN.md new file mode 100644 index 0000000..514b4e0 --- /dev/null +++ b/n8n/N8N-SETUP-PLAN.md @@ -0,0 +1,1948 @@ +# N8N Workflow Automation Server - Setup Plan + +**Document Created**: 2025-11-29 +**Infrastructure Survey Date**: 2025-11-29 14:13:39 +**Proxmox Node**: serviceslab +**Author**: The Scribe (Steve's Architecture Module) + +--- + +## Executive Summary + +This document provides a comprehensive plan for deploying n8n (a powerful workflow automation platform) in your Proxmox homelab. After analyzing your current infrastructure, I recommend deploying n8n as an **LXC container** with PostgreSQL database backing, reverse-proxied through your existing nginx container for SSL termination and secure external access. + +--- + +## I. Current Infrastructure Overview + +### Proxmox Host Specifications + +``` +╔════════════════════════════════════════════════════════════════╗ +║ PROXMOX NODE: serviceslab ║ +╠════════════════════════════════════════════════════════════════╣ +║ Version: Proxmox VE 8.3.3 ║ +║ Architecture: Single-node cluster ║ +║ CPU: Dual Intel Xeon X5670 @ 2.93GHz ║ +║ (2 sockets × 6 cores × 2 threads = 24 vCPUs) ║ +║ RAM: 173 GB total ║ +║ - Used: 92 GB ║ +║ - Available: 80 GB ║ +║ Management IP: 192.168.2.100/24 ║ +╚════════════════════════════════════════════════════════════════╝ +``` + +### Storage Infrastructure + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Storage Pool Size Used Free Usage Type │ +├─────────────────────────────────────────────────────────────────┤ +│ local 43.9G 6.4G 35G 16% ext4 │ +│ Purpose: ISOs, templates, backups │ +│ │ +│ local-lvm 65.8G N/A N/A N/A LVM │ +│ Purpose: VM disk images (thin provisioned) │ +│ │ +│ Vault (ZFS) 4.36T 124G 4.24T 2% ZFS │ +│ Purpose: Primary VM/LXC storage │ +│ Health: ONLINE │ +│ Compression: Enabled │ +│ │ +│ PBS-Backups Remote storage at 192.168.2.151 │ +│ Purpose: Proxmox Backup Server repository │ +│ │ +│ iso-share (NFS) 3.1T at 192.168.2.150 │ +│ Purpose: Shared ISO library │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Network Configuration + +``` +┌──────────────────────────────────────────────────────────────────┐ +│ Network Topology │ +├──────────────────────────────────────────────────────────────────┤ +│ │ +│ Physical NIC: eno1 (f0:4d:a2:04:0c:17) │ +│ │ │ +│ └──► vmbr0 (Primary Bridge) │ +│ │ │ +│ ├─ IP: 192.168.2.100/24 │ +│ ├─ Gateway: 192.168.2.1 │ +│ ├─ VLAN-aware: yes │ +│ └─ VLAN range: 2-4094 │ +│ │ +│ vmbr1 (Isolated Network) │ +│ └─ IP: 192.168.3.0/24 │ +│ Purpose: Internal services network │ +│ │ +│ VLAN 5: Used for web servers and ansible control │ +└──────────────────────────────────────────────────────────────────┘ +``` + +### Existing Virtual Machines (QEMU/KVM) + +``` +┌────────┬─────────────────────┬───────┬──────────┬──────────┬────────────┐ +│ VM ID │ Name │ vCPU │ RAM (GB) │ Disk │ Network │ +├────────┼─────────────────────┼───────┼──────────┼──────────┼────────────┤ +│ 100 │ docker-hub │ 4 │ 8.2 │ 100G │ vmbr0 │ +│ 101 │ gitlab │ 4 │ 17.0 │ 50G │ vmbr0 │ +│ 104 │ ubuntu-dev │ N/A │ N/A │ N/A │ vmbr0 │ +│ 105 │ dev │ N/A │ N/A │ N/A │ vmbr0 │ +│ 106 │ Ansible-Control │ 2 │ 4.0 │ 32G │ vmbr0.5 │ +│ 107 │ ubuntu-docker │ 2 │ 4.0 │ 50G │ vmbr0 │ +│ │ (Template) │ │ │ │ │ +│ 108 │ CML │ N/A │ N/A │ N/A │ vmbr0 │ +│ 109 │ web-server-01 │ 1 │ 2.0 │ 32G │ vmbr0.5 │ +│ 110 │ web-server-02 │ N/A │ N/A │ 32G │ vmbr0.5 │ +│ 111 │ db-server-01 │ 1 │ 4.0 │ 32G │ vmbr0.5 │ +└────────┴─────────────────────┴───────┴──────────┴──────────┴────────────┘ + +Total VM Resources: ~11 vCPUs, ~40 GB RAM +``` + +### Existing LXC Containers + +``` +┌────────┬─────────────────────┬───────┬──────────┬──────────┬────────────┐ +│ CT ID │ Name │ Cores │ RAM (GB) │ Disk │ IP Address │ +├────────┼─────────────────────┼───────┼──────────┼──────────┼────────────┤ +│ 102 │ nginx │ 1 │ 2.0 │ 2G │ 192.168. │ +│ │ (Reverse Proxy) │ │ │ │ 2.101/24 │ +│ │ │ │ │ │ │ +│ 103 │ netbox │ N/A │ N/A │ N/A │ DHCP │ +│ │ (IPAM/Docs) │ │ │ │ │ +│ │ │ │ │ │ │ +│ 112 │ Anytype │ 2 │ 4.0 │ 25G │ DHCP │ +│ │ (Knowledge Mgmt) │ │ │ │ │ +└────────┴─────────────────────┴───────┴──────────┴──────────┴────────────┘ + +Features: All containers have nesting=1 (Docker support) +``` + +### Resource Availability Assessment + +``` +╔═══════════════════════════════════════════════════════════════╗ +║ AVAILABLE RESOURCES ║ +╠═══════════════════════════════════════════════════════════════╣ +║ CPU: ~13 vCPUs available (24 total - ~11 allocated) ║ +║ RAM: ~80 GB available ║ +║ Disk: 4.12 TB available on Vault ZFS pool ║ +║ ║ +║ Verdict: EXCELLENT capacity for n8n deployment ║ +╚═══════════════════════════════════════════════════════════════╝ +``` + +--- + +## II. N8N Architecture Decision: LXC vs VM + +### The Verdict: **LXC Container (Recommended)** + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ LXC vs VM Comparison │ +├──────────────────┬──────────────────────┬───────────────────────┤ +│ Factor │ LXC Container │ Virtual Machine │ +├──────────────────┼──────────────────────┼───────────────────────┤ +│ Boot Time │ < 5 seconds │ 30-60 seconds │ +│ RAM Overhead │ ~100 MB │ ~500 MB │ +│ Disk Space │ ~2-5 GB │ ~10-15 GB │ +│ Performance │ Near-native │ Slight overhead │ +│ Snapshot Speed │ Instant (ZFS) │ 5-30 seconds │ +│ Backup Size │ Smaller │ Larger │ +│ Docker Support │ Yes (with nesting) │ Native │ +│ Resource Isol. │ Good │ Excellent │ +│ Complexity │ Simple │ Moderate │ +└──────────────────┴──────────────────────┴───────────────────────┘ +``` + +### Why LXC for n8n? + +**Advantages:** +1. **Efficiency**: n8n is a Node.js application with modest requirements. LXC provides near-native performance with minimal overhead. +2. **Fast Deployment**: Container creation takes seconds vs minutes for VMs. +3. **Resource Conservation**: Uses ~500 MB less RAM than a VM, leaving more resources for workflows. +4. **ZFS Snapshots**: Instant snapshots before updates or configuration changes. +5. **Consistency**: Your existing nginx reverse proxy (CT 102) is already an LXC container. +6. **Docker Compatibility**: With `nesting=1` feature, the container can run Docker if needed for custom nodes. + +**Considerations:** +- n8n doesn't require kernel-level isolation that VMs provide +- No exotic hardware requirements +- Perfectly suited for containerized deployment + +### When to Use a VM Instead: + +You would only need a VM if: +- Running n8n with Windows-specific integrations +- Requiring complete kernel isolation for security +- Testing multiple OS environments simultaneously +- Planning to run heavy compute workloads within workflows + +**For typical n8n workflow automation: LXC is the superior choice.** + +--- + +## III. N8N Server Specifications + +### Recommended LXC Configuration + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ N8N LXC Container Specs │ +├─────────────────────────────────────────────────────────────────┤ +│ Container ID: 113 (next available) │ +│ Hostname: n8n │ +│ OS Template: Debian 12 (bookworm) or Ubuntu 24.04 LTS │ +│ │ +│ vCPU Cores: 2 (scalable to 4 if needed) │ +│ RAM: 4096 MB (4 GB) │ +│ Swap: 2048 MB (2 GB) │ +│ │ +│ Root Disk: 20 GB (Vault ZFS pool) │ +│ - OS: ~2 GB │ +│ - n8n: ~1 GB │ +│ - PostgreSQL: ~2 GB │ +│ - Workflow data: ~5 GB │ +│ - Growth buffer: ~10 GB │ +│ │ +│ Network: vmbr0 (Primary bridge) │ +│ IP Address: 192.168.2.113/24 (static) │ +│ Gateway: 192.168.2.1 │ +│ DNS: 8.8.8.8, 8.8.4.4 │ +│ │ +│ Features: nesting=1 (Docker support) │ +│ Start on Boot: Yes (onboot: 1) │ +│ Unprivileged: Yes (security best practice) │ +│ Firewall: Yes (Proxmox firewall enabled) │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Performance Scaling Guide + +``` +Usage Level vCPU RAM Disk Notes +─────────────────────────────────────────────────────────────── +Light (< 10 workflows) 2 4 GB 20 GB Recommended start +Medium (10-50) 2 6 GB 30 GB Most homelabs +Heavy (50-100) 4 8 GB 50 GB Power users +Enterprise (100+) 4 12 GB 100 GB Consider VM/K8s +``` + +--- + +## IV. Network Architecture & Integration + +### Network Diagram + +``` + Internet + │ + ▼ + ┌─────────────────┐ + │ Router/Firewall │ + │ 192.168.2.1 │ + └────────┬─────────┘ + │ + ┌──────────┴───────────┐ + │ vmbr0 Bridge │ + │ 192.168.2.0/24 │ + └──────────┬───────────┘ + │ + ┌──────────────────┼──────────────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌──────────┐ ┌──────────┐ ┌──────────┐ + │ nginx │ │ n8n │ │ GitLab │ + │ CT: 102 │ │ CT: 113 │ │ VM: 101 │ + │ .101:80 │◄─────┤ .113:5678│ │ DHCP │ + │ .101:443 │ └──────────┘ └──────────┘ + └──────────┘ │ + │ ▼ + │ ┌──────────────┐ + │ │ PostgreSQL │ + │ │ (in CT 113) │ + │ │ localhost: │ + │ │ 5432 │ + │ └──────────────┘ + │ + ▼ + SSL Termination + n8n.yourdomain.com ──► https://192.168.2.113:5678 +``` + +### IP Address Allocation + +``` +┌──────────────────┬────────────────────┬──────────────────────┐ +│ IP Address │ Hostname │ Service │ +├──────────────────┼────────────────────┼──────────────────────┤ +│ 192.168.2.1 │ router │ Gateway │ +│ 192.168.2.100 │ serviceslab │ Proxmox Host │ +│ 192.168.2.101 │ nginx │ Reverse Proxy │ +│ 192.168.2.113 │ n8n │ N8N Server (NEW) │ +│ 192.168.2.150 │ NAS │ NFS Storage │ +│ 192.168.2.151 │ PBS │ Backup Server │ +└──────────────────┴────────────────────┴──────────────────────┘ +``` + +### Port Mapping Strategy + +``` +┌────────────────────────────────────────────────────────────────┐ +│ Port Configuration │ +├────────────────────────────────────────────────────────────────┤ +│ Internal n8n Container: │ +│ 5678/tcp ──► n8n Web Interface (HTTP) │ +│ 5432/tcp ──► PostgreSQL (localhost only) │ +│ │ +│ Nginx Reverse Proxy (CT 102): │ +│ 443/tcp ──► HTTPS (proxies to n8n:5678) │ +│ 80/tcp ──► HTTP (redirects to HTTPS) │ +│ │ +│ External Access: │ +│ https://n8n.yourdomain.com ──► nginx:443 ──► n8n:5678 │ +└────────────────────────────────────────────────────────────────┘ +``` + +### Firewall Rules + +``` +# Proxmox Firewall Configuration for CT 113 (n8n) + +Direction Protocol Source Dest Port Action Comment +───────────────────────────────────────────────────────────────── +IN TCP 192.168.2.101 5678 ACCEPT nginx proxy +IN TCP 192.168.2.0/24 22 ACCEPT SSH admin +IN TCP 0.0.0.0/0 5678 DROP Block direct +OUT TCP any 80,443 ACCEPT Updates/webhooks +OUT TCP any 25,587 ACCEPT Email (SMTP) +OUT UDP any 53 ACCEPT DNS +``` + +--- + +## V. Database Architecture + +### PostgreSQL vs SQLite Decision Matrix + +``` +┌───────────────────┬─────────────────────┬─────────────────────┐ +│ Factor │ PostgreSQL (Rec.) │ SQLite │ +├───────────────────┼─────────────────────┼─────────────────────┤ +│ Performance │ Excellent │ Good │ +│ Concurrent Access │ Full support │ Limited │ +│ Data Integrity │ ACID compliant │ ACID compliant │ +│ Backup/Restore │ Standard tools │ File copy │ +│ Scaling │ Unlimited │ Limited to 1-2 GB │ +│ HA/Clustering │ Yes │ No │ +│ Setup Complexity │ Moderate │ Zero config │ +│ Production Ready │ Yes │ For small setups │ +└───────────────────┴─────────────────────┴─────────────────────┘ +``` + +**Recommendation**: **PostgreSQL** for production reliability and future growth. + +### PostgreSQL Configuration + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ PostgreSQL Database Specifications │ +├─────────────────────────────────────────────────────────────────┤ +│ Version: PostgreSQL 16 (latest stable) │ +│ Installation: Same container as n8n (CT 113) │ +│ Database Name: n8n_db │ +│ Database User: n8n_user │ +│ Listen Address: 127.0.0.1 (localhost only) │ +│ Port: 5432 │ +│ Max Connections: 20 │ +│ Shared Buffers: 512 MB │ +│ Maintenance Work: 128 MB │ +│ WAL Level: replica (for backups) │ +│ │ +│ Backup Strategy: │ +│ - Daily pg_dump to /var/backups/n8n/ │ +│ - Proxmox container backup to PBS (weekly) │ +│ - ZFS snapshots before updates (manual) │ +└─────────────────────────────────────────────────────────────────┘ +``` + +--- + +## VI. Integration with Existing Services + +### A. Nginx Reverse Proxy (CT 102) + +Your existing nginx container will handle: + +1. **SSL/TLS Termination** - Let's Encrypt certificates +2. **HTTPS Enforcement** - HTTP to HTTPS redirect +3. **Security Headers** - HSTS, CSP, X-Frame-Options +4. **Rate Limiting** - Prevent abuse +5. **Access Logging** - Centralized logging + +**Nginx Configuration Snippet:** + +```nginx +# /etc/nginx/sites-available/n8n.yourdomain.com + +upstream n8n_backend { + server 192.168.2.113:5678; + keepalive 32; +} + +server { + listen 80; + server_name n8n.yourdomain.com; + return 301 https://$server_name$request_uri; +} + +server { + listen 443 ssl http2; + server_name n8n.yourdomain.com; + + # SSL Configuration (Let's Encrypt) + ssl_certificate /etc/letsencrypt/live/n8n.yourdomain.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/n8n.yourdomain.com/privkey.pem; + ssl_protocols TLSv1.2 TLSv1.3; + ssl_ciphers HIGH:!aNULL:!MD5; + + # Security Headers + add_header Strict-Transport-Security "max-age=31536000" always; + add_header X-Frame-Options "SAMEORIGIN" always; + add_header X-Content-Type-Options "nosniff" always; + + # Proxy Settings + location / { + proxy_pass http://n8n_backend; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # Timeouts for long-running workflows + proxy_connect_timeout 300s; + proxy_send_timeout 300s; + proxy_read_timeout 300s; + } + + # Health check endpoint + location /healthz { + proxy_pass http://n8n_backend/healthz; + access_log off; + } +} +``` + +### B. GitLab Integration (VM 101) + +N8N can automate GitLab workflows: + +- **CI/CD Triggers** - Start pipelines on external events +- **Issue Management** - Auto-create/update issues from tickets +- **Merge Request Automation** - Code review workflows +- **Repository Monitoring** - Track commits, tags, releases +- **Deployment Notifications** - Slack/email on deployments + +**Connection:** +- GitLab API available at VM 101's IP (configure in n8n) +- Create Personal Access Token in GitLab with API scope +- Store token in n8n credentials manager + +### C. Docker Hub (VM 100) + +Integration possibilities: + +- **Image Update Notifications** - Alert on new image versions +- **Automated Pulls** - Trigger container updates +- **Build Monitoring** - Track image builds + +### D. Ansible Control (VM 106) + +N8N can trigger Ansible playbooks: + +- **Infrastructure Automation** - Run playbooks via webhook +- **Configuration Management** - Scheduled config updates +- **Orchestration** - Multi-system deployment workflows + +**Setup Method:** +- n8n HTTP Request node → Ansible AWX/Tower REST API +- Or: n8n Execute Command node via SSH to Ansible Control VM + +--- + +## VII. Step-by-Step Setup Plan + +### Phase 1: Container Creation (15 minutes) + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Create LXC Container │ +└─────────────────────────────────────────────────────────────────┘ + +Option A: Proxmox Web UI Method +───────────────────────────────── +1. Login to Proxmox at https://192.168.2.100:8006 +2. Click "Create CT" button +3. Configure container: + + General: + ├─ Node: serviceslab + ├─ CT ID: 113 + ├─ Hostname: n8n + ├─ Unprivileged: ☑ (checked) + └─ Password: [Set strong password] + + Template: + └─ Storage: local + Template: debian-12-standard (or ubuntu-24.04-standard) + + Disks: + ├─ Storage: Vault + └─ Disk size: 20 GB + + CPU: + └─ Cores: 2 + + Memory: + ├─ Memory: 4096 MB + └─ Swap: 2048 MB + + Network: + ├─ Bridge: vmbr0 + ├─ IPv4: Static + ├─ IPv4/CIDR: 192.168.2.113/24 + ├─ Gateway: 192.168.2.1 + └─ IPv6: SLAAC (or disable) + + DNS: + ├─ DNS domain: apophisnetworking.net + └─ DNS servers: 8.8.8.8 8.8.4.4 + +4. Click "Confirm" then "Finish" +5. Enable features: + ├─ Right-click CT 113 → Options + ├─ Features → Edit + └─ Check: "nesting" ☑ + +6. Start container + + +Option B: CLI Method (Proxmox SSH) +────────────────────────────────── +# SSH to Proxmox host +ssh root@192.168.2.100 + +# Download template (if not present) +pveam update +pveam download local debian-12-standard_12.7-1_amd64.tar.zst + +# Create container +pct create 113 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \ + --hostname n8n \ + --cores 2 \ + --memory 4096 \ + --swap 2048 \ + --rootfs Vault:20 \ + --net0 name=eth0,bridge=vmbr0,ip=192.168.2.113/24,gw=192.168.2.1 \ + --nameserver 8.8.8.8 \ + --features nesting=1 \ + --unprivileged 1 \ + --onboot 1 + +# Start container +pct start 113 + +# Enter container +pct enter 113 +``` + +### Phase 2: System Preparation (10 minutes) + +```bash +# Inside CT 113 (after pct enter 113) + +# Update system +apt update && apt upgrade -y + +# Install prerequisites +apt install -y \ + curl \ + wget \ + git \ + gnupg2 \ + ca-certificates \ + lsb-release \ + postgresql-16 \ + postgresql-contrib \ + nginx-light \ + certbot \ + ufw + +# Configure timezone +timedatectl set-timezone America/New_York # Adjust to your TZ +``` + +### Phase 3: PostgreSQL Setup (10 minutes) + +```bash +# Switch to postgres user +sudo -u postgres psql + +-- Execute these SQL commands: +CREATE DATABASE n8n_db; +CREATE USER n8n_user WITH ENCRYPTED PASSWORD 'YourSecurePassword123!'; +GRANT ALL PRIVILEGES ON DATABASE n8n_db TO n8n_user; +\q + +# Configure PostgreSQL +cat >> /etc/postgresql/16/main/postgresql.conf << 'EOF' + +# N8N Optimizations +max_connections = 20 +shared_buffers = 512MB +effective_cache_size = 2GB +maintenance_work_mem = 128MB +checkpoint_completion_target = 0.9 +wal_buffers = 16MB +default_statistics_target = 100 +random_page_cost = 1.1 +effective_io_concurrency = 200 +work_mem = 26214kB +min_wal_size = 1GB +max_wal_size = 4GB +EOF + +# Restart PostgreSQL +systemctl restart postgresql +systemctl enable postgresql + +# Test connection +PGPASSWORD='YourSecurePassword123!' psql -U n8n_user -d n8n_db -h localhost -c "SELECT version();" +``` + +### Phase 4: Node.js & n8n Installation (15 minutes) + +```bash +# Install Node.js 20.x (LTS - recommended for n8n) +curl -fsSL https://deb.nodesource.com/setup_20.x | bash - +apt install -y nodejs + +# Verify Node.js installation +node --version # Should show v20.x.x +npm --version # Should show 10.x.x + +# Install n8n globally +npm install -g n8n + +# Create n8n system user +useradd -r -m -d /opt/n8n -s /bin/bash n8n + +# Create directory structure +mkdir -p /opt/n8n/{.n8n,backups,logs} +chown -R n8n:n8n /opt/n8n +``` + +### Phase 5: N8N Configuration (10 minutes) + +```bash +# Create environment configuration +cat > /opt/n8n/.env << 'EOF' +# Database Configuration +DB_TYPE=postgresdb +DB_POSTGRESDB_HOST=localhost +DB_POSTGRESDB_PORT=5432 +DB_POSTGRESDB_DATABASE=n8n_db +DB_POSTGRESDB_USER=n8n_user +DB_POSTGRESDB_PASSWORD=YourSecurePassword123! + +# n8n Configuration +N8N_PROTOCOL=https +N8N_HOST=n8n.yourdomain.com +N8N_PORT=5678 +N8N_PATH=/ +WEBHOOK_URL=https://n8n.yourdomain.com/ + +# Execution +EXECUTIONS_DATA_SAVE_ON_ERROR=all +EXECUTIONS_DATA_SAVE_ON_SUCCESS=all +EXECUTIONS_DATA_SAVE_MANUAL_EXECUTIONS=true + +# Performance +N8N_PAYLOAD_SIZE_MAX=16 +EXECUTIONS_TIMEOUT=300 +EXECUTIONS_TIMEOUT_MAX=3600 + +# Timezone +GENERIC_TIMEZONE=America/New_York + +# Security +N8N_BASIC_AUTH_ACTIVE=false +N8N_JWT_AUTH_ACTIVE=true +N8N_ENCRYPTION_KEY=$(openssl rand -hex 32) + +# Paths +N8N_USER_FOLDER=/opt/n8n/.n8n +N8N_LOG_LOCATION=/opt/n8n/logs/ +N8N_LOG_LEVEL=info +EOF + +# Secure environment file +chown n8n:n8n /opt/n8n/.env +chmod 600 /opt/n8n/.env +``` + +### Phase 6: Systemd Service Creation (5 minutes) + +```bash +# Create systemd service +cat > /etc/systemd/system/n8n.service << 'EOF' +[Unit] +Description=n8n - Workflow Automation +Documentation=https://docs.n8n.io +After=network.target postgresql.service +Wants=postgresql.service + +[Service] +Type=simple +User=n8n +Group=n8n +WorkingDirectory=/opt/n8n +EnvironmentFile=/opt/n8n/.env +ExecStart=/usr/bin/n8n start + +# Restart configuration +Restart=always +RestartSec=10 +StartLimitInterval=60 +StartLimitBurst=5 + +# Security +NoNewPrivileges=true +PrivateTmp=true +ProtectSystem=strict +ReadWritePaths=/opt/n8n +ProtectHome=true + +# Logging +StandardOutput=append:/opt/n8n/logs/n8n.log +StandardError=append:/opt/n8n/logs/n8n-error.log + +[Install] +WantedBy=multi-user.target +EOF + +# Reload systemd and start n8n +systemctl daemon-reload +systemctl enable n8n +systemctl start n8n + +# Check status +systemctl status n8n + +# View logs +journalctl -u n8n -f +``` + +### Phase 7: Nginx Reverse Proxy Configuration (20 minutes) + +```bash +# On nginx container (CT 102) +# SSH or pct enter 102 + +# Install certbot if not present +apt update && apt install -y certbot python3-certbot-nginx + +# Create nginx configuration +cat > /etc/nginx/sites-available/n8n.yourdomain.com << 'EOF' +upstream n8n_backend { + server 192.168.2.113:5678; + keepalive 64; +} + +server { + listen 80; + listen [::]:80; + server_name n8n.yourdomain.com; + + # Allow certbot challenges + location /.well-known/acme-challenge/ { + root /var/www/html; + } + + location / { + return 301 https://$server_name$request_uri; + } +} + +server { + listen 443 ssl http2; + listen [::]:443 ssl http2; + server_name n8n.yourdomain.com; + + # SSL certificates (will be configured by certbot) + ssl_certificate /etc/letsencrypt/live/n8n.yourdomain.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/n8n.yourdomain.com/privkey.pem; + + # SSL configuration + ssl_protocols TLSv1.2 TLSv1.3; + ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384; + ssl_prefer_server_ciphers off; + ssl_session_cache shared:SSL:10m; + ssl_session_timeout 10m; + ssl_stapling on; + ssl_stapling_verify on; + + # Security headers + add_header Strict-Transport-Security "max-age=63072000" always; + add_header X-Content-Type-Options "nosniff" always; + add_header X-Frame-Options "SAMEORIGIN" always; + add_header X-XSS-Protection "1; mode=block" always; + add_header Referrer-Policy "no-referrer-when-downgrade" always; + + # Logging + access_log /var/log/nginx/n8n-access.log; + error_log /var/log/nginx/n8n-error.log; + + # Client settings + client_max_body_size 50M; + + location / { + proxy_pass http://n8n_backend; + proxy_http_version 1.1; + + # WebSocket support + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + + # Proxy headers + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Port $server_port; + + # Timeouts + proxy_connect_timeout 300; + proxy_send_timeout 300; + proxy_read_timeout 300; + send_timeout 300; + + # Buffering + proxy_buffering off; + proxy_request_buffering off; + } + + # Health check + location /healthz { + proxy_pass http://n8n_backend/healthz; + access_log off; + } +} +EOF + +# Enable site +ln -sf /etc/nginx/sites-available/n8n.yourdomain.com /etc/nginx/sites-enabled/ + +# Test nginx configuration +nginx -t + +# Obtain SSL certificate +certbot --nginx -d n8n.yourdomain.com --non-interactive --agree-tos -m your@email.com + +# Reload nginx +systemctl reload nginx + +# Setup auto-renewal +systemctl enable certbot.timer +systemctl start certbot.timer +``` + +### Phase 8: Firewall Configuration (5 minutes) + +```bash +# On n8n container (CT 113) +ufw default deny incoming +ufw default allow outgoing +ufw allow from 192.168.2.0/24 to any port 22 comment 'SSH from LAN' +ufw allow from 192.168.2.101 to any port 5678 comment 'nginx proxy' +ufw enable + +# On Proxmox host (configure Proxmox firewall) +# Via Web UI: +# Datacenter → Firewall → Add rules for CT 113 +# Or via CLI: +cat >> /etc/pve/firewall/113.fw << 'EOF' +[OPTIONS] +enable: 1 + +[RULES] +IN ACCEPT -source 192.168.2.101 -dport 5678 -proto tcp -log nolog +IN ACCEPT -source 192.168.2.0/24 -dport 22 -proto tcp -log nolog +IN DROP -dport 5678 -proto tcp -log warning +EOF + +# Apply firewall +pve-firewall compile +``` + +### Phase 9: Initial n8n Setup (10 minutes) + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ First-Time Access │ +└─────────────────────────────────────────────────────────────────┘ + +1. Open browser to: https://n8n.yourdomain.com + +2. Create owner account: + ├─ Email: your@email.com + ├─ First Name: [Your Name] + ├─ Last Name: [Your Last Name] + └─ Password: [Strong password] + +3. Setup tour will guide you through: + ├─ Creating your first workflow + ├─ Adding credentials + └─ Testing a simple automation + +4. Configure critical settings: + Settings → General: + ├─ Timezone: Match your location + ├─ Execution timeout: 300 seconds (default) + └─ Save data: All executions (for debugging) + +5. Setup credentials for integrations: + Credentials → Add Credential: + ├─ GitLab API (for VM 101) + ├─ SMTP (for email notifications) + ├─ Webhook authentication + └─ Any other services you use +``` + +### Phase 10: Backup Configuration (15 minutes) + +```bash +# On n8n container (CT 113) + +# Create backup script +cat > /opt/n8n/backup.sh << 'EOF' +#!/bin/bash +# N8N Backup Script + +BACKUP_DIR="/opt/n8n/backups" +DATE=$(date +%Y%m%d_%H%M%S) +BACKUP_NAME="n8n_backup_${DATE}" + +# Create backup directory +mkdir -p ${BACKUP_DIR}/${BACKUP_NAME} + +# Backup PostgreSQL database +PGPASSWORD='YourSecurePassword123!' pg_dump \ + -U n8n_user \ + -h localhost \ + -d n8n_db \ + -F c \ + -f ${BACKUP_DIR}/${BACKUP_NAME}/n8n_db.dump + +# Backup n8n user folder +tar -czf ${BACKUP_DIR}/${BACKUP_NAME}/n8n_data.tar.gz \ + -C /opt/n8n/.n8n . + +# Backup environment file +cp /opt/n8n/.env ${BACKUP_DIR}/${BACKUP_NAME}/ + +# Create backup manifest +cat > ${BACKUP_DIR}/${BACKUP_NAME}/MANIFEST.txt << MANIFEST +N8N Backup +Date: $(date) +Database: n8n_db.dump +User Data: n8n_data.tar.gz +Environment: .env +MANIFEST + +# Compress backup +cd ${BACKUP_DIR} +tar -czf ${BACKUP_NAME}.tar.gz ${BACKUP_NAME} +rm -rf ${BACKUP_NAME} + +# Keep last 7 daily backups +find ${BACKUP_DIR} -name "n8n_backup_*.tar.gz" -mtime +7 -delete + +echo "Backup completed: ${BACKUP_DIR}/${BACKUP_NAME}.tar.gz" +EOF + +chmod +x /opt/n8n/backup.sh + +# Create daily backup cron job +cat > /etc/cron.d/n8n-backup << 'EOF' +# Daily n8n backup at 2 AM +0 2 * * * n8n /opt/n8n/backup.sh >> /opt/n8n/logs/backup.log 2>&1 +EOF + +# Test backup +sudo -u n8n /opt/n8n/backup.sh + +# Configure Proxmox container backup +# On Proxmox host: +# Datacenter → Backup → Add +# Schedule: Weekly, Sunday 1:00 AM +# Storage: PBS-Backups +# Mode: Snapshot +# Compression: ZSTD +# Include: CT 113 +``` + +--- + +## VIII. Post-Installation Checklist + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Verification Checklist │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ □ Container CT 113 created and running │ +│ □ Static IP 192.168.2.113 configured │ +│ □ PostgreSQL installed and accessible │ +│ □ n8n service running (systemctl status n8n) │ +│ □ n8n accessible at https://n8n.yourdomain.com │ +│ □ SSL certificate valid and auto-renewing │ +│ □ Nginx reverse proxy forwarding correctly │ +│ □ Firewall rules in place │ +│ □ Owner account created in n8n │ +│ □ Daily database backups configured │ +│ □ Proxmox container backups scheduled │ +│ □ Can create and execute test workflow │ +│ □ Webhook URL working (test with curl) │ +│ □ Email notifications configured (optional) │ +│ □ GitLab integration tested (optional) │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +--- + +## IX. Testing & Validation + +### Test 1: Basic Connectivity + +```bash +# From any machine on 192.168.2.0/24 network +curl -I http://192.168.2.113:5678 +# Expected: HTTP/1.1 200 OK + +# HTTPS through nginx +curl -I https://n8n.yourdomain.com +# Expected: HTTP/2 200 (or 301 → 200) +``` + +### Test 2: Create Sample Workflow + +``` +1. Login to n8n web interface +2. Click "+ Add workflow" +3. Add nodes: + ├─ Schedule Trigger (every 1 hour) + ├─ HTTP Request (GET https://api.github.com/zen) + └─ Send Email (to your address) +4. Configure email credentials +5. Execute workflow manually +6. Check email received +7. Enable workflow +8. Monitor executions tab +``` + +### Test 3: GitLab Integration + +``` +1. In GitLab (VM 101): + └─ Settings → Access Tokens + Create token with 'api' scope + +2. In n8n: + └─ Credentials → Add → GitLab API + Enter: GitLab URL and Access Token + +3. Create workflow: + ├─ Webhook Trigger (on /webhook/gitlab) + ├─ GitLab node (Get issue details) + └─ Log to file + +4. Configure GitLab webhook: + └─ Project → Settings → Webhooks + URL: https://n8n.yourdomain.com/webhook/gitlab + Events: Issues + +5. Test: Create issue in GitLab +6. Verify: Check n8n execution log +``` + +### Test 4: Database Performance + +```bash +# On CT 113 +sudo -u postgres psql -d n8n_db -c " +SELECT + schemaname, + tablename, + pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size +FROM pg_tables +WHERE schemaname = 'public' +ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC +LIMIT 10; +" +``` + +### Test 5: Backup & Restore + +```bash +# Test backup +sudo -u n8n /opt/n8n/backup.sh + +# Simulate disaster: Stop n8n, drop database +systemctl stop n8n +sudo -u postgres psql -c "DROP DATABASE n8n_db;" + +# Restore from backup +sudo -u postgres psql -c "CREATE DATABASE n8n_db OWNER n8n_user;" +LATEST_BACKUP=$(ls -t /opt/n8n/backups/*.tar.gz | head -1) +tar -xzf $LATEST_BACKUP -C /tmp/ +PGPASSWORD='YourSecurePassword123!' pg_restore \ + -U n8n_user \ + -d n8n_db \ + /tmp/*/n8n_db.dump + +# Restart n8n +systemctl start n8n +# Verify workflows still exist +``` + +--- + +## X. Monitoring & Maintenance + +### Health Monitoring + +```bash +# Create monitoring script +cat > /usr/local/bin/n8n-health-check << 'EOF' +#!/bin/bash +# N8N Health Check Script + +# Check if n8n service is running +if ! systemctl is-active --quiet n8n; then + echo "CRITICAL: n8n service is not running" + exit 2 +fi + +# Check if n8n responds to HTTP +HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:5678/healthz) +if [ "$HTTP_CODE" != "200" ]; then + echo "WARNING: n8n healthcheck returned $HTTP_CODE" + exit 1 +fi + +# Check PostgreSQL +if ! PGPASSWORD='YourSecurePassword123!' psql -U n8n_user -d n8n_db -h localhost -c "SELECT 1;" > /dev/null 2>&1; then + echo "CRITICAL: PostgreSQL connection failed" + exit 2 +fi + +# Check disk space +DISK_USAGE=$(df -h /opt/n8n | awk 'NR==2 {print $5}' | sed 's/%//') +if [ "$DISK_USAGE" -gt 80 ]; then + echo "WARNING: Disk usage at ${DISK_USAGE}%" + exit 1 +fi + +echo "OK: All checks passed" +exit 0 +EOF + +chmod +x /usr/local/bin/n8n-health-check + +# Schedule health check every 5 minutes +cat > /etc/cron.d/n8n-health << 'EOF' +*/5 * * * * root /usr/local/bin/n8n-health-check >> /var/log/n8n-health.log 2>&1 +EOF +``` + +### Log Rotation + +```bash +# Configure logrotate +cat > /etc/logrotate.d/n8n << 'EOF' +/opt/n8n/logs/*.log { + daily + rotate 14 + compress + delaycompress + missingok + notifempty + create 0640 n8n n8n + sharedscripts + postrotate + systemctl reload n8n > /dev/null 2>&1 || true + endscript +} +EOF +``` + +### Update Procedure + +```bash +# Create update script +cat > /opt/n8n/update-n8n.sh << 'EOF' +#!/bin/bash +# N8N Update Script + +set -e + +echo "Starting n8n update process..." + +# Backup before update +echo "Creating backup..." +/opt/n8n/backup.sh + +# Stop n8n service +echo "Stopping n8n..." +systemctl stop n8n + +# Update n8n +echo "Updating n8n..." +npm update -g n8n + +# Clear npm cache +npm cache clean --force + +# Start n8n service +echo "Starting n8n..." +systemctl start n8n + +# Wait for service to be ready +sleep 10 + +# Check health +if /usr/local/bin/n8n-health-check; then + echo "Update completed successfully!" + n8n --version +else + echo "ERROR: Health check failed after update!" + echo "Restoring from backup may be necessary" + exit 1 +fi +EOF + +chmod +x /opt/n8n/update-n8n.sh +``` + +### Maintenance Schedule + +``` +┌────────────────────────────────────────────────────────────────┐ +│ Maintenance Calendar │ +├────────────────────────────────────────────────────────────────┤ +│ │ +│ Daily: │ +│ └─ 02:00 AM - Database backup │ +│ │ +│ Weekly: │ +│ └─ Sunday 01:00 AM - Proxmox container backup │ +│ │ +│ Monthly: │ +│ ├─ Review execution logs │ +│ ├─ Check disk space usage │ +│ ├─ Update n8n to latest version │ +│ ├─ Review and optimize workflows │ +│ └─ Test backup restoration │ +│ │ +│ Quarterly: │ +│ ├─ Review security headers │ +│ ├─ Update SSL certificates (automatic via Let's Encrypt) │ +│ ├─ Audit user access │ +│ └─ PostgreSQL VACUUM ANALYZE │ +│ │ +└────────────────────────────────────────────────────────────────┘ +``` + +--- + +## XI. Troubleshooting Guide + +### Common Issues & Solutions + +#### Issue 1: n8n Won't Start + +```bash +# Check service status +systemctl status n8n + +# Check logs +journalctl -u n8n -n 50 --no-pager + +# Common causes: +# 1. PostgreSQL not running +systemctl status postgresql +systemctl start postgresql + +# 2. Database connection error +PGPASSWORD='YourSecurePassword123!' psql -U n8n_user -d n8n_db -h localhost + +# 3. Port already in use +ss -tulpn | grep 5678 + +# 4. Permission issues +chown -R n8n:n8n /opt/n8n +``` + +#### Issue 2: Can't Access via HTTPS + +```bash +# Check nginx status +systemctl status nginx + +# Test nginx configuration +nginx -t + +# Check SSL certificate +certbot certificates + +# Renew if needed +certbot renew --dry-run + +# Check firewall +ufw status +# Ensure 443 is open on nginx container + +# Test backend connectivity +curl http://192.168.2.113:5678 +``` + +#### Issue 3: Workflows Timeout + +```bash +# Increase timeout in /opt/n8n/.env +EXECUTIONS_TIMEOUT=600 +EXECUTIONS_TIMEOUT_MAX=7200 + +# Restart n8n +systemctl restart n8n + +# Also check nginx timeout in proxy config +# proxy_read_timeout 600; +``` + +#### Issue 4: Database Connection Pool Exhausted + +```bash +# Edit /etc/postgresql/16/main/postgresql.conf +max_connections = 50 + +# Restart PostgreSQL +systemctl restart postgresql + +# Monitor connections +sudo -u postgres psql -c " +SELECT count(*) +FROM pg_stat_activity +WHERE datname = 'n8n_db'; +" +``` + +#### Issue 5: High Memory Usage + +```bash +# Check memory usage +free -h +top -o %MEM + +# Reduce n8n memory if needed in /opt/n8n/.env +N8N_PAYLOAD_SIZE_MAX=8 + +# Restart n8n +systemctl restart n8n + +# Consider increasing container RAM +# On Proxmox: pct set 113 -memory 6144 +``` + +--- + +## XII. Security Hardening + +### Security Checklist + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Security Hardening Checklist │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ □ Use strong, unique passwords (20+ characters) │ +│ □ Enable HTTPS only (HTTP → HTTPS redirect) │ +│ □ Configure HSTS header (max-age=31536000) │ +│ □ Implement rate limiting in nginx │ +│ □ Use unprivileged LXC container │ +│ □ Firewall blocks direct access to port 5678 │ +│ □ PostgreSQL listens on localhost only │ +│ □ Regular security updates (apt update && upgrade) │ +│ □ Backup encryption for sensitive workflows │ +│ □ Multi-factor authentication (n8n Cloud feature) │ +│ □ IP whitelist for admin access (optional) │ +│ □ Credential encryption enabled (N8N_ENCRYPTION_KEY) │ +│ □ Audit logs enabled and monitored │ +│ □ Fail2ban for SSH brute-force protection │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Additional Security Measures + +```bash +# Install fail2ban (on CT 113) +apt install -y fail2ban + +# Configure fail2ban for SSH +cat > /etc/fail2ban/jail.local << 'EOF' +[sshd] +enabled = true +port = ssh +logpath = /var/log/auth.log +maxretry = 3 +bantime = 3600 +EOF + +systemctl enable fail2ban +systemctl start fail2ban + +# Implement nginx rate limiting +# Edit /etc/nginx/sites-available/n8n.yourdomain.com +# Add before server blocks: +limit_req_zone $binary_remote_addr zone=n8n_limit:10m rate=10r/s; + +# Inside server block: +limit_req zone=n8n_limit burst=20 nodelay; + +# Reload nginx +nginx -t && systemctl reload nginx +``` + +--- + +## XIII. Advanced Configurations + +### High Availability Setup (Future Expansion) + +If you need HA in the future: + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ High Availability Architecture │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ Load Balancer (HAProxy/Keepalived) │ +│ │ │ +│ ┌───────────┴───────────┐ │ +│ │ │ │ +│ ┌────▼─────┐ ┌─────▼────┐ │ +│ │ n8n-01 │ │ n8n-02 │ │ +│ │ CT: 113 │ │ CT: 114 │ │ +│ └────┬─────┘ └─────┬────┘ │ +│ │ │ │ +│ └───────────┬───────────┘ │ +│ │ │ +│ ┌────────▼────────┐ │ +│ │ PostgreSQL HA │ │ +│ │ (Replication) │ │ +│ └─────────────────┘ │ +│ │ +│ Requirements: │ +│ - Shared PostgreSQL with streaming replication │ +│ - Queue mode enabled (requires Redis) │ +│ - Session persistence at load balancer │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Custom Node Development + +```bash +# Setup development environment +# On CT 113 or separate dev container + +# Install n8n development tools +npm install -g n8n-node-dev + +# Create custom node +n8n-node-dev new + +# Follow prompts to create node +# Test node locally +# Package and install to production n8n +``` + +--- + +## XIV. Integration Examples + +### Example 1: GitLab CI/CD Automation + +``` +Workflow: "GitLab Deploy Notification" + +Trigger: GitLab Webhook (on pipeline completion) + │ + ├─ Filter: status == "success" + │ + ├─ Get Deployment Details (GitLab API) + │ + ├─ Update Netbox (HTTP Request to VM 103) + │ + ├─ Send Slack Notification + │ + └─ Create Jira Ticket (if production deploy) +``` + +### Example 2: Infrastructure Monitoring + +``` +Workflow: "Service Health Monitor" + +Trigger: Schedule (every 5 minutes) + │ + ├─ Check GitLab Health (HTTP Request) + │ + ├─ Check nginx Status (SSH to CT 102) + │ + ├─ Check Docker Hub (HTTP Request to VM 100) + │ + ├─ Query Netbox for Status (HTTP Request) + │ + ├─ Aggregate Results + │ + └─ If any failed: + ├─ Send Email Alert + ├─ Post to Slack + └─ Create PagerDuty Incident +``` + +### Example 3: Automated Backups + +``` +Workflow: "Backup Verification" + +Trigger: Schedule (daily at 3 AM) + │ + ├─ SSH to each VM/CT + │ + ├─ Run backup script + │ + ├─ Verify backup file exists + │ + ├─ Check backup size (ensure not 0 bytes) + │ + ├─ Update Google Sheets log + │ + └─ Send summary email +``` + +--- + +## XV. Performance Optimization + +### Tuning PostgreSQL for n8n + +```sql +-- Run these on PostgreSQL for better performance + +-- Increase work memory for complex queries +ALTER SYSTEM SET work_mem = '50MB'; + +-- Optimize for SSD (ZFS) +ALTER SYSTEM SET random_page_cost = 1.1; + +-- Enable parallel query execution +ALTER SYSTEM SET max_parallel_workers_per_gather = 2; +ALTER SYSTEM SET max_parallel_workers = 4; + +-- Reload configuration +SELECT pg_reload_conf(); + +-- Create indexes for common queries +CREATE INDEX IF NOT EXISTS idx_execution_data_workflow_id +ON public.execution_entity (workflow_id); + +CREATE INDEX IF NOT EXISTS idx_execution_data_finished +ON public.execution_entity (finished); + +-- Vacuum and analyze +VACUUM ANALYZE; +``` + +### n8n Performance Settings + +```bash +# Edit /opt/n8n/.env + +# Increase concurrent executions +EXECUTIONS_PROCESS=main +EXECUTIONS_MODE=regular + +# Queue mode for high throughput (requires Redis) +# EXECUTIONS_MODE=queue +# QUEUE_BULL_REDIS_HOST=localhost +# QUEUE_BULL_REDIS_PORT=6379 + +# Increase payload limits +N8N_PAYLOAD_SIZE_MAX=32 + +# Cache settings +CACHE_TTL=3600 +``` + +--- + +## XVI. Cost Analysis & Resource Planning + +### Resource Usage Projection + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Resource Usage Forecast │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ Current (0-10 workflows): │ +│ ├─ CPU: 5-10% avg (0.1-0.2 vCPU) │ +│ ├─ RAM: 600-800 MB │ +│ └─ Disk: 5 GB │ +│ │ +│ Growth (10-50 workflows): │ +│ ├─ CPU: 15-25% avg (0.3-0.5 vCPU) │ +│ ├─ RAM: 1.5-2 GB │ +│ └─ Disk: 15 GB │ +│ │ +│ Heavy Use (50-100 workflows): │ +│ ├─ CPU: 30-50% avg (0.6-1.0 vCPU) │ +│ ├─ RAM: 3-4 GB │ +│ └─ Disk: 30 GB │ +│ │ +│ Scaling Recommendations: │ +│ - At 50 workflows: Increase RAM to 6 GB │ +│ - At 100 workflows: Increase vCPU to 4 cores │ +│ - At 200 workflows: Consider queue mode + Redis │ +│ - At 500+ workflows: Migrate to dedicated VM or HA setup │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +--- + +## XVII. Documentation & Knowledge Base + +### Essential Resources + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ N8N Documentation Links │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ Official Documentation: │ +│ └─ https://docs.n8n.io/ │ +│ │ +│ Community Forum: │ +│ └─ https://community.n8n.io/ │ +│ │ +│ GitHub Repository: │ +│ └─ https://github.com/n8n-io/n8n │ +│ │ +│ Workflow Templates: │ +│ └─ https://n8n.io/workflows │ +│ │ +│ YouTube Channel: │ +│ └─ https://www.youtube.com/@n8n-io │ +│ │ +│ Integration Nodes: │ +│ └─ https://docs.n8n.io/integrations/ │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Internal Documentation + +Create these documents in your homelab repo: + +1. **/home/jramos/homelab/docs/n8n-workflows.md** + - Document all active workflows + - Purpose, trigger, dependencies + - Maintenance notes + +2. **/home/jramos/homelab/docs/n8n-credentials.md** (encrypted) + - Credential inventory + - Service API endpoints + - Renewal/rotation schedule + +3. **/home/jramos/homelab/docs/n8n-runbook.md** + - Emergency procedures + - Escalation contacts + - Restore procedures + +--- + +## XVIII. Migration Path from SQLite (If Needed) + +If you start with SQLite and later migrate to PostgreSQL: + +```bash +# Export from SQLite +n8n export:workflow --all --output=/tmp/workflows.json +n8n export:credentials --all --output=/tmp/credentials.json + +# Stop n8n +systemctl stop n8n + +# Update .env to use PostgreSQL +# (already configured in Phase 5) + +# Import to PostgreSQL +n8n import:workflow --input=/tmp/workflows.json +n8n import:credentials --input=/tmp/credentials.json + +# Start n8n +systemctl start n8n +``` + +--- + +## XIX. Final Architecture Diagram + +``` +╔══════════════════════════════════════════════════════════════════╗ +║ ║ +║ N8N HOMELAB INFRASTRUCTURE - FINAL STATE ║ +║ ║ +╠══════════════════════════════════════════════════════════════════╣ +║ ║ +║ Internet ║ +║ │ ║ +║ ▼ ║ +║ ┌──────────────────┐ ║ +║ │ Router/Firewall │ ║ +║ │ 192.168.2.1 │ ║ +║ │ Port 443 → 101 │ ║ +║ └────────┬─────────┘ ║ +║ │ ║ +║ ┌───────────────────────┼─────────────────────────┐ ║ +║ │ Proxmox Node: serviceslab │ ║ +║ │ 192.168.2.100/24 (vmbr0) │ ║ +║ │ │ │ ║ +║ │ ┌───────────────────┼───────────────────────┐ │ ║ +║ │ │ │ │ │ ║ +║ │ │ ┌────────────────▼──────┐ ┌──────────▼┐ │ ║ +║ │ │ │ nginx (CT 102) │ │ n8n │ │ ║ +║ │ │ │ 192.168.2.101 │ │ (CT 113) │ │ ║ +║ │ │ ├───────────────────────┤ │ .113 │ │ ║ +║ │ │ │ - SSL Termination │ │ │ │ ║ +║ │ │ │ - Reverse Proxy │───┤ n8n App │ │ ║ +║ │ │ │ - Let's Encrypt │ │ :5678 │ │ ║ +║ │ │ │ - Rate Limiting │ │ │ │ ║ +║ │ │ └───────────────────────┘ │ Postgres │ │ ║ +║ │ │ │ :5432 │ │ ║ +║ │ │ └─────┬─────┘ │ ║ +║ │ │ │ │ ║ +║ │ │ ┌─────────────────────────────────┼─────┐ │ ║ +║ │ │ │ Integration Services │ │ │ ║ +║ │ │ │ │ │ │ ║ +║ │ │ │ ┌──────────┐ ┌──────────┐ │ │ │ ║ +║ │ │ │ │ GitLab │ │ Docker │ │ │ │ ║ +║ │ │ │ │ VM 101 │ │ Hub │ │ │ │ ║ +║ │ │ │ │ 17GB RAM │ │ VM 100 │ │ │ │ ║ +║ │ │ │ └────┬─────┘ └────┬─────┘ │ │ │ ║ +║ │ │ │ │ │ │ │ │ ║ +║ │ │ │ ┌────▼─────┐ ┌────▼─────┐ │ │ │ ║ +║ │ │ │ │ Netbox │ │ Ansible │ │ │ │ ║ +║ │ │ │ │ CT 103 │ │ VM 106 │◄── ─┼─────┘ │ ║ +║ │ │ │ │ IPAM │ │ Control │ │ │ ║ +║ │ │ │ └──────────┘ └──────────┘ │ │ ║ +║ │ │ └───────────────────────────────────────┘ │ ║ +║ │ │ │ ║ +║ │ │ ┌──────────────────────────────────────┐ │ ║ +║ │ │ │ Storage: Vault (ZFS) │ │ ║ +║ │ │ │ 4.36 TB Total / 124 GB Used │ │ ║ +║ │ │ └──────────────────────────────────────┘ │ ║ +║ │ │ │ ║ +║ │ └─────────────────────────────────────────────┘ ║ +║ │ ║ +║ │ Backups: PBS (192.168.2.151) ║ +║ └─────────────────────────────────────────────────────────── ║ +║ ║ +╚══════════════════════════════════════════════════════════════════╝ + +Workflow Examples: +───────────────── +• GitLab Pipeline → n8n → Slack Notification +• Schedule → n8n → Ansible Playbook → Email Report +• Webhook → n8n → Update Netbox → GitLab Issue +• Cron → n8n → Backup Verification → Dashboard Update +``` + +--- + +## XX. Conclusion & Next Steps + +### Summary + +You now have a comprehensive blueprint for deploying n8n in your Proxmox homelab. The LXC container approach provides: + +- **Efficiency**: Minimal resource overhead (~600 MB RAM, < 5 GB disk) +- **Performance**: Near-native execution speed +- **Reliability**: PostgreSQL backend with automated backups +- **Security**: Reverse proxy with SSL, firewall rules, unprivileged container +- **Scalability**: Easy to upgrade resources as workflow count grows +- **Integration**: Seamless connection to your existing infrastructure + +### Immediate Next Steps + +1. **Create CT 113** using Phase 1 instructions +2. **Install PostgreSQL** (Phase 3) +3. **Deploy n8n** (Phase 4-6) +4. **Configure nginx proxy** (Phase 7) +5. **Test connectivity** (Phase 9) +6. **Setup backups** (Phase 10) + +### Week 1 Goals + +- [ ] Complete installation (Phases 1-7) +- [ ] Create first workflow (scheduled health check) +- [ ] Integrate with GitLab +- [ ] Configure backups +- [ ] Test disaster recovery + +### Month 1 Goals + +- [ ] Deploy 5-10 production workflows +- [ ] Setup monitoring and alerting +- [ ] Document all workflows +- [ ] Optimize PostgreSQL performance +- [ ] Implement advanced security measures + +### Future Considerations + +**3 Months:** +- Evaluate workflow performance +- Consider queue mode if > 50 workflows +- Implement custom nodes if needed + +**6 Months:** +- Review HA requirements +- Integrate with additional services +- Expand automation scope + +**1 Year:** +- Consider n8n Cloud migration (if needed) +- Evaluate container→VM migration +- Implement disaster recovery testing + +--- + +## Appendix A: Quick Reference Commands + +```bash +# Container Management +pct list # List all containers +pct start 113 # Start n8n container +pct stop 113 # Stop n8n container +pct enter 113 # Enter container shell + +# Service Management +systemctl status n8n # Check n8n status +systemctl restart n8n # Restart n8n +journalctl -u n8n -f # Follow n8n logs + +# Database +sudo -u postgres psql -d n8n_db # Connect to database +/opt/n8n/backup.sh # Run backup + +# Health Checks +/usr/local/bin/n8n-health-check # Run health check +curl http://localhost:5678/healthz # Test n8n API + +# Updates +/opt/n8n/update-n8n.sh # Update n8n +apt update && apt upgrade -y # Update system packages + +# Backups +ls -lh /opt/n8n/backups/ # List backups +tar -tzf backup.tar.gz | head # View backup contents +``` + +--- + +## Appendix B: Environment Variables Reference + +```bash +# Core Configuration +DB_TYPE=postgresdb # Database type +N8N_PROTOCOL=https # Protocol (http/https) +N8N_HOST=n8n.yourdomain.com # Public hostname +N8N_PORT=5678 # n8n listen port +WEBHOOK_URL=https://... # Webhook base URL + +# Database +DB_POSTGRESDB_HOST=localhost +DB_POSTGRESDB_PORT=5432 +DB_POSTGRESDB_DATABASE=n8n_db +DB_POSTGRESDB_USER=n8n_user +DB_POSTGRESDB_PASSWORD=... + +# Execution +EXECUTIONS_TIMEOUT=300 # Default timeout (seconds) +EXECUTIONS_TIMEOUT_MAX=3600 # Maximum timeout +EXECUTIONS_DATA_SAVE_ON_ERROR=all # Save failed executions +EXECUTIONS_DATA_SAVE_ON_SUCCESS=all # Save successful + +# Performance +N8N_PAYLOAD_SIZE_MAX=16 # Max payload size (MB) +EXECUTIONS_PROCESS=main # Execution mode + +# Timezone +GENERIC_TIMEZONE=America/New_York + +# Security +N8N_ENCRYPTION_KEY=... # Credential encryption key +N8N_JWT_AUTH_ACTIVE=true # Use JWT auth +``` + +--- + +## Appendix C: Support & Contact + +### Getting Help + +1. **N8N Community Forum**: https://community.n8n.io/ +2. **GitHub Issues**: https://github.com/n8n-io/n8n/issues +3. **Discord**: https://discord.gg/n8n +4. **Documentation**: https://docs.n8n.io/ + +### This Document + +- **Location**: /home/jramos/homelab/N8N-SETUP-PLAN.md +- **Version**: 1.0 +- **Created**: 2025-11-29 +- **Last Updated**: 2025-11-29 +- **Author**: The Scribe (Homelab Infrastructure) + +--- + +**Remember**: This is a living document. Update it as your infrastructure evolves, workflows expand, and requirements change. Good infrastructure documentation is your future self's best friend. + +Now, let's build something remarkable with n8n. + +*— The Scribe*