feat(infrastructure): enhance TrueNAS collection with comprehensive Docker/apps support
- Added collect-truenas-apps.sh script for standalone app/container collection - Enhanced collect-truenas-config.sh with Docker container, image, network, and volume collection - Fixed JSON format issues (converted newline-delimited JSON to proper arrays using jq/sed) - Added dynamic SSH user detection (tries root, admin, truenas_admin) - Implemented file size validation to prevent false success messages - Added container logs collection (last 500 lines per container) - Added Docker Compose file extraction from running containers - Added individual app configs collection from /mnt/.ix-apps/app_configs/ - Updated CLAUDE.md to reflect TrueNAS repository scope and strict agent routing rules - Restored sub-agent definitions (backend-builder, lab-operator, librarian, scribe) - Added SCRIPT_UPDATES.md with detailed changelog and testing instructions - Updated .gitignore to exclude Windows Zone.Identifier files These changes enable complete disaster recovery exports including all Docker/app configurations, logs, and metadata that were previously missing from TrueNAS infrastructure snapshots. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
221
SCRIPT_UPDATES.md
Normal file
221
SCRIPT_UPDATES.md
Normal file
@@ -0,0 +1,221 @@
|
||||
# TrueNAS Collection Scripts - Updates Summary
|
||||
|
||||
**Date**: 2024-12-16
|
||||
**Updated Scripts**:
|
||||
- `/home/jramos/truenas/scripts/collect-truenas-config.sh`
|
||||
- `/home/jramos/truenas/scripts/collect-truenas-apps.sh`
|
||||
|
||||
**Backups Created**:
|
||||
- `collect-truenas-config.sh.backup`
|
||||
- `collect-truenas-apps.sh.backup`
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Enhanced both TrueNAS collection scripts to properly capture Docker containers and TrueNAS SCALE apps. The scripts now collect comprehensive app/container information that was previously missing from disaster recovery exports.
|
||||
|
||||
---
|
||||
|
||||
## Changes to `collect-truenas-config.sh`
|
||||
|
||||
### 1. Dynamic SSH User Detection (Lines 294-301)
|
||||
**Before**: Hardcoded `truenas_admin@` username
|
||||
**After**: Auto-detects SSH user by trying `root`, `admin`, and `truenas_admin`
|
||||
|
||||
**Benefit**: Works on more TrueNAS configurations without modification
|
||||
|
||||
### 2. JSON Format Correction (All Docker API calls)
|
||||
**Before**: `--format json` (produces invalid newline-delimited JSON)
|
||||
**After**: `--format '{{json .}}' | jq -s '.'` with fallback to `sed` conversion
|
||||
|
||||
**Benefit**: Produces valid JSON arrays parseable by `jq` and other tools
|
||||
|
||||
### 3. File Size Validation
|
||||
**Before**: Logs success even if output files are empty
|
||||
**After**: Checks `[[ -s "$file" ]]` before logging success
|
||||
|
||||
**Benefit**: Accurate reporting of what was actually collected
|
||||
|
||||
### 4. Enhanced Data Collection
|
||||
**Added**:
|
||||
- Docker containers (JSON + text formats)
|
||||
- Docker images
|
||||
- Docker networks
|
||||
- Docker volumes
|
||||
- Docker storage sizes
|
||||
|
||||
**Benefit**: Complete Docker infrastructure snapshot
|
||||
|
||||
---
|
||||
|
||||
## Changes to `collect-truenas-apps.sh`
|
||||
|
||||
### 1. JSON Format Correction (Lines 30-73)
|
||||
**Before**: `--format json` (invalid newline-delimited JSON)
|
||||
**After**: `--format '{{json .}}' | jq -s '.'` with sed fallback
|
||||
|
||||
**Applies to**:
|
||||
- Docker containers
|
||||
- Docker images
|
||||
- Docker networks
|
||||
- Docker volumes
|
||||
- Docker Compose projects
|
||||
|
||||
### 2. Container Logs Collection (Lines 128-145) - NEW
|
||||
**Collects**: Last 500 lines of logs from each container
|
||||
**Output**: `exports/apps/logs/{container-name}.log`
|
||||
|
||||
**Benefit**: Troubleshooting information without needing SSH access later
|
||||
|
||||
### 3. Docker Compose Files Collection (Lines 147-171) - NEW
|
||||
**Collects**: Compose files from containers with `com.docker.compose.project.config_files` label
|
||||
**Output**: `configs/apps/compose/{project-name}.yml`
|
||||
|
||||
**Benefit**: Reconstruct compose stacks from running containers
|
||||
|
||||
### 4. Individual App Configs Collection (Lines 173-198) - NEW
|
||||
**Collects**: YAML, JSON, ENV, CONF files from `/mnt/.ix-apps/app_configs/`
|
||||
**Output**: `configs/apps/app_configs/{app-name}/`
|
||||
|
||||
**Benefit**: Full TrueNAS app configuration backup
|
||||
|
||||
---
|
||||
|
||||
## New Directory Structure
|
||||
|
||||
After running the updated scripts, exports will include:
|
||||
|
||||
```
|
||||
truenas-export-YYYYMMDD-HHMMSS/
|
||||
├── configs/
|
||||
│ └── apps/
|
||||
│ ├── metadata.yaml # TrueNAS app metadata
|
||||
│ ├── user_config.yaml # TrueNAS user config
|
||||
│ ├── compose/ # Docker Compose files
|
||||
│ │ ├── arr-stack.yml
|
||||
│ │ └── *.yml
|
||||
│ └── app_configs/ # Individual app configs
|
||||
│ ├── plex/
|
||||
│ ├── dockge/
|
||||
│ └── */
|
||||
└── exports/
|
||||
└── apps/
|
||||
├── docker-containers.json # JSON array of all containers
|
||||
├── docker-containers.txt # Human-readable container list
|
||||
├── docker-images.json # JSON array of images
|
||||
├── docker-networks.json # JSON array of networks
|
||||
├── docker-volumes.json # JSON array of volumes
|
||||
├── docker_sizes.txt # Storage usage
|
||||
├── app_configs_list.txt # Directory listing
|
||||
├── containers/ # Individual container details
|
||||
│ ├── plex.json
|
||||
│ ├── dockge.json
|
||||
│ └── *.json
|
||||
└── logs/ # Container logs (last 500 lines)
|
||||
├── plex.log
|
||||
├── dockge.log
|
||||
└── *.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Instructions
|
||||
|
||||
### On TrueNAS Server (Direct Collection)
|
||||
|
||||
```bash
|
||||
# Copy the script to TrueNAS
|
||||
scp /home/jramos/truenas/scripts/collect-truenas-apps.sh truenas_admin@192.168.2.150:/tmp/
|
||||
|
||||
# SSH to TrueNAS
|
||||
ssh truenas_admin@192.168.2.150
|
||||
|
||||
# Run the collection script
|
||||
bash /tmp/collect-truenas-apps.sh
|
||||
|
||||
# Verify JSON validity (if jq installed)
|
||||
jq empty /tmp/truenas-apps-export-*/exports/apps/docker-containers.json
|
||||
jq empty /tmp/truenas-apps-export-*/exports/apps/docker-images.json
|
||||
|
||||
# Copy back to workstation
|
||||
# On your workstation:
|
||||
scp truenas_admin@192.168.2.150:/tmp/truenas-apps-export-*.tar.gz /home/jramos/truenas/disaster-recovery/
|
||||
```
|
||||
|
||||
### From WSL (Via SSH)
|
||||
|
||||
```bash
|
||||
# Set up SSH key authentication first (if not already done)
|
||||
ssh-copy-id truenas_admin@192.168.2.150
|
||||
|
||||
# Run the collection script
|
||||
cd /home/jramos/truenas
|
||||
bash scripts/collect-truenas-config.sh
|
||||
|
||||
# Check for apps data
|
||||
ls -la disaster-recovery/truenas-exports/exports/apps/
|
||||
ls -la disaster-recovery/truenas-exports/configs/apps/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Critical Fixes Applied
|
||||
|
||||
### 1. **Invalid JSON Bug** (High Priority)
|
||||
- **Issue**: Docker commands with `--format json` produce newline-delimited JSON, not arrays
|
||||
- **Impact**: Files couldn't be parsed by `jq` or JSON parsers
|
||||
- **Fix**: Pipe through `jq -s '.'` or use `sed` to convert to proper JSON arrays
|
||||
|
||||
### 2. **Hardcoded SSH User** (Medium Priority)
|
||||
- **Issue**: Script assumed `truenas_admin@` user exists
|
||||
- **Impact**: Failed on systems with different admin usernames
|
||||
- **Fix**: Auto-detect by trying multiple common usernames
|
||||
|
||||
### 3. **Empty File False Positives** (Medium Priority)
|
||||
- **Issue**: Scripts logged "success" even when output files were empty
|
||||
- **Impact**: Misleading collection statistics
|
||||
- **Fix**: Added `[[ -s "$file" ]]` size checks
|
||||
|
||||
### 4. **Missing Critical Data** (High Priority)
|
||||
- **Issue**: No logs, compose files, or individual app configs collected
|
||||
- **Impact**: Incomplete disaster recovery information
|
||||
- **Fix**: Added three new collection sections
|
||||
|
||||
---
|
||||
|
||||
## Known Limitations
|
||||
|
||||
1. **SSH Access Required**: For remote collection from WSL, SSH key auth must be configured
|
||||
2. **Large Logs**: Container logs are limited to last 500 lines to prevent huge files
|
||||
3. **Compose File Detection**: Only works for containers with proper Docker Compose labels
|
||||
4. **Sudo Required**: All Docker commands require sudo access on TrueNAS
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Test** the scripts on your TrueNAS server
|
||||
2. **Verify** JSON files are valid with `jq`
|
||||
3. **Review** collected data for completeness
|
||||
4. **Document** the apps inventory (similar to Proxmox VMs/CTs)
|
||||
5. **Update** TrueNAS README with new collection capabilities
|
||||
|
||||
---
|
||||
|
||||
## Your TrueNAS Environment
|
||||
|
||||
**Running Containers** (from your sudo docker ps output):
|
||||
- **TrueNAS Apps**: Plex, Dockge
|
||||
- **Compose Stack (arr-stack)**: Radarr, Sonarr, Bazarr
|
||||
- **VPN/Networking**: Gluetun, NordLynx (2x)
|
||||
- **Download/Media**: Deluge, Prowlarr, FlareSolverr
|
||||
- **Monitoring**: Beszel-agent
|
||||
|
||||
The updated scripts will now capture all of these properly!
|
||||
|
||||
---
|
||||
|
||||
**Review completed by**: backend-builder agent
|
||||
**Applied by**: Main agent
|
||||
**Status**: ✓ All fixes applied successfully
|
||||
Reference in New Issue
Block a user