truenas/SCRIPT_UPDATES.md

# TrueNAS Collection Scripts - Updates Summary

**Date**: 2024-12-16
**Updated Scripts**:
- `/home/jramos/truenas/scripts/collect-truenas-config.sh`
- `/home/jramos/truenas/scripts/collect-truenas-apps.sh`

**Backups Created**:
- `collect-truenas-config.sh.backup`
- `collect-truenas-apps.sh.backup`

---

## Overview

Enhanced both TrueNAS collection scripts to properly capture Docker containers and TrueNAS SCALE apps. The scripts now collect comprehensive app/container information that was previously missing from disaster recovery exports.

---

## Changes to `collect-truenas-config.sh`

### 1. Dynamic SSH User Detection (Lines 294-301)
**Before**: Hardcoded `truenas_admin@` username
**After**: Auto-detects SSH user by trying `root`, `admin`, and `truenas_admin`

**Benefit**: Works on more TrueNAS configurations without modification

### 2. JSON Format Correction (All Docker API calls)
**Before**: `--format json` (produces invalid newline-delimited JSON)
**After**: `--format '{{json .}}' | jq -s '.'` with fallback to `sed` conversion

**Benefit**: Produces valid JSON arrays parseable by `jq` and other tools

### 3. File Size Validation
**Before**: Logs success even if output files are empty
**After**: Checks `[[ -s "$file" ]]` before logging success

**Benefit**: Accurate reporting of what was actually collected

### 4. Enhanced Data Collection
**Added**:
- Docker containers (JSON + text formats)
- Docker images
- Docker networks
- Docker volumes
- Docker storage sizes

**Benefit**: Complete Docker infrastructure snapshot

---

## Changes to `collect-truenas-apps.sh`

### 1. JSON Format Correction (Lines 30-73)
**Before**: `--format json` (invalid newline-delimited JSON)
**After**: `--format '{{json .}}' | jq -s '.'` with sed fallback

**Applies to**:
- Docker containers
- Docker images
- Docker networks
- Docker volumes
- Docker Compose projects

### 2. Container Logs Collection (Lines 128-145) - NEW
**Collects**: Last 500 lines of logs from each container
**Output**: `exports/apps/logs/{container-name}.log`

**Benefit**: Troubleshooting information without needing SSH access later

### 3. Docker Compose Files Collection (Lines 147-171) - NEW
**Collects**: Compose files from containers with `com.docker.compose.project.config_files` label
**Output**: `configs/apps/compose/{project-name}.yml`

**Benefit**: Reconstruct compose stacks from running containers

### 4. Individual App Configs Collection (Lines 173-198) - NEW
**Collects**: YAML, JSON, ENV, CONF files from `/mnt/.ix-apps/app_configs/`
**Output**: `configs/apps/app_configs/{app-name}/`

**Benefit**: Full TrueNAS app configuration backup

---

## New Directory Structure

After running the updated scripts, exports will include:

```
truenas-export-YYYYMMDD-HHMMSS/
├── configs/
│   └── apps/
│       ├── metadata.yaml          # TrueNAS app metadata
│       ├── user_config.yaml       # TrueNAS user config
│       ├── compose/               # Docker Compose files
│       │   ├── arr-stack.yml
│       │   └── *.yml
│       └── app_configs/           # Individual app configs
│           ├── plex/
│           ├── dockge/
│           └── */
└── exports/
    └── apps/
        ├── docker-containers.json     # JSON array of all containers
        ├── docker-containers.txt      # Human-readable container list
        ├── docker-images.json         # JSON array of images
        ├── docker-networks.json       # JSON array of networks
        ├── docker-volumes.json        # JSON array of volumes
        ├── docker_sizes.txt           # Storage usage
        ├── app_configs_list.txt       # Directory listing
        ├── containers/                # Individual container details
        │   ├── plex.json
        │   ├── dockge.json
        │   └── *.json
        └── logs/                      # Container logs (last 500 lines)
            ├── plex.log
            ├── dockge.log
            └── *.log
```

---

## Testing Instructions

### On TrueNAS Server (Direct Collection)

```bash
# Copy the script to TrueNAS
scp /home/jramos/truenas/scripts/collect-truenas-apps.sh truenas_admin@192.168.2.150:/tmp/

# SSH to TrueNAS
ssh truenas_admin@192.168.2.150

# Run the collection script
bash /tmp/collect-truenas-apps.sh

# Verify JSON validity (if jq installed)
jq empty /tmp/truenas-apps-export-*/exports/apps/docker-containers.json
jq empty /tmp/truenas-apps-export-*/exports/apps/docker-images.json

# Copy back to workstation
# On your workstation:
scp truenas_admin@192.168.2.150:/tmp/truenas-apps-export-*.tar.gz /home/jramos/truenas/disaster-recovery/
```

### From WSL (Via SSH)

```bash
# Set up SSH key authentication first (if not already done)
ssh-copy-id truenas_admin@192.168.2.150

# Run the collection script
cd /home/jramos/truenas
bash scripts/collect-truenas-config.sh

# Check for apps data
ls -la disaster-recovery/truenas-exports/exports/apps/
ls -la disaster-recovery/truenas-exports/configs/apps/
```

---

## Critical Fixes Applied

### 1. **Invalid JSON Bug** (High Priority)
- **Issue**: Docker commands with `--format json` produce newline-delimited JSON, not arrays
- **Impact**: Files couldn't be parsed by `jq` or JSON parsers
- **Fix**: Pipe through `jq -s '.'` or use `sed` to convert to proper JSON arrays

### 2. **Hardcoded SSH User** (Medium Priority)
- **Issue**: Script assumed `truenas_admin@` user exists
- **Impact**: Failed on systems with different admin usernames
- **Fix**: Auto-detect by trying multiple common usernames

### 3. **Empty File False Positives** (Medium Priority)
- **Issue**: Scripts logged "success" even when output files were empty
- **Impact**: Misleading collection statistics
- **Fix**: Added `[[ -s "$file" ]]` size checks

### 4. **Missing Critical Data** (High Priority)
- **Issue**: No logs, compose files, or individual app configs collected
- **Impact**: Incomplete disaster recovery information
- **Fix**: Added three new collection sections

---

## Known Limitations

1. **SSH Access Required**: For remote collection from WSL, SSH key auth must be configured
2. **Large Logs**: Container logs are limited to last 500 lines to prevent huge files
3. **Compose File Detection**: Only works for containers with proper Docker Compose labels
4. **Sudo Required**: All Docker commands require sudo access on TrueNAS

---

## Next Steps

1. **Test** the scripts on your TrueNAS server
2. **Verify** JSON files are valid with `jq`
3. **Review** collected data for completeness
4. **Document** the apps inventory (similar to Proxmox VMs/CTs)
5. **Update** TrueNAS README with new collection capabilities

---

## Your TrueNAS Environment

**Running Containers** (from your sudo docker ps output):
- **TrueNAS Apps**: Plex, Dockge
- **Compose Stack (arr-stack)**: Radarr, Sonarr, Bazarr
- **VPN/Networking**: Gluetun, NordLynx (2x)
- **Download/Media**: Deluge, Prowlarr, FlareSolverr
- **Monitoring**: Beszel-agent

The updated scripts will now capture all of these properly!

---

**Review completed by**: backend-builder agent
**Applied by**: Main agent
**Status**: ✓ All fixes applied successfully