truenas/START-HERE-DOCS/TRUENAS_COLLECTION_README.md

# TrueNAS Scale Infrastructure Collection System

**Version:** 1.0.0
**TrueNAS Scale API:** v2.0
**Target Server:** 192.168.2.150
**Collection Method:** Hybrid API + SSH

---

## Table of Contents

1. [Overview](#overview)
2. [Quick Start](#quick-start)
3. [Prerequisites](#prerequisites)
4. [Collection Levels](#collection-levels)
5. [Directory Structure](#directory-structure)
6. [API Endpoint Reference](#api-endpoint-reference)
7. [SSH Command Reference](#ssh-command-reference)
8. [Usage Examples](#usage-examples)
9. [Security Considerations](#security-considerations)
10. [Troubleshooting](#troubleshooting)
11. [Integration with Homelab](#integration-with-homelab)

---

## Overview

This collection system provides comprehensive, READ-ONLY inventory and configuration export capabilities for TrueNAS Scale infrastructure. It follows the same proven pattern as the Proxmox homelab collection script, using a hybrid approach:

- **API-First**: Leverages TrueNAS Scale REST API v2.0 for structured data
- **SSH Fallback**: Uses SSH for system-level commands and file access
- **Non-Invasive**: All operations are read-only - no modifications to your system
- **Documented**: Generates comprehensive exports with README and summary files

**What This System Does:**
- Collects pool, dataset, and zvol configurations
- Exports storage metrics and health status
- Documents sharing configurations (NFS, SMB, iSCSI)
- Captures system information and network settings
- Records service states and configurations
- Generates structured JSON and human-readable text outputs

**What This System Does NOT Do:**
- Modify any TrueNAS configurations
- Export actual data from pools/datasets
- Change permissions or access controls
- Interfere with running services

---

## Quick Start

### Method 1: Remote Collection via API + SSH (Recommended)

```bash
# 1. Set up API authentication
export TRUENAS_HOST="192.168.2.150"
export TRUENAS_API_KEY="your-api-key-here"

# 2. Run collection script (when available)
./collect-truenas-config.sh --level standard --output ./truenas-exports

# 3. Review results
cat ./truenas-exports/truenas-export-$(date +%Y%m%d)/SUMMARY.md
```

### Method 2: Manual Collection via SSH

```bash
# SSH to TrueNAS Scale
ssh admin@192.168.2.150

# Run system commands
midclt call system.info
midclt call pool.query
zpool list -v
```

### Method 3: API-Only Collection

```bash
# Using curl with API key
curl -X GET "https://192.168.2.150/api/v2.0/system/info" \
  -H "Authorization: Bearer ${TRUENAS_API_KEY}" \
  -H "Content-Type: application/json" \
  --insecure
```

---

## Prerequisites

### 1. API Access Configuration

**Generate API Key:**
1. Log in to TrueNAS Scale Web UI (https://192.168.2.150)
2. Navigate to: **Account** → **API Keys**
3. Click **Add** and create new key
4. Copy the key immediately (only shown once)
5. Store securely in password manager

**Set Environment Variable:**
```bash
# Add to ~/.bashrc or ~/.zshrc
export TRUENAS_API_KEY="1-YourActualAPIKeyHere"
export TRUENAS_HOST="192.168.2.150"
```

**Test API Access:**
```bash
curl -X GET "https://${TRUENAS_HOST}/api/v2.0/system/version" \
  -H "Authorization: Bearer ${TRUENAS_API_KEY}" \
  -H "Content-Type: application/json" \
  --insecure  # Use only if self-signed certificate
```

Expected response:
```json
{
  "fullversion": "TrueNAS-SCALE-23.10.1",
  "version": "23.10.1"
}
```

### 2. SSH Access Configuration

**Set Up SSH Key Authentication:**
```bash
# Generate SSH key (if needed)
ssh-keygen -t ed25519 -C "homelab-collection"

# Copy to TrueNAS
ssh-copy-id admin@192.168.2.150

# Test connection
ssh admin@192.168.2.150 'echo "SSH working"'
```

### 3. Required Tools

**On Collection Machine (WSL2/Linux):**
- `curl` (7.68.0+)
- `jq` (1.6+) - JSON processing
- `ssh` (OpenSSH 8.0+)
- `bash` (5.0+)

**Install on Ubuntu/Debian:**
```bash
sudo apt update
sudo apt install curl jq openssh-client
```

### 4. Disk Space Requirements

| Collection Level | Estimated Size | Description |
|-----------------|----------------|-------------|
| **basic** | 5-10 MB | System info, pool summary |
| **standard** | 20-50 MB | + datasets, shares, services |
| **full** | 50-100 MB | + detailed metrics, logs |
| **paranoid** | 100-500 MB | + debug data, full configs |

---

## Collection Levels

### Basic Level
**Use Case:** Quick status check, daily monitoring

**Collected:**
- System version and hardware info
- Pool list and health status
- Dataset tree (top-level only)
- Active services list
- Network interface summary

**Execution Time:** ~30 seconds

### Standard Level (Default)
**Use Case:** Regular documentation, weekly snapshots

**Collected (includes Basic +):**
- Complete dataset hierarchy
- ZFS properties for all datasets
- Share configurations (NFS, SMB)
- User and group listings
- Disk information and SMART status
- System logs (last 1000 lines)

**Execution Time:** ~2-3 minutes

### Full Level
**Use Case:** Pre-maintenance backup, troubleshooting

**Collected (includes Standard +):**
- Complete service configurations
- Network routing and DNS settings
- Sysctl parameters
- Installed packages list
- Certificate information
- Replication and snapshot tasks
- Cloud sync configurations

**Execution Time:** ~5-8 minutes

### Paranoid Level
**Use Case:** Complete disaster recovery documentation

**Collected (includes Full +):**
- System debug data
- Complete middleware configuration
- Historical metrics (if available)
- Boot pool information
- Update history
- Audit logs

**Execution Time:** ~10-15 minutes

---

## Directory Structure

```
truenas-export-YYYYMMDD-HHMMSS/
├── README.md                          # Export overview and instructions
├── SUMMARY.md                         # Quick reference summary
├── collection.log                     # Detailed collection log
│
├── configs/                           # Configuration files
│   ├── system/
│   │   ├── version.json              # TrueNAS version info
│   │   ├── hardware.json             # CPU, memory, motherboard
│   │   ├── advanced-config.json      # Advanced system settings
│   │   └── general-config.json       # General system configuration
│   │
│   ├── storage/
│   │   ├── pools.json                # Pool configurations
│   │   ├── datasets.json             # Dataset hierarchy
│   │   ├── zvols.json                # ZVol configurations
│   │   ├── disks.json                # Disk inventory
│   │   └── smart-tests.json          # SMART test configurations
│   │
│   ├── sharing/
│   │   ├── nfs-shares.json           # NFS export configurations
│   │   ├── smb-shares.json           # SMB/CIFS shares
│   │   ├── iscsi-config.json         # iSCSI targets and extents
│   │   └── afp-shares.json           # AFP shares (if any)
│   │
│   ├── network/
│   │   ├── interfaces.json           # Network interface configs
│   │   ├── static-routes.json        # Static route table
│   │   ├── dns-config.json           # DNS resolver settings
│   │   └── openvpn-config.json       # VPN configurations
│   │
│   ├── services/
│   │   ├── service-status.json       # All service states
│   │   ├── ssh-config.json           # SSH service configuration
│   │   ├── snmp-config.json          # SNMP settings
│   │   └── ups-config.json           # UPS monitoring config
│   │
│   └── tasks/
│       ├── cron-jobs.json            # Scheduled tasks
│       ├── rsync-tasks.json          # Rsync jobs
│       ├── cloud-sync-tasks.json     # Cloud sync configurations
│       ├── replication-tasks.json    # ZFS replication
│       └── smart-tasks.json          # Scheduled SMART tests
│
├── exports/                           # System state exports
│   ├── storage/
│   │   ├── zpool-list.txt            # Pool status summary
│   │   ├── zpool-status.txt          # Detailed pool health
│   │   ├── zfs-list-all.txt          # All datasets/zvols
│   │   ├── zfs-get-all.txt           # All ZFS properties
│   │   └── disk-list.txt             # Disk inventory
│   │
│   ├── system/
│   │   ├── uname.txt                 # Kernel version
│   │   ├── uptime.txt                # System uptime
│   │   ├── df.txt                    # Filesystem usage
│   │   ├── free.txt                  # Memory usage
│   │   ├── dmesg.txt                 # Kernel messages
│   │   └── sysctl.txt                # Kernel parameters
│   │
│   ├── network/
│   │   ├── ip-addr.txt               # IP address assignments
│   │   ├── ip-route.txt              # Routing table
│   │   ├── ss-listening.txt          # Listening ports
│   │   └── netstat-interfaces.txt    # Network interface stats
│   │
│   └── logs/
│       ├── middleware.log            # TrueNAS middleware log (recent)
│       ├── syslog.txt                # System log (recent)
│       ├── messages.txt              # General messages
│       └── scrub-history.txt         # ZFS scrub history
│
└── metrics/                           # Performance and health metrics
    ├── pool-usage.json               # Pool capacity metrics
    ├── dataset-usage.json            # Per-dataset usage
    ├── disk-temps.json               # Disk temperatures
    ├── smart-status.json             # SMART health status
    └── system-stats.json             # CPU, memory, network stats
```

---

## API Endpoint Reference

### System Information Endpoints

| Endpoint | Method | Description | Collection Level |
|----------|--------|-------------|------------------|
| `/api/v2.0/system/info` | GET | System hardware and version | basic |
| `/api/v2.0/system/version` | GET | TrueNAS version string | basic |
| `/api/v2.0/system/advanced` | GET | Advanced system settings | standard |
| `/api/v2.0/system/general` | GET | General configuration | standard |
| `/api/v2.0/system/state` | GET | System state and uptime | basic |

### Storage Endpoints

| Endpoint | Method | Description | Collection Level |
|----------|--------|-------------|------------------|
| `/api/v2.0/pool` | GET | List all pools | basic |
| `/api/v2.0/pool/dataset` | GET | List all datasets | standard |
| `/api/v2.0/pool/snapshots` | GET | List all snapshots | full |
| `/api/v2.0/pool/scrub` | GET | Scrub status and history | standard |
| `/api/v2.0/disk` | GET | Disk inventory | standard |
| `/api/v2.0/disk/temperature` | GET | Disk temperatures | standard |
| `/api/v2.0/smart/test/results` | GET | SMART test results | full |

### Sharing Endpoints

| Endpoint | Method | Description | Collection Level |
|----------|--------|-------------|------------------|
| `/api/v2.0/sharing/nfs` | GET | NFS share configurations | standard |
| `/api/v2.0/sharing/smb` | GET | SMB share configurations | standard |
| `/api/v2.0/sharing/iscsi/target` | GET | iSCSI targets | standard |
| `/api/v2.0/sharing/iscsi/extent` | GET | iSCSI extents | standard |

### Network Endpoints

| Endpoint | Method | Description | Collection Level |
|----------|--------|-------------|------------------|
| `/api/v2.0/interface` | GET | Network interfaces | standard |
| `/api/v2.0/network/configuration` | GET | Network configuration | standard |
| `/api/v2.0/staticroute` | GET | Static routes | standard |

### Service Endpoints

| Endpoint | Method | Description | Collection Level |
|----------|--------|-------------|------------------|
| `/api/v2.0/service` | GET | All service statuses | basic |
| `/api/v2.0/ssh` | GET | SSH service configuration | standard |
| `/api/v2.0/snmp` | GET | SNMP configuration | full |

### Task Endpoints

| Endpoint | Method | Description | Collection Level |
|----------|--------|-------------|------------------|
| `/api/v2.0/cronjob` | GET | Cron jobs | standard |
| `/api/v2.0/rsynctask` | GET | Rsync tasks | standard |
| `/api/v2.0/cloudsync` | GET | Cloud sync tasks | full |
| `/api/v2.0/replication` | GET | Replication tasks | full |
| `/api/v2.0/pool/snapshottask` | GET | Snapshot tasks | standard |

**Full API documentation:** See `TRUENAS_API_REFERENCE.md` for complete details.

---

## SSH Command Reference

### Storage Commands

| Command | Description | Output Format |
|---------|-------------|---------------|
| `zpool list -v` | Pool list with vdevs | Text table |
| `zpool status -v` | Detailed pool health | Text report |
| `zfs list -t all` | All datasets/zvols | Text table |
| `zfs get all` | All ZFS properties | Text table |
| `midclt call pool.query` | Pool info via middleware | JSON |

### System Commands

| Command | Description | Output Format |
|---------|-------------|---------------|
| `uname -a` | Kernel version | Text |
| `uptime` | System uptime | Text |
| `free -h` | Memory usage | Text table |
| `df -h` | Filesystem usage | Text table |
| `lsblk` | Block device tree | Text tree |

### Network Commands

| Command | Description | Output Format |
|---------|-------------|---------------|
| `ip addr show` | IP addresses | Text |
| `ip route show` | Routing table | Text |
| `ss -tuln` | Listening sockets | Text table |
| `midclt call interface.query` | Interface config | JSON |

### Disk Health Commands

| Command | Description | Output Format |
|---------|-------------|---------------|
| `smartctl -a /dev/sda` | SMART data for disk | Text report |
| `midclt call disk.query` | Disk inventory | JSON |
| `midclt call disk.temperature` | Disk temperatures | JSON |

---

## Usage Examples

### Example 1: Basic Daily Check

```bash
#!/bin/bash
# quick-truenas-check.sh

TRUENAS_HOST="192.168.2.150"
TRUENAS_API_KEY="${TRUENAS_API_KEY}"

# Get system info
curl -s -X GET "https://${TRUENAS_HOST}/api/v2.0/system/info" \
  -H "Authorization: Bearer ${TRUENAS_API_KEY}" \
  --insecure | jq .

# Get pool status
curl -s -X GET "https://${TRUENAS_HOST}/api/v2.0/pool" \
  -H "Authorization: Bearer ${TRUENAS_API_KEY}" \
  --insecure | jq '.[] | {name: .name, status: .status, healthy: .healthy}'
```

### Example 2: Export All NFS Shares

```bash
#!/bin/bash
# export-nfs-config.sh

TRUENAS_HOST="192.168.2.150"
OUTPUT_FILE="nfs-shares-$(date +%Y%m%d).json"

curl -s -X GET "https://${TRUENAS_HOST}/api/v2.0/sharing/nfs" \
  -H "Authorization: Bearer ${TRUENAS_API_KEY}" \
  --insecure | jq . > "${OUTPUT_FILE}"

echo "NFS shares exported to: ${OUTPUT_FILE}"
jq -r '.[] | "\(.path) -> \(.networks[0])"' "${OUTPUT_FILE}"
```

### Example 3: Monitor Pool Health

```bash
#!/bin/bash
# monitor-pool-health.sh

ssh admin@192.168.2.150 << 'EOF'
  echo "=== Pool Status ==="
  zpool status

  echo ""
  echo "=== Pool Capacity ==="
  zpool list
EOF
```

---

## Security Considerations

### API Key Security

**DO:**
- Store API keys in environment variables or secure vaults
- Use separate keys for different purposes
- Set appropriate key expiration dates
- Revoke keys when no longer needed
- Use HTTPS for all API calls

**DON'T:**
- Hardcode API keys in scripts
- Commit keys to version control
- Share keys between users/systems
- Use keys in URLs (query parameters)

### SSH Security

**Best Practices:**
- Use SSH key authentication (not passwords)
- Use Ed25519 keys (more secure than RSA)
- Restrict SSH access by IP if possible
- Use dedicated service accounts for automation
- Regularly rotate SSH keys

### Data Sanitization

**Before sharing exports:**

```bash
# Remove sensitive data from export
find truenas-export-*/ -type f -name "*.json" -exec sed -i \
  's/"api_key": "[^"]*"/"api_key": "<REDACTED>"/g' {} \;

# Remove passwords
find truenas-export-*/ -type f -exec sed -i \
  's/"password": "[^"]*"/"password": "<REDACTED>"/g' {} \;
```

---

## Troubleshooting

### "API authentication failed"

**Symptoms:**
```
HTTP/1.1 401 Unauthorized
{"error": "Invalid API key"}
```

**Solutions:**
1. Verify API key is correct: `echo $TRUENAS_API_KEY`
2. Check key hasn't expired in TrueNAS UI
3. Ensure Authorization header format: `Bearer <key>`
4. Verify user account has API access permissions

### "SSL certificate verify failed"

**Symptoms:**
```
curl: (60) SSL certificate problem: self signed certificate
```

**Solutions:**
```bash
# Option 1: Use --insecure flag (development only)
curl --insecure https://192.168.2.150/api/v2.0/...

# Option 2: Install CA certificate (recommended)
sudo cp /path/to/truenas-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
```

### "Permission denied" on SSH commands

**Symptoms:**
```
sudo: a password is required
```

**Solutions:**
1. Configure passwordless sudo for specific commands
2. Use middleware API instead: `midclt call ...`
3. SSH as root user (if allowed)
4. Add user to appropriate groups

### "Rate limit exceeded"

**Symptoms:**
```
HTTP/1.1 429 Too Many Requests
```

**Solutions:**
1. Add delays between API calls: `sleep 1`
2. Use batch endpoints when available
3. Cache results instead of repeated queries

---

## Integration with Homelab

### Current Homelab Context

**TrueNAS Server:**
- **IP Address:** 192.168.2.150
- **Primary Pool:** Vauly (3.0T capacity)
- **NFS Export:** `/mnt/Vauly/iso-vault` → mounted on Proxmox

**Proxmox Integration:**
```
# On serviceslab (192.168.2.200)
nfs: iso-share
	export /mnt/Vauly/iso-vault
	path /mnt/pve/iso-share
	server 192.168.2.150
	content iso
```

### Coordinated Collection Strategy

**Recommended workflow:**
1. **Weekly:** Run Proxmox collection (existing script)
2. **Weekly:** Run TrueNAS collection (this system)
3. **Archive:** Both exports to same timestamp directory
4. **Document:** Cross-reference in CLAUDE_STATUS.md

**Directory structure:**
```
homelab-exports/
├── export-20251214-030000/
│   ├── proxmox-export/          # From collect-homelab-config.sh
│   └── truenas-export/          # From collect-truenas-config.sh
└── export-20251207-030000/
    ├── proxmox-export/
    └── truenas-export/
```

---

## Next Steps

### Phase 1: Foundation (Current)
- [x] Document API endpoints and SSH commands
- [x] Create reference documentation
- [ ] Generate API key and test access
- [ ] Test individual API endpoints
- [ ] Test SSH command execution

### Phase 2: Script Development
- [ ] Create `collect-truenas-config.sh` script
- [ ] Implement collection levels
- [ ] Add error handling and logging
- [ ] Create directory structure generation
- [ ] Add sanitization functions

### Phase 3: Integration
- [ ] Integrate with homelab collection workflow
- [ ] Add to cron for automated collection
- [ ] Create unified export archive structure
- [ ] Update CLAUDE_STATUS.md automatically

### Phase 4: Monitoring
- [ ] Set up Prometheus exporters
- [ ] Create Grafana dashboards
- [ ] Configure alerting rules

---

## Reference Links

**Official Documentation:**
- TrueNAS Scale Documentation: https://www.truenas.com/docs/scale/
- TrueNAS API Reference: https://www.truenas.com/docs/api/

**Related Homelab Documentation:**
- Proxmox Collection: `/home/jramos/homelab/scripts/crawlers-exporters/collect-homelab-config.sh`
- Homelab Index: `/home/jramos/homelab/INDEX.md`
- Status File: `/home/jramos/homelab/CLAUDE_STATUS.md`

**Community Resources:**
- TrueNAS Forums: https://forums.truenas.com/
- r/truenas: https://reddit.com/r/truenas

---

**Document Version:** 1.0.0
**Last Updated:** 2025-12-14
**Maintained By:** Scribe Agent
**Homelab Repository:** /home/jramos/homelab