Add API usage tracking and dynamic task reloading

Features:
- Usage tracking system (usage_tracker.py)
  - Tracks input/output tokens per API call
  - Calculates costs with support for cache pricing
  - Stores data in usage_data.json (gitignored)
  - Integrated into llm_interface.py

- Dynamic task scheduler reloading
  - Auto-detects YAML changes every 60s
  - No restart needed for new tasks
  - reload_tasks() method for manual refresh

- Example cost tracking scheduled task
  - Daily API usage report
  - Budget tracking ($5/month target)
  - Disabled by default in scheduled_tasks.yaml

Improvements:
- Fixed tool_use/tool_result pair splitting bug (CRITICAL)
- Added thread safety to agent.chat()
- Fixed N+1 query problem in hybrid search
- Optimized database batch queries
- Added conversation history pruning (50 messages max)

Updated .gitignore:
- Exclude user profiles (memory_workspace/users/*.md)
- Exclude usage data (usage_data.json)
- Exclude vector index (vectors.usearch)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
2026-02-13 23:38:44 -07:00
parent ab3a5afd59
commit 8afff96bb5
16 changed files with 1096 additions and 244 deletions

51
test_hybrid_search.py Normal file
View File

@@ -0,0 +1,51 @@
"""Test hybrid search implementation."""
from memory_system import MemorySystem
print("Initializing memory system with hybrid search...")
memory = MemorySystem()
print("\nRe-syncing all memories to generate embeddings...")
# Force re-index by clearing the database
memory.db.execute("DELETE FROM chunks")
memory.db.execute("DELETE FROM chunks_fts")
memory.db.execute("DELETE FROM files")
memory.db.commit()
# Re-sync to generate embeddings
memory.sync()
print("\n" + "="*60)
print("TESTING HYBRID SEARCH")
print("="*60)
# Test 1: Semantic search (should work even with different wording)
print("\n1. Testing semantic search for 'when was I born' (looking for birthday):")
results = memory.search_hybrid("when was I born", max_results=3)
for i, result in enumerate(results, 1):
print(f"\n Result {i} (score: {result['score']:.3f}):")
print(f" {result['path']}:{result['start_line']}-{result['end_line']}")
print(f" {result['snippet'][:100]}...")
# Test 2: Technical keyword search
print("\n2. Testing keyword search for 'SQLite FTS5':")
results = memory.search_hybrid("SQLite FTS5", max_results=3)
for i, result in enumerate(results, 1):
print(f"\n Result {i} (score: {result['score']:.3f}):")
print(f" {result['path']}:{result['start_line']}-{result['end_line']}")
print(f" {result['snippet'][:100]}...")
# Test 3: Conceptual search
print("\n3. Testing conceptual search for 'cost optimization':")
results = memory.search_hybrid("cost optimization", max_results=3)
for i, result in enumerate(results, 1):
print(f"\n Result {i} (score: {result['score']:.3f}):")
print(f" {result['path']}:{result['start_line']}-{result['end_line']}")
print(f" {result['snippet'][:100]}...")
print("\n" + "="*60)
print(f"Vector index size: {len(memory.vector_index)} embeddings")
print("="*60)
memory.close()
print("\nTest complete!")