Local-First AI Systems
A production-grade personal AI stack where every system enforces human governance by design. Built by a testing practitioner, tested like production software.
The Governing Principle
I build and maintain a bespoke, local-first AI environment for knowledge management, evaluation, and agentic tooling. Not a product or a startup. It's personal infrastructure designed for privacy, control, and long-term leverage.
This stack is my daily working environment. Retrieving rich context across conversations, emails, and past decisions. Storing structured knowledge about ongoing work. Drafting responses with full history available. It has fundamentally changed how I work, and the patterns directly inform my evaluation workshops and conference talks. What follows is the architecture and thinking, not the implementation.
The Systems
Personal Knowledge Base
A unified knowledge store that syncs Gmail, Calendar, Slack, WhatsApp, ChatGPT conversations, GitHub issues, and handwritten notes into a single searchable index. Not a RAG app. A memory system with a Data Constitution.
Retrieval uses hybrid search (BM25 keyword + vector semantic + cross-encoder reranking) with a two-stage architecture: LLM-generated doc cards provide compressed summaries for first-pass scanning before loading full documents. This keeps context manageable for local models while maintaining recall.
The Data Constitution enforces three validation lanes (CLEAN, REPAIR, QUARANTINE). Corrupted data fails visibly, never silently. Every piece of knowledge has traceable provenance. All content lives as plain-text org-mode files, editable in any text editor, with no vendor lock-in.
Personal Assistant
An action layer that converts knowledge into suggestions: email reply drafts, Slack responses, receipt downloads, task creation. The hard boundary: the PA drafts, you decide. No external action without explicit human approval.
Most agent systems collapse the suggestion and decision layers. This system maintains separation by design. Batch processing uses a mix of local and API-based models depending on the task. A false-positive minimizer filters on confidence, commitment language, and ambiguity before anything reaches the review queue.
Evaluation Framework
The question: can 100% local open-source models match Claude for my actual workflows? Not benchmarks. Real tasks extracted from daily knowledge work, email triage, and receipt processing.
Tests run in bubblewrap filesystem isolation against a frozen fixture corpus. Train/validation/holdout splits prevent overfitting. A meta-optimizer improves vendor matching patterns but is constrained by governance gates: validation must not regress, false positives are penalized 2x, and changes are limited per iteration.
Key Finding
Multi-step tool use is the blocker. Local models execute single tools well but fail at chained operations. The gap is measurable (~78% vs ~52% pass rate), specific (tool chaining, not reasoning), and being tracked. The smaller 14B model outperforms the 32B and 72B variants. Scale isn't everything.
Training Pipeline
LoRA fine-tuning of local models using training data extracted from real workflows: Claude Code sessions, receipt processing, email and Slack draft responses. The pipeline extracts, trains via Unsloth, exports to GGUF format for Ollama, and validates through the eval framework before promotion to production.
Daily Diary
Answers "what did I do yesterday?" by aggregating audit logs, git commits, emails, calendar events, Slack threads, and more into a structured daily journal. The LLM organizes data, it doesn't judge productivity. A witness, not a coach. Built in a weekend because the frustration was fresh and the infrastructure was ready.
Audit Logger
Structured JSONL logging across all systems with span tracing and daily rotation. Every tool call, every decision, every action is traceable. Zero external dependencies. The foundation that makes the diary, debugging, and cost tracking possible.
How It Connects
Data flows from ingestion (Gmail, Calendar, Slack, WhatsApp, ChatGPT, Claude sessions) through the knowledge base indexer (hybrid search with LanceDB) into the personal assistant for classification and suggestion generation. Everything passes through human review before execution. The audit logger captures the full trace.
The human interface is Emacs with custom Elisp: reviewing PA suggestions, running morning routines, browsing the daily diary, navigating the knowledge base, and reviewing Data Constitution quarantine items. Ten bespoke modes that turn a text editor into a control surface.
The evaluation framework sits alongside, testing whether local models can handle specific workflow families. When they can, those tasks graduate to local execution. When they can't, the gap is measured and tracked rather than ignored.
Why This Matters
The testing background isn't incidental to the AI work. It's why the AI work is rigorous. Every system has validation gates, not because it's fun to build them, but because I've spent 20 years learning what happens when you ship without them.
This infrastructure represents a way of thinking about AI systems: governed iteration with explicit memory and human judgment beats autonomy. The patterns transfer to any team building LLM-based tools, which is why this work directly informs my evaluation workshops and conference talks.
Interested?
I write and speak about the architectural patterns, failure modes, and design trade-offs from building these systems. If you're working on similar problems, whether evaluating LLM-based tools, building governed AI workflows, or designing privacy-first infrastructure, I'd enjoy the conversation.