Writing Evals for AI Tools
Test automation applied to a new domain. Build your own evaluation framework for AI/LLM systems.
First delivered as a full-day tutorial at HUSTEF 2026 in Budapest.
The Inspection Problem
Everyone's adding AI to their products, but how do you test it? Most teams are "vibe-checking" outputs or relying on manual review. That doesn't scale, and it doesn't catch regressions when the underlying model changes.
The Insight
LLM evaluation is essentially test automation applied to a new domain. If you can write test cases, you can write evals. The skills transfer directly. You just need to learn the new patterns for non-deterministic systems.
What You'll Build
In this workshop, participants build their own evaluation framework from scratch:
- Design test cases for non-deterministic LLM-based systems
- Build scoring rubrics (structural assertions, not vibes)
- Handle non-determinism with statistical approaches
- Run real evaluations and interpret results
- Walk away with a framework applicable to any LLM-based feature
Agenda
Available as one or two days (09:00 to 16:30 each). The full day covers code-based evals end to end. The second day climbs to the techniques that catch what code-based checks miss.
Day one: code-based evals
- Morning: why evals are test automation in a new domain; meet the system under test; build up your first assertions, exclusion rules, factual-accuracy checks, and weighted scoring
- Afternoon: comparing variants, A/B testing, where evals sit in delivery (governance and regression gates), and taking the framework back to your own feature
Day two: beyond code-based
- Model-as-judge, done properly: compare candidates instead of scoring 0 to 10, and prove the judge can tell good from bad before you trust it
- Trajectory evals: judge the path an agent takes, not just its final answer
- Multi-turn evaluation: synthetic users that drive real conversations
- Human evaluation that isn't vibes: blind rating, rubrics, inter-rater agreement
- Assembling a layered strategy for your own system
For in-house delivery, the system under test and examples are tuned to your domain on request.
Who Is This For?
- Testers and QA engineers working with LLM-based tools
- Development teams adding LLM capabilities to products
- Anyone responsible for LLM-based system quality
No AI experience required. Testing experience is the foundation. Participants work in pre-configured cloud environments with AI coding assistants (currently Claude Code), so the focus stays on test design and judgment, not setup. The workshop is tool-agnostic. The patterns apply to any LLM-based system.
Pricing
Full day: €6,000. Two days: €10,000.
In-house delivery across Europe, travel included. Conference tutorials, North American, and custom-scoped engagements are quoted on request.
Interest Inquiry
This workshop is available for conference tutorials and in-house training.