AI evaluation and safety · Case study 02
Testing a production AI agent before a risky sentence reaches a customer.
A public Python evaluation harness for an unsupervised rental-lead agent operating in a federally regulated workflow.
- Status
- Public, tested, and actively extended
- Role
- Evaluation design, domain cases, requirements, and validation
- Stack
- Python, Pydantic, pytest, YAML, and model adapters
The risk
A plausible answer can still be unlawful.
The agent reads inbound rental inquiries, looks up unit data, and drafts replies without a person reviewing every response first. That makes spot-checking inadequate: a rare failure can still create fair-housing exposure, misstate availability, or promise authority the agent does not have.
The evaluation harness converts those risks into repeatable cases and explicit criteria. The goal is not to claim the agent is flawless. It is to make known failures reproducible and prevent them from quietly returning.
Evaluation design
Evidence at each layer, with safety held apart.
Representative cases
Golden and adversarial datasets cover routing, duplicate handling, grounding, availability, fair housing, lead-paint disclosure, authority limits, and screening behavior.
Separate quality and safety
Quality criteria use release thresholds. Safety criteria are hard gates, because averaging a compliance failure into a strong tone score would hide the result that matters.
Scorers ranked by trust
Deterministic and structural assertions come first, contextual policy gates second, and LLM-as-judge only for subjective qualities such as tone and responsiveness.
Tests for the tests
Meta-tests feed deliberately bad and good answers through every safety gate. A scorer must prove it can catch its violation class without rejecting a lawful response.
What the work demonstrated
The evaluation can be wrong, too.
Across repeated evaluation rounds, several apparent agent failures were actually problems in the evaluation: overly broad assertions, missing route categories, mocked-backend behavior, and pattern gaps. Correcting those failures made the harness more trustworthy than simply celebrating a clean run.
The project also shows why production evidence matters. Real inbox patterns exposed duplicate notifications, unexpected source formats, and a recipient-address trap that the original specification did not capture.
Inspect the evidence
The implementation and full write-up are public.
The repository includes the cases, scorers, adapters, meta-tests, PII safeguards, setup instructions, and a candid account of the harness failures that changed the design.