Test Sets, Rubrics, and Auto-Evals

You've crafted what feels like the perfect prompt. It works brilliantly on the three examples you tested. You deploy it to production, and suddenly you're getting complaints about inconsistent outputs, hallucinations, and responses that completely miss the mark. Sound familiar?

The problem isn't your prompt—it's that you tested it like you'd test a light switch. Turn it on, it works, ship it. But prompts aren't binary. They're probabilistic systems that need rigorous, systematic evaluation before they're ready for real users.

Let me show you how to evaluate prompts like a professional.

Building Your Test Set: The Foundation

Before you evaluate anything, you need a proper test set. Think of this as your prompt's obstacle course—a collection of inputs that represent the full spectrum of what you'll encounter in production.

Start with diversity. If you're building a customer support classifier, don't just test happy-path questions. Include edge cases: misspellings, multiple issues in one message, passive-aggressive tone, requests in different languages, and completely irrelevant queries.

Aim for 50-100 test cases minimum. Yes, that sounds like a lot, but here's the reality: three examples tell you almost nothing about how your prompt will perform at scale. Organize your test set into categories: typical cases (60%), edge cases (30%), and adversarial cases (10%).

Document the expected output for each test case. This is your ground truth. Without it, you're just generating outputs and hoping they look good.

Creating Human Rating Rubrics

Now you need a scoring system. Subjective "this looks good" evaluations are useless when you're comparing prompt versions or debugging failures.

Build a rubric with specific, measurable criteria. For a customer support prompt, you might evaluate: accuracy (does it correctly identify the issue?), tone appropriateness (is it empathetic and professional?), completeness (does it address all parts of the question?), and safety (does it avoid making promises the company can't keep?).

Use a consistent scale—typically 1-5 or 1-10. Define what each rating means. "5 for accuracy" should mean "correctly identifies the primary issue and all secondary concerns" not just "pretty accurate I guess."

Create evaluation guidelines with examples. Show what a 5 looks like versus a 3. This turns subjective judgment into a repeatable process. When multiple people can apply your rubric and get similar scores, you know it's working.

LLM-as-Judge: Automating Evaluation

Here's where it gets powerful: using AI to evaluate AI. The "LLM-as-judge" technique lets you scale evaluation beyond what humans can reasonably review.

The concept is simple. You give a language model your test input, the generated output, and a detailed rubric, then ask it to score the response. The prompt might look like: "Evaluate this customer support response on accuracy (1-5). A score of 5 means the response correctly identifies all issues mentioned. A score of 1 means it misses the primary issue entirely."

The key is specificity. Generic prompts like "rate this response" produce garbage scores. Give the judge model concrete criteria, examples of good and bad outputs, and clear scoring thresholds.

Run correlation studies between human ratings and LLM-judge scores. If they align 80%+ of the time, you've got a reliable automated evaluator. If not, refine your judge prompt until they do.

Beware of judge model biases. Some models rate verbose outputs higher regardless of quality. Others prefer formal tone even when casual is appropriate. Always validate your judge against human ratings periodically.

Continuous Evaluation in Production

Testing before deployment is just the beginning. Real users will always find scenarios you never imagined.

Implement sampling-based evaluation. Randomly select 1-5% of production outputs daily and run them through your automated evaluators. Track metrics over time: accuracy trends, tone consistency, failure patterns.

Set up threshold alerts. If your accuracy score drops below 4.0 for three consecutive days, something's wrong. Maybe users shifted their behavior, or a model update changed response patterns.

Create feedback loops. When users report issues, add those cases to your test set. Your evaluation system should grow smarter with every failure.

Use A/B testing for prompt changes. Run your new prompt on 10% of traffic while keeping the old version as control. Compare evaluation scores across both groups. Only promote changes that show measurable improvement.

Making It Practical

Start small. Pick your most critical prompt. Build a 20-case test set. Create a simple three-criteria rubric. Run it manually once. Then automate one piece at a time.

The difference between amateur and professional prompt engineering isn't creativity—it's measurement. You can't improve what you don't measure, and you can't measure what you haven't systematically defined.

Stop guessing. Start evaluating.