AI tutorial: build a repeatable evaluation for structured answers

Before connecting an AI assistant to a workflow, decide what a usable answer looks like. This tutorial creates a small evaluation harness for an assistant that extracts an order ID and issue category from a message. You will test JSON structure, allowed labels and exact answers using synthetic fixtures.

1. Write the task contract

The assistant must return a JSON object containing exactly two keys: `order_id` and `issue`. Allowed issue values are `delivery`, `refund` and `other`. An absent order ID must be `null`, not a guessed number. A useful prompt is: “Extract only facts present in the message. Return exactly the two required keys. Do not add an explanation or invent an order ID.”

This contract is intentionally narrow. It gives you observable checks instead of asking whether an answer merely sounds helpful. Keep real customer messages out of this exercise.

2. Create a local evaluation harness

Save the following as `evaluate_answers.py`. It does not call an AI provider or require an API key. The candidate responses are hand-written test fixtures; replace them with actual model outputs when you are ready to compare prompts.

import json

cases = [
    ({"order_id": "A17", "issue": "delivery"},
     '{"order_id":"A17","issue":"delivery"}'),
    ({"order_id": None, "issue": "refund"},
     '{"order_id":"UNKNOWN","issue":"refund"}'),
    ({"order_id": None, "issue": "other"}, 'not JSON'),
]

def evaluate(expected, raw):
    try:
        answer = json.loads(raw)
    except json.JSONDecodeError:
        return {"valid": False, "correct": False}
    valid = (
        isinstance(answer, dict)
        and set(answer) == {"order_id", "issue"}
        and answer["issue"] in ["delivery", "refund", "other"]
        and (answer["order_id"] is None or isinstance(answer["order_id"], str))
    )
    return {"valid": valid, "correct": valid and answer == expected}

results = [evaluate(expected, raw) for expected, raw in cases]
print(results)
print("Exact matches:", sum(x["correct"] for x in results), "/", len(results))
assert sum(x["valid"] for x in results) == 2
assert sum(x["correct"] for x in results) == 1

3. Interpret the failures

Run `python evaluate_answers.py`. Two fixtures pass the structure check, but only one matches the expected answer. The second response demonstrates why valid JSON is not the same as a correct extraction. The third cannot be parsed at all.

Add the input message to each case in your own version, then collect responses from a chosen model using the same settings. Store the model identifier, prompt version and evaluation date with each run. Keep a separate set of examples that you do not use while tuning the prompt.

4. Expand the evaluation deliberately

Add cases with no order ID, more than one order ID, misspellings and messages that contain instructions to ignore the extraction task. Decide how ambiguity should be handled before judging outputs. Consider reviewing privacy and harmful-action risks separately; exact-match accuracy does not measure every kind of failure.

This three-case exercise teaches the mechanics of testing. It is far too small to establish production reliability. The useful result is a repeatable process that makes improvements and regressions visible.

References

Python JSON decoding · Python unittest guide