What is an AI evaluation?
An evaluation is a repeatable check of whether the assistant still does the job. You keep a small set of real inputs and the answers you would accept, and you run them again after you change a prompt, a file, or a model.
Also called: eval, evals, red teaming
In plain English
A vibe check is reading one nice draft and turning it on. An eval is the opposite: ten messy messages you have seen before, with a note on what a good reply must include and must not include. When you edit the prompt, you run the ten again. Red teaming is the hostile cousin: you add messages that try to break the rules, not just the ordinary Tuesday mail.
How an owner would use it
Save the emails that went wrong and the ones that were fine. After any change, rerun them before the next customer sees the new version. If you cannot name a failure, you are not ready to automate the send.
Common mistakes
Testing on the happy path only, then calling it an eval, hides the weird messages that matter. Changing the model and the prompt and the files on the same day means you will not know what helped. Do not borrow someone else's score. Do not red team by trying to jailbreak a system you do not own. Test your own bot with rude and tricky notes you wrote.
A worked example
A home cleaning company
Ten old texts, rerun after the prompt change
This is an illustration, not a customer result. The owner keeps ten inquiries: a pet fee question, a same-day ask, a complaint about a missed fridge. The accepted replies mention only fees that are on the current card. After they tighten the prompt, they rerun the ten. One draft still invents a weekend rate. They do not turn on auto-send.
Related terms
Prompts and guides
Questions owners ask
What is an AI evaluation?
It is a fixed set of examples you rerun to see if a change made the assistant better or worse on the jobs you actually get.
How do I evaluate an assistant without a data science team?
Use ten to twenty real messages with the private details removed. Write what must appear and what must not. Rerun them by hand after each change until the volume forces a tool.
What is red teaming in this context?
It is testing your own assistant with hostile or tricky inputs, such as a customer email that tells the bot to ignore your prices. It is not permission to attack someone else's system.
Product guides are on AI apps for business.