Evaluation¶
Ecosystem reference
This page covers evaluation from inside agents, with the agents_eval library. For scoring agents and models against labelled datasets as queued, containerised jobs, see Evaluate2.ai (GitHub) and Offline Benchmarks with Evaluate2.ai below.
Evaluation answers one question: is this output good enough? On AgentGr.id you describe what a good result looks like as an expectation, check outputs against it with an evaluator, and let agents self-refine: try, get evaluated, and try again with the feedback until the expectation is met or a budget runs out.
Everything on this page is in the agents_eval package (libs/agents-eval):
pip install ./libs/agents-eval
Expectations¶
An expectation is a named list of criteria with a pass score:
from agents_eval import Expectation, Criterion
summary_expectation = Expectation(
name="contract-summary",
description="A short, neutral summary that names the governing law and lists risks.",
criteria=[
Criterion.required_fields(["summary", "risks"], required=True), # must pass
Criterion.max_length("summary", 1200),
Criterion.min_items("risks", 1),
Criterion.regex("summary", r"(?i)governing law"),
Criterion.custom("no placeholders", lambda out, ctx: "TODO" not in out["summary"]),
Criterion.llm_judge("Is the summary accurate to the source, neutral and complete?",
threshold=0.7, weight=2.0),
],
pass_score=0.85,
)
Criteria¶
| Constructor | Passes when |
|---|---|
required_fields(fields) |
Every field exists and is not empty. Dotted paths are allowed ("result.items") |
type(path, type_name) |
The value is a string, number, integer, boolean, object, array or null |
regex(path, pattern) |
The pattern is found in the value |
contains(path, value) |
value in the field (substring or list member) |
max_length(path, n) / min_length(path, n) |
Length limits |
min_items(path, n) |
A list has at least n items |
range(path, minimum=None, maximum=None) |
A number is within bounds |
equals(path, value) |
Exact match |
custom(name, fn) |
fn(output, context) returns True, a score ≥ threshold (default 0.5), or (bool_or_score, message) |
llm_judge(rubric, threshold=0.7) |
A model scores the output from 0 to 1 against the rubric, and the score is ≥ threshold |
benchmark(dataset_id, category_id, metric="pass_rate", minimum=None, maximum=None) |
The output, a set of predictions, is evaluated on Evaluate2.ai and the metric is within bounds. See Benchmark Criteria |
Every constructor also takes name=, weight= (default 1.0) and required= (default False).
Scoring¶
- Each criterion scores 1 or 0, except
llm_judge,benchmarkand score-returningcustomcriteria, which score between 0 and 1. - The evaluation score is the weighted average of the criterion scores.
- The output passes when the score is ≥
pass_scoreand everyrequired=Truecriterion passed.
Expectations made of data-only criteria (everything except custom) can be stored and shared as JSON with expectation.to_dict() and Expectation.from_dict(...). For example, a requester can send the expectation along with the task.
Evaluating Output¶
from agents_eval import Evaluator
evaluator = Evaluator(llm=Evaluator.llm_from_client(client, name="openai")) # model for llm_judge
report = evaluator.evaluate(output, summary_expectation, context={"source": contract_text})
report.passed # True / False
report.score # 0.0 – 1.0
report.failures # failed CriterionResult objects (name, score, message)
report.feedback # text listing what to fix, ready to send back to the agent
report.to_dict() # for logs, traces or job_output_trace_data
context is shown to the judge model with the output, so include whatever it needs to check accuracy, for example the source document or the original question. The judge runs at temperature 0 and must answer with {"score": ..., "reason": ...}. The reason becomes the failure message.
Self-Evaluation Loops¶
self_refine lets an agent work towards an expectation. It calls your produce(feedback, attempt) function, evaluates the output, and calls it again with the evaluator's feedback until the output passes, or until a limit is reached:
from agents_eval import self_refine
def on_data(self, task):
source = task.job_data["contract"]
def produce(feedback, attempt):
prompt = f"Summarise this contract as JSON with 'summary' and 'risks'.\n\n{source}"
if feedback:
prompt += f"\n\n{feedback}\nFix these problems."
return self.ask_model_for_json(prompt)
result = self_refine(
produce,
summary_expectation,
evaluator=self.evaluator,
max_attempts=4, # compute limit: number of tries
time_budget_s=90, # wall-clock limit
patience=2, # stop after 2 attempts in a row without a better score
context={"source": source},
)
return AgentResult(
task_id=task.task_id,
job_output=result.output, # best output seen
is_error=not result.passed,
job_output_trace_data={"evaluation": result.report.to_dict(),
"attempts": result.history,
"stopped_because": result.stopped_because},
)
RefineResult field |
Meaning |
|---|---|
output |
The passing output, or the best-scoring one if none passed |
report |
Its EvaluationReport |
passed |
Whether the expectation was met |
attempts |
How many outputs were produced |
stopped_because |
passed, max_attempts, time_budget or no_improvement |
history |
Per attempt: score, passed, feedback, elapsed time |
Use on_attempt=lambda n, out, report: ... to observe each attempt, for example to send progress events on a live task stream.
Offline Benchmarks with Evaluate2.ai¶
Everything above runs online: inside the agent, on one output, in milliseconds. Evaluate2.ai adds the offline half. It scores a whole dataset of outputs as a queued job in its own container, against labelled answers, and returns aggregate metrics.
Online (agents_eval) |
Offline (Evaluate2.ai) | |
|---|---|---|
| When | Every task, while the agent works | Before a release, nightly, or when an agent tests a change |
| Question | Is this one output good enough? | How good is this agent, prompt or model across a dataset? |
| Ground truth | An expectation | A labelled dataset, plus optionally the same expectation |
| Speed | Milliseconds to seconds, in-process | A queued container job, seconds to minutes |
The two share one definition of "good". The agent_task_eval evaluator for Evaluate2.ai runs the same Expectation objects with agents_eval inside its container. The same expectation therefore gates single outputs at runtime and whole datasets offline.
dataset (tasks + expected answers) agent Evaluate2.ai
─────────────────────────────── ───── ────────────
upload once, with the expectation ──► runs every task ──► predictions.jsonl ──► agent_task_eval job
│
Criterion.benchmark / CI gate ◄────────── pass_rate, f1, tool_calls, failures ◄──────┘
Three pieces make this work:
| Piece | Where | What it does |
|---|---|---|
Evaluate2Client |
agents_eval.evaluate2 |
Uploads predictions, runs evaluations and waits for results |
Criterion.benchmark |
agents_eval |
Turns an Evaluate2.ai metric into a criterion, for release gates and refinement loops |
agent_task_eval |
Evaluate2.ai repository, agent-metrics/agent_task_eval/ |
The evaluator image that scores agent tasks |
One-Time Setup¶
1. Build the evaluator image and push it to a registry that the Evaluate2.ai scheduler can pull from. The build bundles the Evaluate2.ai eval framework, agents_eval and the evaluator:
cd Evaluate2.ai/agent-metrics/agent_task_eval
./build.sh /path/to/agents-v1/services/agents/libs/agents-eval registry.example.com/agent_task_eval:1.0
docker push registry.example.com/agent_task_eval:1.0
2. Create a client and register the image as a category:
import os
from agents_eval import Evaluate2Client
e2 = Evaluate2Client(
users_url="http://evaluate2-users:5000",
jobs_url="http://evaluate2-jobs:5001",
username="admin",
password=os.environ["EVALUATE2_PASSWORD"],
admin=True, # see Current Limitations
)
e2.create_category("agent-tasks", image_name="registry.example.com/agent_task_eval:1.0",
pkg_name="agent_task_eval", description="AgentGr.id agent task evaluation")
Evaluate2Client.from_env() reads the same settings from EVALUATE2_USERS_URL, EVALUATE2_JOBS_URL, EVALUATE2_USERNAME, EVALUATE2_PASSWORD and EVALUATE2_ADMIN.
Creating a Dataset¶
A dataset is a JSONL file (or JSON array) of tasks. Every field except id is optional:
{"id": "t1", "input": {"question": "Which state's law governs the RetailCo agreement?"},
"expected": {"answer": "Delaware"},
"expected_tool_calls": [{"name": "search_contracts", "arguments": {"party": "RetailCo"}}],
"tags": ["governing-law"]}
| Task field | Meaning |
|---|---|
id |
Unique task ID; predictions are matched to tasks by it |
input |
What the agent is given. It is also passed to llm_judge criteria as context |
expected |
The expected answer. A dict is compared field by field; anything else is compared as a whole |
expected_tool_calls |
Tool calls the agent should make: names, or {"name", "arguments"}. Only the listed arguments are checked |
context |
Extra context for llm_judge criteria, such as a source document |
expectation |
An expectation for this task only, overriding the dataset's |
tags |
Labels; the result reports the pass rate per tag |
Upload the file, and register the dataset with its parameters: the expectation and the scoring options.
from agents_eval import Expectation, Criterion
answer_expectation = Expectation(
name="contract-answer",
criteria=[
Criterion.required_fields(["answer", "citations"], required=True),
Criterion.min_items("citations", 1),
Criterion.range("confidence", 0, 1),
],
)
file_id = e2.upload_dataset_file("agent-tasks", "contract_qa_v1.jsonl")
e2.create_dataset("contract-qa-v1", "agent-tasks", file_id, parameters={
"expectation": answer_expectation.to_dict(),
"match": {"fields": ["answer"]},
"pass_requires": ["expectation", "exact_match", "tool_calls"],
})
| Parameter | Default | Meaning |
|---|---|---|
expectation |
none | An Expectation.to_dict(), applied to every output |
match.fields |
the keys of expected |
Dotted paths compared with expected |
match.normalize |
true |
Compare strings case- and punctuation-insensitively, ignoring articles |
pass_requires |
see below | Which checks a task must pass: any of expectation, exact_match, tool_calls |
judge |
none | Model for llm_judge criteria: {"base_url", "model", "api_key_env"}, for any OpenAI-compatible endpoint |
max_failures |
20 |
How many failing tasks to list in the result |
include_tasks |
false |
Also return a per-task task_results list |
By default, a task passes when its output meets the expectation, or, without an expectation, matches expected exactly. When the task lists expected_tool_calls, the agent must also have made each of them with the expected arguments. Custom (Criterion.custom) criteria are code and cannot be sent to Evaluate2.ai; use data criteria or llm_judge.
Check a dataset before uploading it
python run_local.py tasks.jsonl predictions.jsonl params.json, in agent-metrics/agent_task_eval/, runs the evaluator on local files and prints the result, without Evaluate2.ai. Install agents_eval first.
Collecting Predictions¶
Run every task through the agent and write one record per task:
{"id": "t1", "output": {"answer": "Delaware", "citations": ["retailco-msa.pdf#p12"], "confidence": 0.93},
"tool_calls": [{"name": "search_contracts", "arguments": {"party": "RetailCo"}}], "latency_s": 2.1}
| Prediction field | Meaning |
|---|---|
id |
The task ID |
output |
The agent's result |
tool_calls |
Calls the agent made: {"name", "arguments"} or OpenAI-style {"function": {"name", "arguments"}} |
latency_s |
Time taken; reported as mean, p50, p95 and max |
error |
Set when the agent failed; the task counts as failed |
For example, sending the tasks to a deployed agent:
import json, time
def collect_predictions(tasks, call_agent):
predictions = []
for task in tasks:
start = time.time()
try:
result = call_agent(task["input"]) # e.g. a Delegate API call
predictions.append({"id": task["id"], "output": result["output"],
"tool_calls": result.get("tool_calls", []),
"latency_s": round(time.time() - start, 3)})
except Exception as e:
predictions.append({"id": task["id"], "error": str(e),
"latency_s": round(time.time() - start, 3)})
return predictions
Running an Evaluation¶
result = e2.run("contract-qa-v1", "agent-tasks", predictions, timeout_s=600)
result.succeeded # True when the job finished with status "success"
result.output["pass_rate"] # 0.0 – 1.0
result.output["failures"] # the lowest-scoring failed tasks, with feedback
result.metrics # Evaluate2.ai phase timings (admin clients)
predictions can be a list of records (sent as JSONL), a dict, bytes or a file path. To queue the job and continue, for example in a workflow step, use eval_id = e2.submit(...) and later e2.wait(eval_id, "agent-tasks"). If the job fails, run and wait raise Evaluate2Error with Evaluate2.ai's failure message; pass raise_on_failure=False to get the result with status and log instead.
The result of an agent_task_eval job:
| Field | Meaning |
|---|---|
tasks, passed, pass_rate |
How many tasks passed |
mean_score |
Average task score: the expectation score, the answer F1, or their average when both apply |
criteria |
Pass rate of each expectation criterion, by name |
exact_match, f1 |
Answer accuracy against expected (token-overlap F1 for strings) |
tool_calls |
precision, recall, f1, arguments_accuracy and sequence_accuracy |
latency_s |
mean, p50, p95, max |
by_tag |
Tasks and pass rate per tag |
missing_predictions, agent_errors |
Tasks with no prediction, and tasks where the agent reported an error |
failures |
Failed tasks with id, score and feedback, lowest score first |
Benchmark Criteria: Release Gates and Refinement¶
Criterion.benchmark turns an Evaluate2.ai metric into a criterion. The output being evaluated is the set of predictions; the criterion uploads it, runs the job, and checks the metric. Give the client to the Evaluator, in the same way as the model for llm_judge:
release_gate = Expectation(name="release-gate", criteria=[
Criterion.benchmark("contract-qa-v1", "agent-tasks", metric="pass_rate", minimum=0.9, required=True),
Criterion.benchmark("contract-qa-v1", "agent-tasks", metric="latency_s.p95", maximum=5.0),
])
report = Evaluator(evaluate2=e2).evaluate(predictions, release_gate)
if not report.passed:
raise SystemExit(report.feedback) # e.g. fail the CI job
metricis a dotted path into the result, such aspass_rate,tool_calls.f1orby_tag.dates.pass_rate.- A failing criterion scores partial credit (how close the value came to the bound), and its message names the job and the worst failing tasks, so
report.feedbacksays what to fix. report.results[i].detailsholds the job'seval_id,status,valueand fulloutput.- Each benchmark criterion runs its own job; criteria on the same dataset do not share one.
Because it is an ordinary criterion, a benchmark works with self_refine. This suits slower, offline improvement loops, for example an agent tuning its own prompt against a dataset:
def produce(feedback, attempt):
prompt = improve_prompt(current_prompt, feedback) # feedback lists the failing tasks
return collect_predictions(tasks, lambda inp: run_with_prompt(prompt, inp))
result = self_refine(produce, release_gate, evaluator=Evaluator(evaluate2=e2),
max_attempts=5, time_budget_s=3600)
Keep benchmarks out of per-task loops: every evaluation is a container job.
Current Limitations¶
These come from the current Evaluate2.ai code. The client works around them where it can.
Evaluate2.ai fixes needed before use
- Evaluation creation:
createEvaluationlooks the prediction file up byuniqueid, but the field is stored asuniqueId(frontend-apis/jobs-service/src/controls.js), so every evaluation is rejected with "Invalid evaluation file". Change the query key touniqueId. - Service startup: the Users and Jobs Service
index.jsfiles usePORT, which is defined only insrc/server.js. Define it inindex.jsfromUSERS_SVC_PORT/JOBS_SVC_PORT. - Node.js version: the service images are built
FROM node:latest. On Node.js 26 thejsonwebtokendependency fails to load. Pinnode:18(tested) or upgrade the dependency.
| Limitation | Effect | What to do |
|---|---|---|
| Only the root account can create datasets and evaluations (non-root access checks fail, see the Evaluate2.ai API reference) | Regular users are refused | Use admin=True with a dedicated deployment's root credentials, kept in a secret |
| Results are polled; there is no completion callback | wait polls getStatus every poll_s seconds |
Use submit and wait to keep the caller free |
| Tokens expire (5 minutes by default) and are per service | The client logs in to each service and logs in again when a token is rejected | |
| The scheduler passes only its own settings into evaluation containers | llm_judge inside agent_task_eval cannot receive an API key from the environment |
Use a judge endpoint that needs no key inside your network, or bake the key into the image as AGENT_EVAL_JUDGE_API_KEY. Avoid judge.api_key in dataset parameters, which are stored in plain text |
| The framework's base image is Python 3.6 | agent_task_eval cannot be uploaded as a dynamic zip package |
Use the container image built by build.sh (Python 3.8) |
Where Evaluation Fits¶
| Use | How |
|---|---|
| Testing an agent before release | Run a dataset through the agent and score it on Evaluate2.ai, with a benchmark release gate. Track the pass rate per version (see Testing Agents) |
| Self-checking inside the agent | self_refine, as above |
| Checking another agent's work | Evaluate delegated results before using them, and escalate to the Verification System or a human when they fail |
| Admission / output guardrails | Call an evaluator from a post_processor policy (see Governance) |
| Learning from outcomes | Store reports with the run's trace and feed them to the Learner (see Loop Improvement) |
| Comparing models or prompts | Run the same dataset with each variant and compare their Evaluate2.ai results |
Ecosystem reference
This page covers evaluation from inside agents, with the agents_eval library. For scoring agents and models against labelled datasets as queued, containerised jobs, see Evaluate2.ai (GitHub) and Offline Benchmarks with Evaluate2.ai below.