Observability¶
Ecosystem reference
This page covers how agents emit and use telemetry through the agents SDK. For more dedicated and comprehensive reading, see LedgerGr.id (docs, GitHub): the telemetry platform for metrics, tracing, logging and transaction ledgers, with its own instrumentation SDKs and Kubernetes deployments.
Observability means being able to tell what an agent did, why it did it, and how well it went, from the signals it emits: logs, metrics and traces. In a multi-agent system you also need to follow one task as it moves between agents, tools, models and workflows, and to keep the record of a run together with the feedback it received.
| Signal | What it answers | Where it comes from |
|---|---|---|
| Logs | What happened, step by step | Standard Python logging in every service and agent |
| Metrics | How many, how fast, how often | Agent application metrics on port 9090. See Metrics |
| Traces | How one task moved through steps and agents | job_output_trace_data on every result; workflow-engine trace events |
| Live task streams | What the agent is doing right now: tokens, tool calls, tool results, progress | context.stream(task). See Agents Streaming |
| Evaluations | Whether the result met its expectation, and why not | EvaluationReport. See Evaluation |
| Feedback | Whether the result was good | Verification results, human responses, Incorporator feedback |
Logs¶
| Component | Control |
|---|---|
| Agent instances (SDK) | LOG_LEVEL (default INFO); format %(asctime)s %(levelname)s %(name)s: %(message)s |
| Agent specification | execution.logging_level (CRITICAL, ERROR, WARNING, INFO, DEBUG) |
| Platform services (subjects service, deployer, scouter, Delegate API, chat server) | LOG_LEVEL |
Useful log lines to watch for:
| Message | Meaning |
|---|---|
on_preprocess returned None (task_id=…) |
The agent dropped a task |
on_data failed (subtask_id=…) |
The agent raised. The result carries is_error and error_data.stage = "data" |
[p2p …] BLOCKED session_id=… |
An admission policy blocked a mesh task |
failed to route result (type=…) |
A result could not be delivered to its output_ptr destination. See Output Routing |
Recovered N unacknowledged task(s) |
Tasks left in flight by a previous run of the instance were requeued |
Max deliveries reached; moving task to dead-letter list |
A task failed REDIS_MAX_DELIVERIES times and was parked in jobs:inbox:dead |
[admission] … policy failed (fail-closed) |
A policy error blocked a task (POLICY_FAIL_MODE=closed) |
[AutoScaler] scale_up / scale_down |
The auto-scaler changed the instance count |
Guardrail policy '…' loaded |
An SDK guardrail policy is active on the instance |
Exhausted retries for exchange_id=… |
A result could not be reported to the exchange |
Logs go to standard output, so any Kubernetes log stack can collect them.
Metrics¶
Agents expose application metrics on port 9090, and the platform includes a metrics collector (k8s/global-services/metrics-collector.yaml). See Metrics for defining and publishing metrics from agent code.
Traces¶
Per-Result Trace Data¶
Every AgentResult has a job_output_trace_data dictionary. It travels with the result to the exchange, the calling agent or a Redis queue. Use it to record how the result was produced:
return AgentResult(
task_id=task.task_id,
job_output={"answer": answer},
job_output_trace_data={
"model": "gpt-4o-mini",
"tool_calls": [{"tool": "search", "args": {"q": q}, "ms": 412}],
"delegations": [{"to": "fact-checker", "task_id": sub_id}],
"latency_ms": elapsed_ms,
},
)
Use the same task_id and session_id when you delegate, so a run can be reassembled across agents.
Workflow-Engine Trace Events¶
AgentWorkflowEngine emits workflow_start, plan_cache_hit / plan_cache_miss, converter_executed, replan_triggered, replan_limit_reached, workflow_completed and workflow_failed. With enable_tracing=True the built-in tracer prints them. To send them to your tracing backend, pass your own tracer:
class OTelTracer:
def __init__(self, tracer):
self.tracer = tracer # e.g. an OpenTelemetry tracer
def log(self, event_type, payload):
with self.tracer.start_as_current_span(event_type) as span:
for k, v in payload.items():
span.set_attribute(k, str(v))
engine = AgentWorkflowEngine(llm_client=client, modules=modules, tracer=OTelTracer(otel_tracer))
The final WorkflowState.to_dict() contains every node's input, output, status, attempts and timings.
LLM Latency¶
The LLM interface keeps per-backend latency statistics (count and last, minimum, maximum and average latency). See Agent × AI Model Interface.
Feedback Paths¶
To improve agents, keep the record of a run together with the feedback on it:
- Return trace data with every result, as shown above.
- Store results and traces, for example in the Persistent Store or agent memory, keyed by
task_id. - Attach feedback as it arrives: verification outcomes (Verification System), human answers (Human-in-the-Loop) and lesson feedback (Incorporator & Feedback).
- Attach evaluation reports (
report.to_dict()) to the trace data of results that were checked against an expectation. - Feed the combined record to the Learner so it becomes lessons. See Loop Improvement.
Connecting to Your Observability Stack¶
| Signal | How to export it |
|---|---|
| Logs | Standard output of every pod. Collect them with your cluster's log stack (Fluent Bit, Loki, Elasticsearch, ...) |
| Metrics | Scrape port 9090 of agent instances, or use the platform metrics collector |
| Workflow traces | A custom tracer= object, as above |
| Task traces | job_output_trace_data on every result. Results routed to Redis (output_ptr.type = "redis") can be consumed by an exporter |
| Live activity | Subscribe to live task streams on Redis |
Ecosystem reference
This page covers how agents emit and use telemetry through the agents SDK. For more dedicated and comprehensive reading, see LedgerGr.id (docs, GitHub): the telemetry platform for metrics, tracing, logging and transaction ledgers, with its own instrumentation SDKs and Kubernetes deployments.