Persistence, Checkpointing & Recovery¶
Long-running agent tasks — multi-step plans, workflows that call other agents, processes waiting on humans — must survive restarts, crashes and redeployments. This chapter explains what the platform persists for you, what it does not, and how to checkpoint and resume your own long-running work.
What Is Persisted Automatically¶
| State | Where | Survives |
|---|---|---|
| Agent specification (Subject) | Subjects service → MongoDB | Everything |
| Where each agent runs (Runtime Subject) | Subjects service → MongoDB | Everything |
| Delegation records and status | Delegation system | Agent restarts. See Task Delegation |
| Verification tasks and results | Verification system → MongoDB | Agent restarts. See Verification System |
| Human-intervention requests | HIS → MongoDB | Agent restarts. See Human-in-the-Loop |
| Behavior-controller tasks and results | Behavior controller DB | Agent restarts |
| Queued and in-flight tasks | Instance Redis: queue jobs:inbox, processing list jobs:inbox:processing:<instance>, dead letters jobs:inbox:dead |
Instance restarts. Unacknowledged tasks are redelivered (at-least-once) |
| Rate counters and backlogs | Executor — memory, or Redis with RATE_LIMITER_BACKEND=redis / BACKLOG_BACKEND=redis |
Executor restarts with the Redis backends |
| Memory, context, objects | Memory, Context Cache, Persistent Store | Depends on the backend you configure |
What Lives Only in the Process¶
- Results waiting to be reported to an exchange. The output reporter sends them from an in-memory queue, retrying 3 times. Because the task is acknowledged only after
on_datareturns and the result is queued, a crash in between redelivers the task, and the result is produced again. - State held in your agent object. Keep anything you need after a restart in a checkpoint (below).
Because a redelivered task runs on_data again, design long-running work to resume from a checkpoint and make side effects idempotent per task_id.
Checkpointing Workflow State¶
The planning engine's WorkflowState (agent_workflows.state) records everything about a run — goal and global context, every node's input, output, status, error, attempt count and timing, and the execution order — and serialises to a plain dict:
from agent_workflows.state import WorkflowState
from agent_core_libs.storage import S3Storage # the Persistent Store
store = S3Storage(access_key_id="...", secret_access_key="...", region_name="us-east-1")
# save
store.put_json("agent-checkpoints", "report-run-42.json", state.to_dict())
# restore
state = WorkflowState.from_dict(store.get_json("agent-checkpoints", "report-run-42.json"))
state.get_completed_nodes() # nodes that do not need to run again
state.get_pending_nodes()
state.get_failed_nodes()
Save a snapshot after every node completes (or every few nodes) to any durable store — the Persistent Store, the Profile Manager for small state, or memory.
Checkpointing Your Own Long Tasks¶
The same pattern works in plain agent code:
- Give the task a stable ID (
task.task_id) and use it as the checkpoint key. - At the start of
on_data, load the checkpoint if one exists and skip the steps already done. - After each expensive step, save the checkpoint with the step's output.
- Make steps idempotent — running a step twice with the same input must be safe, because a crash can happen after the step but before the checkpoint is written.
- Delete the checkpoint when the final result has been returned.
def on_data(self, task):
bucket, key = "agent-checkpoints", f"{task.task_id}.json"
ckpt = self.store.get_json(bucket, key) if self.store.exists(bucket, key) else {"done": {}}
for step in ("collect", "analyse", "draft"):
if step not in ckpt["done"]:
ckpt["done"][step] = self.run_step(step, task.job_data, ckpt["done"])
self.store.put_json(bucket, key, ckpt) # checkpoint after each step
self.store.delete_object(bucket, key)
return AgentResult(task_id=task.task_id, job_output=ckpt["done"]["draft"])
Recovery¶
| Failure | Recovery strategy |
|---|---|
| Instance crash mid-task | The task is redelivered when the instance restarts. With checkpoints, it resumes from the last completed step instead of starting over |
| Task fails on every attempt | After REDIS_MAX_DELIVERIES attempts it is moved to jobs:inbox:dead. Inspect it with redis-cli LRANGE jobs:inbox:dead 0 -1 and requeue it with LMOVE jobs:inbox:dead jobs:inbox RIGHT LEFT once fixed |
| Step fails repeatedly | In AgentWorkflowEngine, modules retry up to max_retries, then the engine replans around the failure (up to max_replans times) with the completed and failed nodes as context. See Prebuilt Operators |
| Waiting on a human across restarts | Store the HIS request ID in the checkpoint and resume with check_and_get_response(id) |
| Long external jobs | Use job-based workflow policies, which submit to an external executor and poll for completion. See Workflow System |
| Results must not be lost | Have results verified or acknowledged — the Verification System persists every verification |