Skip to content

Persistence, Checkpointing & Recovery

Long-running agent tasks — multi-step plans, workflows that call other agents, processes waiting on humans — must survive restarts, crashes and redeployments. This chapter explains what the platform persists for you, what it does not, and how to checkpoint and resume your own long-running work.

What Is Persisted Automatically

State Where Survives
Agent specification (Subject) Subjects service → MongoDB Everything
Where each agent runs (Runtime Subject) Subjects service → MongoDB Everything
Delegation records and status Delegation system Agent restarts. See Task Delegation
Verification tasks and results Verification system → MongoDB Agent restarts. See Verification System
Human-intervention requests HIS → MongoDB Agent restarts. See Human-in-the-Loop
Behavior-controller tasks and results Behavior controller DB Agent restarts
Queued and in-flight tasks Instance Redis: queue jobs:inbox, processing list jobs:inbox:processing:<instance>, dead letters jobs:inbox:dead Instance restarts. Unacknowledged tasks are redelivered (at-least-once)
Rate counters and backlogs Executor — memory, or Redis with RATE_LIMITER_BACKEND=redis / BACKLOG_BACKEND=redis Executor restarts with the Redis backends
Memory, context, objects Memory, Context Cache, Persistent Store Depends on the backend you configure

What Lives Only in the Process

  • Results waiting to be reported to an exchange. The output reporter sends them from an in-memory queue, retrying 3 times. Because the task is acknowledged only after on_data returns and the result is queued, a crash in between redelivers the task, and the result is produced again.
  • State held in your agent object. Keep anything you need after a restart in a checkpoint (below).

Because a redelivered task runs on_data again, design long-running work to resume from a checkpoint and make side effects idempotent per task_id.


Checkpointing Workflow State

The planning engine's WorkflowState (agent_workflows.state) records everything about a run — goal and global context, every node's input, output, status, error, attempt count and timing, and the execution order — and serialises to a plain dict:

from agent_workflows.state import WorkflowState
from agent_core_libs.storage import S3Storage        # the Persistent Store

store = S3Storage(access_key_id="...", secret_access_key="...", region_name="us-east-1")

# save
store.put_json("agent-checkpoints", "report-run-42.json", state.to_dict())

# restore
state = WorkflowState.from_dict(store.get_json("agent-checkpoints", "report-run-42.json"))
state.get_completed_nodes()   # nodes that do not need to run again
state.get_pending_nodes()
state.get_failed_nodes()

Save a snapshot after every node completes (or every few nodes) to any durable store — the Persistent Store, the Profile Manager for small state, or memory.

Checkpointing Your Own Long Tasks

The same pattern works in plain agent code:

  1. Give the task a stable ID (task.task_id) and use it as the checkpoint key.
  2. At the start of on_data, load the checkpoint if one exists and skip the steps already done.
  3. After each expensive step, save the checkpoint with the step's output.
  4. Make steps idempotent — running a step twice with the same input must be safe, because a crash can happen after the step but before the checkpoint is written.
  5. Delete the checkpoint when the final result has been returned.
def on_data(self, task):
    bucket, key = "agent-checkpoints", f"{task.task_id}.json"
    ckpt = self.store.get_json(bucket, key) if self.store.exists(bucket, key) else {"done": {}}

    for step in ("collect", "analyse", "draft"):
        if step not in ckpt["done"]:
            ckpt["done"][step] = self.run_step(step, task.job_data, ckpt["done"])
            self.store.put_json(bucket, key, ckpt)  # checkpoint after each step

    self.store.delete_object(bucket, key)
    return AgentResult(task_id=task.task_id, job_output=ckpt["done"]["draft"])

Recovery

Failure Recovery strategy
Instance crash mid-task The task is redelivered when the instance restarts. With checkpoints, it resumes from the last completed step instead of starting over
Task fails on every attempt After REDIS_MAX_DELIVERIES attempts it is moved to jobs:inbox:dead. Inspect it with redis-cli LRANGE jobs:inbox:dead 0 -1 and requeue it with LMOVE jobs:inbox:dead jobs:inbox RIGHT LEFT once fixed
Step fails repeatedly In AgentWorkflowEngine, modules retry up to max_retries, then the engine replans around the failure (up to max_replans times) with the completed and failed nodes as context. See Prebuilt Operators
Waiting on a human across restarts Store the HIS request ID in the checkpoint and resume with check_and_get_response(id)
Long external jobs Use job-based workflow policies, which submit to an external executor and poll for completion. See Workflow System
Results must not be lost Have results verified or acknowledged — the Verification System persists every verification