TL;DR - Durable execution helps an AI agent recover its progress after a crash. It can avoid repeating expensive model calls, research, or work a human already approved. But that does not mean the agent’s actions happen exactly once. An external system may successfully create a ticket, send a message, or update a record just before the worker crashes. If the workflow never records that success, recovery may run the action again. The agent has gone back. The outside world has not. AI agents make this harder because a restarted model may generate a slightly different tool request instead of replaying the original one.
For production agents, builders need to design these as two separate problems: recovering computation and recovering external side effects. Give every important action a stable identity before the model generates the request, understand what duplicate protection the receiving system actually provides, persist the exact artifacts later decisions depend on, and crash-test the integration at the point where the external action succeeds but your workflow has not recorded it yet. That boundary is where durable execution stops and distributed-systems design begins.
A bank runs an AI agent to review new business partners. The agent gathers public information, reads documents, writes an assessment, waits for an analyst to approve it, and then creates a review ticket in the bank’s case-management system.
The example is made up. The architecture is not unusual though.
Now imagine the worker running that agent dies during a deployment. A new worker comes up and resumes the workflow.
For a short question-answering task, restarting may be cheap. Here it is not. The agent may repeat paid searches and model calls. It may regenerate an assessment that differs from the one the analyst approved. It may also create a second ticket for work that already exists.
Those are two different recovery problems.
The first is preserving completed work. The agent should not redo everything it already finished.
The second is safely resolving actions whose outcome is uncertain. The agent may have changed another system just before it crashed, without recording that the action succeeded.
Durable execution is very good at the first problem. The second problem is where things get really interesting.
What durable execution actually promises
Durable execution saves enough state so that a crashed program can continue from where it stopped. That is enormously useful for long-running agents, but the guarantee is narrower.
DBOS saves workflow inputs and the output of each step in Postgres. When a workflow is interrupted, DBOS finds the interrupted workflow, starts it again with the same saved inputs, and checks each step for an existing result. Completed steps are skipped. The first step without a saved result is treated as the point where execution stopped, so that step runs again.
The important guarantee is in the wording. Steps are attempted at least once, and a completed step is never run again. Database transactions managed by the runtime can commit exactly once.
The phrase at least once is what matters for calls to systems outside the workflow engine.
Suppose a step creates a ticket in a customer’s case system. The case system saves the ticket successfully. A few milliseconds later, the worker crashes before the workflow records that the step completed. When recovery begins, there is no saved result for that step. The runtime has no proof that the ticket already exists, so it runs the step again.
Temporal documents the same boundary from a different angle. A worker can successfully perform an activity and then crash before reporting completion. The server’s history never records the success, so the activity is scheduled again.
Temporal makes a useful distinction here. The activity is reported as completed exactly once, but the activity code itself may execute more than once. It may even partially succeed multiple times.
That difference sounds small until the activity changes a customer system. Your workflow runtime counts recorded completion. The customer’s system counts actual executions.
No workflow engine can make someone else’s API safe to call twice. The runtime can preserve your state, replay your code, and give you stable identifiers to work with. The receiving system still decides what a repeated request means.
This also explains why deterministic workflow code matters. Anything unpredictable, including API calls, random values, or reading the current time, belongs inside a step. The workflow itself needs to replay consistently.
Unhandled failures matter too. In DBOS, an error that escapes the workflow is recorded and the workflow moves into ERROR. It is not automatically recovered. Calls that are expected to fail temporarily should therefore live inside steps with intentional retry behavior.
Retry configuration creates its own failure mode. The Pydantic AI integration notes that DBOS does not classify model errors as final in the same way some engines do. A poorly configured agent can therefore consume its full retry budget before stopping.
Durability gives us a powerful recovery model. It does not remove the need to reason about what happened outside that model.
The Acknowledgment Gap
There is a small but important period between an outside system accepting an action and your workflow durably recording that the action succeeded.
I call this The Acknowledgment Gap.
In the bank example, the ticket already exists in the case system, but the agent has not recorded that fact yet. The outside world has moved forward. The agent’s durable state has not caught up.
That gap exists in every long-running agent that performs real actions.
Saving progress more frequently can make the window smaller, but it cannot make the window disappear. The outside action and your local acknowledgment are two different operations. Something can always fail between them.
That is why checkpoint frequency is only part of the design.
LangGraph makes the tradeoff visible through its durability modes. With exit, progress is written only when the graph finishes, so intermediate progress is not durable. With async, persistence happens while later work continues, leaving a small window where the save may not land. With sync, the save completes before execution proceeds, which reduces that window at the cost of latency.
The strongest mode makes the window smaller. It still does not make an external write exactly once.
The price of finer-grained durability is measurable. DBOS performs one database write for every step, plus two writes per workflow for the workflow inputs and final result. DBOS reports more than 40,000 workflows or steps per second on a single Postgres database, but the practical cost also depends on how much data each step returns. That is why the same documentation recommends returning a pointer to large objects rather than writing files directly into workflow state.
This is a useful engineering tradeoff because we can reason about it directly. More save points mean less work is lost after a crash. They also mean more persistence overhead.
What they do not provide is certainty about an external action whose success has not yet been recorded.
Why AI agents make ordinary idempotency harder
The traditional solution to repeated requests is idempotency.
An idempotency key is a stable identifier attached to an operation. When the receiving service sees the same identifier again, it knows that the caller is retrying earlier work rather than requesting a new action.
Stripe is a good example. Stripe stores the first result associated with an idempotency key and can return that result when the request is retried.
But there are two important details.
The first is retention. Stripe may remove keys after 24 hours. If a workflow waits three days for human approval and then resumes, a 24-hour duplicate-protection window may no longer help.
The second is payload consistency. Stripe compares a retried request with the original request and rejects the reuse of the same key when the parameters differ.
This is where AI agents introduce a new complication.
A traditional application often retries the same serialized request. A restarted language-model agent may generate the request again. The intent may be identical while the generated arguments differ slightly.
Independent research has started to measure this problem. The ACRFence paper reviewed twelve major agent frameworks and found none that enforced exactly-once behavior at the tool boundary. The paper discusses failures including duplicate payments and credential reuse.
The underlying idea is simple: the agent can go back to an earlier execution state, while the outside world cannot.
That changes how action identity should be designed.
The identifier for a business action should be assigned by application code before the model generates the request. For the bank example, the ticket operation could be identified as:
review-123:create-ticket
That value comes from stable workflow state: the review ID and the operation being performed. It does not come from the model’s output, and it does not change because the process restarted.
The exact request sent under that action identity should also be stored. If recovery retries the action, the system reuses the same identity and the same committed payload instead of asking the model to invent the request again.
DBOS provides a related mechanism at the workflow boundary. A workflow ID can prevent the same workflow from being started twice. That is useful, but it does not automatically make every API call inside the workflow idempotent.
The mistake to avoid is generating a fresh identifier during recovery. If a tool creates a new UUID every time it executes, or if the identifier comes from the model, then every retry looks like a completely new action.
When the receiving API supports idempotency, the retention period becomes part of the architecture. When it does not support idempotency, the workflow needs another way to determine what happened. Sometimes that means querying by a stable business ID. Sometimes it means reconciling against an external record. In the worst case, it means stopping the workflow and asking a human to resolve the uncertain action before the agent writes again.
Where save points belong
Save points determine how much work a failure can destroy or repeat, so they should be designed rather than added randomly.
Temporal’s guidance makes the tradeoff easy to see. If three pieces of work live inside a single activity, a failure near the end can cause all three to run again. If those pieces become three activities, only the unfinished one has to execute during recovery. The tradeoff is a longer event history.
For agentic workflows, model output often deserves its own durable boundary.
Imagine an analyst approves an AI-generated assessment. If the workflow crashes and the assessment is regenerated, the new output may not match what the analyst actually reviewed. The system still has an approval record, but that approval now refers to different content.
The safe approach is to persist the exact artifact that later decisions depend on.
That usually means putting durable boundaries around expensive work, decision-bearing work, and external side effects. Paid research that would be costly to reproduce belongs there. The model output that a human approved belongs there. An operation that changes a customer system belongs there.
Framework integrations can still provide a useful default. Wrapping an agent for durable execution can turn the run loop into a workflow and model calls into durable steps. That is reasonable, but it is still an architectural choice. Teams should know where those boundaries are rather than inheriting them without looking.
The message queue underneath the agent deserves the same scrutiny.
A queue gives you reliable delivery of work. It does not guarantee that the work itself happens once.
SQS standard queues can deliver the same message more than once, and AWS explicitly tells applications to handle repeated processing safely.
FIFO queues reduce some duplicate deliveries, but their deduplication window is five minutes. That may handle a short network retry. It does not protect a long-running workflow that resumes an hour later.
Human approval is durable state too
Long-running agents often include people in the execution path. That creates another kind of recovery problem.
Imagine an analyst approves an assessment and the workflow waits several hours before taking action. During that time, the source evidence changes.
The workflow still has an approval. But the world that approval referred to has changed.
A production system therefore needs to persist more than a boolean such as approved=true. It should preserve the version of the assessment that was reviewed, the action the reviewer authorized, the identity of the reviewer, and the policy that determined why approval was required.
DBOS’s examples show the same pattern in simpler form. A refund can be sent to manual approval above a configured threshold.
The waiting mechanism itself must also survive failure. DBOS stores human-in-the-loop waits durably, so the deadline and incoming message survive restarts. Temporal provides similar behavior using signals and durable timers, where waiting survives disruptions.
One useful operational pattern is an inbox of agents waiting on humans. That turns a vague support problem into something observable. Operations teams can see which workflows are blocked, how long they have been waiting, and what action is required.
Code deployment creates a related form of state drift.
A workflow started yesterday may have persisted results from one sequence of steps. Today’s code may expect a different sequence. If the runtime cannot reconcile the two, recovery can fail even though all the data is still there.
DBOS supports patching and versioning for this reason.
For long-running agents, application upgrades are part of the recovery design.
What this looks like in production
The closest public example is a DBOS case study about Yutori. DBOS describes Yutori’s Scouts as always-on agents that monitor the web on a schedule. Their workflows are generated from model output, and subtasks can run across many processes.
The stated product requirement maps directly to this problem: a user should be notified exactly once.
Missing a notification is a failure. Sending it twice is also a failure.
At scale, host failure becomes another part of the design. DBOS recommends connecting production applications to its control plane. A dead server can be detected through a closed websocket, after which its workflows can be reassigned to a healthy server.
The control plane does not execute the workflows itself. It coordinates recovery.
The broader lesson matters more than any individual product.
Scheduling a run, preserving progress inside the run, and safely committing a business action are three different mechanisms.
A platform can solve the first two extremely well and still leave the third to your application architecture.
Where this architecture is unnecessary
Not every agent needs durable execution.
A short, read-only task with cheap inputs may only need a timeout and a retry policy. If the entire task costs a few cents to rerun and produces no external side effects, adding a workflow engine may create more operational complexity than value.
The framework already in the stack may also provide enough persistence. If an agent is built as a graph with a database-backed checkpointer, progress recovery may already be solved at the level the workload needs.
Adding another workflow system underneath becomes a migration rather than a requirement.
That migration is also getting easier. Durable runtimes now ship as drop-in replacements for parts of agent SDKs, including the OpenAI Agents SDK runner.
The important distinction is that this argument changes as soon as the agent writes to an external system.
A single duplicated payment, support case, account change, approval, notification, or infrastructure action can matter even when the agent handles only ten requests per day.
External side effects make recovery an application-design problem, not simply a runtime feature.
Production failure modes
The quiet duplicate
The most dangerous duplicate is often the one the workflow runtime believes was a successful recovery.
The agent creates a record in the outside system. The worker crashes before storing the result. Recovery runs the unfinished step again and creates another record. From the runtime’s point of view, everything worked: the workflow recovered and finished.
The customer sees two tickets.
This is why recovery status alone cannot tell you whether the business action was correct.
The rewritten request
A restarted agent may regenerate a tool call rather than replay the exact request it sent before the crash.
The meaning can stay the same while the body changes slightly. The model may choose different wording, reorder fields, select a different optional argument, or generate a fresh identifier.
A receiving service can interpret that as a new action.
This is especially easy to miss in development because healthy runs never exercise the recovery path. The problem appears only after a crash, which is exactly when teams are least likely to be looking at model-generated argument differences.
The stale approval
A human approves an AI-generated assessment and the workflow waits.
Hours later, the evidence used to generate the assessment changes. The workflow still has a valid approval record, but the approval applies to an older version of the evidence or output.
If the system does not bind the approval to a specific version, the agent may act on authority that nobody actually granted for the current state.
For regulated or high-impact workflows, this becomes an audit problem rather than simply an execution bug.
The dead workflow
A step throws an error the system treats as permanent. The workflow moves into a terminal error state and stops recovering.
That behavior may be correct.
The operational problem begins when nobody owns those failures. A workflow can remain dead for days while dashboards continue to show plenty of healthy traffic. The first signal may be a customer asking why an operation never finished.
Terminal failure is not only a runtime state. It needs an operational owner and an escalation path.
The stranded workflow after a deployment
A long-running workflow can survive for days or weeks. During that time, the application code changes.
A deployment may add a step, remove one, or change their order. Workflows that started under the previous version still contain persisted state from the old execution path.
Without a versioning strategy, those workflows may no longer be able to continue.
This is why deployment compatibility belongs in the architecture for long-running agents. It is not something to discover after a release strands live customer work.
Operational metrics that tell you whether recovery is actually working
Traditional agent metrics such as latency, token usage, and success rate are not enough for long-running systems that perform side effects.
The most useful recovery metrics describe what happened after something went wrong.
Duplicate external effects after recovery
Track how many business actions are duplicated after an injected or real crash.
Read this metric from the receiving system whenever possible. If the agent creates tickets, count tickets in the ticket system. If it sends notifications, count deliveries in the notification system. If it triggers payments, use the payment provider’s records.
The workflow runtime may report that a step completed once even though the outside system observed two executions.
A non-zero value is not a generic reliability problem. It points to a specific integration where action identity or duplicate protection is missing.
That makes the metric directly actionable.
Completion rate for recovered workflows
Track recovered workflows separately from workflows that succeed on the first attempt.
A single blended success rate can hide recovery bugs because first attempts usually dominate the volume.
If 99.9% of all workflows finish but recovered workflows succeed only 80% of the time, the production system has a recovery problem that the headline metric is hiding.
When this number drops, inspect replay-sensitive steps first. Look for fresh UUID generation, reading the current time, model-regenerated tool arguments, non-durable local state, and APIs called without stable action identity.
Age of workflows in terminal error
Count terminal failures, but also measure how long they have remained unresolved.
The age is often more useful than the total.
Ten failures from the last five minutes may be part of a short incident. One workflow that has been dead for seven days may represent a customer request that quietly disappeared from the system.
This metric should connect directly to operations. Old terminal workflows need ownership, investigation, and either recovery or explicit closure.
External actions with no known outcome
This is the operational view of the Acknowledgment Gap.
Track actions that were sent to another system but do not yet have a durable recorded result.
Do not treat them only as a single aggregate number. Keep the stable action identity with each one so the operation can be reconciled against the receiving system.
A slowly increasing backlog means the system is accumulating uncertain side effects faster than it can resolve them.
That is a warning that the recovery design and the real behavior of the integration no longer match.
Durability overhead per workflow
Measure how much persistence the recovery design adds.
Track step count, save-point writes, and the size of persisted outputs.
This gives teams evidence when deciding whether another save point is worth adding. If a step returns a 30 MB object when it could store a blob reference, that will be obvious in the data. If a workflow saves hundreds of tiny intermediate values that are cheap to recompute, the design can be simplified.
Durability should be treated as an engineering budget rather than as a binary setting.
Lessons from the field
The same misunderstanding appears repeatedly in system-design conversations.
A customer hears recovery and assumes the business operation will finish correctly. A runtime vendor may use recovery to mean the program can resume from durable state.
Both statements can be true, but they describe different guarantees.
AI engineers and AI FDEs should separate those guarantees early in a design review. The discussion becomes much clearer once the team distinguishes preserving computation from safely settling side effects.
The second recurring gap is testing.
Teams often prove recovery by killing a process, restarting it, and watching the workflow reach COMPLETED.
That validates durable progress.
It does not prove that the customer’s systems contain the right number of tickets, notifications, payments, updates, or records.
The receiving system has to be part of the failure test.
What builders should do next
For an AI engineer, this work starts in the system-design review, before implementation details disappear behind an agent framework.
Draw the agent workflow and mark every place where execution crosses into a system the workflow runtime does not control. Model calls matter, but the more important boundaries are actions that change durable business state: creating a case, sending a message, updating a CRM, approving access, modifying infrastructure, charging a card, or writing into a customer’s database.
For each of those boundaries, assign the business action an identity that comes from application state rather than model output. A case ID plus an operation name is often enough. That identity should exist before the model writes the tool arguments, and it should survive process restarts, model changes, and retries.
Then look at the actual client code for the integration. Find where request IDs are created. Find where tool arguments are generated. Find whether a helper uses uuid4() each time the function runs. Find whether the model is allowed to generate the identifier itself. These are small implementation details that determine whether recovery is safe.
The next place to look is the receiving API. Read its documentation for idempotency behavior, duplicate handling, lookup capabilities, and retention windows. Do not stop at “supports idempotency.” A key remembered for five minutes solves a different problem from one remembered for seven days. A service that rejects changed payloads behaves differently from one that silently treats them as new actions.
Bring those findings back into the design review. The discussion should connect the maximum lifetime of the workflow, the longest human wait, the runtime’s retry policy, and the receiving system’s duplicate-protection window. This is where application architecture and vendor guarantees become one design instead of two separate documents.
During implementation, persist the outputs that future decisions depend on. The model response a person approved should be stored as the exact artifact they reviewed. Expensive research should not be regenerated unnecessarily. External actions should be associated with their stable action identities and recorded payloads.
Then build one deliberate crash into the development environment.
Do not kill the worker at a random point. Kill it at the dangerous point: immediately after the external system commits the action and immediately before the workflow records success.
Restart the worker and inspect the external system.
That test should become part of the way the team validates every high-risk integration. When the framework is upgraded, when checkpoint behavior changes, when retry settings change, or when a new side-effecting tool is added, run the crash again.
Keep the result visible in the repository. A simple side-effect table is enough. Record each external action the agent can perform, what identifies that action, whether the receiving service honors the identity, how long it remembers it, and what happens if the action executes twice.
This artifact is particularly useful for AI FDEs because it turns an abstract reliability discussion into something the customer can inspect. It also exposes integration gaps before they become production incidents.
There is a deeper skill underneath all of this that AI engineers should deliberately build.
Learn to read runtime guarantees as distributed-systems guarantees.
When a framework says “durable,” understand what is persisted. When it says “exactly once,” understand whether that refers to scheduling, recording completion, database transactions, or the external effect itself. When it says “retry,” understand what state is replayed and what state is regenerated.
The classical distributed-systems concept behind this is the output commit problem. Once a process changes the outside world, rolling the process back does not roll the outside world back with it.
Temporal’s activity semantics and DBOS’s workflow guarantees are good practical places to study this boundary. The ACRFence work adds the agent-specific twist: language models can generate a different request after recovery, so techniques built around replaying an identical payload are no longer enough on their own.
Keep learning this through real failure injection rather than only through documentation. Every production agent you build gives you another chance to identify one external side effect, understand its semantics, crash the system around it, and observe what survives.
That is how durable execution becomes more than a framework feature.
It becomes part of how you design production AI systems.
