Guide

What Is Durable Execution?

How durable workflows recover from failures, what exactly-once execution guarantees, and the trade-offs between different architectures.

Ask ChatGPTAsk Claude

Durable execution definition

Durable execution is a way of running programs that preserves their progress across crashes, restarts, and other infrastructure failures. It records completed operations in persistent storage, allowing interrupted programs to recover without repeating previously checkpointed work.

Applications using durable execution organize their operations into durable workflows. Many frameworks follow a workflow-as-code model, where developers define workflows using ordinary code with branches, loops, and function calls. Others use declarative models such as DAGs or state machines. Within a workflow, individual operations, often called steps, may invoke APIs, call LLMs, or update databases. The runtime persists their outcomes or equivalent execution history to support recovery.

Progress is usually recorded at explicit operation boundaries, not after every line of code. Durable execution provides recovery from infrastructure failures, but it does not eliminate application errors or automatically guarantee exactly-once external side effects.

Why durable execution matters

Throughout this guide, we’ll use a checkout service as a running example. When a customer places an order, the service notifies a reviewer and waits up to seven days for approval. When the reviewer responds, or the seven days run out, the service records the outcome in the database: approved, rejected, or timed out. Only an approved order is charged.

  1. Customer places an order
  2. Notify reviewer
  3. Wait for approval
    Up to 7 days
  4. Record decision
    Approved, rejected, or timed out
  5. Approved
    Charge customer
    Rejected or timed out
    Finish without charging

Many other workflows have the same shape. An AI agent makes a series of LLM and tool calls, and a business process may wait days for human input. Any of them can be interrupted by a process crash, a deployment, a server restart, or a failure in an external service.

After a restart, the application may not know which operations completed. Starting over can notify the reviewer twice, repeat expensive LLM calls, or charge the customer twice. Abandoning the workflow can leave an order stuck halfway. Teams often handle these problems with status tables, outboxes, reconciliation jobs, and custom retry logic. These approaches work, but applying them consistently across an application is difficult.

Durable execution moves progress tracking and recovery into the runtime. Developers still control checkpoint boundaries, retry policies, how to handle permanent failures, and how to make external side effects safe to repeat.

How durable execution works

Durable execution systems differ in how they store execution state and recover from failures. A common approach, particularly in workflow-as-code frameworks, is checkpoint and replay:

  1. Persist the workflow. Store a workflow ID, its inputs, and the metadata needed for recovery.
  2. Checkpoint completed steps. Record each step’s result in durable storage before proceeding to the next operation.
  3. Detect interrupted workflows. After a failure, a worker or coordination service identifies unfinished workflows and initiates recovery.
  4. Replay and continue. Re-execute the workflow function from the beginning, returning saved results for completed steps instead of running them again. The workflow then continues from where it left off, at the first step without a checkpoint.

Deterministic replay

Deterministic replay is an important concept in durable execution because it avoids the overhead of checkpointing the entire program state or taking memory snapshots. Instead, the runtime reconstructs the workflow’s state by re-executing its code with previously recorded step results. As long as execution is deterministic, local variables and control flow can be reconstructed without repeating completed operations.

For replay to work, workflow execution must be deterministic. Given the same inputs and recorded results, the workflow must follow the same control flow and invoke the same steps in the same order. Otherwise, replay may diverge from the recorded execution history.

Operations such as reading the current time, generating random numbers, or querying mutable data can produce different results during replay. These operations should run inside checkpointed steps or use runtime-provided durable primitives, so their results can be reused consistently during recovery. Changes to workflow code can also affect determinism, which is why long-running workflows need versioning or other mechanisms to support code updates without breaking replay.

Exactly-once guarantees and limitations

Exactly-once execution is one of the most commonly misunderstood concepts in durable execution. Checkpointing makes recovery possible, but it does not automatically guarantee exactly-once side effects. Consider three points at which a process can crash:

The crash happens…What happens during recovery
Before the step executesNo side effect has occurred. Recovery executes the step.
After the result is checkpointedRecovery reuses the recorded result without executing the step again.
After the side effect, but before the checkpointThe outcome is unknown to the runtime. Recovery may execute the step again, potentially repeating the side effect.

The third case is the challenging one. Suppose a payment API successfully charges a card, but the process crashes before recording the response. The runtime cannot determine from its checkpoints whether the payment succeeded. Retrying the step could charge the customer twice.

This is why durable execution systems typically require steps to be idempotent. A step may execute more than once during recovery, but repeated executions should have the same effect as a single execution. Combined with at-least-once execution, idempotency can provide effectively exactly-once behavior.

There are several common techniques for achieving exactly-once effects:

  • Idempotency keys. Send a stable key with each external request. If the request is repeated, the service recognizes the key and returns the original result instead of performing the operation again. This depends on the service’s idempotency guarantees and how long it retains the key.
  • Atomic database transactions. When an application write and its workflow checkpoint are committed in the same database transaction, either both succeed or neither does. This provides exactly-once effects for database operations without requiring separate idempotency checks.
  • Deduplicated delivery. For external systems without idempotent APIs, a durable operation record or transactional outbox can track pending effects. The receiving system must also support deduplication to prevent repeated operations.

Workflow identity

Many durable execution systems support exactly-once workflow starts using unique workflow IDs. By deriving a workflow ID from an order or event ID, repeated requests start the workflow only once and return the existing execution instead. However, this does not prevent duplicate external side effects within individual steps, which still require the idempotency mechanisms described above.

The checkout example at the end of this guide shows these techniques in code.

Architectures for durable execution

Every durable execution system must persist execution state and recover interrupted workflows. Where these responsibilities live affects transaction guarantees, latency, infrastructure complexity, data ownership, and scalability. Two common architectures are:

External orchestration service

A separate orchestration service coordinates workflow execution and maintains execution history in its own storage. Application workers execute tasks and report their results to the orchestrator, which tracks progress and manages recovery.

Database-native durable execution

A library runs inside the application, storing workflow state directly in a database such as Postgres. Application processes execute workflows and recover interrupted executions from the database. Workflow state can share the application’s database or use a separate one.

Atomicity with application data

With external orchestration, workflow checkpoints and application data typically reside in separate stores. A database write may succeed before the orchestrator records its completion, creating an ambiguous outcome after a crash. Applications must handle this with idempotency, outboxes, or other coordination mechanisms.

In contrast, database-native durable execution can commit application writes and workflow checkpoints in the same database transaction, providing atomicity without additional coordination.

Network overhead and deployment

External orchestration adds a service boundary between application workers and workflow state, introducing network overhead for task scheduling and checkpointing. The orchestration service must also be self-hosted or consumed as a managed service.

Database-native durable execution writes checkpoints directly to a database, eliminating the need for a separate orchestration service. Both approaches incur database I/O overhead for persisting workflow state.

Recovery ownership and high availability

An external orchestrator detects failed workers and reassigns their work. The orchestrator and its storage must also be highly available.

With database-native durable execution, application processes can recover their workflows on restart, while other executors or a control plane can recover workflows from failed processes. Database high availability can rely on existing replication and failover mechanisms.

Scaling, data ownership, and cost

External orchestrators can scale workflow coordination independently of the application but require additional infrastructure, often including multiple services and data stores. Managed services reduce operational overhead but introduce an external dependency on the application’s critical path and typically require sending workflow data to a third-party service. Some platforms support encrypting sensitive data or storing payloads in application-controlled storage, though the orchestration dependency remains.

Database-native durable execution can reuse the application’s existing database, reducing infrastructure complexity and cost while keeping workflow data under the application’s control. However, workflow traffic competes with application queries for database resources, and growing execution history adds storage and maintenance overhead. A separate database or sharding can improve isolation and scalability at the cost of additional infrastructure.

How to choose a durable execution framework

Start with your application’s requirements and the problems you’re trying to solve. When evaluating durable execution frameworks, consider these questions:

  • What are you building? Are you running background jobs, processing events, coordinating services, or building AI agents? Do your workflows need to run for extended periods or wait for human input and external events?
  • How easily can you adopt it? Can you make existing code durable without major changes? Does the framework support your programming languages, databases, and deployment environment?
  • What happens when something fails? Can workflows recover from crashes, restarts, and infrastructure failures? What happens when an operation fails permanently?
  • Can you safely retry operations? How does the framework prevent duplicate payments, database writes, or API calls? What guarantees does it actually provide?
  • What infrastructure do you need? Can you use your existing infrastructure, or do you need additional services? What are the implications for deployment, availability, and data ownership?
  • How does it scale? Can it handle your expected workload? What are the performance implications of checkpointing, concurrency limits, queueing, and growing workflow history?
  • How do you operate it in production? Can you inspect failures, troubleshoot and recover workflows, and deploy code changes without disrupting running executions?
  • What does it cost? Consider infrastructure, managed service fees, storage, and operational overhead. How do costs change as your workload grows?

When should you not use durable execution?

Durable execution adds storage and I/O overhead to record progress, typically requiring at least one durable write per checkpointed step. That overhead isn’t always justified. You may not need durable execution for:

  • Simple, idempotent background tasks. If a job can safely restart from the beginning, a task queue with retries may be sufficient.
  • Short-lived operations. If an operation completes quickly and can be safely retried, restarting it after a failure may be simpler than checkpointing its progress.
  • High-volume, low-value operations. If occasional failures or repeated work are acceptable, the cost of checkpointing may outweigh the benefits.

Durable execution with DBOS

DBOS is an open-source, database-native durable execution library for Python, TypeScript, Go, Java (with Kotlin interoperability), and Rust. It runs workflows directly inside your application and stores execution state in Postgres, without a separate orchestration server.

By using Postgres for durable execution, DBOS can share your application’s existing database or use a separate one for isolation and scalability. When using DBOS datasources, application writes and workflow checkpoints can commit in the same transaction, providing exactly-once database effects.

  • Durable execution primitives: Workflows, steps, queues, scheduling, messaging, and timers.
  • Postgres-native state: Workflow history stored in Postgres and queryable with SQL.
  • AI framework integrations: OpenAI Agents SDK, Pydantic AI, Vercel AI SDK, and others.
  • Production management: DBOS Conductor provides workflow observability, management, and recovery across executors.
  • Horizontal scalability: A published benchmark demonstrated over one million workflows per second across 50 Postgres shards. Actual throughput depends on application logic, step I/O, and infrastructure.

Example: A durable workflow with DBOS

Here is the checkout workflow introduced earlier, implemented in Python with DBOS. It notifies a reviewer, waits for approval, records the decision in Postgres, and charges the customer if approved.

The code assumes an existing orders table and notification and payment services that support idempotency keys.

import os
from dbos import DBOS, SetWorkflowID, SQLAlchemyDatasource
from sqlalchemy import text

# Connect to the application's Postgres database.
orders_db = SQLAlchemyDatasource.create(os.environ["APP_DATABASE_URL"])

@DBOS.step(retries_allowed=True, max_attempts=3)
def notify_reviewer(order_id: str) -> None:
    reviewers.notify(order_id, idempotency_key=f"{order_id}:review")

@orders_db.transaction()
def record_decision(order_id: str, status: str) -> None:
    # Commit the order update and checkpoint atomically.
    orders_db.sql_session().execute(
        text("UPDATE orders SET status = :status WHERE id = :id"),
        {"status": status, "id": order_id},
    )

@DBOS.step(retries_allowed=True, max_attempts=5)
def charge_customer(order_id: str) -> str:
    return payments.charge(order_id, idempotency_key=f"{order_id}:charge")

@DBOS.workflow()
def checkout(order_id: str) -> str:
    notify_reviewer(order_id)

    # Wait up to seven days for human approval.
    decision = DBOS.recv("approval", timeout_seconds=7 * 86400)

    if decision is None:
        status = "timed_out"
    elif decision is True:
        status = "approved"
    else:
        status = "rejected"

    record_decision(order_id, status)
    if status != "approved":
        return status

    return charge_customer(order_id)

# Start one workflow per order, after DBOS.launch().
def start_checkout(order_id: str):
    with SetWorkflowID(f"checkout:{order_id}"):
        return DBOS.start_workflow(checkout, order_id)

# Called by an authenticated, authorized reviewer.
def review_order(order_id: str, approved: bool):
    DBOS.send(
        f"checkout:{order_id}",
        approved,
        topic="approval",
        idempotency_key=f"{order_id}:decision",
    )

This example illustrates several important properties of durable execution:

  • Durable workflows and steps. @DBOS.workflow() and @DBOS.step() mark functions as durable workflows and steps. DBOS persists workflow inputs and metadata, and checkpoints completed step results so they can be reused during recovery instead of re-executing the steps. Each step can also have its own retry policy.
  • Durable messaging. DBOS.recv() waits up to seven days for the reviewer’s decision. Messages and the receive deadline persist in Postgres, so the workflow can recover from crashes or restarts without restarting the wait period.
  • Atomic database transactions. record_decision() uses a DBOS datasource to commit the order update and its checkpoint in the same database transaction. If the process crashes, either both changes committed or neither did, preventing duplicate database effects during recovery.
  • Idempotent external calls. Payment and notification APIs cannot share a database transaction with their workflow checkpoints. Stable idempotency keys prevent duplicate charges or notifications when a step is retried, provided the external services honor those keys. The approval message also uses an idempotency key to deduplicate repeated submissions.
  • Exactly-once workflow starts. The workflow ID is derived from the order ID. Starting checkout multiple times for the same order returns the existing workflow rather than starting another one. The reviewer sends the decision to that same workflow ID.

What happens when something fails?

Failure scenarioRecovery behavior
The process crashes while waiting for approvalThe workflow recovers and resumes waiting with its original deadline. The completed notification step is not repeated.
The reviewer responds before the workflow starts waitingDBOS stores the approval message in Postgres. The workflow receives it when it reaches DBOS.recv().
The process crashes while recording the decisionThe database update and checkpoint commit atomically. Recovery reuses the committed result or retries the transaction without duplicating the database effect.
Payment succeeds, but the process crashes before checkpointingDBOS may execute the payment step again. The same idempotency key allows the payment provider to recognize the previous charge and return its result rather than charging twice.

See the DBOS Transact overview, architecture documentation, or quickstart.

FAQs

Durable execution FAQs

Durable execution is the capability to preserve a program’s progress across failures. A durable workflow is a program or sequence of operations that uses durable execution. Workflows can be defined using ordinary code or declarative models such as DAGs and state machines. The terms are closely related and often used interchangeably.

Not by itself. Durable execution guarantees that completed, checkpointed steps are not re-executed. But if a process crashes after a step’s side effect and before its checkpoint, recovery may run the step again, so steps execute at least once. Exactly-once effects also require idempotent steps, for example using idempotency keys, or committing database writes in the same transaction as the step’s checkpoint.

Read more about exactly-once guarantees and limitations

AI agents often make multiple LLM and tool calls that can be slow, expensive, or prone to failures. Durable execution records completed calls, allowing agents to recover from crashes without repeating previously checkpointed work. It also supports long-running agents that need to wait for human approval or external events. The exact recovery behavior depends on how the agent framework integrates with the durable execution system.

Yes. Because workflow progress is persisted, execution is not tied to a single process. With support for durable timers and messaging, workflows can wait for hours, days, or months and recover after restarts or deployments. Long-running workflows also require consideration of code updates, data retention, and resource consumption while waiting.

In replay-based systems, changing a workflow’s control flow or the order of its steps can cause recovery to diverge from the recorded execution history. Changes within steps generally don’t affect previously checkpointed results, but may affect steps that haven’t completed.

Common approaches include running multiple workflow versions simultaneously, using version markers to preserve existing execution paths, or using framework-provided mechanisms for workflow upgrades. The available options and compatibility guarantees vary by framework.

Durable execution adds storage and I/O overhead to record workflow progress. In checkpoint-based systems, this typically means at least one durable write per step, along with workflow metadata and execution history.

For workflows dominated by slow LLM or external API calls, checkpointing overhead is often small relative to execution time. It becomes more significant for high-frequency, low-latency operations. Developers can reduce overhead by adjusting checkpoint boundaries and storing large payloads externally, with references in the workflow history.

Add durable execution to your application.

Install the open-source DBOS library, connect to Postgres, and add durable execution to existing application code.