Often, durable workflows need to wait for something to happen in the real world: for a reviewer to approve an action, or for a customer to submit a form. These workflows can wait for a long time, sometimes days, weeks, or even months. As a result, it’s not uncommon for even a small application to have thousands of workflows waiting at the same time, and for a large application to have millions.
To make it easier to manage large numbers of long-waiting workflows, we recently released workflow suspension. This allows a workflow that’s sleeping or waiting for a notification to “suspend” itself until a deadline is reached or the notification arrives. A suspended workflow exists only as rows in your database; it doesn’t consume any resources like threads, memory, or queue slots. In this blog post, we’ll explain how workflow suspension works.
How DBOS Suspension Works
Using workflow suspension is easy: you pass suspend=True to DBOS.recv (which waits with a
timeout for a notification) or DBOS.sleep (which sleeps for some length of time). For example, if you
need to suspend a workflow until it receives human approval (waiting up to a timeout, for example one week), you call
DBOS.recv on the “approval” topic with a timeout of one week and suspend=True:
Under the hood, workflow suspension works by leveraging workflow durability. When a workflow is suspended, it first updates its own database record: it sets its status to “SUSPENDED”, its “delay deadline” to its scheduled wakeup time (the time at which its timeout expires) and its “suspend key” to the topic on which it is waiting for a notification, if any. Then, the workflow exits execution by throwing a special “suspension exception.” After exiting, the workflow exists only as database entries; it does not occupy a thread or consume memory.
The most challenging part of implementing workflow suspend is figuring out when to resume suspended workflows. If there are millions of suspended workflows in the database at once, we can’t simply poll all of them to see which ones are ready to resume execution. Instead, we need to build two different mechanisms for efficiently identifying workflows that are ready to resume: one for workflows whose waits have timed out, and another for workflows whose notifications have arrived.
To find workflows whose waits have timed out, we use a special database index. Recall that when a workflow suspends itself, it sets its “delay deadline” in the database to its wakeup time. We build a database index on that column, which essentially sorts all workflows from those that will wake up the soonest to those that will wake up the latest. A background thread periodically polls that index, finding the next workflows to wake up and waking them up. After waking up a workflow, it clears that workflow’s delay deadline, keeping the index small so that queries for the next workflows to wake up are always fast.
To find workflows woken up by a notification, we include some special logic in DBOS.send, the call that
sends a notification. When sending a notification to a workflow, we also check if the workflow is suspended waiting
for that notification: if its suspend_key in the database matches the topic of the notification. If it
does, we set the workflow’s delay deadline to 0 (to avoid a race with this, workflows transactionally check if a
notification has arrived before suspending themselves). This causes the background thread described previously to
immediately wake the workflow up.
When a workflow is ready to wake up, we use its own durability to resume it by updating its status from SUSPENDED to ENQUEUED, re-enqueueing it for execution. The workflow then recovers from checkpoints and continues execution from where it was suspended, the same way it would recover from a crash, restart, or any other failure. This is the power of durable execution: a workflow’s state is completely described by its database records, so workflows can freely move between executing in memory and suspended in the database.
Learn More
If you like building scalable, reliable systems, we’d love to hear from you. At DBOS, our goal is to make Postgres-backed durable execution as simple and performant as possible. Check it out:
- Quickstart: https://docs.dbos.dev/quickstart
- GitHub: https://github.com/dbos-inc
- Discord community: https://discord.gg/eMUHrvbu67





