Microsoft Open-Sources PostgreSQL Extension for In-Database Durable Execution

Recently open-sourced by Microsoft, pg_durable is a PostgreSQL extension that enables durable workflows to run natively inside the database, eliminating the need for external orchestration systems.

According to Microsoft, pg_durable lets developers avoid stitching together cron jobs, background workers, message queues, and external orchestrators to implement long-running, fault-tolerant SQL functions. Instead, workflows are expressed directly in SQL, with the extension handling execution concerns such as retries, fan-out, and recovery.

With pg_durable, workflows are defined directly in SQL, with retry state, progress tracking, and checkpointing managed entirely within PostgreSQL rather than in external applications. A significant advantage of this, according to Microsoft, is that "some app-tier workers, queue consumers, or scheduler glue can disappear entirely".

A pg_durable function is a graph of SQL steps that PostgreSQL executes and checkpoints as it goes. If the database crashes, restarts, or a step fails, execution resumes from the last durable checkpoint instead of making you reconstruct state by hand.

👁 Image

The extension persists function execution state in PostgreSQL tables, allowing workflows to survive crashes, restarts, and failovers. It exposes a DSL with first-class primitives for scheduling, conditions, and parallel execution.

This is a short example of a durable function that processes data in discrete steps:

SELECT df.start(
 'SELECT id FROM documents WHERE processed = false LIMIT 100' |=> 'batch'
 ~> 'UPDATE documents SET processed = true WHERE id = ANY($batch)'
);

The snippet above shows ~> and |=>, two special operators defined by pg_durable that enable sequential execution of nodes and binding of results to variables, respectively. The df.start function starts the execution of a durable function. The next example shows how to execute two nodes in parallel and wait for their completion using the df.join function, or its equivalent & operator:

df.join('SELECT count(*) FROM a', 'SELECT count(*) FROM b')\
'SELECT 1' & 'SELECT 2'

Microsoft positions the extension for vector embedding pipelines, where data must be chunked, sent to an embedding API, and then upserted into pgvector,as well as for scheduled maintenance tasks, such as detecting bloat, triggering notifications, awaiting approval, and executing follow-up actions. Furthermore, it is suitable for workflows that depend on external APIs.

Architecturally, pg_durable is intentionally minimal, as it includes only a Postgres extension plus a background worker with no external control plane. The worker responsible for running durable functions is built on two Rust libraries: duroxide, which provides the orchestration runtime, including deterministic replay, checkpoints, sub-orchestrations, and timers; and duroxide-pg, which persists instances, history, work queues, and other runtime state in a dedicated schema owned by duroxide.

Durable execution is a software paradigm that enables long-running workflows to automatically resume from failure points, eliminating complex manual state recovery. This model simplifies distributed systems, facilitating robust agent architectures and cloud control planes. InfoQ has previously explored durable execution as supported by platforms like Temporal and Cloudflare.

Related Topics:

• Development
• AI, ML & Data Engineering
• Postgres
• Microsoft
• Open Source

We protect your privacy.