Transaction Boundaries in CDC
Transaction Boundaries in CDC
A customer places an order. Your application inserts a row into the orders table, two rows into order_items, and updates the inventory table for each item. All five writes happen inside a single database transaction. They either all commit or none of them do.
Your CDC pipeline reads these changes from the WAL and delivers them to a downstream system. If it delivers the events one at a time as they arrive, the downstream system briefly sees an order with no line items, then an order with one line item, then two, and so on. For a few hundred milliseconds, every query against the downstream copy returns wrong data.
This is the transaction boundary problem. The source database guarantees atomicity. The CDC pipeline, if it delivers events individually, does not.
What Are Transaction Boundaries?
A transaction boundary is the BEGIN and COMMIT that wrap a set of database operations. Everything between BEGIN and COMMIT is one atomic unit: it succeeds together or fails together.
When decoded via the logical replication protocol, a transaction appears as a sequence of messages:
BEGIN (transaction ID: 4817)
INSERT orders {id: 1001, customer: 42, total: 89.99}
INSERT order_items {order_id: 1001, product: "Widget A", qty: 2}
INSERT order_items {order_id: 1001, product: "Widget B", qty: 1}
UPDATE inventory {product: "Widget A", stock: 98} (was 100)
UPDATE inventory {product: "Widget B", stock: 49} (was 50)
COMMIT (transaction ID: 4817)
A CDC pipeline reading the WAL sees these records in order. The question is whether it delivers them to the sink as a group or one at a time.
What Breaks Without Transaction Grouping
When a pipeline delivers events individually, downstream systems see intermediate states that never existed in the source database.
Partial Reads
A downstream analytics system queries the replicated orders and order_items tables. The pipeline has delivered the orders INSERT but not yet the order_items INSERTs. The query returns an order with a total of $89.99 and zero line items. This state never existed in the source.
Referential Inconsistency
A downstream application joins orders and order_items on order_id. If the order_items rows arrive before the orders row, the join fails. The application either errors or silently produces incorrect results.
Double-Counting
An event-driven system triggers on each UPDATE to inventory. It sees two separate stock decrements and fires two separate alerts. If the system had seen both decrements as part of a single transaction, it could have fired one consolidated alert.
The Window of Inconsistency
The duration of inconsistency depends on how quickly the pipeline delivers subsequent events. Under low load, the gap between the first and last event in a transaction might be milliseconds. Under high load (large batches, slow sinks), it can stretch to seconds.
For many use cases, this window is acceptable. For others, it is not. The decision depends on what the downstream system does with the data and whether it can tolerate seeing intermediate states.
How Transaction Grouping Works
A pipeline that respects transaction boundaries buffers events between BEGIN and COMMIT, then delivers the complete set as a single batch.
The process:
- The pipeline sees a BEGIN event. It creates a buffer for this transaction.
- Subsequent events (INSERTs, UPDATEs, DELETEs) with the same transaction ID are added to the buffer instead of being delivered immediately.
- When the pipeline sees a COMMIT, it takes all buffered events and delivers them as a single batch to the sink.
- Rolled-back transactions are never delivered. In standard logical decoding (
pgoutput), aborted transactions are simply never emitted by the server. In streaming mode (protocol version 2, used for large in-progress transactions), the server can stream partial data before COMMIT and then send a Stream Abort if the transaction rolls back; the pipeline discards the buffer.
The sink receives all events from the transaction in one write call. If the sink is a database, it can apply them within its own transaction. If the sink is Kafka, they arrive as a contiguous group of messages.
Ordering
Within a transaction, events are delivered in WAL order: the order in which the database wrote them. Across transactions, events are delivered in commit order: if transaction A committed before transaction B, all of A's events are delivered before any of B's.
This matches the order in which changes became visible in the source database.
The Memory Problem: Large Transactions
Transaction grouping requires holding all events in memory until COMMIT arrives. For a transaction that inserts 10 rows, this is trivial. For a transaction that updates 5 million rows during a batch migration, it is a problem.
If the pipeline buffers all 5 million events in memory, it may exhaust available RAM and be killed by the operating system. This is why many CDC pipelines do not group by transaction: the memory requirement is unbounded, determined entirely by the source database's transaction size.
Spill to Disk
The standard solution is to set a threshold. Events are buffered in memory up to a configurable limit. When the limit is exceeded, accumulated events are written to a temporary file on disk. The pipeline continues appending to the file until COMMIT arrives.
On COMMIT, events are streamed from the file into batches and delivered to sinks. The pipeline never holds the entire transaction in memory at once, only the current batch being written.
The trade-off is disk I/O. Serializing events to disk and reading them back adds latency to large transactions. For normal-sized transactions (tens to hundreds of events), the in-memory path is used and the overhead is zero.
Not Truly Atomic for Large Transactions
When a transaction exceeds the spill threshold and is split across multiple batches, the sink receives multiple write calls for a single source transaction. The events arrive in order and completely, but they are not delivered in a single atomic operation.
This means a downstream system reading between batch writes can still see partial transaction state for very large transactions. The window is shorter than with per-event delivery (it is bounded by the number of batches, not the number of events), but it exists.
If strict atomicity is required for very large transactions, the sink itself must provide a mechanism for grouping writes (for example, a database sink can wrap all batch writes from the same source transaction in a single destination transaction).
What CDC Tools Do Today
Transaction boundary handling varies across CDC tools.
Debezium provides transaction metadata via the provide.transaction.metadata option. When enabled, Debezium emits BEGIN and END events to a dedicated transaction topic that logs transaction IDs, event counts, and the list of affected topics. DML events still go to individual table topics as separate messages. The consumer is responsible for using the transaction metadata to reassemble grouped delivery. Debezium does not buffer and deliver transactions as atomic batches.
Kafka Streams can group events by transaction at the consumer level, reading from Kafka topics and reassembling transactions using the metadata Debezium provides. This pushes the grouping responsibility downstream.
Flink CDC reads directly from databases via an embedded Debezium engine (not from Kafka topics). It has access to transaction boundaries at the source level, and Flink's changelog stream model can represent grouped transactions internally. However, whether the final sink write is atomic depends on the sink connector implementation.
Most polling-based and trigger-based CDC does not preserve transaction boundaries. Timestamp polling queries individual tables on a schedule. Trigger-based capture writes to a changelog table row by row. Neither mechanism has access to transaction boundary information.
Log-based CDC tools that read the WAL have access to BEGIN/COMMIT markers and can implement grouping at the pipeline level. Whether they do depends on the tool's design priorities. Grouping adds memory overhead and implementation complexity. Streaming row-by-row is simpler and works for many use cases.
| Approach | Grouping layer | Atomic delivery? | Large transaction handling |
|---|---|---|---|
| Polling / triggers | None | No | N/A |
| Debezium (default) | Consumer (via metadata) | Consumer's responsibility | Consumer's responsibility |
| Pipeline-level grouping | Pipeline | Yes (in-memory transactions) | Spill to disk, multi-batch |
When Transaction Grouping Is Not Needed
Transaction boundary preservation is not always necessary. Several common use cases work fine without it.
Append-only analytics. If the downstream system aggregates events over time windows (per-hour counts, daily sums), the order and grouping of individual events within a window does not affect the final result.
Idempotent sinks with eventual consistency. If the sink uses upserts (INSERT ... ON CONFLICT DO UPDATE), every event eventually produces the correct final state regardless of delivery order. Intermediate states exist briefly, but the final state is consistent.
Single-table replication. If a CDC pipeline replicates one table, every transaction touching that table produces exactly one event (or a set of events all targeting the same table). There is no cross-table inconsistency to worry about.
Search index sync. Elasticsearch, Solr, and similar search engines are eventually consistent by design. A brief delay between related document updates is expected and does not affect search quality.
Disabling transaction grouping reduces memory usage and simplifies the pipeline. Events are delivered as they arrive without buffering, which lowers latency for individual events at the cost of not guaranteeing cross-table consistency.
Where to Go from Here
This article covers what transaction boundaries are, why they matter for CDC, and the trade-offs involved in preserving them. The related articles go deeper into connected topics:
- CDC Delivery Guarantees: at-least-once and exactly-once semantics, and how they interact with transaction grouping
- Checkpointing and Crash Recovery: how a pipeline resumes after a crash without re-delivering committed transactions
- Backpressure in Data Pipelines: how large transactions interact with buffering and memory pressure
- What is Change Data Capture: the broader context of how log-based CDC captures changes
For the broader picture of how transaction boundaries fit into transaction log processing, see What is Transaction Log Processing.
We built Remac with transaction boundary preservation enabled by default. Events are buffered in memory between BEGIN and COMMIT and delivered as a single batch on commit. Rolled-back transactions are discarded and never reach sinks. For transactions that exceed a configurable event threshold, events spill to a temporary file on disk to prevent unbounded memory growth, then stream from disk into batches on commit. Orphaned transactions (a BEGIN without a matching COMMIT within a configurable timeout) are evicted automatically. Remac currently supports PostgreSQL, with MySQL, MongoDB, SQL Server, and Oracle on the roadmap.
Take a look at remac.io.
Further Reading:
- PostgreSQL Logical Replication: Message Formats. Official documentation on BEGIN, COMMIT, INSERT, UPDATE, DELETE message structure in the logical replication protocol.
- Debezium: Transaction Metadata. How Debezium exposes transaction boundary information via a dedicated topic.
- Designing Data-Intensive Applications, Chapter 7: Transactions. Martin Kleppmann's analysis of transaction isolation, atomicity, and their implications for data systems.