CDC Delivery Guarantees: At-Least-Once vs Exactly-Once
CDC Delivery Guarantees: At-Least-Once vs Exactly-Once
A row changes in your database. Your CDC pipeline reads the change, delivers it to Kafka, and then crashes before recording that it delivered it. When the pipeline restarts, does it know the change was already sent? Or does it send it again?
The answer depends on the delivery guarantee the pipeline implements. There are three options, and understanding them is necessary for building a CDC pipeline that behaves correctly under failure.
The Three Guarantees
Distributed systems theory defines three delivery semantics. Every CDC tool implements one of them as its worst-case guarantee.
At-most-once. Each event is delivered zero or one times. If the pipeline fails before confirming delivery, the event is lost. No duplicates, but possible data loss.
At-least-once. Each event is delivered one or more times. If the pipeline fails, it replays events from its last saved position. No data loss, but possible duplicates.
Exactly-once. Each event is delivered precisely one time. No data loss, no duplicates. Achieving this across independent systems is, in the general case, not possible (more on this below).
| Guarantee | Events lost? | Duplicates? | Requires |
|---|---|---|---|
| At-most-once | Possible | Never | Nothing special |
| At-least-once | Never | Possible | Checkpoint + replay |
| Exactly-once | Never | Never | Sink-specific transaction support |
At-Most-Once: Fire and Forget
At-most-once is the simplest mode. The pipeline reads a change, acknowledges it to the source database (allowing the WAL to advance past that position), then attempts delivery to the sink. If delivery fails, the change is gone. The source has already moved past that portion of the log.
This is acceptable for approximate metrics, development environments, or analytics where a few missing data points do not affect conclusions. It is not acceptable for financial data, audit logs, replication, or any system where a missing event creates a permanent inconsistency between the source and the destination.
In PostgreSQL terms, at-most-once means advancing the replication slot position before confirming the sink has received the data. If the pipeline crashes between the acknowledgment and delivery, those events are not recoverable.
At-Least-Once: The Industry Standard
At-least-once inverts the order. The pipeline delivers events to the sink first, then saves a checkpoint (the position in the WAL it has processed up to). If the pipeline crashes after delivery but before the checkpoint is saved, it replays from the last checkpoint on restart. The sink receives some events a second time.
This is the delivery guarantee used by every major CDC tool:
- Debezium: at-least-once as the worst-case guarantee. Under normal operation (no crashes), events are delivered exactly once. During recovery, duplicates are possible.
- AWS DMS: duplicates are possible during CDC, particularly for tables without primary keys.
- Airbyte: at-least-once with optional deduplication.
- Maxwell: at-least-once via Kafka producer semantics.
The reason is a constraint that applies to all of them: they deliver to heterogeneous sinks. A single pipeline might write to Kafka, S3, and a PostgreSQL database. True exactly-once delivery across all three would require a distributed transaction coordinating Kafka, S3, and PostgreSQL in a single atomic commit. No such protocol exists for arbitrary sink combinations.
How Duplicates Happen
Duplicates occur in one specific scenario: the pipeline delivers a batch of events to the sink, but crashes before saving the checkpoint.
On restart, the pipeline loads the last saved checkpoint and asks the source database to replay from that position. Events between the checkpoint and the crash point are delivered again.
The duplicate window is bounded by the checkpoint interval. With a 10-second checkpoint interval, at most 10 seconds of events can be re-delivered after a crash. In normal operation (no crashes), each event is delivered exactly once.
Why This Is Acceptable
Three properties make at-least-once duplicates manageable in practice:
- Duplicates are bounded. The re-delivery window is one checkpoint interval at most. Not the entire event history.
- Duplicates are detectable. Each event has a unique position (the WAL LSN in PostgreSQL, the GTID in MySQL). Consumers can track the highest position they have processed and skip anything at or below it.
- Most sinks can absorb duplicates naturally. PostgreSQL upserts (
INSERT ... ON CONFLICT DO UPDATE) with the same data produce the same result whether executed once or ten times. S3 object writes are idempotent. Kafka consumers can deduplicate by position.
Exactly-Once: Harder Than It Sounds
Exactly-once delivery between two independent systems is a well-studied problem. The Two Generals' Problem proves there is no general solution: two parties communicating over an unreliable channel cannot reach guaranteed mutual agreement in a finite number of messages.
In CDC terms: after the pipeline writes events to a sink, it needs to save a checkpoint. If the sink confirms the write but the checkpoint save fails, the pipeline cannot know on restart whether the sink received the data. It replays. Duplicate.
Where Exactly-Once Is Achievable
Exactly-once is possible in specific configurations where the pipeline and the sink participate in the same transaction:
Within Kafka. Kafka's exactly-once semantics work when both the input and output are Kafka topics. The framework coordinates the consumer offset commit and the producer write in a single Kafka transaction. This is genuine exactly-once, but it only applies within the Kafka ecosystem. As the Kafka documentation notes, exactly-once delivery for other destination systems requires cooperation from those systems.
Transactional database sinks. If the sink is a database that supports transactions, the pipeline can write the events and update its checkpoint in the same database transaction. Either both succeed or both roll back. This provides exactly-once, but it requires the checkpoint to live in the same database as the sink, coupling the pipeline's state to the destination.
Where Exactly-Once Is Not Achievable
When a pipeline writes to multiple sinks simultaneously (Kafka, S3, and PostgreSQL), there is no transaction protocol that spans all three. You would need all three systems to commit atomically, and if any one of them reports an ambiguous failure (a timeout where the write may or may not have succeeded), the pipeline cannot determine the correct action. This is the fundamental constraint that makes exactly-once impossible across heterogeneous sinks.
This is not a limitation of any specific CDC tool. It is a property of distributed systems. Tyler Treat's You Cannot Have Exactly-Once Delivery provides a detailed walkthrough of why this is an impossibility result, not an engineering challenge.
Effectively-Once: The Practical Approach
Since true exactly-once is not achievable across arbitrary sinks, production CDC pipelines aim for "effectively-once" delivery: at-least-once semantics where the consumer handles duplicates so the end result is the same as if each event arrived exactly once.
Three patterns accomplish this.
Idempotent Writes
Design sink writes so that applying the same event twice produces the same result as applying it once.
For database sinks, this means upserts:
INSERT INTO orders (id, status, total)
VALUES (1001, 'shipped', 89.99)
ON CONFLICT (id) DO UPDATE
SET status = EXCLUDED.status, total = EXCLUDED.total;
Whether this runs once or three times, the orders table ends up in the same state. The duplicate is absorbed without application logic.
For S3 sinks, writing to the same object key with the same content is naturally idempotent. The object is identical regardless of how many times it was written.
Position-Based Deduplication
Each CDC event carries a position from the source database (the WAL LSN in PostgreSQL, the GTID in MySQL). Consumers track the highest position they have processed and skip any event at or below it.
This works for ordered streams where events arrive in WAL commit order. It does not work for unordered delivery or when multiple consumers write to the same destination without coordination.
Pipeline-Level Deduplication
The pipeline itself can deduplicate events before they reach sinks. Events are identified by a composite key (source, table, operation, WAL position) and tracked in a bounded window. Events seen within the window are skipped.
This catches duplicates from source-level replay, for example when PostgreSQL replays events after a replication slot reconnection. The trade-off is memory: the deduplication map must be large enough to cover the replay window but bounded to prevent unbounded growth.
Choosing the Right Guarantee
| Use case | Guarantee | Why |
|---|---|---|
| Development, staging | At-most-once | Speed over completeness |
| Approximate metrics | At-most-once | Missing a few data points is acceptable |
| Production CDC | At-least-once | No data loss; sinks handle duplicates |
| Financial data, audit logs | At-least-once + idempotent sinks | Effectively-once through sink-side dedup |
| Kafka-to-Kafka processing | Exactly-once (Kafka EOS) | Same-system transactions are possible |
For most production CDC workloads, at-least-once with idempotent sinks is the right choice. It provides the data safety of no-loss delivery without requiring all sinks to participate in a distributed transaction protocol.
Where to Go from Here
This article covers what delivery guarantees mean and why at-least-once is the industry standard. The related articles go deeper into the mechanics:
- Checkpointing and Crash Recovery: how a CDC pipeline saves its position and resumes after a crash
- Deduplication Strategies: specific patterns for handling duplicates at the sink level
- Transaction Boundaries: why delivering grouped changes atomically matters
- What is Change Data Capture: the broader context of how log-based CDC works
For the broader picture of how delivery guarantees fit into transaction log processing, see What is Transaction Log Processing.
We built Remac with at-least-once delivery as the default, and with the tools to make effectively-once straightforward: pipeline-level deduplication, position metadata on every event, and idempotent writes for supported sinks (PostgreSQL upserts, S3 object overwrites). Remac currently supports PostgreSQL, with MySQL, MongoDB, SQL Server, and Oracle on the roadmap.
Take a look at remac.io.
Further Reading:
- Two Generals' Problem. The foundational impossibility result for reliable agreement between independent systems.
- Kafka: Message Delivery Semantics. Official Kafka documentation on exactly-once processing within the Kafka ecosystem.
- Debezium FAQ. Debezium's explanation of its delivery guarantees under normal and failure conditions.
- You Cannot Have Exactly-Once Delivery. Tyler Treat's detailed argument on why exactly-once delivery is an impossibility result, not an engineering challenge.
- Designing Data-Intensive Applications, Chapter 11. Martin Kleppmann's analysis of exactly-once semantics and stream processing guarantees.