What a Lost ACK Means for Data Delivery
What a Lost ACK Means for Data Delivery
About Remac: Remac is a high-performance, database-agnostic transaction log processor. It turns native database transaction logs into governed data streams, replicas, audit records, recovery data, and replayable events.
Remac defines delivery as each selected event reaching its required sink under a declared delivery policy. That claim depends on the exact effect a sink acknowledgment confirms.
This delivery timeline shows the ambiguity:
02:11:04 a generic processor sends a write
02:11:05 the destination applies it
02:11:05 the connection closes before the response arrives
02:11:12 the processor must choose its next action
The processor knows that it sent the request. It cannot tell from the missing response whether the destination rejected the request, never received it, or applied it before the connection failed.
The safe response depends on the destination, the operation, and the effect of repeating the write.
Name the Observation Boundary
The word “acknowledged” is useful only when it names the system that produced the response and the effect that system confirmed.
| Observation | What it can confirm | What remains outside its scope |
|---|---|---|
| Client API accepted a request | The client admitted the request to its local send path | Broker or destination receipt |
| Destination accepted a write | The destination reached its documented acceptance boundary | Later consumer work or an external side effect |
| Transaction committed | The selected writes committed under that database's transaction rules | A response reaching the caller |
| Consumer saved a position | The consumer recorded where it plans to continue | The durability or correctness of a separate destination effect |
These boundaries can all be valid. They answer different questions. A delivery policy must select the boundary that matches the required outcome.
A Missing Response Does Not Reveal the Result
Several outcomes can produce the same client observation.
| Possible outcome | Destination effect | Response received |
|---|---|---|
| Connection fails before the request arrives | None | No |
| Destination rejects the request, but its response is lost | None | No |
| Destination applies the write and loses the response | Applied | No |
The client sees a timeout or closed connection in every row. The destination state differs.
This makes “retry on error” incomplete as a delivery policy. A timeout says that the client received no usable response before its deadline. It does not identify the destination state.
Kafka Shows What a Successful Send Can Mean
Kafka provides a concrete example because one producer API exposes several completion boundaries.
The Java producer's send() method is asynchronous. It normally returns after adding the record to a client buffer. The returned future completes later with record metadata or an error.
The producer's acks setting defines what the broker must confirm before it considers a request complete.
| Producer observation | What it proves |
|---|---|
send() returns | The client accepted the record into its send path. No broker acknowledgment has arrived yet. |
Completion with acks=0 | The producer did not wait for a broker acknowledgment. Returned metadata uses offset -1. |
Completion with acks=1 | The partition leader wrote the record to its local log. It did not wait for all followers. |
Completion with acks=all | The current in-sync replicas acknowledged the record. |
acks=all is Kafka's strongest producer acknowledgment setting. It refers to the current in-sync replica set, not every replica assigned to the partition. With acks=all, the topic's min.insync.replicas setting can require Kafka to reject the write when too few in-sync replicas remain.
Even acks=all proves a Kafka publication result. Kafka separates producers and consumers, so the acknowledgment does not prove that a consumer received or processed the record.
Kafka's transactional API exposes the same network ambiguity at a later boundary. The commitTransaction() documentation states that a timeout does not mean the commit request failed to reach the broker. The application cannot switch safely to a different operation while that commit might still be completing. Kafka allows the same commit call to be retried.
This is a narrower and more useful rule than “retry all timeouts.” It ties the response to one operation whose completion state Kafka can continue resolving.
Blind Retry Can Repeat the Effect
A retry protects against cases where the first request did not take effect. It can repeat the effect when the first request succeeded.
The consequence depends on the operation. A repeated assignment can end with the same row state. A repeated increment changes the value twice. A repeated append creates two records. A repeated request to an external service can trigger two actions.
The delivery policy should define the operation before it defines the retry.
Classify the effect before retrying
The same network error needs a different response for different operations.
| Operation | Result of an identical retry | Required control |
|---|---|---|
Set row status to shipped | Can end in the same row state | Stable row identity and validation that the repeated request has the same meaning |
| Insert with a unique operation ID | Can resolve to one durable record | A uniqueness rule and a way to read the existing result |
| Increment a counter | Can apply the increment twice | A separate operation identity or reconciliation against the intended total |
| Append a record without a stable ID | Can create another record | A durable idempotency key or a destination query that can identify the first effect |
| Trigger an external action | Can repeat the action | A destination-supported idempotency contract or an explicit recovery procedure |
An operation is safe to repeat when repetition has the same intended effect, or when the destination can recognize and suppress or reconcile repetitions of the same intended operation. Similar payloads are not enough. Two legitimate operations can carry identical values.
A stable operation ID changes the recovery options
One database operation sends an inventory reservation with operation ID reserve-8041. The destination enforces a unique constraint on that ID and stores the reservation in the same transaction that records the ID.
If the commit response disappears, the caller can query reserve-8041. A matching committed record proves that the first attempt took effect. If no matching committed record is visible, the caller can retry with the same ID only when the uniqueness mechanism safely arbitrates against an unresolved first attempt. The retry can wait or return a unique-key conflict if the first attempt commits. That conflict gives the caller a known record to inspect.
This pattern has strict conditions. The ID must identify one intended operation. The destination must preserve it durably with the effect. Reusing one ID for different request content creates another ambiguity, so the system should reject that mismatch.
An idempotency key does not make every operation transactional. It gives the destination a stable identity for detecting the same requested effect.
Idempotence Controls One Retry Boundary
Kafka documents the lost-response problem directly. After a network error, a producer cannot know from that error whether the broker committed the record before the connection failed.
Kafka's idempotent producer prevents qualifying automatic retries from writing duplicate entries to the Kafka log. In Kafka 4.2, idempotence is enabled by default when no conflicting configuration disables it.
The guarantee depends on configuration. Kafka requires acks=all, retries above zero, and no more than five in-flight requests per connection for idempotence. If idempotence is explicitly enabled with a conflict, Kafka raises a configuration error. If it is not explicit, a conflicting setting can disable it.
Idempotence does not deduplicate an application's separate send calls. Kafka also limits the producer's idempotence guarantee to messages sent within one producer session. The guarantee controls duplicate records in the Kafka log. It does not prove that a consumer processed the record once or coordinate the Kafka write with an effect in another system.
The destination's public contract must define the scope. A statement such as “the sink acknowledged it” omits the configuration and the effect that the response confirms.
Source Progress Adds Another Boundary
A data pipeline also tracks progress at the source. Advancing that progress too early can remove the original history before the destination uncertainty is resolved.
The safety question is whether the system still has an approved way to retry, reconcile, or replay the write. The answer can come from source retention, a documented durable copy, or another recovery path.
Source events 101, 102, and 103 are sent in order. The destination confirms 101 and 103, but 102 has an unknown result. Advancing source progress through 103 would hide the unresolved range. Releasing the only retained copy of 102 would remove the safest repair input.
Later success is useful evidence for later work. It cannot settle an earlier unknown effect.
Resolve an Unknown Result Deliberately
A delivery procedure for an unknown result should follow a defined order:
- Preserve the exact operation identity, payload, and source position.
- Query the destination when it offers an authoritative lookup for that operation.
- Retry only when the operation and destination contract define safe repeat behavior.
- Keep the required source or replay history until the result is confirmed, repaired, or accepted as loss under an explicit policy.
Some destinations cannot answer whether a prior operation happened. Some side effects cannot be reversed or queried reliably. In those cases, the delivery policy must expose the uncertainty to an operator or compensating process. Calling the request successful would invent evidence that the destination did not provide.
Reconciliation can help when the intended final state is known. It compares the destination with a declared source boundary and repairs a difference. Reconciliation does not change the original acknowledgment, but it can create new evidence that the required state now exists.
A Retry Rule Needs a Defined Boundary
A retry rule must name the operation it repeats and the effect a successful response proves under the current configuration. It must also define what a timeout leaves unknown, whether that operation can repeat safely, and which data remains available for repair. Destination storage, visibility, and downstream processing are separate boundaries unless the destination contract explicitly connects them.
These facts differ by destination. A universal retry slogan cannot replace them.
The Remac Contract connects delivery, resumability, and recoverability. A lost ACK turns a network failure into a data question. The correct response starts with the effect that might already exist.
Join the Remac waitlist for product updates.
External reference note: The external technical information and linked references in this article were current when we published it. External systems, documentation, and defaults can change after publication.
Further Reading:
- KafkaProducer 4.2 API. Official documentation for asynchronous sends, producer sessions, and transactional timeouts.
- Kafka 4.2 Producer Configuration. Official definitions for
acks, retries, idempotence, and in-flight request limits. - Kafka Design: Message Delivery Semantics. Official explanation of network ambiguity, idempotent production, and consumer processing boundaries.
- When Can a Data Pipeline Safely Delete History?. How destination uncertainty affects retention and repair.