Seven explicit guarantees.
One declared contract.
Every transaction log processor makes implicit promises about your data. Remac makes those promises explicit and maps them to tests, failure injection, traces, invariants, and recovery checks.
Completeness
Remac preserves every selected event required by the declared processing policy.
Ordering
Each required sink receives events in source commit order.
Delivery
Each selected event reaches its required sink under the declared delivery policy.
Resumability
Processing continues from the last safe confirmed position after an interruption.
Consistency
A replication sink converges to the selected source row state. Other sink types match their declared stream or projection.
Liveness
The processor makes progress when its required systems can make progress. It exposes unsafe or impossible progress.
Recoverability
The processor preserves enough durable state to recover its declared work from a known safe position.
The contract applies across logical and physical processing. Each mode has its own evidence, mechanisms, and recovery boundaries.
The cost of getting it wrong
One enterprise survey shows how quickly pipeline failures become a business cost.
74+ failure modes. Cataloged and growing.
Remac catalogs known ways a transaction log processor can break. The catalog guides design, test coverage, and release evidence. It is not exhaustive, and it continues to grow.
Source failures
9 modesConnection drops, slot invalidation, disk pressure, DDL mid-stream, failover races, snapshot degradation.
Pipeline failures
10 modesTransaction buffer overflow, orphaned transactions, deduplication exhaustion, backpressure deadlock, thread leaks.
Sink failures
8 modesWrite timeouts, partial batch writes, schema mismatches, connection pool exhaustion, broker unavailability.
Middleware failures
20 modesFilter false-drops, transform mutations, enrichment staleness, routing conflicts, hot-reload races.
Durable state failures
6 modesCorruption, unsafe frontier advancement, concurrent ownership, lease expiry, storage exhaustion.
Storage failures
6 modesPartial uploads, read-after-write inconsistency, credential expiry, bit-rot, prefix collision.
Recovery failures (physical)
6 modesArchive gaps, unreachable restore targets, snapshot corruption, timeline divergence, config conflicts.
Recovery failures (logical)
5 modesReplay ordering violations, sink failure during replay, event format evolution, cross-database type mismatches.
Configuration failures
4 modesInvalid hot-reload, config drift, secret exposure in logs, default value mismatches.
Total: 74+ cataloged failure modes across 9 categories. This number grows with every release.
Real incidents. Real consequences.
These documented incidents come from production data, replication, and distributed systems.
Six hours of production data lost
PostgreSQL replication lag caused the secondary to fall behind. WAL segments were removed before the secondary could consume them. During recovery, an engineer accidentally deleted the primary database. Multiple backup mechanisms had silently failed, including unusable database exports and unconfigured cloud snapshots.
43-second network split causes 24 hours of degraded service
A routine maintenance task severed connectivity for 43 seconds, triggering an automated MySQL failover. Both data centers accepted writes during the partition, creating divergent datasets. Reconciling these writes took over 24 hours of degraded service.
Two-day outage causes unrecoverable data gaps
A control plane outage lasted approximately 40 hours. Datasets that were not replicated across regions experienced persistent gaps. Log push features suffered unrecoverable data loss for the majority of the event.
Storage pathology contributes to a 73-hour outage
High read and write load on Consul triggered a pathological performance issue in its BoltDB storage. Deleted entries left free pages that were not reclaimed, and write times grew from milliseconds to seconds. A single Consul cluster supported several workloads, which increased the blast radius.
24-hour global outage with confirmed customer data loss
An unsupervised global update restarted infrastructure, disconnecting 50–60% of production Kubernetes nodes. Intake pipelines lacked disk-based persistence, so data existed only in memory or local disk. Losing nodes meant losing data. Replicated stores were unable to accept writes, causing memory buffers to overflow. Datadog confirmed "a limited but non-zero amount of customer data" was lost in unrecoverable ways.
Bulk user removal triggers replication lag cascade and repeated OOM kills
A customer removed a large number of users from their workspace, overloading the database cluster with queries. Replication lag appeared as replicas fell behind the primary. The high write load exhausted memory on the shard primary, triggering an OOM kill. The promoted replica also OOM-killed under the same load, and the cycle repeated.
Failure doesn't come one at a time.
Compound failures expose gaps that isolated fault tests miss. The cases below are selected examples from a wider program that tests how source, pipeline, sink, durable state, storage, and recovery failures interact.
Schema change during active streaming
Columns are added, dropped, retyped, or renamed while events stream. The decoder must pick up each new RELATION message so that every insert, update, delete, and truncate afterward keeps decoding against the current schema.
Durable progress failure + slow sink
The durable progress path becomes unavailable while a sink is slow. The processor must bound backlog growth and must not confirm source progress past its last contiguous safe position. If safe progress cannot resume within the declared limits, the data path must stop without advancing that position.
Primary failover + slot loss
The primary database fails over to a standby. If the replication slot was not synchronized, events between the old slot position and the failover point are at risk. Detection must be immediate, not discovered days later in a data audit.
Multi-sink partial delivery
An event is routed to two sinks. The first sink accepts it, but the second sink fails mid-transfer. Each sink must retain its own contiguous progress state. Success at one sink must not hide unresolved delivery at another sink.
Credential expiry during long archive job
An authentication token expires mid-upload of a large archive segment to cloud storage. Partial uploads must be aborted cleanly. Archiving must stop before it confirms unsafe progress and resume after valid credentials are restored, without orphaned data or archive gaps.
Reliability needs evidence.
The Remac Contract defines named, testable boundaries. Tests, failure injection, invariants, traces, and recovery checks provide the evidence.
We took part of our testing philosophy from SQLite, which publishes hundreds of lines of test code for each line of production code.
We don't need that ratio, but we share the same responsibility: we handle other people's data. A transaction log processor that silently drops events, delivers them out of order, or fails to resume after a crash is worse than having no processor at all.
Engineering Principles
- Every cataloged failure mode is a tracked test target; coverage is measured, not assumed.
- No bug is fixed without a failing test that reproduces it first.
- Failure test cases outnumber success test cases 70/30 for I/O code.
- Crash recovery is tested by SIGKILLing the process mid-stream and verifying that it resumes from its last safe durable position with no loss.
- Network partitions, latency, and disconnects are injected between the pipeline and every dependency with a real fault-injection proxy.
- A flaky test is treated as a production bug, investigated and fixed, never skipped.
- Tests are the specification. If a behavior is not tested, it is not guaranteed.
One processor. Six transaction log functions.
Move committed data through one controlled path, with explicit ordering, delivery, and recovery boundaries.