What is Change Data Capture (CDC)?

Change Data Capture is the process of detecting changes made to data in a database and delivering those changes to downstream systems in near real time.

A row gets inserted, updated, or deleted in PostgreSQL. Within milliseconds, that change appears in Kafka, Elasticsearch, a data warehouse, or another database. No batch jobs. No nightly syncs. No polling the source database for "what changed since last time."

That's CDC.

---

Why CDC Exists

Data rarely stays in one place. The database that stores your orders also needs to feed your analytics dashboard, your search index, your recommendation engine, and maybe a second database in a different region. The question is how to keep all of these in sync.

The traditional answer was batch ETL: a scheduled job runs every hour (or every night), queries the source database for recent changes, transforms them, and loads them into the destination. This works until it doesn't. Hourly batches mean up to 59 minutes of stale data. Full table scans tax the source database. Deletes get missed because you can't query for rows that no longer exist. And transaction context (which changes belong together) is lost entirely.

CDC solves this by reading changes as they happen, in order, with full context, including deletes. For the detailed breakdown of how batch ETL falls short, see What is Transaction Log Processing.

---

Three Ways to Capture Changes

Not all CDC is created equal. There are three common approaches, and they differ significantly in reliability, performance, and completeness.

Timestamp-Based (Query Polling)

The simplest approach. Add an updated_at column to every table, update it on every write, then periodically query for rows where updated_at > last_poll_time.

SELECT * FROM orders WHERE updated_at > '2025-06-10 14:00:00';

It's easy to set up and doesn't require special database permissions. But it has real limitations:

• Misses deletes. A deleted row can't be queried. It's gone. Unless you implement soft deletes (a deleted_at flag), deletions are invisible.
• Misses intermediate changes. If a row is updated three times between polls, you only see the final state. The intermediate values are lost.
• Requires schema changes. Every tracked table needs an updated_at column that is reliably set on every write. In practice, many tables don't have one, and retrofitting it across a large schema is a project in itself.
• Adds query load. Each poll runs a query against the source database, which competes with production traffic for CPU, memory, and I/O.
• No transaction context. You get individual row states, not grouped transactions. Five rows that changed together in a single commit arrive as five independent results with no indication they're related.

Timestamp-based CDC can work for simple, append-only tables where deletes aren't a concern and minute-level latency is acceptable. For anything beyond that, it falls short.

Trigger-Based

Database triggers fire custom procedures on every INSERT, UPDATE, or DELETE. In a trigger-based CDC setup, each trigger writes the change to a separate changelog table, which downstream consumers then read.

CREATE TRIGGER capture_order_changes
AFTER INSERT OR UPDATE OR DELETE ON orders
FOR EACH ROW
EXECUTE FUNCTION log_change_to_changelog();

This captures everything, including deletes. The trigger itself fires immediately, but end-to-end latency depends on how quickly downstream consumers poll the changelog table. It comes with costs:

• Write amplification. Every write to the source table triggers a second write to the changelog table. The exact overhead depends on hardware, schema, and write volume, but in high-throughput environments it can be significant. Doubling every write is inherently expensive.
• Schema coupling. Triggers are tightly coupled to the table schema. When columns are added, renamed, or removed, triggers break. Managing triggers across dozens or hundreds of tables becomes a maintenance burden.
• DDL blind spots. Triggers fire on data changes (DML), not schema changes (DDL). A table rename, column type change, or index modification won't appear in the changelog.
• Database-specific. Trigger syntax and behavior differ between PostgreSQL, MySQL, and other databases. A trigger-based CDC solution doesn't port cleanly.

Trigger-based CDC made sense before log-based alternatives were widely available. Today, it's rarely the right choice for new implementations.

Log-Based

Log-based CDC reads the database's transaction log directly. Instead of querying tables or installing triggers, it connects to the log that the database is already writing on every commit and decodes the raw entries into structured change events.

This is the approach used by production CDC tools like Debezium, Fivetran, and Remac. Here's why:

• Captures everything. Inserts, updates, deletes, and (depending on the database) DDL changes. Nothing is missed.
• No query load. The CDC tool reads the log via the database's replication protocol. It doesn't run SELECT queries against your tables.
• Preserves transaction context. Changes that occurred in the same database transaction can be grouped and delivered together, so downstream consumers see atomic operations, not fragments.
• Minimal overhead. Reading the log adds no write amplification to the source database. The overhead is limited to maintaining a replication connection and (in PostgreSQL) a replication slot that prevents the database from discarding unread log segments. This needs monitoring but is far lighter than triggers.
• Sub-second latency. Changes are available in the log as soon as the transaction commits. A well-configured CDC tool delivers them within milliseconds.

The trade-off is setup complexity. Log-based CDC requires database configuration (wal_level = logical in PostgreSQL, binlog_format = ROW in MySQL, a replica set in MongoDB) and understanding of concepts like replication slots, delivery guarantees, and checkpointing. But once configured, it's the most reliable and lowest-impact approach available.

Comparison

---

What a CDC Event Looks Like

Regardless of approach, the goal of CDC is to produce structured change events. In log-based CDC, these events are decoded from the transaction log and normalized into a consistent format.

An INSERT:

{
  "operation": "INSERT",
  "table": "orders",
  "timestamp": "2025-06-10T14:30:22Z",
  "after": {
    "id": 1001,
    "customer_id": 42,
    "total": 89.99,
    "status": "pending"
  }
}

An UPDATE (with before and after):

{
  "operation": "UPDATE",
  "table": "orders",
  "timestamp": "2025-06-10T14:35:10Z",
  "before": {
    "id": 1001,
    "status": "pending"
  },
  "after": {
    "id": 1001,
    "status": "shipped"
  }
}

A DELETE:

{
  "operation": "DELETE",
  "table": "orders",
  "timestamp": "2025-06-10T15:00:00Z",
  "before": {
    "id": 1001,
    "customer_id": 42,
    "total": 89.99,
    "status": "shipped"
  }
}

The before image shows what the row looked like before the change. The after image shows what it looks like now. For deletes, there is no after because the row no longer exists. For inserts, there is no before because the row didn't exist yet.

How much detail the before image contains depends on the database and its configuration. MySQL includes all columns by default (binlog_row_image = FULL). PostgreSQL, by default, only includes the primary key in the before image for UPDATEs and DELETEs. To get the full before image shown above, you need to set REPLICA IDENTITY FULL on the table. The examples here assume full before images are enabled.

This before/after structure is what makes CDC useful beyond simple replication. Downstream consumers can see exactly what changed, compute diffs, trigger alerts on specific field transitions, or build complete audit trails.

---

Common Use Cases

Real-time analytics. Stream changes from your transactional database to a data warehouse or OLAP system. Dashboards update in seconds instead of waiting for the next batch job. When an order is placed, the revenue dashboard reflects it immediately.

Search index synchronization. Keep Elasticsearch or Solr in sync with the source database. When a product's price changes in PostgreSQL, the search index updates within milliseconds. No more selling items at yesterday's price.

Microservice data propagation. In a microservice architecture, services often need access to data owned by other services. CDC lets you propagate changes between service boundaries without tight coupling or synchronous API calls. The orders service publishes change events - the shipping service, the billing service, and the analytics service each consume what they need.

Audit logging and compliance. CDC captures every change with a timestamp and the full before/after state. This creates an immutable audit trail for regulatory requirements (GDPR, SOX, HIPAA) without modifying application code or relying on developers to remember to log changes manually.

Cache invalidation. Instead of expiring caches on a timer and hoping for the best, use CDC to invalidate or refresh cache entries the moment the underlying data changes. No stale reads, no unnecessary cache misses.

---

Where to Go from Here

This article covers what CDC is and how the three approaches differ. The articles that follow go deeper into the mechanics:

• CDC Delivery Guarantees: at-least-once, exactly-once, and why at-most-once is unacceptable for CDC
• Transaction Boundaries: why atomic delivery matters and what happens when it's missing
• Checkpointing and Crash Recovery: how a CDC pipeline remembers where it stopped
• Deduplication Strategies: handling the duplicates that at-least-once delivery introduces

For the broader picture of how CDC fits into transaction log processing, see What is Transaction Log Processing. For a deep dive into the logs that log-based CDC reads, see What is a Transaction Log.

We built Remac to make log-based CDC straightforward. A single binary that reads your database's transaction log, preserves transaction boundaries, and delivers change events to Kafka, S3, Elasticsearch, PostgreSQL, and several other destinations. Remac currently supports PostgreSQL, with MySQL, MongoDB, SQL Server, and Oracle on the roadmap.

Take a look at remac.io.

---

Further Reading:

• What is Change Data Capture?. Confluent's overview of CDC concepts and architecture.
• CDC Capture Techniques: Log, Trigger, and Query-Based. Detailed comparison of the three CDC approaches with performance data.
• Change Data Capture: Stop Copying 50M Rows to Move 5K Changes. Practical breakdown of why batch ETL breaks down at scale.
• Kafka CDC: The Complete Guide. In-depth guide on CDC with Kafka as the event backbone.