When Can a Data Pipeline Safely Delete History?

About Remac: Remac is a high-performance, database-agnostic transaction log processor. It turns native database transaction logs into governed data streams, replicas, audit records, recovery data, and replayable events.

These outputs can depend on retained history for delivery, audit, replay, or recovery. The safe retention period depends on the source, each destination, and the declared policy.

History can mean source WAL retained for a consumer, a logical event archive kept for replay, audit records held under a retention policy, or archived native WAL kept for physical recovery. Deleting one copy settles only the obligation that copy served.

Keeping history has a cost. Old transaction logs use source storage. Event archives continue to grow. Longer retention periods add monitoring and recovery work.

Early deletion creates a different risk. A destination can miss data that no remaining source can reproduce. A restarted consumer can request a position that the source no longer has. A recovery job can have a valid base backup but lack the later log files it needs.

The safe deletion point depends on evidence. A pipeline must know what the source still retains, what each destination has confirmed, and what every recovery policy still needs.

---

Start With the Obligation

Retention decisions become clearer when every stored copy has one named purpose.

The same bytes can support more than one purpose. A WAL archive might serve recovery and help a standby catch up. Deletion must wait until every declared purpose has ended or moved to another approved copy.

Age alone cannot prove this. A file can be one day old and still be the only copy of a required recovery range. Another file can be older but safe to remove because a newer complete recovery chain replaced it.

PostgreSQL Slots Define a Source Retention Boundary

PostgreSQL replication slots provide a concrete example.

A replication slot can stop PostgreSQL from removing WAL that its consumer might still need. PostgreSQL reports the oldest WAL that might still be required in restart_lsn. For a logical slot, confirmed_flush_lsn reports the position through which the consumer confirmed receipt.

The two positions have different meanings. PostgreSQL can retain WAL older than the consumer's confirmed position because logical decoding can still need it. Operators should inspect both values instead of treating one LSN as a complete account of slot health.

Retention can also put the source at risk. PostgreSQL warns that replication slots can retain enough WAL to fill the space allocated to pg_wal. The max_slot_wal_keep_size setting limits the WAL that replication slots can retain at checkpoint time. It does not cap total pg_wal use because PostgreSQL can retain WAL for other requirements.

A finite limit changes the slot failure mode. If restart_lsn falls too far behind, PostgreSQL can remove WAL that the consumer still needs. In pg_replication_slots, wal_status can become unreserved while required files are due for removal. A value of lost means that the slot is no longer usable.

Deleting or recycling source history is safe only when no declared consumer or recovery policy still needs it. Another approved copy can satisfy the condition only when it covers the same source range and recovery purpose, and a supported, verified procedure can use it.

Measure the retained range

On a PostgreSQL 18 source primary, the following query measures the source-side range held for each slot:

SELECT
    slot_name,
    slot_type,
    active,
    restart_lsn,
    confirmed_flush_lsn,
    wal_status,
    safe_wal_size,
    pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS wal_distance_from_restart
FROM pg_replication_slots
ORDER BY slot_name;

wal_distance_from_restart measures the distance from restart_lsn to the current WAL write location. It is not the disk space occupied by WAL files in pg_wal. It also does not measure destination application or prove that every required record reached the destination.

safe_wal_size answers another question when max_slot_wal_keep_size is finite. It estimates how many more bytes PostgreSQL can write before the slot is in danger of becoming lost. PostgreSQL reports it as NULL for a lost slot and when max_slot_wal_keep_size is -1.

One query result is a snapshot. Operators also need the rate at which the retained distance changes. If a workload generates 20 GB of WAL per hour and the allowed slot headroom is 160 GB, the simple capacity estimate is eight hours. This is an illustrative calculation, not a safe alert threshold. Bursts, checkpoints, other WAL retention needs, archive delays, and response time reduce the usable intervention window.

The practical alert should fire early enough for the team to repair the consumer, add safe capacity, or rebuild from another approved source before the slot loses required WAL.

Destination Success Needs a Precise Meaning

A source position describes source-side progress. It does not automatically describe the destination.

A destination can acknowledge a network request, commit a database transaction, store an object, or accept a message under a documented durability setting. Each result confirms a different boundary.

Before a pipeline uses a destination result to release retained history, it must tie the response to an exact event or transaction. The confirmed effect must also survive the failures covered by the delivery policy. That evidence comes from the destination's current documentation and configuration. A generic “write succeeded” metric does not provide enough evidence.

One Pipeline Can Have Several Deletion Conditions

A pipeline sends the same selected source changes to an operational replica and an audit archive.

The replica has confirmed every required transaction through source position P900. The audit archive contains the same range, but its policy requires those records for another 90 days. A replay copy also starts at P500 and is approved for rebuilding the replica through P900.

The replica's delivery condition is complete through P900. That result does not end the audit obligation. It also does not prove that history before P500 is unnecessary because the replay copy cannot rebuild state that its baseline and event range do not cover.

Now assume the replica has an unresolved result at P760, followed by individually confirmed work through P900. The latest individually confirmed result still cannot authorize deletion of the range around P760. The earlier unknown result controls the repair requirement.

Each condition uses different evidence. Delivery needs a confirmed destination effect for the complete required range. Replay depends on a compatible baseline, the complete event range, and a defined final result. Audit retention follows its declared evidence period, while physical recovery requires a compatible base backup and continuous native WAL for the required range.

The pipeline can remove a stored copy only when all obligations attached to its exact range have ended or moved to an approved equivalent copy.

A Later Success Can Leave an Earlier Result Unknown

Three records show the retention problem. The source produces records 41, 42, and 43. A consumer sends all three to the destination. The destination confirms records 41 and 43, but the response for record 42 disappears after the write is sent.

Record 43 proves that later work succeeded. It does not resolve record 42. If the source removes the history needed to reproduce record 42 and no replay copy remains, the team can lose its only practical repair path.

Concurrency makes this harder to see. Several records can complete in a different order from their source order. The retention rule must still account for every required earlier record, not only the highest position observed.

Replay and Recovery Use Separate Retention Rules

Delivery completion does not always end retention.

An audit policy can require records for years. A replay policy can keep events long enough to rebuild a declared projection. A physical recovery policy can require a base backup and the complete WAL range after that backup. Each policy needs its own deletion condition.

For physical PostgreSQL recovery, the team needs a compatible cluster-level base backup and the required native WAL through the selected recovery target. PostgreSQL requires a continuous sequence of archived WAL files that reaches back at least to the start of the backup.

For logical replay, the team needs a compatible starting state and the event history required to reach the intended result. Retaining event files without a known starting point does not prove that the destination can be rebuilt.

Delivery, audit, and recovery can use different retention periods. One global age limit cannot express every obligation safely.

Delete Against a Proved Boundary

A deletion policy should produce an auditable decision, not only a retention duration.

1. Identify the exact history range and every declared purpose it serves.
2. Record the source and destination evidence for that range.
3. Stop if an earlier delivery, replay, audit, or recovery obligation remains unresolved.
4. Verify that any replacement copy covers the same source history, interpretation, and recovery purpose, and that a supported procedure can consume it.
5. Delete only through the greatest range for which every required condition is complete.

This procedure also defines what operators must preserve during an incident. Storage pressure can justify an emergency policy change, but it does not make an unresolved range safe. The system should record any explicit decision to accept loss or reduce a recovery window.

The Remac Contract connects completeness, delivery, resumability, and recoverability. This makes retention part of the product contract. It cannot be reduced to a storage setting.

History becomes safe to delete when the declared work and recovery needs no longer depend on it. No single recent position can settle every retention obligation.

Join the Remac waitlist for product updates.

External reference note: The external technical information and linked references in this article were current when we published it. External systems, documentation, and defaults can change after publication.

---

Further Reading:

• Replication Slots. Official PostgreSQL documentation on slot retention and source-storage risk.
• pg_replication_slots. Official definitions for restart_lsn, confirmed_flush_lsn, wal_status, and related slot fields.
• max_slot_wal_keep_size. Official PostgreSQL configuration reference for the slot retention limit.
• Continuous Archiving and Point-in-Time Recovery. Official PostgreSQL documentation on base backups, archived WAL, and physical recovery.
• System Administration Functions. Official definitions for pg_current_wal_lsn() and pg_wal_lsn_diff().
• PostgreSQL WAL: How It Works. Remac's guide to WAL records, checkpoints, replication slots, and monitoring.