Change Data Capture (CDC)
Implementing Real-Time Cache Invalidation with CDC Events
Learn how to use database modification events to automatically update or purge Redis entries, ensuring perfect consistency between your database and cache.
In this article
The Distributed State Dilemma
Modern applications frequently rely on Redis to reduce database load and provide sub-millisecond response times for frequently accessed data. However, maintaining synchronization between a relational database and a high-speed cache introduces significant architectural challenges. Most developers initially attempt to solve this by updating both systems directly from the application layer, a strategy often called the dual-write pattern.
The dual-write approach is inherently fragile because it lacks atomicity across two different network services. If the database update succeeds but the application crashes before it can update Redis, the cache becomes stale and serves incorrect data to users. Even with retry logic, network partitions or concurrent requests can lead to race conditions that leave the system in an inconsistent state for long periods.
Change Data Capture provides a more resilient alternative by treating the database as the single source of truth. Instead of relying on the application to update the cache, we observe the database transaction log to identify changes as they happen. This shift in perspective ensures that every committed change is eventually propagated to Redis, regardless of application-level failures.
Consistency in distributed systems is not a feature you add later but a property you design into the foundation of your data flow from the beginning.
The Limitations of TTL and Manual Eviction
Time To Live settings are a common safety net used to expire old data, but they represent a compromise rather than a solution. Setting a short TTL increases database pressure, while a long TTL allows stale data to persist longer than business requirements might permit.
Manual eviction logic embedded within service code often becomes complex as the number of cache keys grows. As business requirements change, developers must remember to update every single code path that modifies a specific table to ensure the corresponding Redis keys are cleared.
Why Log-Based CDC Wins
Log-based CDC works by reading the internal transaction logs of the database, such as the PostgreSQL Write Ahead Log or the MySQL Binary Log. Because these logs record every mutation committed to the disk, they provide a perfect history of the data state without requiring changes to the application code.
By capturing changes at the storage engine level, we can bypass the overhead of database triggers and avoid the performance penalties associated with frequent polling. This allows for a clean separation of concerns where the primary application focuses on business logic while a dedicated pipeline handles synchronization.
Architecting the Synchronization Pipeline
A robust CDC pipeline usually involves three main components: the database, a capture connector, and a message broker. Tools like Debezium serve as the connector, monitoring the database logs and streaming row-level modifications as structured events. These events are then pushed into a distributed streaming platform like Apache Kafka or Redpanda for reliable processing.
Using a message broker acts as a buffer between the source database and the Redis cache. If Redis experiences high latency or goes offline for maintenance, the messages remain in the queue until the synchronization worker can process them. This decoupling prevents performance issues in the caching layer from impacting the availability of the primary database.
Defining the Event Structure
Each CDC event typically contains a payload representing the state of the row before and after the change. This information is critical for intelligent cache management, allowing the system to determine exactly which keys need to be invalidated or updated.
1{
2 "before": { "id": 101, "status": "pending" },
3 "after": { "id": 101, "status": "shipped" },
4 "source": { "version": "2.4.0", "table": "orders" },
5 "op": "u",
6 "ts_ms": 1672531200000
7}Choosing Between Eviction and Pre-fetching
When a change event arrives, the synchronization worker must decide whether to delete the existing Redis key or update it with new data. Deleting the key, also known as cache invalidation, is often safer because it forces the next read request to fetch the most current data from the database.
Pre-fetching, or updating the value directly in Redis, can reduce latency for the next read but risks a race condition if events are processed out of order. For complex data structures like nested JSON objects, simple invalidation is usually the preferred strategy to maintain high data integrity.
Implementing the Sync Worker
The synchronization worker is a lightweight service that consumes events from the message broker and translates them into Redis commands. This service must be designed for high throughput and horizontal scalability to keep up with high-volume database writes. In a typical implementation, the worker maps the table name and primary key from the event to a specific Redis key format.
To ensure reliability, the worker should use idempotent operations when interacting with Redis. For instance, using the DEL command to remove a key is naturally idempotent because deleting a key that no longer exists has no side effects. This property allows the worker to safely retry operations if the network connection drops during processing.
Developing the Logic in Python
The following example demonstrates how to process an incoming stream of database events and perform the corresponding actions in Redis. We focus on handling inserts, updates, and deletes while ensuring that we only react to events from relevant tables.
1import json
2from redis import Redis
3from kafka import KafkaConsumer
4
5# Initialize connections
6redis_client = Redis(host='localhost', port=6379, decode_responses=True)
7consumer = KafkaConsumer('db_server.public.inventory', bootstrap_servers=['kafka:9092'])
8
9def process_cdc_stream():
10 for message in consumer:
11 event = json.loads(message.value)
12 operation = event.get('op')
13 data = event.get('after') if operation in ['c', 'u'] else event.get('before')
14
15 # Construct a unique Redis key using table metadata
16 primary_key = data.get('id')
17 redis_key = f"product_cache:{primary_key}"
18
19 if operation in ['u', 'd']:
20 # Invalidate cache on update or delete to ensure consistency
21 redis_client.delete(redis_key)
22 print(f"Invalidated key: {redis_key}")
23
24 elif operation == 'c':
25 # Optionally pre-warm the cache for new records
26 redis_client.set(redis_key, json.dumps(data))
27 print(f"Pre-warmed key: {redis_key}")
28
29if __name__ == '__main__':
30 process_cdc_stream()Managing Transactional Boundaries
One common pitfall is ignoring the order of operations when multiple updates occur for the same record in a short window. If the message broker does not guarantee strict ordering based on the partition key, an older event could potentially overwrite a newer one in Redis.
By using the primary key of the database row as the partition key in Kafka, we guarantee that all changes for a specific record are processed by the same worker instance in the exact order they occurred. This preserves the serializability of the database updates in our distributed cache.
Operational Resilience and Trade-offs
While CDC significantly improves consistency, it introduces additional infrastructure components that must be monitored and maintained. The complexity of managing Kafka and Debezium is a trade-off for the increased reliability of the caching layer. Engineering teams must weigh the operational overhead against the cost of data stale-ness in their specific business context.
Monitoring the lag between the database commit and the Redis update is essential for maintaining a healthy system. High lag indicates that the synchronization worker cannot keep up with the write volume, which may require scaling the consumer group or optimizing the Redis write patterns.
- Eventual Consistency: Updates to Redis are not instantaneous and follow the database commit with a small delay.
- Infrastructure Overhead: Requires managing message brokers and connector services like Debezium.
- Network Partition Handling: Must account for scenarios where the broker or Redis is temporarily unreachable.
- Schema Evolution: Changes to the database table structure must be coordinated with the synchronization logic.
Handling Out-of-Order Delivery
In rare failure scenarios, message brokers might deliver events out of order or multiple times. To handle this, we can include a version field or a timestamp from the database source in the Redis value. Before updating, the worker checks if the incoming event is newer than the data currently stored in the cache.
Using a Lua script in Redis allows for atomic check-and-set operations. This ensures that the version comparison and the write happen as a single unit, preventing race conditions between multiple worker threads.