Implementing Idempotency to Prevent Duplicate Event Processing

Distributed systems rarely offer the luxury of exactly-once delivery due to the inherent instability of network communications. Most modern webhook providers utilize an at-least-once delivery policy to ensure that critical information eventually reaches the destination. This means your application must be prepared to receive the same event multiple times without side effects.

Consider a scenario where a payment provider successfully processes a transaction and sends a webhook to your server. Your server updates the user balance and prepares to send a success response. If the connection drops at this exact moment the provider will interpret the lack of a response as a failure and retry the request.

Without an idempotency strategy your system might credit the user balance a second time when the retried request arrives. This leads to data corruption and financial discrepancies that are difficult to reconcile manually. Building a resilient consumer requires a mental shift from assuming unique requests to verifying them explicitly.

Idempotency is not a feature you add to a single function but a core architectural property that ensures your system state remains consistent regardless of how many times a specific operation is invoked.

The most effective way to handle duplicate events is to implement an idempotency layer that tracks processed message identifiers. Every webhook payload from a reputable provider includes a unique event ID in the header or the body. Your system must record these IDs in a persistent store to check against incoming requests.

When a new webhook arrives your first action should be to query this store to see if the ID has already been marked as completed. If the ID exists you can safely return a success response immediately without re-running any business logic. This pattern effectively shields your core services from the noise of repeated network attempts.

• Event ID persistence: Storing the unique identifier provided by the source system.
• Atomic status updates: Ensuring the record is created only when processing is guaranteed to start.
• Expiration policies: Cleaning up old event IDs to prevent the database from growing indefinitely.
• Response caching: Storing the original response status to return the same result on subsequent attempts.

A common pitfall is the check-then-act race condition where two identical requests are processed by different worker threads simultaneously. If both threads check the database and find no record they will both proceed to execute the business logic. To prevent this you must use atomic operations provided by your database engine.

Using a unique constraint on the event ID column in your database is the simplest way to enforce atomicity. When you attempt to insert a new record for an incoming webhook the database will reject the second attempt if the ID already exists. This built-in mechanism is far more reliable than manual checks in your application code.

1async function handleWebhook(event) {
2  const eventId = event.id;
3
4  try {
5    // Use a transaction to ensure both operations succeed or fail together
6    await db.transaction(async (trx) => {
7      // Attempt to insert the event ID to lock it
8      // This will throw an error if the ID already exists due to unique constraints
9      await trx('processed_events').insert({
10        event_id: eventId,
11        processed_at: new Date()
12      });
13
14      // Execute actual business logic (e.g., updating a subscription)
15      await trx('subscriptions')
16        .where('customer_id', event.customer_id)
17        .update({ status: 'active' });
18    });
19
20    return { status: 200, message: 'Processed' };
21  } catch (error) {
22    // Handle unique constraint violation specifically
23    if (error.code === 'ER_DUP_ENTRY') {
24      console.log('Duplicate event received, skipping logic');
25      return { status: 200, message: 'Already processed' };
26    }
27    throw error; // Rethrow other errors for retry logic
28  }
29}

As your application grows the idempotency table can become a bottleneck if not managed properly. Indexing the event ID column is mandatory to keep lookups fast as the table grows to millions of rows. You should also consider the storage costs of keeping event logs for long periods when most duplicates occur within the first twenty-four hours.

Implementing a background cleanup job to remove records older than thirty days is a standard practice. Most webhook providers only retry events for a few days so there is little value in keeping identifiers from six months ago. This keeps your indexes lean and ensures that your webhook endpoint remains responsive under heavy load.

1import redis
2import time
3
4cache = redis.Redis(host='localhost', port=6379)
5
6def process_event_with_lock(event_id, logic_fn):
7    lock_key = f"webhook_lock:{event_id}"
8    
9    # Try to acquire a lock that expires in 30 seconds
10    # nx=True ensures only one client can set the key
11    if cache.set(lock_key, "locked", nx=True, ex=30):
12        try:
13            # Verify the event hasn't been completed in the main DB
14            if not db.is_processed(event_id):
15                logic_fn()
16                db.mark_as_processed(event_id)
17            return "Success"
18        finally:
19            # Always release the lock after processing
20            cache.delete(lock_key)
21    else:
22        # Another worker is already processing this event
23        return "Retry Later"