Design a Webhook System

A webhook system receives real-time event notifications from external services (Stripe, Shopify, GitHub) and processes them reliably. Unlike polling-based APIs where you repeatedly ask "did anything change?", webhooks push data immediately when events occur—making them efficient for real-time integrations.

This walkthrough follows the Interview Framework and focuses on what you'd actually present in a 45-60 minute interview.

Phase 1: Requirements (~5 minutes)

Functional Requirements

1. Receive event notifications — Accept HTTP POST requests from external services (e.g., payment confirmed, order shipped)
2. Process events reliably — Execute business logic and persist results for tracking and auditing
3. Handle retries gracefully — External services retry failed deliveries; our system must handle duplicates

Out of Scope

• Managing downstream state changes triggered by webhooks (that's application-specific business logic)
• Building a webhook delivery system (sending webhooks to customers)

Non-Functional Requirements

Capacity Estimation

Throughput:

1M events/day ÷ 86,400 seconds ≈ 12 events/second (average)
Peak (5x): ~60 events/second

Storage (30-day retention):

1M events/day × 5KB per event × 30 days = 150 GB

Phase 2: Data Model (~5 minutes)

Core Entities

WebhookEvent
├── id (UUID, PK)
├── idempotency_key (string, unique) — "source:event_id" for deduplication
├── source (string) — "stripe", "shopify", "github"
├── event_type (string) — "payment.succeeded", "order.created"
├── payload (JSONB) — Raw event data
├── status: processing | completed | failed
├── attempts (int) — Processing attempt count
├── error_message (string, nullable)
├── processed_at (timestamp, nullable)
├── created_at (timestamp)
└── updated_at (timestamp) — For detecting stale processing

WebhookSource
├── id (PK)
├── name (string) — "stripe", "shopify"
├── signing_secret (encrypted) — For HMAC verification
├── ip_allowlist (array) — Optional IP restrictions
└── is_active (boolean)

Phase 3: API Design (~5 minutes)

Receive Webhook

POST /webhooks/{source}
Headers:
  Content-Type: application/json
  Stripe-Signature: t=1234567890,v1=abc123...  (varies by provider)

Body:
{
  "id": "evt_1234",
  "type": "payment_intent.succeeded",
  "data": {
    "object": {
      "id": "pi_abc123",
      "amount": 2000,
      "currency": "usd"
    }
  }
}

Response: 200 OK
{
  "status": "accepted"
}

Why return 200 immediately?

External services like Stripe have strict timeout policies (typically 5-20 seconds). If we don't respond quickly, they'll:

1. Consider the delivery failed
2. Retry (causing duplicates)
3. Eventually disable our webhook endpoint

We acknowledge receipt immediately, then process asynchronously.

Get Event Status (Internal API)

GET /api/webhook-events/{id}

Response:
{
  "id": "evt_1234",
  "source": "stripe",
  "event_type": "payment_intent.succeeded",
  "status": "completed",
  "processed_at": "2024-01-15T10:30:00Z"
}

Phase 4: High-Level Design (~15-25 minutes)

Why Not Process Synchronously?

Let's start with the naive approach and understand why it fails:

Problems:

1. If the handler crashes after processing but before responding, the provider retries and we double-process
2. If the database is slow, we timeout and the provider retries
3. Handler does too much: HTTP handling + business logic + persistence
4. No buffering during traffic spikes

Adding a Message Queue

Benefits:

• Faster acknowledgment: Handler only validates and enqueues, returns 200 quickly
• Reliability: Queue persists events even if consumers are down
• Scalability: Add more consumers during peak load
• Failure isolation: Consumer crashes don't affect request handling

Complete Architecture

Request Handler Flow

def handle_webhook(request, source):
    # 1. Verify signature (security)
    if not verify_signature(request, source):
        return Response(status=401)

    # 2. Parse and validate payload
    event = parse_event(request.body)

    # 3. Quick deduplication check (Redis)
    if redis.exists(f"webhook:{source}:{event.id}"):
        return Response({"status": "duplicate"}, status=200)

    # 4. Enqueue for processing
    try:
        queue.publish({
            "source": source,
            "event_id": event.id,
            "event_type": event.type,
            "payload": event.data,
            "signature": request.headers.get("signature"),
            "received_at": now()
        })
    except QueueUnavailable:
        return Response(status=503)  # Provider will retry

    # 5. Mark as seen (short TTL for dedup window)
    redis.setex(f"webhook:{source}:{event.id}", 3600, "1")

    # 6. Return 200 immediately
    return Response({"status": "accepted"}, status=200)

Consumer Flow

MAX_RETRIES = 5
PROCESSING_TIMEOUT = timedelta(minutes=5)
RETRY_DELAYS = [1, 5, 30, 120, 600]  # seconds - exponential backoff

def process_webhook(message):
    event = message.payload
    idempotency_key = f"{event.source}:{event.event_id}"

    # 1. Try to claim this event atomically
    db_event, action = claim_event(idempotency_key, event)

    if action == "ack":
        message.ack()      # Already completed
        return
    elif action == "nack":
        message.nack()     # Retry later
        return
    elif action == "dlq":
        message.send_to_dlq()
        message.ack()      # Remove from main queue
        return

    # 2. We claimed it - process the event
    try:
        execute_business_logic(event)

        db.update("""
            UPDATE webhook_events SET status = 'completed', processed_at = NOW()
            WHERE id = ?
        """, db_event.id)
        message.ack()

    except Exception as e:
        db.update("""
            UPDATE webhook_events SET status = 'failed', error_message = ?
            WHERE id = ?
        """, str(e), db_event.id)

        if db_event.attempts >= MAX_RETRIES:
            message.send_to_dlq()
            message.ack()  # Remove from main queue
        else:
            # Exponential backoff before retry
            delay = RETRY_DELAYS[min(db_event.attempts - 1, len(RETRY_DELAYS) - 1)]
            message.nack(requeue_delay=delay)


 ():
    
    
    result = db.execute(, idempotency_key, event.source, event.event_type, event.payload)

     result:
         (result, )  

    
    existing = db.query(, idempotency_key)

     existing.status == :
         (, )

     existing.status == :
        
         existing.updated_at < now() - PROCESSING_TIMEOUT:
            
            claimed = db.execute(, existing.)
             claimed:
                 (claimed, )
         (, )  

    
     existing.attempts >= MAX_RETRIES:
         (, )

    
    claimed = db.execute(, existing.)

     claimed:
         (claimed, )  

     (, )  

Phase 5: Deep Dives (~15-20 minutes)

1. Security: Verifying Webhook Authenticity

External services sign their webhooks using HMAC. Without verification, attackers could send fake events to trigger unintended actions (e.g., fake "payment succeeded" events).

How HMAC Signing Works:

def verify_stripe_signature(payload, signature_header, secret):
    # Use the raw request body bytes (Stripe requires exact payload)
    # Parse header: "t=1234567890,v1=abc123..."
    # Use split("=", 1) to handle base64 signatures containing "="
    parts = dict(p.split("=", 1) for p in signature_header.split(","))
    timestamp = parts["t"]
    expected_sig = parts["v1"]

    # Recreate the signed payload
    signed_payload = f"{timestamp}.{payload}"

    # Calculate our signature
    computed_sig = hmac.new(
        secret.encode(),
        signed_payload.encode(),
        hashlib.sha256
    ).hexdigest()

    # Constant-time comparison (prevents timing attacks)
    return hmac.compare_digest(computed_sig, expected_sig)

Additional Security Layers:

2. Handling Duplicate Events

Duplicates are inevitable in webhook systems:

• Provider retries after timeout
• Network issues cause retransmission
• Provider's own retry logic

Two-Layer Deduplication:

CREATE TABLE webhook_events (
    id UUID PRIMARY KEY,
    idempotency_key VARCHAR(255) UNIQUE NOT NULL,  -- source:event_id
    -- ... other fields
);

The unique constraint on idempotency_key ensures that even if Redis misses a duplicate, the database will reject the insert.

3. Handling Out-of-Order Events

Events may arrive out of order due to network delays or provider retry behavior. For example, Stripe might send:

1. invoice.paid (arrives first)
2. invoice.created (arrives second)

Strategies:

Option 1: Fetch Latest State from Source

Don't rely on event data alone—fetch the current state from the provider's API:

def process_stripe_invoice_event(event):
    invoice_id = event.data.object.id

    # Always fetch latest state from Stripe
    current_invoice = stripe.Invoice.retrieve(invoice_id)

    # Update our database with current state
    db.upsert("""
        INSERT INTO invoices (stripe_id, status, amount, updated_at)
        VALUES (?, ?, ?, ?)
        ON CONFLICT (stripe_id) DO UPDATE SET
            status = EXCLUDED.status,
            amount = EXCLUDED.amount,
            updated_at = EXCLUDED.updated_at
        WHERE invoices.updated_at < EXCLUDED.updated_at
    """, invoice_id, current_invoice.status, current_invoice.amount, current_invoice.updated)

Option 2: Use Event Timestamps

Compare timestamps and skip stale events:

def process_event(event):
    existing = db.query("SELECT * FROM orders WHERE id = ?", event.order_id)

    if existing and existing.updated_at >= event.timestamp:
        # Event is stale, skip processing
        return

    # Process event...

4. Failure Handling at Each Layer

Request Handler Failures:

Circuit Breakers for Dependencies:

If the message queue becomes slow or unavailable, the handler shouldn't hang indefinitely. Use a circuit breaker:

@circuit_breaker(failure_threshold=5, recovery_timeout=30)
def enqueue_event(event):
    queue.publish(event, timeout=500)  # 500ms timeout

When the circuit is "open" (queue consistently failing), immediately return 503 Service Unavailable. The provider will retry later when the queue recovers.

Message Queue Failures:

Consumer Failures:

Database Failures:

5. Dead Letter Queue (DLQ) Strategy

After multiple failed attempts, move messages to a DLQ rather than retrying forever. The consumer code handles this with exponential backoff:

RETRY_DELAYS = [1, 5, 30, 120, 600]  # seconds

# After processing fails:
if db_event.attempts >= MAX_RETRIES:
    message.send_to_dlq()  # Stop retrying, needs human intervention
    message.ack()          # Remove from main queue
else:
    delay = RETRY_DELAYS[min(db_event.attempts - 1, len(RETRY_DELAYS) - 1)]
    message.nack(requeue_delay=delay)  # Retry after delay

Why exponential backoff? If a downstream service is failing, hammering it with retries makes things worse. Increasing delays (1s → 5s → 30s → 2m → 10m) give the system time to recover.

DLQ Handling:

• Alert on-call when messages arrive in DLQ
• Provide tooling to inspect and replay DLQ messages
• Track DLQ depth as a health metric
• Investigate root cause before replaying (don't just blindly retry)

6. Monitoring & Observability

Common Pitfalls

Interview Checklist

Summary

The webhook receiver is a great interview question because it appears simple but reveals depth in reliability engineering. The key insight is that webhooks are inherently unreliable (network failures, timeouts, retries), so your system must be defensive: persist before acknowledging, deduplicate aggressively, and never assume events arrive in order.