Deduplicating Webhook Retries with Idempotency Keys

This guide is part of the Webhook Event Ingestion section within the broader Data Ingestion & Grant Parsing Workflows framework, and it solves one narrow problem: a funder portal re-delivers the same webhook on every non-2xx response or timeout, so your handler sees one award notification three, five, or a dozen times — how do you process it exactly once without dropping the genuinely new events that look similar?

Grant portals treat webhook delivery as at-least-once. If your endpoint is slow, returns a 500, or the TCP connection drops after you committed but before the ACK flushed, the portal assumes failure and queues a redelivery. Point a naive handler at that stream and every retry re-inserts a disbursement row, re-fires an email, or re-advances a workflow. This guide builds a deduplication gate that claims a stable idempotency key before doing any work, acknowledges duplicates fast so the retries stop, and still lets a legitimate new state change for the same grant through.

When to Use This Approach #

Reach for a store-first idempotency gate whenever all three of these hold:

• The portal retries on non-2xx or timeout. Grants.gov, Fluxx, Submittable, and most foundation portals follow at-least-once delivery: any response outside 2xx, or no response inside their timeout, schedules a redelivery with backoff. If you cannot guarantee your handler is the only thing that ever writes the resulting state, deduplication has to live in the handler, because the portal will not stop until it sees a 200.
• Processing has side effects that must not repeat. Advancing a grant to “awarded”, inserting a disbursement, or emitting a notification are effects you can never take twice. Deduplication here is an internal control over the accuracy of financial records, which 2 CFR §200.303 requires a recipient to establish and maintain; a double-posted award is a control failure, not a cosmetic bug.
• The same subject legitimately changes state more than once. A grant fires application.submitted, then application.approved, then award.disbursed — three real events about one grant id. A dedupe scheme that keys on the grant id would swallow the second and third. You need a key that separates a redelivery of one event from a new event about the same subject.

Signature verification, payload schema validation, and downstream fan-out are explicitly out of scope here. Authenticating that a webhook actually came from the funder belongs to the sibling guide Verifying HMAC Signatures on Grant Portal Webhooks and must run before this gate — never deduplicate an unauthenticated body. Durable re-processing of the claimed event, once accepted, is handled by Async Batch Processing Pipelines. This stage does exactly one thing: decide whether the request in front of it has been seen.

Step-by-Step Implementation #

The reference implementation targets Python 3.10+ with redis for the shared claim store. Redis is the natural home for the idempotency set because SET ... NX EX is a single atomic round trip that both claims the key and stamps its expiry — the two operations you need to be indivisible.

pip install "redis==5.0.7"

One contract to internalize: the claim happens before the work, not after. If you process first and record the key second, a crash in between leaves the effect applied with no key stored, and the next retry re-applies it. Store-first inverts that: you lose the ability to process only when you have already durably claimed the key.

Step 1: Derive a stable idempotency key #

The provider’s own event id is the best key because the provider guarantees it is stable across redeliveries of the same event. Read it from the payload (or a header like X-Event-Id). When a portal omits one — some legacy foundation webhooks do — fall back to a SHA-256 digest of the canonical payload: keys sorted, whitespace collapsed, so byte-identical redeliveries hash identically.

import hashlib
import json
import logging
from typing import Any, Mapping

logger = logging.getLogger("webhook.dedup")


def derive_idempotency_key(payload: Mapping[str, Any], headers: Mapping[str, str]) -> str:
    """Return a stable dedup key: provider event id, else a canonical payload digest."""
    event_id = payload.get("id") or headers.get("X-Event-Id")
    if isinstance(event_id, str) and event_id.strip():
        return f"evt:{event_id.strip()}"

    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
    digest = hashlib.sha256(canonical.encode("utf-8")).hexdigest()
    logger.warning("No provider event id; falling back to payload digest %s", digest[:12])
    return f"dig:{digest}"

The evt:/dig: prefix records which strategy produced the key, so an auditor reading the store can tell a provider-supplied id from a computed digest. Sorting keys is what makes the digest deterministic — without sort_keys=True, a re-serialized payload can reorder fields and defeat the match.

Step 2: Claim the key store-first with SETNX and a TTL #

Claim the key with a single SET key value NX EX ttl. NX makes the write succeed only if the key is absent, so exactly one concurrent request wins the claim; EX sets the expiry in the same atomic call. Size the TTL to the provider’s retry horizon — how long it keeps retrying a failed delivery — plus a margin. If Grants.gov retries for 72 hours, a 96-hour TTL guarantees the key still exists when the last retry lands, while letting it expire afterward so the store does not grow without bound.

import redis

RETRY_HORIZON_SECONDS: int = 96 * 60 * 60  # provider retries for ~72h; hold longer


class IdempotencyStore:
    """Store-first claim gate backed by Redis SET NX EX."""

    def __init__(self, client: redis.Redis, ttl_seconds: int = RETRY_HORIZON_SECONDS) -> None:
        self._client = client
        self._ttl = ttl_seconds

    def claim(self, key: str, event_ts: str) -> bool:
        """Attempt to claim key. Return True if newly claimed, False if already seen."""
        try:
            was_set = self._client.set(name=key, value=event_ts, nx=True, ex=self._ttl)
        except redis.RedisError as exc:
            logger.error("Claim store unavailable for %s: %s", key, exc)
            raise RuntimeError("idempotency store unavailable") from exc
        claimed = bool(was_set)
        logger.info("Claim %s -> %s", key, "NEW" if claimed else "DUPLICATE")
        return claimed

Note the explicit raise on RedisError: if the claim store is unreachable you must not silently process, because you can no longer prove the event is new. Surface it as a RuntimeError and return a 5xx so the provider retries later — a retry is safe, a blind double-process is not.

Step 3: Acknowledge duplicates with an immediate 200 #

When claim returns False, the event has already been processed (or is in flight under another worker that won the claim). Do no work and return 200 at once. The 200 is not cosmetic — it is the signal that tells the portal to stop retrying. A duplicate that you answer with a 500 will simply come back again.

from http import HTTPStatus
from typing import Callable, Tuple

Payload = Mapping[str, Any]


def handle_webhook(
    payload: Payload,
    headers: Mapping[str, str],
    store: IdempotencyStore,
    process: Callable[[Payload], None],
) -> Tuple[int, str]:
    """Deduplicate then process. Every branch returns a 2xx so retries stop."""
    key = derive_idempotency_key(payload, headers)
    event_ts = str(payload.get("occurred_at", ""))

    if not store.claim(key, event_ts):
        logger.info("Duplicate %s acked without side effects", key)
        return HTTPStatus.OK, "duplicate ignored"

    try:
        process(payload)
    except Exception as exc:
        logger.error("Processing failed for %s: %s", key, exc)
        return HTTPStatus.INTERNAL_SERVER_ERROR, "processing failed; safe to retry"
    return HTTPStatus.OK, "processed"

The 500 path deliberately leaves the claimed key in place. Because the same event id produces the same key, the provider’s retry re-enters claim, finds the key present, and is acked as a duplicate — so a transient processing failure does not strand the event, and the durable retry is delegated to Async Batch Processing Pipelines rather than the funder’s redelivery loop.

Step 4: Separate a true retry from a new state change #

The key insight of the whole scheme: dedupe on the event id, never the subject id. A redelivery of award.disbursed carries the same event id and is correctly suppressed. A brand-new award.amended for the same grant carries a different event id, wins its own claim, and is processed. Keying on the grant id would collapse both into one and lose the amendment.

def classify(payload: Payload, headers: Mapping[str, str], store: IdempotencyStore) -> str:
    """Return 'retry' for a repeat event id, 'new_change' for a new event on a known subject."""
    key = derive_idempotency_key(payload, headers)
    subject_id = str(payload.get("grant_id", "unknown"))

    if not store.claim(key, str(payload.get("occurred_at", ""))):
        logger.info("Retry of %s for grant %s", key, subject_id)
        return "retry"

    logger.info("New state change %s for grant %s (%s)", key, subject_id, payload.get("type"))
    return "new_change"

Because the claim is per event id, two distinct events about one grant never contend for the same key, and the grant_id travels through only as audit context — it is never part of the dedup decision.

Step 5: Sequence out-of-order deliveries on the provider timestamp #

At-least-once delivery is also out-of-order delivery: a redelivered submitted can land after the approved that superseded it. Deduplication alone would happily apply the stale event because it has a genuinely new event id. Guard state transitions by recording the highest provider timestamp applied per subject and refusing any event that is not strictly newer.

from datetime import datetime


class SequenceGuard:
    """Reject events older than the last applied state for a given subject."""

    def __init__(self, client: redis.Redis) -> None:
        self._client = client

    def apply_if_newer(self, subject_id: str, occurred_at: str) -> bool:
        """Return True if this event advances the subject; False if it is stale."""
        try:
            incoming = datetime.fromisoformat(occurred_at).timestamp()
        except ValueError as exc:
            logger.error("Unparseable timestamp %r for %s: %s", occurred_at, subject_id, exc)
            raise ValueError(f"invalid occurred_at: {occurred_at!r}") from exc

        watermark_key = f"seq:{subject_id}"
        stored = self._client.get(watermark_key)
        if stored is not None and incoming <= float(stored):
            logger.info("Stale event for %s (%.0f <= %s) dropped", subject_id, incoming, stored)
            return False

        self._client.set(watermark_key, incoming)
        return True

Sequence on the provider’s occurred_at, never your own receive time, because receive order is exactly what redelivery scrambles. The watermark makes the apply step monotonic: a stale replay is dropped even though it passed the dedup gate as a new event id.

Verification #

Confirm the gate behaves correctly with four checks:

1. A redelivery is suppressed. Call handle_webhook twice with the identical payload and assert the first returns (200, "processed") and the second returns (200, "duplicate ignored"), and that process ran exactly once. This proves the store-first claim collapses retries.
2. A missing event id still deduplicates. Strip the id field and assert both deliveries derive the same dig: key from the canonical digest, so the second is still caught.
3. A new event on the same grant is processed. Send two payloads with the same grant_id but different id values and assert both win their claim and both call process — the amendment is not swallowed.
4. A stale replay is dropped by the sequence guard. Apply an approved event, then feed an older submitted for the same grant and assert apply_if_newer returns False.

A processed event emits one INFO claim line reading NEW; a duplicate emits DUPLICATE; a stale replay emits a drop line. Ship those to a write-once tier so the dedup decision is itself part of the internal-control record under 2 CFR §200.303.

seen: list[str] = []
store = IdempotencyStore(redis.Redis(), ttl_seconds=3600)
event = {"id": "evt_918", "grant_id": "G-42", "type": "award.disbursed", "occurred_at": "2026-07-09T14:00:00"}

assert handle_webhook(event, {}, store, seen.append) == (HTTPStatus.OK, "processed")
assert handle_webhook(event, {}, store, seen.append) == (HTTPStatus.OK, "duplicate ignored")
assert len(seen) == 1  # side effect applied exactly once

Common Errors & Fixes #

Related #

• Parent section: Webhook Event Ingestion
• Authenticate the webhook before you deduplicate it: Verifying HMAC Signatures on Grant Portal Webhooks
• Where the claimed event is durably processed and retried: Async Batch Processing Pipelines
• The same idempotency-key idea applied to internal pipeline stages: Exactly-Once Processing with Idempotency Keys in Grant Pipelines