The Webhook Ingestion Vault: How to Guarantee Zero Dropped Webhooks From Phone APIs

At 8:59 on a Monday morning a medical practice’s phone queue is empty. By 9:04 it is holding forty calls, and every one of them is firing ringing, answered, and completed events at your server inside the same minute. The handler that looked fine on Friday is now doing a database write, an owner lookup, and an outbound API call before it answers the provider. That is where dropped webhooks come from. Not from a crash, not from an outage, but from a handler that took longer to think than the provider was willing to wait.

I built the call operations dashboard that a multi-site medical practice watches its phones on, and the rule that keeps it honest is boring. The endpoint that receives a webhook is not allowed to think.

The short version: Telephony providers hold your endpoint to a short clock and a small retry budget. Twilio’s default connect timeout is 5,000 milliseconds with a default retry count of 1, and a hard 15-second ceiling applies to every call-related request. Telnyx retries when your app does not answer inside 2,000 milliseconds. RingCentral goes further and can blacklist the whole subscription. The fix is an ingestion buffer: verify the signature, write the raw body to one table keyed on the provider’s own event id, return 200, and do every piece of real work in a separate worker that claims rows with FOR UPDATE SKIP LOCKED. A burst then costs you queue depth for ninety seconds instead of a permanently missing call record.

What causes dropped webhooks during a call burst?

Two things, and neither one is your database falling over.

The first is arithmetic. Your handler’s worst-case latency has to fit inside the provider’s timeout, and under burst it will not. A write that takes 40 milliseconds when one call is in flight takes 400 when forty are, because forty connections are competing for the same pool. Add a lookup against a third-party API and you are past the window with no warning, because nothing in your logs says “we were too slow”. The request simply ends.

The second is that the retry budget is smaller than most people assume. Twilio’s connection override documentation sets the default connect timeout at 5,000 milliseconds and the default retry count at 1. You can raise the retry count to a maximum of 5 and stretch the read timeout to 15,000 milliseconds, but a hard 15-second upper limit still applies to every call-related HTTP request, and it exists to protect call quality rather than your pipeline. One slow response, one retry, then the event is gone.

Phone APIs are stricter than payment APIs here, and it catches people who learned webhooks from Stripe. Stripe retries a failed delivery for up to three days with exponential backoff in live mode. A telephony provider is streaming call lifecycle state in real time and has no interest in replaying it to you tomorrow.

How long does a phone API give your endpoint to respond?

Less time than your handler thinks, and the penalty for missing it varies wildly by provider.

Provider	Response window	Retry budget	What a failure costs you
Twilio	5,000 ms connect timeout by default, 15 s hard ceiling on call-related requests	1 by default, 5 maximum with overrides	The event, once the retries are spent
Telnyx	2,000 ms before a retry fires	Retries, then delivery shifts to your failover URL	The primary endpoint, after two consecutive Voice failures
RingCentral	Not published as a fixed number	Retries, then the subscription is suspended	The entire subscription, permanently
Stripe (for contrast)	Not published, stated as “quickly”	Up to three days of exponential backoff	Very little, if you recover the same week

RingCentral is the one that turns a slow morning into a silent week. Its webhook troubleshooting guide states that when an application fails to return a 200 response or suffers a prolonged outage, the platform flags the subscription as invalid and permanently suspends it. The subscription object then carries a blacklistedData block with a reason such as I/O operation is failed. Details: [Read timed out] and a blacklistedAt timestamp. Nothing gets delivered after that. The remedy is to delete the subscription and register a new one, which means your call data stops arriving until a human notices and re-subscribes.

That is the real cost of a slow endpoint. Not one lost event. Every event, until someone looks.

Telnyx documents the middle path: it retries when your application does not respond inside 2,000 milliseconds, and it supports a failover URL, with two consecutive failed Voice deliveries to the primary sending traffic to the backup. Useful, and still not a reason to be slow.

The vault: write it down before you understand it

The design is one table and one unique index. Call it the vault because that is what it does. It accepts custody of the raw payload and refuses to interpret it.

-- The vault. One table, one unique index, zero business logic.
create table webhook_events (
  id           bigserial   primary key,
  provider     text        not null,
  event_id     text        not null,   -- the provider's id, never one you generate
  payload      jsonb       not null,   -- the raw body, untouched
  received_at  timestamptz not null default now(),
  processed_at timestamptz,
  attempts     int         not null default 0,
  last_error   text
);

-- The whole guarantee lives on this line.
create unique index webhook_events_provider_event
  on webhook_events (provider, event_id);

The unique index on (provider, event_id) is what makes the design safe. Providers redeliver. Stripe says plainly that endpoints might receive the same event more than once and that it does not guarantee delivery in the order events were generated, and every telephony provider behaves the same way under retry. If a duplicate arriving twice can create a duplicate record, you do not have a buffer, you have a slower bug.

The handler that sits in front of it does three things and stops.

// Verify, write, answer. Anything else belongs in the worker.
export async function POST(req) {
  const raw = await req.text();

  // Signature first. An unverified payload never touches the table.
  if (!verifySignature(req.headers, raw)) {
    return new Response('bad signature', { status: 403 });
  }

  const body = JSON.parse(raw);

  await db.query(
    `insert into webhook_events (provider, event_id, payload)
       values ($1, $2, $3::jsonb)
     on conflict (provider, event_id) do nothing`,
    ['telephony', body.CallSid ?? body.id, raw],
  );

  return new Response('', { status: 200 });
}

on conflict do nothing means a redelivery is a no-op that still answers 200, which is exactly what the provider wants to hear. The whole path is a signature check, one insert, and a response. It runs in tens of milliseconds under load because there is nothing in it that can get slower.

Stripe’s own guidance says the same thing in one sentence: your endpoint must quickly return a successful 2xx status code prior to any complex logic that could cause a timeout. Phone APIs give you less room to ignore it.

How the worker drains the vault without double-processing

A queue that two workers can read is a queue that will process the same row twice. PostgreSQL solved this in version 9.5 with SKIP LOCKED, which lets a worker pass over rows another transaction already holds instead of waiting in line behind them.

-- One worker claims a batch. The others skip these rows instead of queuing.
with claimed as (
  select id
    from webhook_events
   where processed_at is null
     and attempts < 5
   order by received_at
   limit 20
   for update skip locked
)
update webhook_events e
   set attempts = e.attempts + 1
  from claimed c
 where e.id = c.id
returning e.id, e.payload;

Run four of these and they divide the backlog between them with no coordination, no broker, and no extra service to pay for. The worker parses the payload, applies the business logic, writes the real call record, and stamps processed_at. A failure increments attempts and leaves last_error behind, so the row stays visible instead of vanishing into a log.

Two details earn their keep here. The attempts < 5 bound stops one poisoned payload from being retried forever. Keeping processed_at null until the work is genuinely finished means a worker that dies mid-batch releases its locks on rollback and another worker picks the rows up on the next pass.

You do not need Redis for this. You do not need a hosted queue. A table you already have, an index you already understand, and a cron that runs every few seconds will absorb a burst that would have killed the inline handler.

What about the events that never arrive at all?

The vault protects you from being too slow. It does not protect you from a provider that never sent the event, or sent it during a window when your domain’s certificate had expired. Buffering is not reconciliation, and treating it as reconciliation is how a dashboard ends up confidently wrong.

The answer is the same one I use everywhere a live feed drives a screen. The feed writes fast, and a scheduled job compares what you hold against what the provider’s own record says, then corrects the difference. A live event is a partial snapshot, so the first signal to arrive is often the wrong one and the settled record has to win. Pair that with a dashboard that says when its data cannot be trusted rather than showing a calm morning that never happened.

Buffer for the burst. Reconcile for the gap. They are two different problems and each one needs its own machinery.

What to check before you call it done

• Time the p99, not the average. Your handler’s slowest response under concurrency is the number that has to clear the provider’s timeout. The average tells you nothing about the Monday morning.
• Key on the provider’s id, never your own. A UUID you generate on arrival is unique per request, which makes every redelivery a fresh row. The call id is the only value that identifies the event itself.
• Verify the signature before the insert. The vault should hold real events only, otherwise you have built an unauthenticated write endpoint into your database.
• Alarm on queue depth and on age, separately. A hundred unprocessed rows during a burst is healthy. One row unprocessed for twenty minutes is a stuck worker.
• Alarm on silence too. Zero events for an hour during business hours is the shape a suspended subscription makes, and it is invisible to every check that only watches for errors.
• Keep the raw payload forever, or at least for a month. When a provider changes a field, replaying the originals through the fixed parser costs an afternoon. Without them it costs the data.

The endpoint that thinks is the endpoint that drops

Most webhook handlers are written the obvious way, doing the real work at the moment the event lands, and they pass every test because a test never sends forty calls in five minutes. Then the busiest morning of the quarter arrives and the provider’s clock, not your code, decides what gets recorded.

Put a vault in front of it. The handler stops being the place where work happens and becomes the place where custody is taken, and a burst turns into a number on a queue-depth chart instead of a patient who called and left no trace. The same idempotency spine carries a field app that has been offline for six hours and needs to replay its queue in the right order when signal comes back.

Write it down first. Understand it second.

Frequently asked questions

What causes dropped webhooks during high call volume?

Almost always a handler that does its real work before answering the provider. Under burst, database writes and third-party calls slow down together, the response misses the provider’s timeout, and the retry budget runs out. Twilio’s default connect timeout is 5,000 milliseconds with a default retry count of 1, so a handler that gets slow twice has already lost the event. The crash you are looking for in the logs never happened.

How long does Twilio give a webhook endpoint to respond?

By default, 5,000 milliseconds to connect, with a default retry count of 1. Connection overrides let you raise the connect timeout to 10,000 milliseconds, the read timeout and total time to 15,000 milliseconds, and the retry count to 5. A hard 15-second upper limit applies to all call-related HTTP requests regardless of what you configure, because Twilio holds that ceiling to protect call quality.

Should I use a message queue or a database table to buffer webhooks?

Start with a database table. A single table with a unique index on the provider’s event id, plus workers claiming rows with FOR UPDATE SKIP LOCKED, handles the burst volume a phone system produces without adding a broker to operate, pay for, and monitor. Move to a dedicated queue when you need fan-out to several independent consumers or throughput beyond what one PostgreSQL instance is comfortable with, not before.

How do I stop duplicate webhook events from creating duplicate records?

Put a unique index on the provider’s own event id and insert with on conflict do nothing. Providers redeliver events on retry and Stripe documents outright that an endpoint might receive the same event more than once, so duplicates are normal traffic rather than an edge case. The database rejecting the second copy is stronger than application code checking first, because the check cannot be forgotten by the next code path someone adds.

What happens if my webhook endpoint is down for an hour?

It depends entirely on the provider, and telephony is the harsh end. Stripe keeps retrying for up to three days. Telnyx shifts delivery to a failover URL if you configured one. RingCentral flags the subscription as invalid after a prolonged outage and permanently suspends it, recording the reason and timestamp in a blacklistedData block, after which nothing arrives until you delete the subscription and register a new one. That is why an alarm on unexpected silence matters as much as an alarm on errors.