Stop asking Claude to reinvent Stripe billing.

.udria/production-stripe-billing/spec/webhooks.md

# Webhook ingestion

> Excerpt from `.udria/production-stripe-billing/spec/webhooks.md`.

Stripe verifies the signature. Everything after that is yours, and it is where
the bugs live. This section specifies ingestion; entitlement resolution is
specified in `entitlements.md`, and event ordering in
`idempotency-and-ordering.md`.

## Invariants

The agent MUST satisfy all five before this section is considered implemented.

1. **Signature.** Reject any request whose signature fails `constructEvent`,
   with 400, before the body is parsed or persisted.
2. **Idempotency.** Processing the same `event.id` twice MUST leave the
   database byte-identical to processing it once.
3. **Ordering.** Stripe does not guarantee delivery order. Applying an older
   event after a newer one MUST NOT move billing state backwards.
4. **Durability.** An event is acknowledged only once its business effect is
   committed. An event that has been *received* but not *processed* MUST NOT
   be mistaken for one that has been processed.
5. **Reconciliation.** Where local state and Stripe disagree, Stripe wins, and
   the disagreement is recorded rather than silently overwritten.

## Seen is not processed

Invariant 4 is the one that is usually implemented wrongly, and the wrong
implementation is the natural one:

```
insert into billing_events (id, ...) values (...) on conflict (id) do nothing
-- zero rows means we have seen this event; acknowledge with 200 and stop
```

Trace a transient failure through it. The first delivery inserts the row. The
business effect then fails — a pool timeout, a failover, a deadlock. The
endpoint correctly returns non-2xx, so Stripe retries. The retry inserts,
conflicts, gets zero rows, concludes "duplicate", and returns 200. Stripe is
now satisfied and stops retrying, and the event has been processed by nobody.
A subscription is never activated, a customer is never granted what they paid
for, and no error is raised anywhere, because every individual component did
what it was told.

The ledger therefore records two different facts, separately:

```text
received   — this event id has arrived at least once
processed  — this event's business effect is committed
```

A duplicate delivery is only a no-op when the second fact is true.

## Ledger table

```sql
create table billing_events (
  id                    text primary key,          -- Stripe's event.id
  type                  text        not null,
  status                text        not null default 'processing',
  received_at           timestamptz not null default now(),
  processing_started_at timestamptz not null default now(),
  processed_at          timestamptz,
  attempt_count         integer     not null default 1,
  last_error            text,
  last_error_at         timestamptz,
  constraint billing_events_status_check
    check (status in ('processing', 'processed', 'failed_permanent'))
);
```

Three states, and deliberately not more:

| `status` | Meaning |
| --- | --- |
| `processing` | Claimed by an instance. The business effect is not committed. |
| `processed` | The business effect is committed. `processed_at` is set. |
| `failed_permanent` | Retrying cannot help. Terminal, with the reason recorded. |

There is no separate `failed_recoverable`. A handler that failed recoverably
and a handler whose process was killed leave the database in the same
condition, and giving them one state gives them one recovery path instead of
two that have to be kept in step.

## Claiming an event

A claim is a **lease**, not a flag. One statement, and it is atomic:

```sql
insert into billing_events (id, type, status, attempt_count, processing_started_at)
values ($1, $2, 'processing', 1, now())
on conflict (id) do update
   set status                = 'processing',
       attempt_count         = billing_events.attempt_count + 1,
       processing_started_at = now(),
       last_error            = null
 where billing_events.status = 'processing'
   and billing_events.processing_started_at < now() - interval '3 minutes'
returning attempt_count
```

`on conflict do update` takes a row lock, so two deliveries arriving together
serialise on it: the loser re-evaluates its `where` against the winner's
committed row, sees a fresh lease, and matches nothing. That is a database
guarantee, so it holds across every instance you are running. An in-memory
mutex does not, and MUST NOT be used for this.

One row back means the event is yours to process. Zero rows means one of three
things, and the difference decides the response:

| Ledger state | Response | Why |
| --- | --- | --- |
| `processed` | 200 | The genuine duplicate. Do no further work. |
| `failed_permanent` | 200 | Retrying cannot repair it. Stripe must stop. |
| `processing`, lease live | **non-2xx** | Another instance has it. Let Stripe redeliver. |

The lease interval MUST exceed your maximum handler duration. Set it shorter
and a slow-but-healthy handler has its event stolen, and the business effect
runs twice in parallel — the failure this table exists to prevent.

## Completing an event

```text
1. Verify the Stripe signature.
2. Atomically claim the event id, as above.
3. If it is already processed, or terminally failed, return 200 and stop.
4. If another instance holds a live lease, return non-2xx and stop.
5. Apply the business effect, idempotently (see below).
6. Only now mark processed and set processed_at.
7. Return 200.
```

Step 6 is the only write on which correctness depends, and it is on the happy
path. Everything else may fail and self-heal.

**Do not compensate by deleting the claim row.** It is the tempting fix — the
handler failed, so remove the claim and let the retry re-claim it — and it
puts correctness in the hands of the failure path. The most common reason a
handler fails is that the database is unreachable, which is also the reason
the compensating delete will fail, and then the row stays behind unprocessed
and every retry is rejected as a duplicate. You have reintroduced the original
bug by a longer route. Let the lease expire instead: doing nothing is already
correct.

## Failure handling

Return non-2xx **only** when a retry could succeed — a database outage, a
timeout, a rate limit. On that path, record the error and set
`processing_started_at` into the past so the next delivery can reclaim
immediately rather than waiting the lease out. This write is best-effort: if
it fails, the lease expires and recovery happens anyway, just later.

Return 2xx and set `failed_permanent` for anything a retry cannot fix — a
malformed payload, an event about an object that does not exist locally and
never will. Classify conservatively: treating a transient fault as permanent
loses the event, while treating a permanent fault as transient costs three days
of retries and leaves a row you can see. Default to recoverable.

`attempt_count >= 5` MUST raise an operational alert. Deciding what "alert"
means is the host application's choice; the specification only requires that
silence is not an option.

## The ledger is not the only defence

Transport-level deduplication tells you an event arrived twice. It cannot tell
you a handler crashed after writing and before acknowledging, and no amount of
ledger design will, because the business write and the ledger write cannot be
made atomic once a Stripe API call or an email send sits between them.

So every business effect MUST be independently idempotent, keyed on a stable
Stripe identifier:

| Effect | What makes a second run a no-op |
| --- | --- |
| Link customer to account | `unique (stripe_customer_id)` |
| Create local subscription | `unique (stripe_subscription_id)` |
| Record an invoice | `unique (stripe_invoice_id)` |
| Grant seats | Set the absolute quantity; never increment |
| Append a lifecycle row | `unique (stripe_event_id, subscription_id)` |
| Send a customer email | Claim a `unique (subscription_id, kind)` row first |

Written this way, replaying an event is safe whether or not the ledger
remembers it — which is the only defence that survives a process dying at the
worst possible moment.

## Ordering

Stripe does not deliver in creation order, and `event.created` has one-second
resolution — two subscription updates in the same second are common on a plan
change with proration. So:

**Do not** write `if (event.created <= last_event_at) discard`. With
second-resolution timestamps that discards legitimate newer state, and the
symptom is a customer on the wrong plan with nothing in the logs.

Nor is a status guard sufficient on its own. `customer.subscription.updated`
carrying `status: canceled` is a legitimate transition out of `active` and a
stale redelivery of a cancellation that was already reversed — the same event
shape, the same local status, opposite correct outcomes. No rule that looks
only at the event and the current row can separate them.

Order per object, never globally, and pick the weakest mechanism that is
actually sufficient for the transition in front of you:

1. **Re-fetch, for anything with a lifecycle.** For subscription- and
   invoice-shaped events, treat the event as a notification that something
   changed, retrieve the object from Stripe, and apply what comes back.
   Delivery order then stops mattering: whichever event you process last, you
   converge on Stripe's current state, which is the state that is true.
   This is the default for every event in the V1 table below.
2. **A status guard, for transitions that are genuinely one-way.** Where a
   local status can never be legitimately re-entered — a reservation that has
   been charged, an invoice already marked void — name the statuses the
   transition may move *from* and make that the `where` clause. Exact, cheap,
   and no extra round trip. It is only safe because the transition is
   irreversible; do not reach for it otherwise.
3. **A stored object version, where Stripe gives you one.** Some objects carry
   a monotonically increasing marker you can persist on the row and compare.
   Where one exists it is cheaper than a re-fetch and just as ordered.

Record every discarded or superseded event with its reason. A discard you
cannot see later is indistinguishable from a bug.

## Events in scope for V1

| Event | Effect |
| --- | --- |
| `checkout.session.completed` | Link the Stripe customer to the local account |
| `customer.subscription.created` | Create local subscription, grant entitlements |
| `customer.subscription.updated` | Re-resolve plan, period, cancellation and seat allowance |
| `customer.subscription.deleted` | End entitlements at `ended_at`, which is not always now |
| `invoice.paid` | Clear any dunning state, extend the period |
| `invoice.payment_failed` | Enter dunning; see `subscription-lifecycle.md` |

`customer.subscription.deleted` fires in two different situations and they do
not end access at the same moment: a cancellation scheduled with
`cancel_at_period_end` has already served the period the customer paid for,
while an immediate cancellation or a failed final payment has not. Read
`ended_at` from the object rather than assuming either. Keep the subscription
row and its history; end the entitlement, do not delete the record.

Events outside this table MUST be recorded in the ledger and ignored. Handling
an event the specification does not cover is out of scope, not a bonus.

.udria/production-stripe-billing/acceptance/tests/webhook-idempotency.spec.ts

// Excerpt from .udria/production-stripe-billing/acceptance/tests/
//   webhook-idempotency.spec.ts
//
// Ships as a real file. Your agent wires `harness` to your schema, then runs
// the suite before it reports the implementation as finished.

import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { harness, type Harness } from "./_harness";

describe("webhook ingestion", () => {
  let h: Harness;

  beforeEach(async () => {
    h = await harness();
    await h.seed.organisation({ id: "org_1", plan: "free" });
    await h.seed.stripeCustomer({ id: "cus_1", organisationId: "org_1" });
  });

  afterEach(() => h.reset());

  it("grants entitlement on a new subscription", async () => {
    await h.post(
      h.event("customer.subscription.created", {
        id: "sub_1",
        customer: "cus_1",
        status: "active",
        items: [{ price: h.prices.pro, quantity: 3 }],
      }),
    );

    expect(await h.entitlements("org_1")).toMatchObject({ plan: "pro", seats: 3 });
  });

  it("has no duplicate effect when the same event is delivered twice", async () => {
    const event = h.event("customer.subscription.created", {
      id: "sub_1",
      customer: "cus_1",
      status: "active",
      items: [{ price: h.prices.pro, quantity: 3 }],
    });

    await h.post(event);
    const afterFirst = await h.snapshot();

    const second = await h.post(event);

    expect(second.status).toBe(200);
    expect(await h.snapshot()).toEqual(afterFirst);
    expect(await h.rows("billing_events")).toHaveLength(1);
  });

  it("rejects an invalid signature before touching the database", async () => {
    const res = await h.postRaw(
      h.event("customer.subscription.created", { id: "sub_1", customer: "cus_1" }),
      { signature: "t=1,v1=deadbeef" },
    );

    expect(res.status).toBe(400);
    expect(await h.rows("billing_events")).toHaveLength(0);
  });

  // The one that matters. An event recorded in the ledger but not processed is
  // not a duplicate, and a ledger that cannot tell the difference loses it
  // permanently: the retry is answered 200 and Stripe stops trying.
  it("retries an event that failed after it was recorded", async () => {
    const event = h.event("customer.subscription.created", {
      id: "sub_1",
      customer: "cus_1",
      status: "active",
      items: [{ price: h.prices.pro, quantity: 3 }],
    });

    h.failNextWrite("deadlock detected"); // transient, recoverable
    const first = await h.post(event);

    expect(first.status).toBeGreaterThanOrEqual(500); // so Stripe will retry
    expect(await h.row("billing_events", "status")).toBe("processing");
    expect(await h.row("billing_events", "processed_at")).toBeNull();
    expect(await h.entitlements("org_1")).toMatchObject({ plan: "free" });

    const retry = await h.post(event); // same event.id

    expect(retry.status).toBe(200);
    expect(await h.row("billing_events", "status")).toBe("processed");
    expect(await h.row("billing_events", "attempt_count")).toBe(2);
    expect(await h.entitlements("org_1")).toMatchObject({ plan: "pro", seats: 3 });
    expect(await h.rows("billing_events")).toHaveLength(1);
  });

  it("commits one effect when two deliveries arrive at once", async () => {
    const event = h.event("customer.subscription.created", {
      id: "sub_1",
      customer: "cus_1",
      status: "active",
      items: [{ price: h.prices.pro, quantity: 3 }],
    });

    const [a, b] = await Promise.all([h.post(event), h.post(event)]);

    // One processes; the other is told to come back, and does not touch state.
    expect([a.status, b.status].sort()).toEqual([200, 409]);
    expect(await h.ledgerAudit("sub_1")).toHaveLength(1);
    expect(await h.entitlements("org_1")).toMatchObject({ plan: "pro", seats: 3 });
  });

  it("recovers an event stranded by a process that died mid-handler", async () => {
    const event = h.event("customer.subscription.created", {
      id: "sub_1",
      customer: "cus_1",
      status: "active",
      items: [{ price: h.prices.pro, quantity: 3 }],
    });

    await h.crashDuring(event); // claimed, never completed, never released

    // While the lease is live, redelivery must not double-process.
    expect((await h.post(event)).status).toBe(409);

    await h.clock.advance({ minutes: 5 }); // lease expires

    expect((await h.post(event)).status).toBe(200);
    expect(await h.entitlements("org_1")).toMatchObject({ plan: "pro", seats: 3 });
  });

  it("does not move state backwards when events arrive out of order", async () => {
    // Reality: a cancellation is retried and lands after the reactivation that
    // followed it. Note what is NOT being relied on here — not `event.created`
    // (one-second resolution; these can share a second) and not a status guard
    // (`active → canceled` is also what a real cancellation looks like). The
    // handler re-fetches the subscription and applies what Stripe currently
    // says, so the order the two events arrive in stops mattering.
    h.stripe.subscription("sub_1", { status: "active", items: [{ price: h.prices.pro }] });

    const stale = h.event("customer.subscription.updated", {
      id: "sub_1",
      customer: "cus_1",
      status: "canceled",
    });

    await h.post(stale);

    expect(h.stripe.fetched).toContain("sub_1"); // it asked, rather than assuming
    expect(await h.entitlements("org_1")).toMatchObject({ plan: "pro", active: true });
    expect(await h.discards()).toMatchObject([
      { reason: "superseded-by-live-state", event: stale.id },
    ]);
  });

  it("keeps access until period end when a subscription is cancelled", async () => {
    await h.post(
      h.event("customer.subscription.updated", {
        id: "sub_1",
        customer: "cus_1",
        status: "active",
        cancel_at_period_end: true,
        current_period_end: h.daysFromNow(12),
      }),
    );

    expect(await h.entitlements("org_1")).toMatchObject({ plan: "pro", active: true });

    await h.clock.advance({ days: 13 });

    expect(await h.entitlements("org_1")).toMatchObject({ plan: "free", active: false });
  });

  it("acknowledges an unrecoverable event instead of retrying it forever", async () => {
    const res = await h.post(
      h.event("customer.subscription.created", { id: "sub_9", customer: "cus_unknown" }),
    );

    expect(res.status).toBe(200);
    expect(await h.row("billing_events", "status")).toBe("failed_permanent");
    expect(await h.row("billing_events", "last_error")).toContain("cus_unknown");
  });
});