We Installed the Same Billing Capability Into Two Real Open-Source Apps

"Your coding agent adapts it to your codebase" is the load-bearing claim behind every reusable capability, including ours. It is also the easiest claim in the category to make and the least often demonstrated, because demonstrating it requires someone else's application — one with its own schema, its own auth library and its own opinions, written by people who had never heard of you.

So we took two: cal.com and rallly. Both are real, public, inspectable products. Neither was built with us in mind.

This is what each one forced us to change, what passed, and the one check that refused to pass.

Not a simulation

The distinction that matters most is the least visible, so it goes first.

An install test can be written two ways. The easy way approximates the host: hand-write a few tables that look roughly like cal.com's, bind to them, watch it pass. That proves the binding works against your idea of the host.

These suites apply the host's own migrations. All 595 of cal.com's, in order, producing their 104 tables with their column names, types and constraints. All 154 of rallly's, producing 33. The capability's billing migrations then go on top, adding eleven tables of their own with no name collision against either.

The test also verifies the checkout is at a frozen commit before it asserts anything, because a conformance result against a different commit is a result about a different repository.

cal.com   6bc45298226f96ff79e0c070c8b2ce39727e8477
rallly    563742d54a9fe3562263c594f99b16d457959c28

Neither repository is vendored into the package. Point an environment variable at a checkout and the suite runs; leave it unset and the tier fails loudly rather than passing on nothing.

The repositories were chosen before the result was known

This is the part of an experiment that is easiest to fudge quietly, so here is the order things happened in.

The requirement that governs this work names eight things a primary repository must have — App Router on a supported version, auth, users, an organisation or workspace, a membership and role distinction, PostgreSQL, a settings UI, and a test framework. Those eight were fixed before any candidate was assessed.

Three candidates were then checked against the live GitHub API on 14 September: their actual package.json, their actual schema files, their actual tree, rather than their README. All three satisfied all eight. The third, formbricks, was set aside for a split licence — AGPL with a proprietary ee/ directory — not for anything to do with how well it would install.

Two were chosen the following day, their SHAs frozen and recorded with licence and permission basis, and only then was anything installed.

The secondary had one additional job. It had to differ on named host contracts — preferably auth, connection mode and deployment at once — because two similar repositories prove nothing. That constraint is why rallly was picked over an easier second cal.com-alike.

One operational note, recorded because it constrains the experiment rather than decorating it: rallly is AGPL-3.0. Counsel's conditions are that the repository stays internal, is never exposed as a public network service, that no AGPL source is copied into the package, and that any change to those facts triggers a review before proceeding. Conformance runs locally and in CI, never behind a public URL.

The results

Both installs run the same fourteen conformance checks, AT-HOST-001 through 012, with the persistence contract split three ways.

Install	Date	Result
cal.com 6bc45298	15 September 2026	13 passed, 0 failed, 1 skipped
rallly 563742d5	15 September 2026	13 passed, 0 failed, 1 skipped

What passed, in both: identity resolution that returns null outside a request rather than fabricating an actor. Billing authorisation that grants the manager and refuses both the foreign principal and the outsider. Persistence that rolls back a row written before an error, and a database that refuses a duplicate primary key. Migration. A durable scheduler. Notification, cache, observability, clock, configuration. Stripe isolation — verified by checking that no domain module imports the Stripe SDK. And operator recovery, which refuses tenant callers and grants only the configured operator.

The check that refused to pass

AT-HOST-003c is the genuinely-concurrent persistence probe, and it is skipped in both installs.

The engine these suites run against is single-connection. It would execute both halves of a concurrency test in sequence and report a pass — for a property it never exercised. The check declines to do that.

This is worth dwelling on because it is the opposite of how a green tick usually gets produced. A suite that wanted a clean sweep could run the same probe on the same engine, see no conflict, and print PASS. The result would be indistinguishable from a real one to anyone reading the summary line, and it would be worthless.

The property itself is not untested — the billing schema's behaviour under real concurrency is proven against real Postgres by a different tier. What a reference install adds is the binding of identity, authorisation, persistence, scheduler and operator recovery onto a host's own data model, and every one of those passed in both.

Closing it means running this tier against a Postgres that can hold two simultaneous connections, which was tracked as its own release-gate item rather than argued away in a footnote.

It closed on 23 September 2026. The tier ran against a real Postgres and passed, and the gate no longer accepts a configured connection string as evidence — it reads the run's own result file and fails if the suite skipped, failed, or never loaded. That change came out of noticing that the previous rule certified a variable rather than a run: exporting UDRIA_TEST_DATABASE_URL was enough to clear a blocker nobody had satisfied.

The install results above remain the run of 15 September 2026 and are reported as of that date.

What the second install caught that the first could not

This is the finding. Two differences would have broken a binding that passed every check against the primary.

	cal.com	rallly
identity	NextAuth v4	better-auth
principal	Team, integer id	spaces, text id
billing authority	role on the membership row	owner column on the space, plus an ADMIN membership role
email	text	citext, case-insensitive
connection mode	Prisma query engine	Prisma driver adapter over a pg Pool
deployment	Vercel cron	Docker, self-hosted

Identifier type

cal.com's ids are integers. rallly's are text.

A binding that quietly assumed integers would have passed every check against the primary and failed here. Nothing in the primary install could have caught it, because nothing in the primary install disagrees with the assumption.

That is the entire argument for doing this twice, and it is why the requirement asks the secondary to differ rather than to be representative.

Where the authority lives

The more interesting one, because it is a modelling difference rather than a type difference.

In cal.com, owning a team is a membership role. Reading the Membership table answers the whole question: is this person an OWNER or ADMIN of this Team, and have they accepted the invitation?

In rallly, the owner is a column on the space, and holds no membership row at all.

A binding that read only the membership table would therefore have refused the one person who certainly may spend the money. It would also have looked completely correct against cal.com, passed its conformance run, and shipped.

Three assertions beyond the conformance suite exist to pin the mistakes that predicate invites: a MEMBER belongs to the organisation but may not spend its money; an invited OWNER who has not accepted is not yet an authority; and a team owner is not an operator.

The migration that takes 140 others with it

rallly's schema needs the citext extension. One of their migrations converts users.email to it, and without the extension loaded the following 140 migrations cascade into failure.

An install that quietly skipped that migration would still produce a database, still run, and still pass. It would be an install of a different schema — which is the failure mode "we applied the host's migrations" exists to rule out, and the reason the suite loads the extension rather than working around the migration.

Two hosts, two mechanisms, one still unproven

The capability's operator-recovery contract allows two mechanisms: a real instance-admin role, or a protected command-line tool with a configured identity.

Both hosts turned out to have the first — UserPermissionRole in cal.com, user_role in rallly — so in both installs the distinction between may manage this tenant's billing and may declare whether Stripe charged that tenant is proven against a real column rather than asserted.

That is a better outcome than expected and a slightly inconvenient one. Most hosts have no admin role model and will fall back to the second mechanism, which these two installs therefore leave proven only by the package's own bindings. A gap found by succeeding is still a gap.

What this does not prove

Two installs is a sample of two. That sentence is doing real work, and the specific limits matter more than the general disclaimer.

Neither install ran against Supabase, a transaction-mode pooler, or Vercel's runtime. Those rows on the public compatibility matrix stay unverified, and the governing rule is that a claim is promoted when a run proves it — not when a neighbouring claim is proven. A pooler is the sharpest of those gaps: the specification warns that a transaction-mode pooler will hand consecutive statements to different backends and that the failure is silent, which is precisely the kind of thing a passing suite does not notice.

Both hosts are Prisma on PostgreSQL with a Next.js App Router front end. That is the compatibility tier being claimed, so it is the right place to test — but it means these results say nothing about a host that differs more deeply than these two differ from each other.

And both installs were performed by us, on a capability we wrote, with bindings we chose. They demonstrate that the contracts can be satisfied by two real applications with genuinely different shapes. They do not demonstrate what happens when somebody else's agent does it to somebody else's repository, which is a different experiment and a harder one.

Reproduce it

The suites are in the package, and neither repository is vendored:

git init calcom && cd calcom
git remote add origin https://github.com/calcom/cal.com.git
git fetch --depth 1 --filter=blob:none origin 6bc45298226f96ff79e0c070c8b2ce39727e8477
git checkout FETCH_HEAD
export UDRIA_CALCOM_REPO=$PWD

npm run capability:test:install

The same shape works for rallly against its own frozen SHA. The suite checks the commit before it asserts anything, so a run against a different checkout will tell you rather than quietly producing a result about a different repository.

What we take from it

The claim under test was that a capability can adapt to a codebase it was not written for. Two installs is thin evidence for a general claim and reasonable evidence for a narrow one: against two real applications with different auth libraries, different identifier types, different authority models and different deployment shapes, the host contracts held, and the adaptations needed were in the bindings rather than in the capability.

The more useful output is not the pass count. It is that the second install found two ways the first was quietly wrong — and that both were in the binding layer, which is exactly where the model predicts the work lands.

If a third install is added, and it goes badly, that will be published too.