Convo
Live group chat that runs in episodes. A creator goes live, followers get a push, and the room fills in seconds.
Shows can be public, invite-only or paid. Paid shows sell monthly memberships through Stripe Connect, with the creator as merchant of record. The system is built for the moment a show goes live. Thousands of phones may connect at once, and every one expects every message in order, with nothing duplicated and nothing lost.
I was the only engineer. I built the NestJS API, the Socket.IO gateway, the Pub/Sub workers, the GCP infrastructure in OpenTofu, the iOS app in Expo, the Next.js website and checkout, the admin console, the Stripe integration and both test harnesses. AI agents wrote much of the code under my process, and I reviewed every change.
- Load test (synthetic)
- 5,052 test sockets against prod
- Decisions
- 72 ADRs
- Tests
- 3,500+ test cases
- Codebase
- ~221K lines TS · ~7.9K OpenTofu
Key engineering decisions
- Split the WebSocket gateway out of the API and broadcast through Redis.Sockets scale on their own, with no database and no sticky sessions in the gateway.
- Moved push fan-out (notifying every follower) off the request path into a two-stage Pub/Sub scatter.The post request does the same work however many followers a show has.
- Graded load tests on correctness, using synthetic clients against production.A run passes only if every joined client got every message, in order, exactly once.
- Treated the first load-test failure as a production incident and kept testing after the fix.Found that API instances were stalling, and why. The rerun and a 2-hour soak returned zero HTTP 504s.
- Made reconnect recovery re-deliver rather than skip, sequenced behind the room rejoin.A dropped connection costs a few duplicate events the client ignores, never a lost message.
- Superseded my own payments design so creators sell on their own Stripe accounts.Creators are the merchant of record. Stripe owns disputes and losses; Convo takes a 20% fee.




System map: the path of one message
ADR 0010 · 0026 · 0034- iOS app
- API
- Postgres write
- ①Redis emit
- ②every WS gateway
- sockets in the room
- API
- ③one Pub/Sub job
- coordinator
- 500-user chunks
- Expo push
① The API writes the message and emits through Redis. ② Every gateway instance broadcasts to the sockets in that room. ③ The API publishes one fan-out job, and workers handle push off the request path.
Why fan-out became a two-stage scatter
ADR 0034One worker, sequential
Expo accepts 100 messages per request, so one job sends batch after batch.
Coordinator + chunk workers
The coordinator computes the audience once, drops anyone already in the room, and republishes 500-user chunks.
Call counts follow the real batch and chunk sizes, assuming one device per follower. Times assume an illustrative 150 ms per Expo request and enough worker concurrency to run every chunk at once. The in-room broadcast stays on the synchronous post path either way.
A load harness that grades correctness, not latency
Latency is always reported and never a pass/fail gate. A slow but correct run passes.
I wrote bertha to answer one question: does every message reach everyone who should see it? It seeds thousands of synthetic users into a namespaced test population, each with a 48-hour access token. It then drives them as test clients through the real path: REST post, Postgres, the Redis emitter, the socket gateway, receipt on a socket. Teardown deletes only what the run seeded. None of these clients are customers.
Each run is scored on delivery completeness, ordering, duplicates and unexpected disconnects. --storm drops and reopens every socket at once, which is what a gateway redeploy looks like. At the top rung of the run below there were zero duplicates. All 50 out-of-order receipts traced to posts whose client had already timed out, so the server committed them after the client gave up. Each finding became a numbered fix plan.
Debugging wedged API instances in production
The first fix held up under load. A longer soak test then found the real cause.
In September 2026 I ran bertha against the production environment with 5,052 synthetic sockets plus a reconnect storm. The API scaled to 14 instances, and the run returned 422 HTTP 504s. Almost all came from three instances that had silently stopped finishing requests, while the other 11 served normally.
I bounded every Redis command, added an in-process watchdog that exits an instance making no progress, and cut the request timeout from 300 s to 30 s. The rerun at the same rung served 206,079 API requests with zero 504s.
A 90-minute soak test then stalled eight instances anyway, each one the oldest and busiest in the fleet. The real cause was the request log. It was written to an in-memory volume and never rotated, and that memory counts against the container’s 512 MiB. Size-based rotation and a 1 GiB container fixed it. A 2 h 10 min verification soak at 2,022 synthetic sockets served 516,044 requests with zero 504s and zero failed liveness probes.
docs/plans/bertha/13 · 21 · run notes in docs/plans/data/berthaReconnects that never lose a message
Re-deliver, never skip. Duplicates are harmless because every update is idempotent. Gaps are not.
After a reconnect the app gap-fills: it fetches what it missed over REST. Several API instances emit through Redis, so events from different instances can arrive out of order and their clocks can disagree. The cursor is therefore the newest timestamp already in the cache, minus a 10-second lookback applied on the server. The gap-fill waits for the room-rejoin acknowledgement, so a message created mid-reconnect can’t fall between the two.
The simulator smoke tests caught a race where the rejoin failed silently. The timeline looked recovered, but no new broadcasts arrived. Echoing clientMessageId on every message lets the app merge its optimistic bubble with the server’s copy.
Payments, redesigned on the record
I wrote down why the first design was wrong for the account Stripe approved, then replaced it.
The first Stripe Connect design made Convo the merchant of record, the business that legally sells the membership and owns refunds and disputes. In October 2026 I superseded it with ADR 0065, to match the live Connect platform profile. Creators are now the merchants of record, selling through direct charges on their own Stripe accounts. Stripe owns losses, and Convo takes a 20% application fee. The card is collected once and cloned to each creator the member joins.
Memberships move through explicit states: incomplete, active, past due (access kept while payment is retried) and canceled. A first payment asks for 3-D Secure, and a member whose bank blocks it can try again. Checkout happens on the web. The in-app link ships only on the US storefront, where Apple permits it.
ADR 0062 → 0065 · 0060 · 0016 · 0049An LLM judge that has to pass its own check
A model's opinion is opt-in, and a judge that agrees with everything gets caught.
The simulator smoke harness checks almost everything with deterministic assertions. It can’t check whether the photo on screen is the one this run sent. With --judge, a pinned Claude Haiku model checks whether the rendered image carries the run’s own nonce text.
Every judged run first checks control screenshots, taken before anything was sent, which must come back “no image”. If the judge misses a control, the run is reported as a judge failure, never as a pass. --judge-override deliberately gives the judge the wrong text, and then every check must fail.
Agent access over MCP: built, then parked before launch
ADR 0001 · 0002 · 0003 · 0011 · 0053 · modules/mcpMCP (Model Context Protocol) is how an AI client such as Claude Code calls a service’s tools. I built a deliberately narrow MCP server so a host could let an agent post in their own show. It was built and tested against the dev environment. In September 2026 I parked it to focus on launch, and it is not offered in the app or on the website.
- Authorization
- The API is an OAuth resource server, and Clerk is the authorization server. Clerk handles login, consent, client registration and PKCE. The API serves protected-resource metadata and answers a tokenless call with
401plus a discovery header, soclaude mcp add --transport httpworks without pasting a token. Headless clients can use a personal access token instead. It is shown once, stored only as a hash, and can expire. - What the agent can do
- Five tools: list my shows, read a show, read the live episode’s messages, post text (up to 2,000 characters) and delete one of its user’s own messages. It can’t start a show, go live, edit, pin, send images or browse old episodes. Posting into a show with nothing live returns an error instead of starting an episode.
- Scope and limits
- There are no OAuth scopes. An OAuth token’s authority is exactly these five tools, checked by the same access and authoring policies the app uses, so an agent can only post where its user could. Requests are stateless, with a fresh server per request so no context carries between users. The per-user rate limits on every signed-in route apply here too. Each call logs the user and the credential type.
- Revocation
- Deleting a personal access token revokes it and clears its verification cache. Clerk issues, refreshes and revokes OAuth tokens. There is no screen in the app yet for listing or disconnecting agents.
- Identity
- An agent’s post is attributed to its user and looks exactly like a message they typed. There is no bot account and no stored marker of which messages came through MCP. Telling them apart later means correlating with the request log. Retries can post duplicates, a documented limitation.
- Open: disclosure
- ADR 0002 chose no AI label for the first version. The reasoning: the agent is the user’s tool, and the user stays accountable for what it posts. The cost is that other people in the chat can’t tell an agent wrote a message. A per-message marker was considered and rejected for now, with reservations. It is cheap to add today and expensive to backfill. The ADR names what would reopen it: regulation such as California’s bot-disclosure law or the EU AI Act, or a change of product direction. It stays unresolved, and it would need an answer before agents post in real chats.