natemoon.io
Live on the US App Store ↗Role Sole engineerApr 2026 – now

Convo

Live group chat that runs in episodes. A creator goes live, followers get a push, and the room fills in seconds.

Shows can be public, invite-only or paid. Paid shows sell monthly memberships through Stripe Connect, with the creator as merchant of record. The system is built for the moment a show goes live. Thousands of phones may connect at once, and every one expects every message in order, with nothing duplicated and nothing lost.

I was the only engineer. I built the NestJS API, the Socket.IO gateway, the Pub/Sub workers, the GCP infrastructure in OpenTofu, the iOS app in Expo, the Next.js website and checkout, the admin console, the Stripe integration and both test harnesses. AI agents wrote much of the code under my process, and I reviewed every change.

Load test (synthetic)
5,052 test sockets against prod
Decisions
72 ADRs
Tests
3,500+ test cases
Codebase
~221K lines TS · ~7.9K OpenTofu
NestJSPrisma · Postgres 16Socket.IO + RedisPub/Sub workersExpo / React NativeNext.jsStripe ConnectClerkCloud RunDatadogMCP

Key engineering decisions

  1. Split the WebSocket gateway out of the API and broadcast through Redis.Sockets scale on their own, with no database and no sticky sessions in the gateway.
  2. Moved push fan-out (notifying every follower) off the request path into a two-stage Pub/Sub scatter.The post request does the same work however many followers a show has.
  3. Graded load tests on correctness, using synthetic clients against production.A run passes only if every joined client got every message, in order, exactly once.
  4. Treated the first load-test failure as a production incident and kept testing after the fix.Found that API instances were stalling, and why. The rerun and a 2-hour soak returned zero HTTP 504s.
  5. Made reconnect recovery re-deliver rather than skip, sequenced behind the room rejoin.A dropped connection costs a few duplicate events the client ignores, never a lost message.
  6. Superseded my own payments design so creators sell on their own Stripe accounts.Creators are the merchant of record. Stripe owns disputes and losses; Convo takes a 20% fee.
App Store preview “Every word, as it happens.”: a live show with hosts posting play-by-play and reaction counts under each messageApp Store preview “React the way you already do.”: a long-pressed message with the tapback bar openApp Store preview “Everything you follow, in one feed.”: the Following tab, with live shows at the top and recent chats belowApp Store preview “Start your own show tonight.”: the host view of a show with three hosts and the composer open
App Store preview images, rendered from design mockups with demo data. The shows, people, messages and viewer counts are illustrative, not live usage.

System map: the path of one message

ADR 0010 · 0026 · 0034
Clients
iOS app
Expo · React Native
Web + checkout
Next.js on Vercel
Claude / MCP
OAuth via Clerk · parked
Admin console
read-only to Stripe
Edge
Global HTTPS load balancer
serverless NEGs · Cloud Armor · Cloud CDN for cover art
Services / State
API① ③
NestJS · 33 modules · OpenAPI
sole owner of the schema
Postgres 16
Cloud SQL · 25 models
WS gateway ×N②
Socket.IO · websocket-only
no DB, no sticky sessions
Redis①
presence ZSETs · emitter · limits
Fan-out workers
coordinator → 500-user chunks
dedupe · devices · Expo push
Pub/Sub③
two fan-out topics
The path of one message
On the request
  1. iOS app
  2. API
  3. Postgres write
  4. ①Redis emit
  5. ②every WS gateway
  6. sockets in the room
Off the request
  1. API
  2. ③one Pub/Sub job
  3. coordinator
  4. 500-user chunks
  5. Expo push

① The API writes the message and emits through Redis. ② Every gateway instance broadcasts to the sockets in that room. ③ The API publishes one fan-out job, and workers handle push off the request path.

Why fan-out became a two-stage scatter

ADR 0034
83,200
One worker, sequential

Expo accepts 100 messages per request, so one job sends batch after batch.

832
serial HTTP calls · ≈ 2.1 min
Coordinator + chunk workers

The coordinator computes the audience once, drops anyone already in the room, and republishes 500-user chunks.

167
parallel chunks of ≤5 calls · ≈ 0.9 s

Call counts follow the real batch and chunk sizes, assuming one device per follower. Times assume an illustrative 150 ms per Expo request and enough worker concurrency to run every chunk at once. The in-room broadcast stays on the synchronous post path either way.

A load harness that grades correctness, not latency

Latency is always reported and never a pass/fail gate. A slow but correct run passes.

I wrote bertha to answer one question: does every message reach everyone who should see it? It seeds thousands of synthetic users into a namespaced test population, each with a 48-hour access token. It then drives them as test clients through the real path: REST post, Postgres, the Redis emitter, the socket gateway, receipt on a socket. Teardown deletes only what the run seeded. None of these clients are customers.

Each run is scored on delivery completeness, ordering, duplicates and unexpected disconnects. --storm drops and reopens every socket at once, which is what a gateway redeploy looks like. At the top rung of the run below there were zero duplicates. All 50 out-of-order receipts traced to posts whose client had already timed out, so the server committed them after the client gave up. Each finding became a numbered fix plan.

test/bertha · ADR 0053 · docs/plans/bertha (22 fix plans)
Debugging wedged API instances in production

The first fix held up under load. A longer soak test then found the real cause.

In September 2026 I ran bertha against the production environment with 5,052 synthetic sockets plus a reconnect storm. The API scaled to 14 instances, and the run returned 422 HTTP 504s. Almost all came from three instances that had silently stopped finishing requests, while the other 11 served normally.

I bounded every Redis command, added an in-process watchdog that exits an instance making no progress, and cut the request timeout from 300 s to 30 s. The rerun at the same rung served 206,079 API requests with zero 504s.

A 90-minute soak test then stalled eight instances anyway, each one the oldest and busiest in the fleet. The real cause was the request log. It was written to an in-memory volume and never rotated, and that memory counts against the container’s 512 MiB. Size-based rotation and a 1 GiB container fixed it. A 2 h 10 min verification soak at 2,022 synthetic sockets served 516,044 requests with zero 504s and zero failed liveness probes.

docs/plans/bertha/13 · 21 · run notes in docs/plans/data/bertha
Reconnects that never lose a message

Re-deliver, never skip. Duplicates are harmless because every update is idempotent. Gaps are not.

After a reconnect the app gap-fills: it fetches what it missed over REST. Several API instances emit through Redis, so events from different instances can arrive out of order and their clocks can disagree. The cursor is therefore the newest timestamp already in the cache, minus a 10-second lookback applied on the server. The gap-fill waits for the room-rejoin acknowledgement, so a message created mid-reconnect can’t fall between the two.

The simulator smoke tests caught a race where the rejoin failed silently. The timeline looked recovered, but no new broadcasts arrived. Echoing clientMessageId on every message lets the app merge its optimistic bubble with the server’s copy.

ADR 0026 · 0037 · docs/plans/bertha/22 · test/smoke reconnect-recovery
Payments, redesigned on the record

I wrote down why the first design was wrong for the account Stripe approved, then replaced it.

The first Stripe Connect design made Convo the merchant of record, the business that legally sells the membership and owns refunds and disputes. In October 2026 I superseded it with ADR 0065, to match the live Connect platform profile. Creators are now the merchants of record, selling through direct charges on their own Stripe accounts. Stripe owns losses, and Convo takes a 20% application fee. The card is collected once and cloned to each creator the member joins.

Memberships move through explicit states: incomplete, active, past due (access kept while payment is retried) and canceled. A first payment asks for 3-D Secure, and a member whose bank blocks it can try again. Checkout happens on the web. The in-app link ships only on the US storefront, where Apple permits it.

ADR 0062 → 0065 · 0060 · 0016 · 0049
An LLM judge that has to pass its own check

A model's opinion is opt-in, and a judge that agrees with everything gets caught.

The simulator smoke harness checks almost everything with deterministic assertions. It can’t check whether the photo on screen is the one this run sent. With --judge, a pinned Claude Haiku model checks whether the rendered image carries the run’s own nonce text.

Every judged run first checks control screenshots, taken before anything was sent, which must come back “no image”. If the judge misses a control, the run is reported as a judge failure, never as a pass. --judge-override deliberately gives the judge the wrong text, and then every check must fail.

test/smoke/src/judge.ts · ADR 0033

Agent access over MCP: built, then parked before launch

ADR 0001 · 0002 · 0003 · 0011 · 0053 · modules/mcp

MCP (Model Context Protocol) is how an AI client such as Claude Code calls a service’s tools. I built a deliberately narrow MCP server so a host could let an agent post in their own show. It was built and tested against the dev environment. In September 2026 I parked it to focus on launch, and it is not offered in the app or on the website.

Authorization
The API is an OAuth resource server, and Clerk is the authorization server. Clerk handles login, consent, client registration and PKCE. The API serves protected-resource metadata and answers a tokenless call with 401 plus a discovery header, so claude mcp add --transport http works without pasting a token. Headless clients can use a personal access token instead. It is shown once, stored only as a hash, and can expire.
What the agent can do
Five tools: list my shows, read a show, read the live episode’s messages, post text (up to 2,000 characters) and delete one of its user’s own messages. It can’t start a show, go live, edit, pin, send images or browse old episodes. Posting into a show with nothing live returns an error instead of starting an episode.
Scope and limits
There are no OAuth scopes. An OAuth token’s authority is exactly these five tools, checked by the same access and authoring policies the app uses, so an agent can only post where its user could. Requests are stateless, with a fresh server per request so no context carries between users. The per-user rate limits on every signed-in route apply here too. Each call logs the user and the credential type.
Revocation
Deleting a personal access token revokes it and clears its verification cache. Clerk issues, refreshes and revokes OAuth tokens. There is no screen in the app yet for listing or disconnecting agents.
Identity
An agent’s post is attributed to its user and looks exactly like a message they typed. There is no bot account and no stored marker of which messages came through MCP. Telling them apart later means correlating with the request log. Retries can post duplicates, a documented limitation.
Open: disclosure
ADR 0002 chose no AI label for the first version. The reasoning: the agent is the user’s tool, and the user stays accountable for what it posts. The cost is that other people in the chat can’t tell an agent wrote a message. A per-message marker was considered and rejected for now, with reservations. It is cheap to add today and expensive to backfill. The ADR names what would reopen it: regulation such as California’s bot-disclosure law or the EU AI Act, or a change of product direction. It stays unresolved, and it would need an answer before agents post in real chats.
Next case studyIntuitive Orton →A literacy platform for tutors who teach reading with the Orton-Gillingham method, an approach designed for dyslexic readers.
Contact

Hiring a senior or staff engineer?

I’m looking for senior and staff roles where system design, production ownership and AI-native engineering matter, especially at companies building with frontier models. Email is the fastest way to reach me.

Email meResume (PDF)
Email
nathaniel.c.moon@gmail.com
GitHub
github.com/nathanielmoon ↗
LinkedIn
in/nathanielmoon ↗