Rehearsal

The problem

Distributed-system failures are taught as diagrams and experienced as outages. The dangerous case — a request that succeeded on the server but whose reply was lost — is invisible until it double-charges someone. There is no safe place to rehearse it.

How it was built

1. 01

   Built sandbox inventory, payment and delivery services so failures happen over real HTTP, not mocked function calls.

2. 02

   Implemented a fault-injection layer that reproduces timeouts, duplicate events and rate limits on demand and reproducibly.

3. 03

   Wrote idempotent recovery strategies — idempotency keys, retry with backoff, deduplication — as the thing under test rather than as boilerplate.

4. 04

   Added a baseline-versus-recovery comparison view so the difference a strategy makes is measured, not asserted.

5. 05

   Exported every run as JSON evidence, which makes a run citable in a report or a postmortem.

6. 06

   Split hosting: Next.js frontend on Vercel, Node.js backend on a DigitalOcean VPS, with Playwright covering the flows.

What shipped

• Live at rehearsal.tharinda.dev with a public source repository.
• Every scenario produces exportable JSON evidence of baseline versus recovered behaviour.
• Playwright end-to-end coverage over the fault-injection flows.

In one paragraph

Rehearsal is an API reliability workbench built with TypeScript, Next.js, React, Node.js and SQLite. It injects faults into real HTTP requests across sandbox inventory, payment and delivery services to reproduce timeouts, duplicate events and rate limits, implements idempotent recovery strategies, and shows a baseline-versus-recovery comparison with JSON evidence export. The frontend is hosted on Vercel and the backend runs on a DigitalOcean VPS.