Rebasely Beta: data may be purged

justintsus-experiment Public

tsus-experiment

Files

README.md

TSUS experiment: plain TypeScript against TypeScript-Ultra-Strict

Two Claude agents built the same Rebasely feature from the same brief. Agent A wrote plain strict TypeScript. Agent B wrote TypeScript-Ultra-Strict (TSUS), a stricter dialect with no null, undefined, as, !, any, class or throw, plus Option, Result and an exhaustive match. A judge agent then scored both builds against a rubric written before either agent started.

The feature: a GitHub-style search query language (is:open label:bug author:@me sort:updated) for Rebasely's Issues and Pull requests lists. It parses the query, prints a canonical form, and turns it into parameterized SQL for D1, scoped to the repository and organization.

The write-up is on the Rebasely blog: https://rebasely.com/blog/strict-typescript-ai-code

How it was run

  1. One brief, two copies. brief/BRIEF-ts.md and brief/BRIEF-tsus.md differ only in their "Language" section (brief/BRIEF.md is the template). Each agent worked alone in its own git worktree of Rebasely at commit 903f762f, on the same model (Claude Opus 5.5), with no access to the other's work or to the tests.
  2. 118 hidden tests (hidden-tests/), written before the builds from the brief alone: parsing, canonical form and round trips, semantics on SQLite, dates, SQL injection and scope. run.sh <worktree> runs them against a build.
  3. A judge agent scored each build against brief/RUBRIC.md: hidden tests (40), code review (30), UI (20), cost (10), minus security penalties. A gap under 5 points is a tie. The judge also fuzzed both builds against each other and reviewed the page wiring for security (judge/JUDGE.md, probes in judge/probes/).
  4. The harness recorded each agent's tokens, tool calls and wall time (usage.json).

The judge knew which build was which: .tsus files cannot be hidden. The rubric tells it not to reward or penalize the language itself.

Results

A, plain TypeScript B, TSUS
Hidden tests passed 118 / 118 118 / 118
Judge's total (of 100) 98.0 95.9
Code review (of 30) 28.5 27.0
UI (of 20) 19.5 20.0
Cost (of 10) 10.0 8.86
Security findings 0 0
Tokens 277,747 290,102 (+4.4%)
Output tokens 35.7k 57.7k (+61%)
Tool calls 117 150 (+28%)
Wall time 25.9 min 25.1 min
Search module lines 626 1,033 (+65%)
as casts 11 0
New unit tests 17 28
TSUS diagnostics during the build n/a 0 in 8 check runs

On 20,400 random inputs plus about 300 targeted probes the two builds returned the same results, errors, canonical forms and SQLite rows. The only differences were outside the brief: B fails on a row whose labels column is not valid JSON (A guards it with json_valid), and B throws on parseQuery(null).

Verdict

A tie on quality, and plain TypeScript was cheaper. The TSUS build had no casts where the plain build had 11, all of them safe. TSUS caught nothing during the build: every check run came back clean, because the agent read the rules first and wrote inside them. In exchange it took 33 more tool calls, 61% more output tokens and 65% more code, including a 51-line adapter between Option and Result and the plain-TypeScript page code. The build that shipped in Rebasely is A, with B's full-width search box.

On this feature, TSUS did not measurably beat strict TypeScript for an AI builder. Where it did help was an earlier pilot that rewrote existing code (Rebasely's on-call watcher) in TSUS: following the Option and match rewrite turned up two real production bugs, described in the blog post.

What is here

Path What
brief/ The template brief, the two briefs the agents got, and the judge's rubric
hidden-tests/ The 118 hidden tests, their SQLite fixture (helpers.mjs) and run.sh; results/ has both runs
build-a-ts/ Build A's new files: apps/web/src/search/*.ts, apps/web/src/ui/search.ts, test/unit/search/
build-b-tsus/ Build B's new files: apps/web/src/search/*.tsus and api.ts, lists.ts, ui/search.ts, test/unit/search/
judge/JUDGE.md The judge's full report and scores
judge/probes/ The judge's differential probes, page renderer and screenshot script, and the probe output
judge/shots/ Six of the judge's 72 screenshots
usage.json Tokens, tool calls and milliseconds per agent, from the harness

The builds are excerpts: only the feature's new files, not the routes, pages and styles they changed in the rest of Rebasely, which is not open source. So they do not run on their own here. The hidden tests only need apps/web/src/search/api.ts of a build, and import it from SEARCH_WORKTREE; B's .tsus files need the TSUS compiler first. The judge's scripts take the two checkouts from WORKTREE_A and WORKTREE_B.

License

MIT, see LICENSE. The two builds are excerpts of Rebasely source, published by its owner for this experiment.

Report abuse