justintsus-experiment Public
Judge report: search query language, plain TypeScript (A) vs TSUS (B)
- A, plain TypeScript:
.claude/worktrees/agent-a7895120081998e14, commit 40d9e6fd - B, TSUS:
.claude/worktrees/agent-a66151373793240e4, commit a69ca436 - Base: 903f762f. Judged 2026-10-17.
Probe scripts, renderers and raw output are in judge/probes/. The judge took 72 screenshots; six are in judge/shots/.
Verdict
It's a tie on quality, and A is the cheaper build. Ship A, but bring over B's search box layout and B's test fix first.
The two builds behave the same everywhere I could measure:
- The hidden tests pass 118/118 for both.
- On 20,400 random (input, target) pairs plus about 300 targeted adversarial probes, they return the same
ok, the same error codes and spans, the same canonical form and the same rows on SQLite. - The page glue produces the same tab hrefs, counts, pager links and label-chip links.
Only two behavioral differences turned up, and neither is in the brief's contract:
- Malformed labels JSON. If a stored
labelsvalue isn't valid JSON, B'slabel:,-label:andno:labelqueries fail in SQLite. A guards the column withjson_valid. The base code had the same unguardedjson_eachthat B uses. - Non-string input. B's
parseQuery(null)andparseQuery(undefined)throw. A coerces them withString(). The contract only covers strings.
The totals are 98.0 for A and 95.9 for B. That gap is inside the rubric's 5-point tie band, and most of it comes from cost: A used 12k fewer tokens and 33 fewer tool calls.
What TSUS bought here: zero casts in B's whole diff, against 11 in A. Every one of A's casts is safe today. The TSUS checker reported zero diagnostics in all 8 of B's check runs. It caught nothing, because the builder read the rules first and wrote inside them. Its value in this experiment was preventive, not corrective. No bug existed in A that TSUS would have stopped.
What TSUS cost:
| A | B | B vs A | |
|---|---|---|---|
| Tool calls | 117 | 150 | +28% |
| Total tokens | 277,747 | 290,102 | +4.4% |
| Output tokens | 35.7k | 57.7k | +61% |
| Search-module lines | 626 | 1,033 | +65% |
| Calls on TSUS docs and checks | 0 | 15 |
B also needed a 51-line adapter and a few awkward idioms to work around banned indexing. Wall clock was a wash: B took 25.1 min and A took 25.9 min.
Scores
| Item | Max | A | B | Evidence |
|---|---|---|---|---|
| Hidden tests | 40 | 40.0 | 40.0 | 118/118 each (ts.out, tsus.out) |
| Correctness beyond tests | 8 | 8.0 | 7.5 | Identical on every probe. B fails on malformed labels JSON, which A guards. |
| Error handling | 6 | 5.5 | 5.5 | A: a dead catch-all in parseQuery would hide a parser bug as a whole-input bad_value. B: an unreachable throw in ui/search.ts, and it throws on non-string input. Neither catches D1 failures, same as base. Both log and survive a RepoDO hydrate failure. |
| Readability and structure | 6 | 5.5 | 5.0 | A is compact (6 files, 486 code lines) but has 5 casts and a closure that mutates the lexer index. B has real tagged unions and exhaustive match, but it's 65% longer, with ban workarounds like parts.slice(0, 1).join("") and a for-loop over a 0-or-1-element array. |
| Edge cases | 5 | 5.0 | 4.5 | Same limits, unicode and whitespace behavior. A guards labels JSON. B guards a null closedAt and mergedAt in rows. |
| Tests the builder wrote | 3 | 2.5 | 3.0 | A: 17 new tests, and test/unit/issues.test.ts still fails. That failure predates both builds (I checked on base 903f762f), but A edited the file and left it red. B: 28 new tests, and it fixed that assertion. |
| Fit with the codebase | 2 | 2.0 | 1.5 | A keeps the one-round-trip DB.batch of base and reuses ISSUE_COLS. B makes two D1 round trips (Promise.all of two statements), duplicates ISSUE_COLS, and puts page glue that imports repoStub and log (lists.ts) in search/. B uses typed .all<R>() instead of casts. |
| Code review subtotal | 30 | 28.5 | 27.0 | |
| UI: search runs the language | 6 | 6 | 6 | Same results for state, label, author, text, date and sort queries |
| UI: errors inline | 5 | 5 | 5 | Both: wavy underline plus a superscript number per span, message list, aria-invalid, no list shown, input kept |
| UI: syntax help | 3 | 3 | 3 | <details> on both lists, only that list's qualifiers, opens with Enter and without JS |
| UI: URL round trip | 3 | 3 | 3 | Canonical q in the box, tabs, pager and chips. Legacy state= and label= links work. |
| UI: design | 3 | 2.5 | 3 | A's box is 460px wide in the filter row, so the canonical query is cut off at 1280 px. B puts a full-width box above the tabs. No overflow at 390 px for either. No uppercase, gradients or em dashes in either. |
| UI subtotal | 20 | 19.5 | 20.0 | |
| Cost: tokens (6) | 6 | 6.00 | 5.74 | 277,747 vs 290,102 |
| Cost: tool calls (4) | 4 | 4.00 | 3.12 | 117 vs 150 |
| Cost subtotal | 10 | 10.0 | 8.86 | Per hidden test passed: 2,354 vs 2,458 tokens |
| Security penalties | 0 | 0 | None found (see below) | |
| Total | 100 | 98.0 | 95.9 | Tie (within 5); A is cheaper |
Preparation checks
| Check | A | B |
|---|---|---|
| Commits on branch | 1 | 1 |
| Stray files committed | None | None. No generated .js/.d.ts, no migration, no edits to scripts/tsus-web.mjs, tools/tsus, wrangler or CI. |
npm run typecheck |
Pass | Pass |
npm run lint |
Pass | Pass |
| Builder's own search tests | 17/17 | 28/28 |
test/unit/issues.test.ts |
2/3, red, inherited from base | 3/3 |
| TSUS constraint (B only) | n/a, no .tsus used |
Honored. All logic is in 7 .tsus files. api.ts (51 lines) only converts Result/Option and wraps edit helpers. |
Code metrics (added code under apps/web/src, comments excluded)
| A | B | |
|---|---|---|
| Source files added or changed | 14 | 15 |
| Search module files | 6 .ts |
7 .tsus, api.ts, lists.ts |
| Search module lines (total / non-blank, non-comment) | 626 / 486 | 1,033 / 776 |
| Lines added in src + css / in tests | 1,002 / 495 | 1,354 / 570 |
| New unit tests | 17 | 28 |
as casts (excluding as const) |
11 | 0 |
Non-null assertions ! |
0 | 0 |
any |
0 | 0 |
try/catch/throw added |
catch-all in parseQuery, .catch on hydrate |
try/catch on hydrate, 1 unreachable throw in ui/search.ts |
A's 11 casts are as Key, as DateOp, as RelUnit, as StateTab, as StateValue, as IssueTab | null, as "open" | "closed" | "merged" | "all", as IssueRow[], as PullIndexRow[] and 2 of as { state: string; n: number }[]. All are safe given the surrounding checks. The last four copy the base code's D1 pattern.
Extra probe results
All probes import each build's apps/web/src/search/api.ts in one Node 24 process and run SQL on the hidden-test fixture (node:sqlite). Scripts: judge-probes/probe.mjs and judge-probes/show.mjs. Raw output: judge-probes/probe.out.
| Probe | A | B | Brief |
|---|---|---|---|
SQL injection. 16 payloads × 2 targets in text, phrases, labels, author, sort, dates and #n ('; DROP TABLE, ") OR 1=1 --, ?1 :name, NUL, U+2028). Plus a hostile ctx: repoId repo-1' OR 1=1 --, orgId with "), viewerId ' OR 1=1 --. |
Inert. No ctx value appears in the SQL text. | Inert. No ctx value appears in the SQL text. | Both match |
| Scope escapes. 20,400 fuzz pairs, every returned row checked against ctx repo and org, ANON included | 0 leaks | 0 leaks | Both match |
Cross-org agent slug. author:fixer: org-1's fixer is FIXER; org-2's fixer (FIXER2) authored issue 6 and PR 6 in this repo. |
Issues [5], pulls [2], never 6. Under orgId org-2: [] | Same | Both match |
review-requested:fixer, -author:fixer on pulls |
[] and [6,1,4,3,5] | Same | Both match |
NULL under negation. -author:@me on pulls (PR 1 authored by ME, PR 2 owned by ME, others have owner NULL) |
[6,4,3,5] | Same | Match |
-author:BOB on pulls (PR 4 authored by Bob, PR 6 owned by Bob) |
[2,1,3,5] | Same | Match |
-author:@me / -review-requested:@me signed out |
Every row in scope | Same | Match |
author:@me signed out |
[] | [] | Match |
LIKE escaping. 100% → [2], a_c → [4], % → [2], _ → [4], \ → [6], a\b → [6], a_b on pulls → [4] |
As listed | Same | Match. No wildcards leak. |
ASCII-only case folding: Ü → [4], ü → [], ÜNÏCODE → [] |
As listed | Same | Match (SQLite LIKE semantics) |
Unicode spans. "🐛 → unterminated 0..3. a 🐛" b → unexpected 2..5. 🐛:x → free text. A lone surrogate \uD83D is free text. |
UTF-16 offsets, as listed | Identical | Match |
Whitespace kinds. NBSP, U+3000, U+2003 and U+FEFF separate terms. U+200B, U+0085 and U+180E do not (JS \s). |
As listed | Same | Match |
Error ordering. -label: / -sort: → empty_value before negation_not_allowed. sort:x sort:y → two bad_value, no conflict. sort:created-asc sort:CREATED-ASC → deduplicated, no conflict. |
As listed | Same | Match |
Dates (48 values × 2 fields × 2 targets). *..*, <0d, <10000d, 7d, =D, 1969-12-31, 2026-04-31, <٧d (Arabic digit), A..B with A after B, >D..E → all bad_date. <0007D → <7d. 9999-12-31 accepted. |
As listed | Same | Match |
Limits. 1025 chars → too_long 0..1025 alone. 33 terms → too_many_terms from the 33rd term to the end, even when an unterminated quote follows. |
As listed | Same | Match |
D1 100-bound-parameter limit, worst case. Most params per term is 2 (text on issues; author: on pulls). 32 distinct heavy terms + scope = 66 params, + LIMIT/OFFSET = 68. Counts query: 66. Hydrate IN list: at most 25. |
68 max | 68 max | Both under 100 |
| SQL text size, worst case | 3.6 KB | 4.5 KB | Fine |
| Speed. 10k random queries, parse + serialize + toSql, median of 5 runs | 13.5 ms | 18.8 ms | Both negligible |
Worst-case 1024-char inputs (escaped quotes, 1024 ", 1023 \, 341 negations), µs per parse |
7 to 37 | 9 to 30 | Both linear, no blowup |
| Never throws, string inputs. 20,400 fuzz + targeted | 0 throws | 0 throws | Match |
Non-string input (outside contract): parseQuery(null) / parseQuery(undefined) |
ok: true (coerces) |
Throws TypeError | Not required |
toSql with now: NaN (outside contract) |
Binds null |
Binds null |
n/a |
Malformed labels JSON row ('not json') with label:bug, -label:bug, no:label |
Works; the row counts as no labels | SQLite "malformed JSON", which would be a 500 on the page | Not in the brief. Base code behaves like B. |
Round trip: parseQuery(serialize(q)) re-serializes identically and gives identical SQL, on every accepted fuzz input |
0 violations | 0 violations | Match |
| Placeholders = params on every accepted fuzz input | 0 violations | 0 violations | Match |
The only behavioral differences between A and B are the last-but-one two rows: non-string input and malformed labels JSON. On every input the brief defines, the builds agree.
Security review of the page wiring
| Area | A | B |
|---|---|---|
| Permission | issuesPage and pullsPage still go through pageOf (prContext, repo.read) before any search. Unchanged. |
Same |
| SQL in the glue | List, counts and pagination queries interpolate only fixed table and column lists plus toSql output. LIMIT ? OFFSET ? are bound. |
Same. The table name comes from a literal union. |
| Scope on counts | Counts query is toSql(withState(q, "all")), so it is scoped |
Counts query is toSql(withState(q, null)), so it is scoped |
| XSS | The error echo, the message list <code> and value= all go through html```, which escapes & < > " '. Payload "> foo: label:"x" is rendered inert (issues-xss.html`). |
Same, verified the same way |
| Open redirects | No new redirects. Every link is searchHref(base, …) with base the repo path and a URLSearchParams-encoded q. |
Same |
| Pagination limits | page clamped to 1..400 (max OFFSET 9,975). 25 + 1 rows per page. |
Same (MAX_PAGE 400) |
| RepoDO hydrate failure | .catch logs pulls.list_hydrate_failed and renders from pull_index |
try/catch logs search.pulls_hydrate_failed and renders from pull_index |
| D1 failure | Not caught (500), same as base listIssueViews |
Not caught (500), same as base |
| Agent slug resolution | Subquery agents.org_id = ? AND slug = ? with ctx.orgId bound |
Same |
No security findings for either build, so no penalties.
Findings by severity
High: none in either build.
Medium:
- A, UI: The search box is 460px wide in the filter row, so at 1280 px the canonical query is cut off:
is:open label:"needs review" sort:upd. For a query language whose canonical form is the point of the box, that hurts. (SeeA-issues-query-1280-light.pngandB-issues-query-1280-light.png.)
Low:
- B:
label:,-label:andno:labelthrow in SQLite if any row in scope has malformedlabelsJSON, which makes the list page return 500. The base code had the same weakness. A'sLABELSCASE guard avoids it. - B:
parseQuery(null)andparseQuery(undefined)throw. This is outside the string contract, but A tolerates it. - B: Two D1 round trips per list (
Promise.allof.all()calls) where base and A use oneDB.batch. - B: Page glue (
search/lists.ts, which importsrepoStubandlog) lives insearch/next to the pure modules, andISSUE_COLSis duplicated instead of exported. - A:
test/unit/issues.test.tsstays red. The failure is inherited from base, but A edited that test and didn't fix it. B did. - A: The catch-all
try/catchinparseQueryis unreachable by design. If a bug ever made it reachable, it would report a whole-inputbad_valueinstead of surfacing the bug. - A: 11 casts. All safe today, but none are checked.
Info:
- B: Hydrated pull rows show the RepoDO's live title and state. A shows
pull_index's state with the DO's comment count. Both are allowed. - B: Old
?label=links with an invalid label show a parse error. A silently drops the label. Both are defensible. - A: On an invalid query, A shows a "Nothing listed" empty state with a Clear button. B shows the error card with a "Clear the search" link and no list card. A's guidance is slightly clearer. A also renders the whole message list in the error color; B uses body text with a red heading, which reads more easily.
UI check
I rendered both builds' pages with each build's own page glue (repo/list-search.ts in A, search/lists.ts in B) against the hidden fixture through a node:sqlite D1 shim and a mocked RepoDO, then wrapped them in the real renderPage layout. The fixture had 30 extra issues so a second page exists. Script: judge-probes/render.mjs.
I screenshotted the pages with Playwright (Chromium) at 1280 and 390 px, in light and dark (judge-probes/shots.mjs). Pages covered:
- issues: default, a query, page 2, a legacy
?state=closed&label=buglink, an invalid query, an XSS payload - pulls: default, a query, an invalid query
- the help popover on both lists
What I checked:
- Links and counts: every tab href, count, pager link and label-chip link is identical between the builds.
- Overflow: no horizontal overflow at 390 px on any page for either build.
- Help:
<details>opens with Enter from keyboard focus, and also with JavaScript disabled. - Help contents: each list shows only its own qualifiers. Pulls never show
label:orno:; issues never showreview-requested:. - Error marks: the offending spans are underlined (wavy) and carry a number, so they don't rely on color alone. A adds a visually hidden "(problem n)"; B marks the echo
aria-hiddenand lists the terms in<code>. B also putsrole="alert"on the error card and pointsaria-describedbyat the help. - DESIGN.md: no uppercase text transforms, gradients or em dashes in the new UI. Inter body text and mono for queries in both.
What TSUS demonstrably caught or prevented, and what it cost
Caught during the build: nothing. All 8 tsus-web.mjs check --json runs in B's transcript returned {"ok": true, "diagnostics": []}. B's only failures came from its own unit tests (2 parse expectations, 2 SQL expectations) and the repo's design test (a raw z-index), the same kinds of failures A hit.
Prevented, by construction: unchecked narrowing. A narrows strings to unions with as after a runtime check. B has to prove the narrowing with a total match:
// A, parse.ts
if (!Object.hasOwn(KEYS, key)) return fail("unknown_qualifier", ...);
const k = key as Key;
...
const op = m[1] as DateOp;
return { kind: "age", op, n, unit: rel[2].toLowerCase() as RelUnit };// B, parse.tsus
const keyOf = (k: string): Option<Key> =>
match (k) {
"is" => Some<Key>("is"),
"label" => Some<Key>("label"),
...
_ => None,
};A design both reached, by different routes: NULL-safe negation. A wraps every negated term generically. B makes each positive form never NULL (IS instead of = on the nullable column). Both are correct; A's is one rule, and B's needs care per term:
// A, sql.ts
parts.push("neg" in t && t.neg ? `NOT IFNULL((${c.sql}), 0)` : `(${c.sql})`);// B, sql.tsus
User(x) => Some(frag("(pull_index.author_id = ? OR pull_index.owner_id IS ?)", [x.id, x.id])),Where B cost more:
- Adapter code.
api.ts(51 lines) converts at every boundary.nullbecomesNone,Resultbecomes{ ok }, and the edit helpers needOptionwrappers:
// B, api.ts
export function toSql(query: SearchQuery, ctx: SqlCtx) {
const r = sql(query, { repoId: ctx.repoId, orgId: ctx.orgId, viewer: ctx.viewerId === null ? None : Some(ctx.viewerId), now: ctx.now });
return { where: r.where, params: [...r.params], orderBy: r.orderBy };
}
export function tabOf(query: SearchQuery): ListState | null {
const t = tabOfQuery(query);
return t.isSome() ? t.value : null;
}- Ban workarounds. These are where plain indexing would have been clearer:
// B, values.tsus: parts[0] and parts[1] without indexing
const parts = v.split("..");
if (parts.length !== 2) return None;
const from = endOf(parts.slice(0, 1).join(""))?;
const to = endOf(parts.slice(1).join(""))?;// B, edit.tsus: "the one element, if exactly one"
const only = states.length === 1 ? states.slice(0, 1) : [];
for (const s of only) if (!s.negated) return Some(s.state);
return None;Compare A:
// A, edit.ts
const [t] = s;
return s.length === 1 && t.kind === "is" && !t.neg ? (t.value as StateTab) : null;- Volume and calls:
| A | B | |
|---|---|---|
| Search module lines | 626 | 1,033 (+65%) |
| Output tokens | 35.7k | 57.7k (+61%) |
| Tool calls | 117 | 150 (+33) |
About 15 of B's extra calls went to TSUS itself: 7 reads of tools/tsus/AGENTS.md, the runtime .d.ts and the templates, plus 8 check runs. The rest went to a longer UI preview session and a WIP-commit-then-soft-reset. The total token gap is small (+4.4%) because input and cache reads dominate.
Where TSUS did not cost more: wall clock (25.1 vs 25.9 min) and behavior (identical). TSUS added no runtime overhead worth mentioning: 18.8 vs 13.5 ms per 10k queries, both far below a request's budget.
Ship recommendation
Ship A: same behavior, cheaper, one D1 round trip, tolerant of bad labels JSON. Before shipping, fix three things:
- Widen the search box so the canonical query is visible on desktop. B's full-width box above the tabs is the better layout; take it.
- Fix the inherited
aria-current="page">Issues<assertion intest/unit/issues.test.ts(B's regex fix works). - Optionally, drop the unreachable catch-all in
parseQuery, or make it log.
If the team prefers B, it's equally shippable. Add the json_valid guard on labels (or accept base's behavior) and batch the two D1 statements.
Notes on the comparison
- Hidden tests didn't separate the builds. They pass 118/118 each, and my fuzzing found no disagreement on any input the brief defines. The two independent implementations agree with each other and with the hidden tests, which is good evidence the brief was unambiguous.
- Environment friction hit both builders. The worktree guard refused compound
ln/gitcommands 7 times for A and 9 times for B. The pre-existing redissues.test.tsaffected both equally; only B spent calls on it, and only B fixed it. - UI was judged from static renders through each build's own glue and the real layout, not a live
wrangler devstack with sessions. Signed-in versus signed-out differences (New issue button, sign-in prompt) weren't exercised beyond thecanOpenflag. Neither build changed them.