justintsus-experiment Public
Rubric: TypeScript vs TypeScript-Ultra-Strict, search query language
You are the judge. Two builders implemented the same brief (BRIEF.md in this directory; builder A
got BRIEF-ts.md, builder B got BRIEF-tsus.md, identical except the "Language" section). Score
each build on its own against this rubric, then compare. Do not reward or penalize the language
itself: judge what each builder produced under its constraints. Give every score with the evidence
(file and line, test name, command output).
You are given, per builder: the worktree path and branch, and from the harness the token count and tool-call count. Run everything yourself; do not trust a builder's own report.
0. Preparation (no points)
- In each worktree:
git log --oneline main..HEADandgit diff main...HEAD --stat. Note anything committed that should not be (generated.jsor.d.tsunderapps/web/src, a migration, edits toscripts/tsus-web.mjs,tools/tsus, the wrangler config or CI, edits to unrelated features). Changing the build wiring or the vendored compiler is a red flag for B and a scope violation for either. npm run typecheckandnpm run lintin each worktree (record pass or fail and the output).- Run the builder's own tests (whatever they added under
test/unit/), and record the count.
1. Hidden tests: 40 points
Run hidden-tests/run.sh <worktree> (it links the worktree's node_modules, runs
scripts/tsus-web.mjs build, then all hidden-tests/*.test.mjs). The last line is
hidden tests: pass N / total M. Score = 40 x N / M, rounded to one decimal.
- If
run.shexits 2 (missingapi.ts, or the.tsusbuild fails), the score is 0. - Do not edit the tests or the build to make them run. If a test fails because of an ambiguity in the brief (not a bug), note it, but keep the score mechanical; the same tests run for both.
- 118 tests: parsing (42), canonical form and round trips (9), semantics on SQLite (30), dates (18),
SQL safety (9, three of them covering 15 injection payloads), scope and generated inputs (10), plus the structural
checks every SQL-running test makes (placeholders equal params, fixed
orderBy, every returned row in ctx's repository and organization). reference/holds a quick reference implementation that passes all 118; it exists only to show the tests are satisfiable. Do not use it to grade code style.
2. Security: automatic severe penalty (up to minus 40, applied after the total)
Look for these yourself in the code (the evaluator and the page glue), not only in test failures. Each confirmed finding:
| Finding | Penalty |
|---|---|
| A value from the query text or from ctx is written into SQL text (string concatenation or a template literal with a value), anywhere: evaluator, page glue, count queries, pagination | -40, and the build cannot win |
| A query (any input) can return rows outside the repository or organization, or the page runs a list query without the scope (for example the counts or "next page" query) | -40, and the build cannot win |
The page lists rows without the route's existing repo.read decision (prContext/pageOf), or the search reaches private data another way (for example resolving an agent slug or user across organizations) |
-30 |
An unescaped LIKE pattern (%, _, \ from the input act as wildcards) |
-5 |
| Error messages or the page echo the raw query into HTML without escaping (XSS) | -20 |
parseQuery, toSql or the page throws (a 500) on some input |
-10 per distinct crash |
Hidden-test failures in the SQL safety or scope groups count here too: any failure in
sql-safety.test.mjs or permission.test.mjs is investigated, and if it is a real injection or
scope bug it takes the penalty above in addition to the lost test points.
3. Code review: 30 points
Read the diff as a senior reviewer on this codebase would. Score each line item and justify it.
| Item | Points | What good looks like |
|---|---|---|
| Correctness beyond the tests | 8 | Grammar and semantics match the brief where the tests do not reach (error positions for odd input, -author: NULL handling, is:closed on pulls including merged, slug resolution scoped to the org, the page's default is:open, legacy state=/label= links). Spot-check with your own inputs. |
| Error handling | 6 | Bad input is data end to end: typed error codes, spans, one error per bad term, no exceptions in the parser or evaluator, D1 failures on the page handled like the rest of the codebase. |
| Readability and structure | 6 | A reader can follow tokenizer, parser, serializer and evaluator; clear types (a real AST, a closed error union); no dead code; names and comments in the repository's style (see neighbouring files). |
| Edge cases | 5 | Limits, unicode offsets, whitespace kinds, empty input, quoting and escaping in both directions, pagination with a sort, counts for the tabs. |
| Tests the builder wrote | 3 | Useful unit tests of their own, covering what they changed. |
| Fit with the codebase | 2 | Follows existing patterns (D1 access, Out/page helpers, DESIGN.md), touches only what the feature needs, no new dependencies. |
For builder B also check the tsus constraint was honored: parser, serializer and evaluator in
.tsus, api.ts a thin adapter with no logic of its own. Logic moved into .ts to dodge the
checker loses up to 6 points from "Readability and structure" and "Fit". For builder A check that no
.tsus was used.
4. UI check: 20 points
Start the stack in each worktree (npm run dev:v2, port 8790; seed a repository with a few issues,
labels and pull requests in several states, signed in and signed out; the repository's test helpers
and test/helpers/seed.ts show how). Check desktop and a 375 px phone width, light and dark.
| Item | Points |
|---|---|
| The search box on both lists runs the language; results are right for a handful of queries you try (state, label, author, text, date, sort) | 6 |
| Errors are shown inline: message per error under the box, offending span marked without relying on color alone, input kept as typed, status 200, no results listed | 5 |
| Syntax help: present on both lists, lists only the qualifiers of that list, works by keyboard and without JavaScript | 3 |
URL round trip: ?q= holds the canonical form; reload, back, the state tabs, a label link and next page all keep the query |
3 |
Design: follows DESIGN.md and the existing list styles (one accent, Inter 450, no stripes, gradients, emoji, uppercase kickers, em dashes in copy); fits on a phone with no horizontal scroll |
3 |
5. Cost: 10 points
The harness supplies, per builder, total tokens (input plus output, cache reads counted as the harness reports them) and tool calls. Score relative to the cheaper build:
- tokens: 6 x (fewer tokens / this build's tokens)
- tool calls: 4 x (fewer calls / this build's calls)
So the cheaper build gets the full 10 and the other gets proportionally less. Report the raw numbers too, and the cost per hidden test passed (tokens / N).
Total and verdict
Total = hidden tests (40) + code review (30) + UI (20) + cost (10) - security penalties, out of 100.
Report a table with every line item for both builds, the totals, and a verdict:
- The winner is the higher total, unless it carries a "cannot win" finding.
- If the totals are within 5 points, say it is a tie on quality and name the cheaper build.
- Then say plainly which build to ship (or that neither is shippable and why), and list what must be fixed before shipping it.
- Note anything about the comparison itself that was unfair or noisy (a hidden test the brief left ambiguous, an environment problem one builder hit and the other did not, a tsus toolchain bug).