Rebasely Beta: data may be purged

justintypescript-ultra-strict Public

Markdown · 5.8 KiB · at 9c48a39

tsus evals

An eval harness for the thesis behind TypeScript-Ultra-Strict: when a model writes the code, the .tsus version is correct more often than the plain TypeScript version.

Results live in RESULTS.md.

How it works

Each task in tasks/<id>/ has:

File Purpose
task.json title, tags, and whether it is in the pilot set
prompt.md the task statement, shared by both languages
signature.ts, signature.tsus the API the model must implement, per language
test.mjs hidden node:test tests, run against the compiled JS of either language
reference.ts, reference.tsus known-good solutions (used by validate.mjs and the self-test providers)
mutant.ts, mutant.tsus plausible buggy solutions the hidden tests must reject

The two signatures express the same contract idiomatically. In .ts, absence is undefined or an optional field and failure is a thrown error class with a kind field. In .tsus, absence is Option<T> and failure is Result<T, E> with a { kind } error. lib/harness.mjs normalizes both, so one test file covers both languages and a crash (an unexpected TypeError, say) always counts as a failure.

run.mjs, for every task and language:

  1. Sends the model a fixed system prompt (prompts/system-ts.md, or prompts/system-tsus.md with LANGUAGE.md inlined) plus the task prompt and signature.
  2. Compiles the reply. .ts: tsc with strict: true, lib ES2022, no DOM or Node types. .tsus: tsus check --json, then tsus build --json, using the compiler in packages/cli at the current commit.
  3. If the compiler rejects it, sends the diagnostics back with the previous attempt and asks for a fix, up to --repairs rounds (default 3).
  4. Runs the hidden tests once, on the first version that compiles. The model never sees test results.

It records, per task and language: whether the first attempt compiled and passed (pass@1), whether any attempt passed (pass after repair), the number of repair rounds, every diagnostic code the compiler raised, failing hidden tests, tokens, cost, and time. Every attempt's code and diagnostics are saved under results/<run>/attempts/.

Running it

From the repository root, after npm install:

Shell
node evals/validate.mjs                      # references pass, mutants fail; no model
node evals/run.mjs --provider selftest       # whole pipeline incl. repair loop; no model
node evals/run.mjs --pilot                   # the 10 pilot tasks, both languages
node evals/run.mjs                           # all tasks
node evals/report.mjs                        # rewrite evals/RESULTS.md from evals/results/

Useful flags: --tasks a,b, --langs tsus, --model <id> (default claude-sonnet-5-5, or TSUS_EVAL_MODEL), --repairs N, --concurrency N (default 4), --max-cost <usd> (stops starting new model calls past this spend), --label <text>, --tsus-bin <path> (test another compiler build).

Model access

  • Claude Code CLI (default when ANTHROPIC_API_KEY is unset): each call is claude -p --model <id> --system-prompt <fixed prompt> --tools "" --strict-mcp-config --no-session-persistence --output-format json, run from an empty temp directory so no project CLAUDE.md is picked up. Cost comes from the CLI's own total_cost_usd. The CLI must be logged in: run claude once interactively and /login if claude -p "hi" reports an authentication error. If you start the eval from inside a Claude Code session, the runner strips the variables that tie a child claude to the parent session.
  • Anthropic API (default when ANTHROPIC_API_KEY is set, or --provider anthropic): needs npm install --prefix evals for @anthropic-ai/sdk. Cost is computed from token usage and the price table in lib/providers.mjs. Add --effort low|medium|high to set the effort level.
  • reference and selftest call no model. They exist to prove the harness works.

Keys are never printed or written to results; both paths read credentials themselves.

Rough cost: a task is about 2k to 4k input tokens (the .tsus side carries the primer) and 1k to 3k output tokens per attempt. At Sonnet 5.5 prices ($2 / $10 per million tokens) a full 30-task, two-language run with few repairs is on the order of $1 to $2; the 10-task pilot is well under $1. Use --max-cost to cap it.

Adding a task

Copy tasks/parse-config/ and keep its conventions. The prompt states the contract precisely but does not list the edge cases; the tests probe them. Every test must follow from the prompt. Then run node evals/validate.mjs <id> until the references pass and the mutants fail.

Caveats

These matter more than any number in RESULTS.md.

  • The tsus checker is younger than tsc. At the time of the first runs, tsus check enforced the five bans and the macro syntax but did no semantic type-checking, so .ts got more compile-time help than .tsus. The harness always uses the compiler at the current commit and records that commit in each run, so results before and after the typed checker lands are comparable side by side.
  • The tasks favor the thesis by design. They were picked because null, undefined, exceptions and unhandled cases cause real bugs in them, and the tasks and tests were written with Claude. A neutral task set (or tasks drawn from real bug reports) would be a fairer test.
  • Prompt asymmetry. The model knows TypeScript from training and learns tsus from a primer in the system prompt. That probably hurts tsus, but it also means the tsus side gets more instructions.
  • One sample per task, no fixed temperature on the CLI path. Re-run to see variance; a difference of a task or two is noise.
  • Contracts differ slightly in shape. .ts throws typed error classes, .tsus returns Err. The tests treat both as the same outcome, but a model can lose a .ts test by throwing a plain Error, which has no .tsus analogue.