evals: harness testing whether AI-written code is more correct in .tsus than .ts
30 tasks (prompt, .ts and .tsus signatures, hidden node:test tests, reference solutions in both languages, and a buggy mutant the tests must reject). run.mjs asks a model for each solution, compiles it (tsc --strict, or tsus check plus tsus build), feeds compiler errors back for up to 3 repair rounds, then runs the hidden tests once. Providers: claude -p headless (default), the Anthropic API when ANTHROPIC_API_KEY is set, and two no-model self-test providers. report.mjs writes RESULTS.md. No model pilot yet: the local Claude Code CLI login had expired and no API key was available, so RESULTS.md holds only the self-test, labeled as such. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Changed files
261 files changed3,366 additions0 deletionsagainst 6dae872
- ChangedREADME.md+6−0
- Addedevals/.gitignore+1−0
- Addedevals/LANGUAGE.md+97−0
- Addedevals/README.md+67−0
- Addedevals/RESULTS.md+78−0
- Addedevals/lib/compile.mjs+172−0
- Addedevals/lib/harness.mjs+185−0
- Addedevals/lib/providers.mjs+210−0
- Addedevals/package-lock.json+102−0
- Addedevals/package.json+15−0
- Addedevals/prompts/system-ts.md+7−0
- Addedevals/prompts/system-tsus.md+9−0
- Addedevals/report.mjs+204−0
- Addedevals/results/2026-10-17-05-27-00-selftest/meta.json+66−0
- Addedevals/results/2026-10-17-05-27-00-selftest/results.jsonl+60−0
- Addedevals/run.mjs+297−0
- Addedevals/tasks/bank-transfer/mutant.ts+42−0
- Addedevals/tasks/bank-transfer/prompt.md+17−0
- Addedevals/tasks/bank-transfer/reference.ts+47−0
- Addedevals/tasks/bank-transfer/reference.tsus+45−0
- Addedevals/tasks/bank-transfer/signature.ts+29−0
- Addedevals/tasks/bank-transfer/signature.tsus+19−0
- Addedevals/tasks/bank-transfer/task.json+1−0
- Addedevals/tasks/bank-transfer/test.mjs+89−0
- Addedevals/tasks/cart-total/mutant.ts+51−0
- Addedevals/tasks/cart-total/prompt.md+18−0
- Addedevals/tasks/cart-total/reference.ts+54−0
- Addedevals/tasks/cart-total/reference.tsus+55−0
- Addedevals/tasks/cart-total/signature.ts+32−0
- Addedevals/tasks/cart-total/signature.tsus+22−0
- Addedevals/tasks/cart-total/task.json+1−0
- Addedevals/tasks/cart-total/test.mjs+100−0
- Addedevals/tasks/csv-rows/mutant.ts+52−0
- Addedevals/tasks/csv-rows/prompt.md+16−0
- Addedevals/tasks/csv-rows/reference.ts+99−0
- Addedevals/tasks/csv-rows/reference.tsus+108−0
- Addedevals/tasks/csv-rows/signature.ts+20−0
- Addedevals/tasks/csv-rows/signature.tsus+11−0
- Addedevals/tasks/csv-rows/task.json+1−0
- Addedevals/tasks/csv-rows/test.mjs+88−0
- Addedevals/tasks/date-range-merge/mutant.ts+37−0
- Addedevals/tasks/date-range-merge/prompt.md+12−0
- Addedevals/tasks/date-range-merge/reference.ts+59−0
- Addedevals/tasks/date-range-merge/reference.tsus+75−0
- Addedevals/tasks/date-range-merge/signature.ts+19−0
- Addedevals/tasks/date-range-merge/signature.tsus+11−0
- Addedevals/tasks/date-range-merge/task.json+1−0
- Addedevals/tasks/date-range-merge/test.mjs+104−0
- Addedevals/tasks/deep-get/mutant.ts+20−0
- Addedevals/tasks/deep-get/prompt.md+15−0
- Addedevals/tasks/deep-get/reference.ts+62−0
- Addedevals/tasks/deep-get/reference.tsus+73−0
- Addedevals/tasks/deep-get/signature.ts+13−0
- Addedevals/tasks/deep-get/signature.tsus+7−0
- Addedevals/tasks/deep-get/task.json+1−0
- Addedevals/tasks/deep-get/test.mjs+106−0
- Addedevals/tasks/feature-flag/mutant.ts+27−0
- Addedevals/tasks/feature-flag/prompt.md+20−0
- Addedevals/tasks/feature-flag/reference.ts+57−0
- Addedevals/tasks/feature-flag/reference.tsus+54−0
- Addedevals/tasks/feature-flag/signature.ts
- Addedevals/tasks/feature-flag/signature.tsus
- Addedevals/tasks/feature-flag/task.json
- Addedevals/tasks/feature-flag/test.mjs
- Addedevals/tasks/free-slot/mutant.ts
- Addedevals/tasks/free-slot/prompt.md
- Addedevals/tasks/free-slot/reference.ts
- Addedevals/tasks/free-slot/reference.tsus
- Addedevals/tasks/free-slot/signature.ts
- Addedevals/tasks/free-slot/signature.tsus
- Addedevals/tasks/free-slot/task.json
- Addedevals/tasks/free-slot/test.mjs
- Addedevals/tasks/inventory-ledger/mutant.ts
- Addedevals/tasks/inventory-ledger/mutant.tsus
- Addedevals/tasks/inventory-ledger/prompt.md
- Addedevals/tasks/inventory-ledger/reference.ts
- Addedevals/tasks/inventory-ledger/reference.tsus
- Addedevals/tasks/inventory-ledger/signature.ts
- Addedevals/tasks/inventory-ledger/signature.tsus
- Addedevals/tasks/inventory-ledger/task.json
- Addedevals/tasks/inventory-ledger/test.mjs
- Addedevals/tasks/json-api-handler/mutant.ts
- Addedevals/tasks/json-api-handler/mutant.tsus
- Addedevals/tasks/json-api-handler/prompt.md
- Addedevals/tasks/json-api-handler/reference.ts
- Addedevals/tasks/json-api-handler/reference.tsus
- Addedevals/tasks/json-api-handler/signature.ts
- Addedevals/tasks/json-api-handler/signature.tsus
- Addedevals/tasks/json-api-handler/task.json
- Addedevals/tasks/json-api-handler/test.mjs
- Addedevals/tasks/lru-cache/mutant.ts
- Addedevals/tasks/lru-cache/mutant.tsus
- Addedevals/tasks/lru-cache/prompt.md
- Addedevals/tasks/lru-cache/reference.ts
- Addedevals/tasks/lru-cache/reference.tsus
- Addedevals/tasks/lru-cache/signature.ts
- Addedevals/tasks/lru-cache/signature.tsus
- Addedevals/tasks/lru-cache/task.json
- Addedevals/tasks/lru-cache/test.mjs
- Addedevals/tasks/manager-chain/mutant.ts
- Addedevals/tasks/manager-chain/prompt.md
- Addedevals/tasks/manager-chain/reference.ts
- Addedevals/tasks/manager-chain/reference.tsus
- Addedevals/tasks/manager-chain/signature.ts
- Addedevals/tasks/manager-chain/signature.tsus
- Addedevals/tasks/manager-chain/task.json
- Addedevals/tasks/manager-chain/test.mjs
- Addedevals/tasks/merge-profile/mutant.ts
- Addedevals/tasks/merge-profile/prompt.md
- Addedevals/tasks/merge-profile/reference.ts
- Addedevals/tasks/merge-profile/reference.tsus
- Addedevals/tasks/merge-profile/signature.ts
- Addedevals/tasks/merge-profile/signature.tsus
- Addedevals/tasks/merge-profile/task.json
- Addedevals/tasks/merge-profile/test.mjs
- Addedevals/tasks/moving-average/mutant.ts
- Addedevals/tasks/moving-average/prompt.md
- Addedevals/tasks/moving-average/reference.ts
- Addedevals/tasks/moving-average/reference.tsus
- Addedevals/tasks/moving-average/signature.ts
- Addedevals/tasks/moving-average/signature.tsus
- Addedevals/tasks/moving-average/task.json
- Addedevals/tasks/moving-average/test.mjs
- Addedevals/tasks/order-state-machine/mutant.ts
- Addedevals/tasks/order-state-machine/prompt.md
- Addedevals/tasks/order-state-machine/reference.ts
- Addedevals/tasks/order-state-machine/reference.tsus
- Addedevals/tasks/order-state-machine/signature.ts
- Addedevals/tasks/order-state-machine/signature.tsus
- Addedevals/tasks/order-state-machine/task.json
- Addedevals/tasks/order-state-machine/test.mjs
- Addedevals/tasks/paginate-cursor/mutant.ts
- Addedevals/tasks/paginate-cursor/prompt.md
- Addedevals/tasks/paginate-cursor/reference.ts
- Addedevals/tasks/paginate-cursor/reference.tsus
- Addedevals/tasks/paginate-cursor/signature.ts
- Addedevals/tasks/paginate-cursor/signature.tsus
- Addedevals/tasks/paginate-cursor/task.json
- Addedevals/tasks/paginate-cursor/test.mjs
- Addedevals/tasks/parse-amount/mutant.ts
- Addedevals/tasks/parse-amount/prompt.md
- Addedevals/tasks/parse-amount/reference.ts
- Addedevals/tasks/parse-amount/reference.tsus
- Addedevals/tasks/parse-amount/signature.ts
- Addedevals/tasks/parse-amount/signature.tsus
- Addedevals/tasks/parse-amount/task.json
- Addedevals/tasks/parse-amount/test.mjs
- Addedevals/tasks/parse-config/mutant.ts
- Addedevals/tasks/parse-config/prompt.md
- Addedevals/tasks/parse-config/reference.ts
- Addedevals/tasks/parse-config/reference.tsus
- Addedevals/tasks/parse-config/signature.ts
- Addedevals/tasks/parse-config/signature.tsus
- Addedevals/tasks/parse-config/task.json
- Addedevals/tasks/parse-config/test.mjs
- Addedevals/tasks/parse-duration/mutant.ts
- Addedevals/tasks/parse-duration/prompt.md
- Addedevals/tasks/parse-duration/reference.ts
- Addedevals/tasks/parse-duration/reference.tsus
- Addedevals/tasks/parse-duration/signature.ts
- Addedevals/tasks/parse-duration/signature.tsus
- Addedevals/tasks/parse-duration/task.json
- Addedevals/tasks/parse-duration/test.mjs
- Addedevals/tasks/parse-log-line/mutant.ts
- Addedevals/tasks/parse-log-line/prompt.md
- Addedevals/tasks/parse-log-line/reference.ts
- Addedevals/tasks/parse-log-line/reference.tsus
- Addedevals/tasks/parse-log-line/signature.ts
- Addedevals/tasks/parse-log-line/signature.tsus
- Addedevals/tasks/parse-log-line/task.json
- Addedevals/tasks/parse-log-line/test.mjs
- Addedevals/tasks/query-params/mutant.ts
- Addedevals/tasks/query-params/prompt.md
- Addedevals/tasks/query-params/reference.ts
- Addedevals/tasks/query-params/reference.tsus
- Addedevals/tasks/query-params/signature.ts
- Addedevals/tasks/query-params/signature.tsus
- Addedevals/tasks/query-params/task.json
- Addedevals/tasks/query-params/test.mjs
- Addedevals/tasks/resolve-path/mutant.ts
- Addedevals/tasks/resolve-path/prompt.md
- Addedevals/tasks/resolve-path/reference.ts
- Addedevals/tasks/resolve-path/reference.tsus
- Addedevals/tasks/resolve-path/signature.ts
- Addedevals/tasks/resolve-path/signature.tsus
- Addedevals/tasks/resolve-path/task.json
- Addedevals/tasks/resolve-path/test.mjs
- Addedevals/tasks/retry-fetch/mutant.ts
- Addedevals/tasks/retry-fetch/mutant.tsus
- Addedevals/tasks/retry-fetch/prompt.md
- Addedevals/tasks/retry-fetch/reference.ts
- Addedevals/tasks/retry-fetch/reference.tsus
- Addedevals/tasks/retry-fetch/signature.ts
- Addedevals/tasks/retry-fetch/signature.tsus
- Addedevals/tasks/retry-fetch/task.json
- Addedevals/tasks/retry-fetch/test.mjs
- Addedevals/tasks/route-match/mutant.ts
- Addedevals/tasks/route-match/prompt.md
- Addedevals/tasks/route-match/reference.ts
- Addedevals/tasks/route-match/reference.tsus
- Addedevals/tasks/route-match/signature.ts
- Addedevals/tasks/route-match/signature.tsus
- Addedevals/tasks/route-match/task.json
- Addedevals/tasks/route-match/test.mjs
- Addedevals/tasks/semver-compare/mutant.ts
- Addedevals/tasks/semver-compare/prompt.md
- Addedevals/tasks/semver-compare/reference.ts
- Addedevals/tasks/semver-compare/reference.tsus
- Addedevals/tasks/semver-compare/signature.ts
- Addedevals/tasks/semver-compare/signature.tsus
- Addedevals/tasks/semver-compare/task.json
- Addedevals/tasks/semver-compare/test.mjs
- Addedevals/tasks/shortest-path/mutant.ts
- Addedevals/tasks/shortest-path/prompt.md
- Addedevals/tasks/shortest-path/reference.ts
- Addedevals/tasks/shortest-path/reference.tsus
- Addedevals/tasks/shortest-path/signature.ts
- Addedevals/tasks/shortest-path/signature.tsus
- Addedevals/tasks/shortest-path/task.json
- Addedevals/tasks/shortest-path/test.mjs
- Addedevals/tasks/signup-validation/mutant.ts
- Addedevals/tasks/signup-validation/prompt.md
- Addedevals/tasks/signup-validation/reference.ts
- Addedevals/tasks/signup-validation/reference.tsus
- Addedevals/tasks/signup-validation/signature.ts
- Addedevals/tasks/signup-validation/signature.tsus
- Addedevals/tasks/signup-validation/task.json
- Addedevals/tasks/signup-validation/test.mjs
- Addedevals/tasks/stats-summary/mutant.ts
- Addedevals/tasks/stats-summary/prompt.md
- Addedevals/tasks/stats-summary/reference.ts
- Addedevals/tasks/stats-summary/reference.tsus
- Addedevals/tasks/stats-summary/signature.ts
- Addedevals/tasks/stats-summary/signature.tsus
- Addedevals/tasks/stats-summary/task.json
- Addedevals/tasks/stats-summary/test.mjs
- Addedevals/tasks/token-bucket/mutant.ts
- Addedevals/tasks/token-bucket/prompt.md
- Addedevals/tasks/token-bucket/reference.ts
- Addedevals/tasks/token-bucket/reference.tsus
- Addedevals/tasks/token-bucket/signature.ts
- Addedevals/tasks/token-bucket/signature.tsus
- Addedevals/tasks/token-bucket/task.json
- Addedevals/tasks/token-bucket/test.mjs
- Addedevals/tasks/topo-sort/mutant.ts
- Addedevals/tasks/topo-sort/prompt.md
- Addedevals/tasks/topo-sort/reference.ts
- Addedevals/tasks/topo-sort/reference.tsus
- Addedevals/tasks/topo-sort/signature.ts
- Addedevals/tasks/topo-sort/signature.tsus
- Addedevals/tasks/topo-sort/task.json
- Addedevals/tasks/topo-sort/test.mjs
- Addedevals/tasks/ttl-cache/mutant.ts
- Addedevals/tasks/ttl-cache/prompt.md
- Addedevals/tasks/ttl-cache/reference.ts
- Addedevals/tasks/ttl-cache/reference.tsus
- Addedevals/tasks/ttl-cache/signature.ts
- Addedevals/tasks/ttl-cache/signature.tsus
- Addedevals/tasks/ttl-cache/task.json
- Addedevals/tasks/ttl-cache/test.mjs
- Addedevals/validate.mjs
ChangedREADME.md+6−0
| @@ -72,6 +72,12 @@ | ||
| - Diagnostics are machine-first: every error has a stable code, a JSON output mode, and a suggested fix, so an agent can consume `tsus check --json` in a loop without parsing prose. Human-readable output is a rendering of the same data. | ||
| - No new package manager and no new registry. npm packages work through the boundary layer, and published `.tsus` packages ship compiled JS plus `.d.ts` files, so consumers don't need to know the source language existed. | ||
| +## Evidence | ||
| + | ||
| +`evals/` holds a harness that tests the thesis directly: a model writes the same 30 small programs (config parsing, an LRU cache, retries with typed errors, a state machine, date ranges, an inventory ledger, and so on) in `.ts` and in `.tsus`, the compiler's errors go back to it for up to three repair rounds, and hidden tests the model never sees decide which version is correct. The tasks were chosen because null, undefined, exceptions and unhandled cases cause real bugs in them. Every task ships a reference solution in both languages and a plausible buggy solution that the hidden tests must catch (`node evals/validate.mjs`). | ||
| + | ||
| +Results are in [evals/RESULTS.md](evals/RESULTS.md). So far that file holds only the harness self-test, which runs the reference solutions through the pipeline and says nothing about the thesis. No model run has been recorded yet; `node evals/run.mjs --pilot` runs the 10-task pilot once the Claude Code CLI is logged in or `ANTHROPIC_API_KEY` is set. Even then, one model, one sample per task and 10 to 30 tasks is a pilot, not proof. [evals/README.md](evals/README.md) lists the caveats, starting with this one: the tsus checker does not type-check yet, so `.ts` currently gets more compile-time help than `.tsus`. | ||
| + | ||
| ## The examples | ||
| `examples/` is the test suite: 29 small projects, each with an `example.json` declaring what must happen. `npm run examples` builds and executes all of them and fails on any drift. | ||
Addedevals/.gitignore+1−0
| @@ -0,0 +1 @@ | ||
| +.work/ | ||
Addedevals/LANGUAGE.md+97−0
| @@ -0,0 +1,97 @@ | ||
| +# TypeScript-Ultra-Strict (tsus) primer | ||
| + | ||
| +This is the language reference the eval harness gives the model for the `.tsus` side. It describes the compiler as it exists in this repository (`packages/cli`, `packages/runtime`), not the full proposal. If the compiler changes (typed `match`, a `?` operator), update this file in the same commit so the primer never teaches a language the checker does not implement. | ||
| + | ||
| +## What a .tsus file is | ||
| + | ||
| +A `.tsus` file is TypeScript with five things removed and one macro added. It compiles to plain ESM JavaScript. | ||
| + | ||
| +Removed, each with a stable diagnostic code: | ||
| + | ||
| +| Code | Banned | Use instead | | ||
| +|---|---|---| | ||
| +| TSUS001 | `class` declarations and expressions | plain object types and functions | | ||
| +| TSUS002 | `null` | `Option<T>` (`Some(v)` / `None`) | | ||
| +| TSUS003 | the identifier `undefined` | `Option<T>`; `fromNullable` at the boundary | | ||
| +| TSUS004 | `var` | `const` (preferred) or `let` | | ||
| +| TSUS005 | `throw` | return `Result<T, E>` (`Ok(v)` / `Err(e)`) | | ||
| + | ||
| +Also do not use optional properties (`x?: T`), optional parameters, or optional chaining (`a?.b`). They produce `undefined` at runtime, which defeats the point. Model absence as an `Option<T>` field instead. The one exception is describing the shape of external data (parsed JSON, a `.ts` dependency) right before you convert it with `fromNullable`. | ||
| + | ||
| +## The runtime | ||
| + | ||
| +```ts | ||
| +import { Some, None, Ok, Err, tag, type Option, type Result, type Tagged } from "@tsus/runtime"; | ||
| +import { fromNullable, wrapSync, wrapAsync } from "@tsus/runtime/boundary"; | ||
| +``` | ||
| + | ||
| +`Option<T>` methods: `isSome()`, `isNone()`, `map(f)`, `andThen(f)` (f returns an Option), `unwrapOr(fallback)`, `okOr(error)` (to a Result). | ||
| + | ||
| +`Result<T, E>` methods: `isOk()`, `isErr()`, `map(f)`, `mapErr(f)`, `andThen(f)` (f returns a Result), `unwrapOr(fallback)`, `ok()` (to an Option). | ||
| + | ||
| +There is no `unwrap()`. To get a value out, use `match` or `unwrapOr`. | ||
| + | ||
| +Constructors: `Some(v)`, `None` (a value, not a function), `Ok(v)`, `Err(e)`, and `tag("Name", payload)` for tagged unions (`Tagged<"Name", Payload>` is `{ tag: "Name"; value: Payload }`). | ||
| + | ||
| +## The match macro | ||
| + | ||
| +```ts | ||
| +const label = match (findUser(id)) { | ||
| + Some(user) => user.name, | ||
| + None => "anonymous", | ||
| +}; | ||
| + | ||
| +const message = match (parsePort(raw)) { | ||
| + Ok(port) => `listening on ${port}`, | ||
| + Err(e) => `bad port: ${e.kind}`, | ||
| +}; | ||
| +``` | ||
| + | ||
| +Rules: | ||
| + | ||
| +- Patterns: `Some(x)`, `None`, `Ok(x)`, `Err(e)`, any tag of a `Tagged` union (`Circle(c)` binds the payload), string or number literals (`"GET" => ...`, `404 => ...`), and the wildcard `_`. | ||
| +- Each arm body is a single expression. Do not write `Pattern(x) => { statements }`; braces are parsed as an object literal. When an arm needs several steps, call a helper function, or use a ternary. | ||
| +- Arms are tried top to bottom. A match with no matching arm crashes at runtime (`TSUS100`), so cover every variant or end with `_`. | ||
| +- If an arm body uses `await`, the whole match becomes a promise: write `await match (x) { ... }`. | ||
| +- `match` only fires as `match (expr) {`. A method call like `str.match(/re/)` is untouched. | ||
| + | ||
| +You do not have to use `match`; `if (opt.isSome())` plus `unwrapOr` or `map` is fine too. Use whatever is clearest. | ||
| + | ||
| +## The boundary: talking to JavaScript APIs | ||
| + | ||
| +Built-in and npm APIs still return `undefined`/`null` and still throw. Convert at the edge: | ||
| + | ||
| +- Nullable values: `fromNullable(map.get(key))`, `fromNullable(arr.find(p))`, `fromNullable(arr[i])` for an index that may be out of range, `fromNullable(obj[key])` for data from JSON, `fromNullable(str.match(re))`. | ||
| +- Throwing calls: `wrapSync(JSON.parse)(text)` returns `Result<any, Error>`; likewise `wrapSync(decodeURIComponent)(s)`. `wrapAsync(fn)` does the same for async functions and returns `Promise<Result<...>>`. | ||
| +- Values typed `unknown` (parsed JSON, request bodies): check with `typeof`, `Array.isArray`, `Number.isInteger`, and `fromNullable` for missing keys. `typeof v === "undefined"` is a string comparison and is allowed, but `fromNullable(v).isNone()` reads better. | ||
| + | ||
| +Inside the boundary, everything you write returns `Option` or `Result`, never `undefined` and never an exception. | ||
| + | ||
| +## A complete example | ||
| + | ||
| +```ts | ||
| +import { Some, None, Ok, Err, type Option, type Result } from "@tsus/runtime"; | ||
| +import { fromNullable, wrapSync } from "@tsus/runtime/boundary"; | ||
| + | ||
| +export type User = { id: string; email: Option<string> }; | ||
| +export type LoadError = { kind: "invalid_json" } | { kind: "not_found"; id: string }; | ||
| + | ||
| +const parseUsers = (raw: string): Result<User[], LoadError> => | ||
| + wrapSync(JSON.parse)(raw) | ||
| + .mapErr((): LoadError => ({ kind: "invalid_json" })) | ||
| + .map((data: Array<{ id: string; email?: string }>) => | ||
| + data.map((u) => ({ id: u.id, email: fromNullable(u.email) })), | ||
| + ); | ||
| + | ||
| +export const emailFor = (raw: string, id: string): Result<Option<string>, LoadError> => | ||
| + parseUsers(raw).andThen((users) => | ||
| + match (fromNullable(users.find((u) => u.id === id))) { | ||
| + Some(user) => Ok(user.email), | ||
| + None => Err({ kind: "not_found", id }), | ||
| + }, | ||
| + ); | ||
| +``` | ||
| + | ||
| +## Checker status | ||
| + | ||
| +`tsus check` enforces the five bans and the macro syntax. At the time this primer was written it did not yet run full semantic type-checking, so type errors in a `.tsus` file show up when the hidden tests run rather than at compile time. Write well-typed code anyway. | ||
Addedevals/README.md+67−0
| @@ -0,0 +1,67 @@ | ||
| +# tsus evals | ||
| + | ||
| +An eval harness for the thesis behind TypeScript-Ultra-Strict: when a model writes the code, the `.tsus` version is correct more often than the plain TypeScript version. | ||
| + | ||
| +Results live in [RESULTS.md](RESULTS.md). | ||
| + | ||
| +## How it works | ||
| + | ||
| +Each task in `tasks/<id>/` has: | ||
| + | ||
| +| File | Purpose | | ||
| +|---|---| | ||
| +| `task.json` | title, tags, and whether it is in the pilot set | | ||
| +| `prompt.md` | the task statement, shared by both languages | | ||
| +| `signature.ts`, `signature.tsus` | the API the model must implement, per language | | ||
| +| `test.mjs` | hidden `node:test` tests, run against the compiled JS of either language | | ||
| +| `reference.ts`, `reference.tsus` | known-good solutions (used by `validate.mjs` and the self-test providers) | | ||
| +| `mutant.ts`, `mutant.tsus` | plausible buggy solutions the hidden tests must reject | | ||
| + | ||
| +The two signatures express the same contract idiomatically. In `.ts`, absence is `undefined` or an optional field and failure is a thrown error class with a `kind` field. In `.tsus`, absence is `Option<T>` and failure is `Result<T, E>` with a `{ kind }` error. `lib/harness.mjs` normalizes both, so one test file covers both languages and a crash (an unexpected `TypeError`, say) always counts as a failure. | ||
| + | ||
| +`run.mjs`, for every task and language: | ||
| + | ||
| +1. Sends the model a fixed system prompt (`prompts/system-ts.md`, or `prompts/system-tsus.md` with [LANGUAGE.md](LANGUAGE.md) inlined) plus the task prompt and signature. | ||
| +2. Compiles the reply. `.ts`: `tsc` with `strict: true`, lib ES2022, no DOM or Node types. `.tsus`: `tsus check --json`, then `tsus build --json`, using the compiler in `packages/cli` at the current commit. | ||
| +3. If the compiler rejects it, sends the diagnostics back with the previous attempt and asks for a fix, up to `--repairs` rounds (default 3). | ||
| +4. Runs the hidden tests once, on the first version that compiles. The model never sees test results. | ||
| + | ||
| +It records, per task and language: whether the first attempt compiled and passed (pass@1), whether any attempt passed (pass after repair), the number of repair rounds, every diagnostic code the compiler raised, failing hidden tests, tokens, cost, and time. Every attempt's code and diagnostics are saved under `results/<run>/attempts/`. | ||
| + | ||
| +## Running it | ||
| + | ||
| +From the repository root, after `npm install`: | ||
| + | ||
| +```sh | ||
| +node evals/validate.mjs # references pass, mutants fail; no model | ||
| +node evals/run.mjs --provider selftest # whole pipeline incl. repair loop; no model | ||
| +node evals/run.mjs --pilot # the 10 pilot tasks, both languages | ||
| +node evals/run.mjs # all tasks | ||
| +node evals/report.mjs # rewrite evals/RESULTS.md from evals/results/ | ||
| +``` | ||
| + | ||
| +Useful flags: `--tasks a,b`, `--langs tsus`, `--model <id>` (default `claude-sonnet-5-5`, or `TSUS_EVAL_MODEL`), `--repairs N`, `--concurrency N` (default 4), `--max-cost <usd>` (stops starting new model calls past this spend), `--label <text>`, `--tsus-bin <path>` (test another compiler build). | ||
| + | ||
| +### Model access | ||
| + | ||
| +- **Claude Code CLI** (default when `ANTHROPIC_API_KEY` is unset): each call is `claude -p --model <id> --system-prompt <fixed prompt> --tools "" --strict-mcp-config --no-session-persistence --output-format json`, run from an empty temp directory so no project `CLAUDE.md` is picked up. Cost comes from the CLI's own `total_cost_usd`. The CLI must be logged in: run `claude` once interactively and `/login` if `claude -p "hi"` reports an authentication error. If you start the eval from inside a Claude Code session, the runner strips the variables that tie a child `claude` to the parent session. | ||
| +- **Anthropic API** (default when `ANTHROPIC_API_KEY` is set, or `--provider anthropic`): needs `npm install --prefix evals` for `@anthropic-ai/sdk`. Cost is computed from token usage and the price table in `lib/providers.mjs`. Add `--effort low|medium|high` to set the effort level. | ||
| +- `reference` and `selftest` call no model. They exist to prove the harness works. | ||
| + | ||
| +Keys are never printed or written to results; both paths read credentials themselves. | ||
| + | ||
| +Rough cost: a task is about 2k to 4k input tokens (the `.tsus` side carries the primer) and 1k to 3k output tokens per attempt. At Sonnet 5.5 prices ($2 / $10 per million tokens) a full 30-task, two-language run with few repairs is on the order of $1 to $2; the 10-task pilot is well under $1. Use `--max-cost` to cap it. | ||
| + | ||
| +## Adding a task | ||
| + | ||
| +Copy `tasks/parse-config/` and keep its conventions. The prompt states the contract precisely but does not list the edge cases; the tests probe them. Every test must follow from the prompt. Then run `node evals/validate.mjs <id>` until the references pass and the mutants fail. | ||
| + | ||
| +## Caveats | ||
| + | ||
| +These matter more than any number in RESULTS.md. | ||
| + | ||
| +- **The tsus checker is younger than tsc.** At the time of the first runs, `tsus check` enforced the five bans and the macro syntax but did no semantic type-checking, so `.ts` got more compile-time help than `.tsus`. The harness always uses the compiler at the current commit and records that commit in each run, so results before and after the typed checker lands are comparable side by side. | ||
| +- **The tasks favor the thesis by design.** They were picked because null, undefined, exceptions and unhandled cases cause real bugs in them, and the tasks and tests were written with Claude. A neutral task set (or tasks drawn from real bug reports) would be a fairer test. | ||
| +- **Prompt asymmetry.** The model knows TypeScript from training and learns tsus from a primer in the system prompt. That probably hurts tsus, but it also means the tsus side gets more instructions. | ||
| +- **One sample per task, no fixed temperature** on the CLI path. Re-run to see variance; a difference of a task or two is noise. | ||
| +- **Contracts differ slightly in shape.** `.ts` throws typed error classes, `.tsus` returns `Err`. The tests treat both as the same outcome, but a model can lose a `.ts` test by throwing a plain `Error`, which has no `.tsus` analogue. | ||
Addedevals/RESULTS.md+78−0
| @@ -0,0 +1,78 @@ | ||
| +# tsus eval results | ||
| + | ||
| +Generated by `node evals/report.mjs` from the runs in `evals/results/`. Method and caveats are in [evals/README.md](README.md). | ||
| + | ||
| +The question: when a model writes the same small program in plain TypeScript (`tsc --strict`) and in TypeScript-Ultra-Strict, which version is correct more often? Each of the 30 tasks has a prompt, a signature per language, and hidden `node:test` tests the model never sees. The compiler's errors are fed back for up to N repair rounds; test results never are. | ||
| + | ||
| +How to read the metrics: | ||
| + | ||
| +- **pass@1**: the first attempt compiled and passed every hidden test. | ||
| +- **Pass after repair**: some attempt within the repair budget compiled, and that version passed every hidden test. | ||
| +- **Compile-time catches**: attempts the compiler rejected. In .tsus these include the ban rules (TSUS001 to TSUS005) and macro errors. A catch is only good news if the repaired version then passes; "tasks rescued by repair" counts those. | ||
| +- **Failing at runtime**: the solution compiled but at least one hidden test failed. These are the bugs that would have shipped. | ||
| + | ||
| +**No model run has been recorded yet.** The sections below come from the harness self-test, which uses the reference solutions instead of a model. See evals/README.md for how to run the pilot. | ||
| + | ||
| +## Harness self-test: 2026-10-17, selftest | ||
| + | ||
| +**No model was called in this run.** The `selftest` provider feeds the hand-written reference solutions (with a planted compile error on the first attempt, to exercise the repair loop) through the full pipeline. It shows the harness and the hidden tests work end to end. It says nothing about the thesis. | ||
| + | ||
| +Run `2026-10-17-05-27-00-selftest` (evals/results/2026-10-17-05-27-00-selftest): provider `selftest`, model `none`, effort default, up to 3 repair rounds, 30 task(s), repo commit `6dae872`, Node v24.17.0. Total cost $0.0000. | ||
| + | ||
| +| Metric | .ts | .tsus | | ||
| +|---|---|---| | ||
| +| Tasks scored | 30 | 30 | | ||
| +| pass@1 (first attempt compiles and passes every hidden test) | 0/30 (0%) | 0/30 (0%) | | ||
| +| Pass after compiler-guided repair | 30/30 (100%) | 30/30 (100%) | | ||
| +| Compiled on first attempt | 0/30 (0%) | 0/30 (0%) | | ||
| +| Tasks where the compiler caught something | 30 | 30 | | ||
| +| Compile-time catches (rejected attempts / diagnostics) | 30 / 30 | 30 / 30 | | ||
| +| Tasks rescued by repair (rejected, then passed) | 30 | 30 | | ||
| +| Never compiled within the repair budget | 0 | 0 | | ||
| +| Average repair rounds | 1.00 | 1.00 | | ||
| +| Tasks failing at runtime (hidden tests) | 0 | 0 | | ||
| +| Failing hidden test cases | 0 of 369 | 0 of 369 | | ||
| +| Cost | $0.0000 | $0.0000 | | ||
| +| Output tokens | 0 | 0 | | ||
| +| Average wall time per task (model calls only) | 0.0 s | 0.0 s | | ||
| + | ||
| +Diagnostic codes seen in .ts attempts: TS2322 x30. | ||
| + | ||
| +Diagnostic codes seen in .tsus attempts: TSUS002 x30. | ||
| + | ||
| +<details><summary>Per-task detail</summary> | ||
| + | ||
| +| Task | .ts | .tsus | | ||
| +|---|---|---| | ||
| +| bank-transfer | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) | | ||
| +| cart-total | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) | | ||
| +| csv-rows | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) | | ||
| +| date-range-merge | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) | | ||
| +| deep-get | pass after repair (2 attempts, tests 11/11) | pass after repair (2 attempts, tests 11/11) | | ||
| +| feature-flag | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) | | ||
| +| free-slot | pass after repair (2 attempts, tests 11/11) | pass after repair (2 attempts, tests 11/11) | | ||
| +| inventory-ledger | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) | | ||
| +| json-api-handler | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) | | ||
| +| lru-cache | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) | | ||
| +| manager-chain | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) | | ||
| +| merge-profile | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) | | ||
| +| moving-average | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) | | ||
| +| order-state-machine | pass after repair (2 attempts, tests 14/14) | pass after repair (2 attempts, tests 14/14) | | ||
| +| paginate-cursor | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) | | ||
| +| parse-amount | pass after repair (2 attempts, tests 10/10) | pass after repair (2 attempts, tests 10/10) | | ||
| +| parse-config | pass after repair (2 attempts, tests 11/11) | pass after repair (2 attempts, tests 11/11) | | ||
| +| parse-duration | pass after repair (2 attempts, tests 11/11) | pass after repair (2 attempts, tests 11/11) | | ||
| +| parse-log-line | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) | | ||
| +| query-params | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) | | ||
| +| resolve-path | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) | | ||
| +| retry-fetch | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) | | ||
| +| route-match | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) | | ||
| +| semver-compare | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) | | ||
| +| shortest-path | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) | | ||
| +| signup-validation | pass after repair (2 attempts, tests 14/14) | pass after repair (2 attempts, tests 14/14) | | ||
| +| stats-summary | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) | | ||
| +| token-bucket | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) | | ||
| +| topo-sort | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) | | ||
| +| ttl-cache | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) | | ||
| + | ||
| +</details> | ||
Addedevals/lib/compile.mjs+172−0
| @@ -0,0 +1,172 @@ | ||
| +// Compile one solution in a scratch project and run the hidden tests on it. | ||
| +// | ||
| +// .ts -> tsc --strict (via a generated tsconfig), emits dist/solution.js | ||
| +// .tsus -> `tsus check --json` then `tsus build --json`, emits dist/solution.js | ||
| +// | ||
| +// Work directories live under evals/.work so the compiled output resolves | ||
| +// @tsus/runtime from the repository's node_modules. | ||
| +import fs from "node:fs"; | ||
| +import path from "node:path"; | ||
| +import { spawnSync } from "node:child_process"; | ||
| +import { pathToFileURL } from "node:url"; | ||
| + | ||
| +export const repoRoot = path.resolve(path.dirname(new URL(import.meta.url).pathname), "../.."); | ||
| +export const evalsDir = path.join(repoRoot, "evals"); | ||
| +const tscBin = path.join(repoRoot, "node_modules/typescript/bin/tsc"); | ||
| +export const defaultTsusBin = path.join(repoRoot, "packages/cli/bin/tsus.js"); | ||
| + | ||
| +export const TS_COMPILER_OPTIONS = { | ||
| + strict: true, | ||
| + target: "ES2022", | ||
| + module: "ESNext", | ||
| + moduleResolution: "Bundler", | ||
| + lib: ["ES2022"], | ||
| + types: [], | ||
| + rootDir: "src", | ||
| + outDir: "dist", | ||
| + skipLibCheck: true, | ||
| + noEmitOnError: true, | ||
| +}; | ||
| + | ||
| +const run = (cmd, args, cwd, timeout = 60_000) => { | ||
| + const r = spawnSync(cmd, args, { cwd, encoding: "utf8", timeout }); | ||
| + return { | ||
| + status: r.status, | ||
| + stdout: r.stdout ?? "", | ||
| + stderr: r.stderr ?? "", | ||
| + timedOut: r.error?.code === "ETIMEDOUT", | ||
| + }; | ||
| +}; | ||
| + | ||
| +const prepare = (workDir, file, code) => { | ||
| + fs.rmSync(workDir, { recursive: true, force: true }); | ||
| + fs.mkdirSync(path.join(workDir, "src"), { recursive: true }); | ||
| + fs.writeFileSync(path.join(workDir, "package.json"), '{ "type": "module", "private": true }\n'); | ||
| + fs.writeFileSync(path.join(workDir, "src", file), code); | ||
| +}; | ||
| + | ||
| +// tsc prints `src/solution.ts(12,5): error TS2322: Type ...` lines. | ||
| +const parseTscOutput = (text) => { | ||
| + const diagnostics = []; | ||
| + for (const line of text.split("\n")) { | ||
| + const m = line.match(/^(.*?)\((\d+),(\d+)\): error (TS\d+): (.*)$/); | ||
| + if (m) { | ||
| + diagnostics.push({ code: m[4], line: Number(m[2]), col: Number(m[3]), message: m[5] }); | ||
| + } else if (line.startsWith(" ") && diagnostics.length > 0) { | ||
| + diagnostics[diagnostics.length - 1].message += `\n${line}`; | ||
| + } else if (/error TS\d+/.test(line)) { | ||
| + diagnostics.push({ code: line.match(/TS\d+/)[0], line: 0, col: 0, message: line }); | ||
| + } | ||
| + } | ||
| + return diagnostics; | ||
| +}; | ||
| + | ||
| +export const compileTs = (workDir, code) => { | ||
| + prepare(workDir, "solution.ts", code); | ||
| + fs.writeFileSync( | ||
| + path.join(workDir, "tsconfig.json"), | ||
| + JSON.stringify({ compilerOptions: TS_COMPILER_OPTIONS, include: ["src"] }, null, 2), | ||
| + ); | ||
| + const r = run(process.execPath, [tscBin, "-p", "tsconfig.json"], workDir); | ||
| + const diagnostics = parseTscOutput(r.stdout + r.stderr); | ||
| + const emitted = path.join(workDir, "dist", "solution.js"); | ||
| + if (r.status !== 0 && diagnostics.length === 0) { | ||
| + diagnostics.push({ code: "TSC", line: 0, col: 0, message: (r.stdout + r.stderr).trim() || "tsc failed" }); | ||
| + } | ||
| + return { ok: r.status === 0 && fs.existsSync(emitted), diagnostics, emitted }; | ||
| +}; | ||
| + | ||
| +const parseTsusJson = (text) => { | ||
| + try { | ||
| + return JSON.parse(text); | ||
| + } catch { | ||
| + return { ok: false, diagnostics: [{ code: "TSUS_CLI", line: 0, col: 0, message: text.trim().slice(0, 2000) }] }; | ||
| + } | ||
| +}; | ||
| + | ||
| +export const compileTsus = (workDir, code, tsusBin = defaultTsusBin) => { | ||
| + prepare(workDir, "solution.tsus", code); | ||
| + const check = run(process.execPath, [tsusBin, "check", workDir, "--json"], workDir); | ||
| + const checked = parseTsusJson(check.stdout || check.stderr); | ||
| + const toDiag = (d) => ({ | ||
| + code: d.code, | ||
| + line: d.line ?? 0, | ||
| + col: d.col ?? 0, | ||
| + message: d.hint ? `${d.message} (hint: ${d.hint})` : d.message, | ||
| + }); | ||
| + if (!checked.ok) { | ||
| + return { ok: false, diagnostics: (checked.diagnostics ?? []).map(toDiag), emitted: "" }; | ||
| + } | ||
| + const build = run(process.execPath, [tsusBin, "build", workDir, "--json"], workDir); | ||
| + const built = parseTsusJson(build.stdout || build.stderr); | ||
| + const emitted = path.join(workDir, "dist", "solution.js"); | ||
| + if (!built.ok || !fs.existsSync(emitted)) { | ||
| + const diagnostics = (built.diagnostics ?? []).map(toDiag); | ||
| + if (diagnostics.length === 0) { | ||
| + diagnostics.push({ code: "TSUS_BUILD", line: 0, col: 0, message: (build.stderr || build.stdout).trim().slice(0, 2000) }); | ||
| + } | ||
| + return { ok: false, diagnostics, emitted: "" }; | ||
| + } | ||
| + return { ok: true, diagnostics: [], emitted }; | ||
| +}; | ||
| + | ||
| +export const compile = (lang, workDir, code, opts = {}) => | ||
| + lang === "ts" ? compileTs(workDir, code) : compileTsus(workDir, code, opts.tsusBin); | ||
| + | ||
| +export const formatDiagnostics = (lang, diagnostics) => | ||
| + diagnostics | ||
| + .map((d) => `solution.${lang}:${d.line}:${d.col} error ${d.code}: ${d.message}`) | ||
| + .join("\n"); | ||
| + | ||
| +// Run the task's hidden node:test file against the compiled solution. | ||
| +// Returns counts plus the names of failing tests. | ||
| +export const runHiddenTests = (lang, testFile, emitted, timeoutMs = 30_000) => { | ||
| + const r = spawnSync( | ||
| + process.execPath, | ||
| + ["--test", "--test-reporter=tap", "--test-timeout=5000", testFile], | ||
| + { | ||
| + cwd: path.dirname(testFile), | ||
| + encoding: "utf8", | ||
| + timeout: timeoutMs, | ||
| + env: { | ||
| + ...process.env, | ||
| + TSUS_EVAL_LANG: lang, | ||
| + TSUS_EVAL_SOLUTION: pathToFileURL(emitted).href, | ||
| + }, | ||
| + }, | ||
| + ); | ||
| + const out = r.stdout ?? ""; | ||
| + const num = (label) => { | ||
| + const m = out.match(new RegExp(`^# ${label} (\\d+)$`, "m")); | ||
| + return m ? Number(m[1]) : 0; | ||
| + }; | ||
| + const failures = []; | ||
| + const lines = out.split("\n"); | ||
| + for (let i = 0; i < lines.length; i += 1) { | ||
| + // Subtests are indented; top-level `not ok N - name` lines name failures. | ||
| + const m = lines[i].match(/^\s*not ok \d+ - (.*)$/); | ||
| + if (m && !/^\s{4,}/.test(lines[i])) { | ||
| + failures.push(m[1].trim()); | ||
| + } | ||
| + } | ||
| + const pass = num("pass"); | ||
| + if (r.error?.code === "ETIMEDOUT") { | ||
| + failures.push("(test process timed out: likely an infinite loop)"); | ||
| + } | ||
| + const fail = num("fail") + num("cancelled"); | ||
| + const timedOut = r.error?.code === "ETIMEDOUT"; | ||
| + // A synchronous infinite loop blocks the test runner's own timers, so the | ||
| + // process is killed with no summary. Count every declared test as failed | ||
| + // rather than reporting 0/0. | ||
| + const declared = (fs.readFileSync(testFile, "utf8").match(/^test\(/gm) ?? []).length; | ||
| + const total = timedOut ? Math.max(num("tests"), declared) : num("tests"); | ||
| + return { | ||
| + pass, | ||
| + fail: timedOut ? Math.max(1, total - pass) : fail, | ||
| + total, | ||
| + timedOut, | ||
| + failures, | ||
| + allPassed: !timedOut && r.status === 0 && fail === 0 && total > 0, | ||
| + rawTail: r.status === 0 ? "" : (out + (r.stderr ?? "")).slice(-3000), | ||
| + }; | ||
| +}; | ||
Addedevals/lib/harness.mjs+185−0
| @@ -0,0 +1,185 @@ | ||
| +// Language-neutral helpers for the hidden tests. | ||
| +// | ||
| +// Each task's test.mjs runs against compiled JS from either a .ts or a .tsus | ||
| +// solution. The two languages express absence and failure differently: | ||
| +// | ||
| +// absence: .ts -> undefined (or null) .tsus -> None / Some(v) | ||
| +// failure: .ts -> throws { kind, ... } .tsus -> Err({ kind, ... }) / Ok(v) | ||
| +// | ||
| +// Tests never look at those representations directly. They build inputs with | ||
| +// h.some/h.none/h.ok/h.err and read outputs with h.opt/h.res, which normalize | ||
| +// both languages to one canonical shape: | ||
| +// | ||
| +// h.opt(v) -> { some: true, value } | { some: false } | ||
| +// h.res(fn) -> { ok: true, value } | { ok: false, error } | { crash: "..." } | ||
| +// | ||
| +// A crash is any exception that is not a typed task error (in .ts) or any | ||
| +// exception at all (in .tsus). Crashes always fail the assertion they reach, | ||
| +// which is the point: an unexpected TypeError is a bug, not an error value. | ||
| + | ||
| +import { Some, None, Ok, Err } from "@tsus/runtime"; | ||
| + | ||
| +export const lang = process.env.TSUS_EVAL_LANG === "tsus" ? "tsus" : "ts"; | ||
| +export const solutionPath = process.env.TSUS_EVAL_SOLUTION; | ||
| + | ||
| +export const load = async () => { | ||
| + if (!solutionPath) { | ||
| + throw new Error("TSUS_EVAL_SOLUTION is not set"); | ||
| + } | ||
| + return import(solutionPath); | ||
| +}; | ||
| + | ||
| +const isTypedError = (e) => e !== null && typeof e === "object" && typeof e.kind === "string"; | ||
| + | ||
| +const describeCrash = (e) => { | ||
| + if (e instanceof Error) { | ||
| + return `${e.name}: ${e.message}`; | ||
| + } | ||
| + try { | ||
| + return `non-error thrown: ${JSON.stringify(e)}`; | ||
| + } catch { | ||
| + return `non-error thrown: ${String(e)}`; | ||
| + } | ||
| +}; | ||
| + | ||
| +const toPlainError = (e) => { | ||
| + // Typed errors in .ts are usually Error subclasses with a kind field. Keep | ||
| + // only own enumerable data fields so deepEqual compares just the contract. | ||
| + // Error bookkeeping (`stack`, `message`, and a `name` set to the class name | ||
| + // by the common `this.name = "FooError"` idiom) is dropped; a contract field | ||
| + // called `name` with any other value is kept. Keys holding `undefined` are | ||
| + // dropped too: with ES2022 class fields, an optional field the constructor | ||
| + // never set is an own property whose value is undefined, and the .tsus error | ||
| + // object simply would not have that key. | ||
| + const out = {}; | ||
| + for (const key of Object.keys(e)) { | ||
| + if (key === "stack" || key === "message" || e[key] === undefined) { | ||
| + continue; | ||
| + } | ||
| + if (key === "name" && e instanceof Error && typeof e.name === "string" && /Error$/.test(e.name)) { | ||
| + continue; | ||
| + } | ||
| + out[key] = e[key]; | ||
| + } | ||
| + return out; | ||
| +}; | ||
| + | ||
| +const normalizeOption = (v) => { | ||
| + if (lang === "ts") { | ||
| + return v === undefined || v === null ? { some: false } : { some: true, value: v }; | ||
| + } | ||
| + if (v !== null && typeof v === "object" && v.tag === "Some") { | ||
| + return { some: true, value: v.value }; | ||
| + } | ||
| + if (v !== null && typeof v === "object" && v.tag === "None") { | ||
| + return { some: false }; | ||
| + } | ||
| + return { invalid: `expected Option, got ${describeValue(v)}` }; | ||
| +}; | ||
| + | ||
| +const describeValue = (v) => { | ||
| + try { | ||
| + return JSON.stringify(v) ?? String(v); | ||
| + } catch { | ||
| + return String(v); | ||
| + } | ||
| +}; | ||
| + | ||
| +const normalizeResultValue = (v) => { | ||
| + if (lang === "ts") { | ||
| + return { ok: true, value: v }; | ||
| + } | ||
| + if (v !== null && typeof v === "object" && v.tag === "Ok") { | ||
| + return { ok: true, value: v.value }; | ||
| + } | ||
| + if (v !== null && typeof v === "object" && v.tag === "Err") { | ||
| + return isTypedError(v.error) | ||
| + ? { ok: false, error: toPlainError(v.error) } | ||
| + : { ok: false, untypedError: describeValue(v.error) }; | ||
| + } | ||
| + return { invalid: `expected Result, got ${describeValue(v)}` }; | ||
| +}; | ||
| + | ||
| +const normalizeThrown = (e) => { | ||
| + if (lang === "ts" && isTypedError(e)) { | ||
| + return { ok: false, error: toPlainError(e) }; | ||
| + } | ||
| + return { crash: describeCrash(e) }; | ||
| +}; | ||
| + | ||
| +export const h = { | ||
| + lang, | ||
| + // --- building inputs --- | ||
| + some: (v) => (lang === "tsus" ? Some(v) : v), | ||
| + none: () => (lang === "tsus" ? None : undefined), | ||
| + // An optional input value: pass `absent` to mean "not provided". | ||
| + maybe: (v, absent = undefined) => (v === absent ? h.none() : h.some(v)), | ||
| + // A dependency that can fail. In .ts it returns or throws; in .tsus it | ||
| + // returns Ok/Err. `error` must be a { kind } object. | ||
| + okValue: (v) => (lang === "tsus" ? Ok(v) : v), | ||
| + failWith: (error) => { | ||
| + if (lang === "tsus") { | ||
| + return Err(error); | ||
| + } | ||
| + throw Object.assign(new Error(error.message ?? error.kind), error); | ||
| + }, | ||
| + // Async dependency outcomes. | ||
| + resolveOk: (v) => Promise.resolve(lang === "tsus" ? Ok(v) : v), | ||
| + resolveErr: (error) => | ||
| + lang === "tsus" | ||
| + ? Promise.resolve(Err(error)) | ||
| + : Promise.reject(Object.assign(new Error(error.message ?? error.kind), error)), | ||
| + | ||
| + // --- reading outputs --- | ||
| + opt: normalizeOption, | ||
| + // Read an optional field off a returned record. | ||
| + field: (obj, key) => normalizeOption(obj?.[key]), | ||
| + // Canonicalize a returned record: every key in optKeys becomes | ||
| + // { some, value } / { some: false }, so `{ a: 1 }` (.ts, b omitted) and | ||
| + // `{ a: 1, b: None }` (.tsus) compare equal. Other keys pass through. | ||
| + shape: (obj, optKeys) => { | ||
| + if (obj === null || typeof obj !== "object") { | ||
| + return { notARecord: describeValue(obj) }; | ||
| + } | ||
| + const out = {}; | ||
| + for (const [key, value] of Object.entries(obj)) { | ||
| + if (!optKeys.includes(key)) { | ||
| + out[key] = value; | ||
| + } | ||
| + } | ||
| + for (const key of optKeys) { | ||
| + out[key] = normalizeOption(obj[key]); | ||
| + } | ||
| + return out; | ||
| + }, | ||
| + res: (fn) => { | ||
| + try { | ||
| + return normalizeResultValue(fn()); | ||
| + } catch (e) { | ||
| + return normalizeThrown(e); | ||
| + } | ||
| + }, | ||
| + resAsync: async (fn) => { | ||
| + try { | ||
| + return normalizeResultValue(await fn()); | ||
| + } catch (e) { | ||
| + return normalizeThrown(e); | ||
| + } | ||
| + }, | ||
| + // Call a function that should neither fail nor crash; returns its value. | ||
| + call: (fn) => { | ||
| + try { | ||
| + return fn(); | ||
| + } catch (e) { | ||
| + return { crash: describeCrash(e) }; | ||
| + } | ||
| + }, | ||
| + // Unwrap an ok result for further inspection, or return a marker that will | ||
| + // fail any deepEqual it reaches. | ||
| + okOf: (r) => (r.ok === true ? r.value : { notOk: r }), | ||
| +}; | ||
| + | ||
| +export const ok = (value) => ({ ok: true, value }); | ||
| +export const err = (error) => ({ ok: false, error }); | ||
| +export const some = (value) => ({ some: true, value }); | ||
| +export const none = { some: false }; | ||
Addedevals/lib/providers.mjs+210−0
| @@ -0,0 +1,210 @@ | ||
| +// Model providers. Each provider exposes | ||
| +// generate({ system, prompt, task, lang, round }) -> { text, usage, costUsd, ms } | ||
| +// | ||
| +// claude-cli `claude -p` headless (default when no ANTHROPIC_API_KEY) | ||
| +// anthropic Messages API through @anthropic-ai/sdk (default when ANTHROPIC_API_KEY is set) | ||
| +// reference returns the task's reference solution (harness self-test, no model) | ||
| +// selftest round 0 returns the reference with a planted compile error, round 1 | ||
| +// the clean reference; proves the repair loop end to end (no model) | ||
| +// | ||
| +// Keys are never logged: the CLI and the SDK read credentials themselves. | ||
| +import fs from "node:fs"; | ||
| +import os from "node:os"; | ||
| +import path from "node:path"; | ||
| +import { spawn } from "node:child_process"; | ||
| + | ||
| +// Price per million tokens, for providers that do not report cost themselves. | ||
| +const PRICES = { | ||
| + "claude-sonnet-5-5": { input: 2, output: 10, cacheRead: 0.2, cacheWrite: 2.5 }, | ||
| + "claude-sonnet-5": { input: 2, output: 10, cacheRead: 0.2, cacheWrite: 2.5 }, | ||
| + "claude-sonnet-4-6": { input: 3, output: 15, cacheRead: 0.3, cacheWrite: 3.75 }, | ||
| + "claude-opus-5-5": { input: 4, output: 20, cacheRead: 0.2, cacheWrite: 5 }, | ||
| + "claude-haiku-5-5": { input: 0.1, output: 0.5, cacheRead: 0.01, cacheWrite: 0.125 }, | ||
| +}; | ||
| + | ||
| +export const DEFAULT_MODEL = "claude-sonnet-5-5"; | ||
| + | ||
| +const emptyUsage = () => ({ inputTokens: 0, outputTokens: 0, cacheReadTokens: 0, cacheWriteTokens: 0 }); | ||
| + | ||
| +const priceUsage = (model, usage) => { | ||
| + const p = PRICES[model]; | ||
| + if (!p) { | ||
| + return 0; | ||
| + } | ||
| + return ( | ||
| + (usage.inputTokens * p.input + | ||
| + usage.outputTokens * p.output + | ||
| + usage.cacheReadTokens * p.cacheRead + | ||
| + usage.cacheWriteTokens * p.cacheWrite) / | ||
| + 1_000_000 | ||
| + ); | ||
| +}; | ||
| + | ||
| +// Environment variables that tie a child `claude` process to a parent Claude | ||
| +// Code session. Inherited, they make the child try to borrow the parent's auth | ||
| +// and fail; the eval wants a standalone, stateless call. | ||
| +const NESTED_SESSION_VARS = [ | ||
| + "CLAUDECODE", | ||
| + "CLAUDE_CODE_ENTRYPOINT", | ||
| + "CLAUDE_CODE_CHILD_SESSION", | ||
| + "CLAUDE_CODE_SESSION_ID", | ||
| + "CLAUDE_CODE_HOST_SESSION_ID", | ||
| + "CLAUDE_CODE_SDK_HAS_HOST_AUTH_REFRESH", | ||
| + "CLAUDE_CODE_MESSAGING_SOCKET", | ||
| + "CLAUDE_CODE_MESSAGING_TOKEN", | ||
| + "CLAUDE_CODE_SIMPLE", | ||
| + "CLAUDE_AGENT_SDK_VERSION", | ||
| +]; | ||
| + | ||
| +const claudeCli = ({ model, timeoutMs = 600_000 }) => ({ | ||
| + name: "claude-cli", | ||
| + model, | ||
| + generate: ({ system, prompt }) => | ||
| + new Promise((resolve, reject) => { | ||
| + const env = { ...process.env }; | ||
| + for (const key of NESTED_SESSION_VARS) { | ||
| + delete env[key]; | ||
| + } | ||
| + const started = Date.now(); | ||
| + const args = [ | ||
| + "-p", | ||
| + "--model", | ||
| + model, | ||
| + "--system-prompt", | ||
| + system, | ||
| + "--tools", | ||
| + "", | ||
| + "--strict-mcp-config", | ||
| + "--no-session-persistence", | ||
| + "--output-format", | ||
| + "json", | ||
| + ]; | ||
| + // Run from an empty temp dir so no project CLAUDE.md leaks into context. | ||
| + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "tsus-eval-")); | ||
| + const child = spawn(process.env.CLAUDE_BIN ?? "claude", args, { cwd, env, stdio: ["pipe", "pipe", "pipe"] }); | ||
| + let stdout = ""; | ||
| + let stderr = ""; | ||
| + const timer = setTimeout(() => child.kill("SIGKILL"), timeoutMs); | ||
| + child.stdout.on("data", (d) => { | ||
| + stdout += d; | ||
| + }); | ||
| + child.stderr.on("data", (d) => { | ||
| + stderr += d; | ||
| + }); | ||
| + child.on("error", reject); | ||
| + child.on("close", () => { | ||
| + clearTimeout(timer); | ||
| + fs.rmSync(cwd, { recursive: true, force: true }); | ||
| + let data; | ||
| + try { | ||
| + data = JSON.parse(stdout); | ||
| + } catch { | ||
| + reject(new Error(`claude -p returned non-JSON output: ${(stdout || stderr).slice(0, 500)}`)); | ||
| + return; | ||
| + } | ||
| + if (data.is_error) { | ||
| + reject(new Error(`claude -p failed: ${String(data.result).slice(0, 500)}`)); | ||
| + return; | ||
| + } | ||
| + const u = data.usage ?? {}; | ||
| + resolve({ | ||
| + text: data.result ?? "", | ||
| + usage: { | ||
| + inputTokens: u.input_tokens ?? 0, | ||
| + outputTokens: u.output_tokens ?? 0, | ||
| + cacheReadTokens: u.cache_read_input_tokens ?? 0, | ||
| + cacheWriteTokens: u.cache_creation_input_tokens ?? 0, | ||
| + }, | ||
| + costUsd: data.total_cost_usd ?? 0, | ||
| + ms: Date.now() - started, | ||
| + servedBy: Object.keys(data.modelUsage ?? {}).join(",") || model, | ||
| + }); | ||
| + }); | ||
| + child.stdin.end(prompt); | ||
| + }), | ||
| +}); | ||
| + | ||
| +const anthropicApi = ({ model, effort }) => { | ||
| + let clientPromise; | ||
| + const getClient = () => { | ||
| + clientPromise ??= import("@anthropic-ai/sdk") | ||
| + .catch(() => { | ||
| + throw new Error("the anthropic provider needs @anthropic-ai/sdk: run `npm install --prefix evals`"); | ||
| + }) | ||
| + .then((mod) => new mod.default()); | ||
| + return clientPromise; | ||
| + }; | ||
| + return { | ||
| + name: "anthropic", | ||
| + model, | ||
| + generate: async ({ system, prompt }) => { | ||
| + const client = await getClient(); | ||
| + const started = Date.now(); | ||
| + const response = await client.messages.create({ | ||
| + model, | ||
| + max_tokens: 16000, | ||
| + system, | ||
| + messages: [{ role: "user", content: prompt }], | ||
| + ...(effort ? { output_config: { effort } } : {}), | ||
| + }); | ||
| + if (response.stop_reason === "refusal") { | ||
| + throw new Error(`model refused (${response.stop_details?.category ?? "no category"})`); | ||
| + } | ||
| + const text = response.content | ||
| + .filter((b) => b.type === "text") | ||
| + .map((b) => b.text) | ||
| + .join("\n"); | ||
| + const usage = { | ||
| + inputTokens: response.usage.input_tokens ?? 0, | ||
| + outputTokens: response.usage.output_tokens ?? 0, | ||
| + cacheReadTokens: response.usage.cache_read_input_tokens ?? 0, | ||
| + cacheWriteTokens: response.usage.cache_creation_input_tokens ?? 0, | ||
| + }; | ||
| + return { text, usage, costUsd: priceUsage(model, usage), ms: Date.now() - started, servedBy: response.model }; | ||
| + }, | ||
| + }; | ||
| +}; | ||
| + | ||
| +const fence = (lang, code) => "```" + lang + "\n" + code + "\n```\n"; | ||
| + | ||
| +// A compile error each checker is guaranteed to reject. | ||
| +const PLANTED_ERROR = { | ||
| + ts: 'const __planted: number = "not a number";\nvoid __planted;\n', | ||
| + tsus: "let __planted = null;\nvoid __planted;\n", | ||
| +}; | ||
| + | ||
| +const referenceProvider = ({ broken }) => ({ | ||
| + name: broken ? "selftest" : "reference", | ||
| + model: "none", | ||
| + generate: async ({ task, lang, round }) => { | ||
| + const code = fs.readFileSync(path.join(task.dir, `reference.${lang}`), "utf8"); | ||
| + const text = broken && round === 0 ? code + "\n" + PLANTED_ERROR[lang] : code; | ||
| + return { text: fence(lang, text), usage: emptyUsage(), costUsd: 0, ms: 0, servedBy: "none" }; | ||
| + }, | ||
| +}); | ||
| + | ||
| +export const makeProvider = ({ provider, model, effort }) => { | ||
| + const chosen = provider ?? (process.env.ANTHROPIC_API_KEY ? "anthropic" : "claude-cli"); | ||
| + switch (chosen) { | ||
| + case "claude-cli": | ||
| + return claudeCli({ model }); | ||
| + case "anthropic": | ||
| + return anthropicApi({ model, effort }); | ||
| + case "reference": | ||
| + return referenceProvider({ broken: false }); | ||
| + case "selftest": | ||
| + return referenceProvider({ broken: true }); | ||
| + default: | ||
| + throw new Error(`unknown provider ${chosen}`); | ||
| + } | ||
| +}; | ||
| + | ||
| +// Pull the solution out of a reply: the longest fenced block wins, since models | ||
| +// sometimes show a snippet before the full file. | ||
| +export const extractCode = (text) => { | ||
| + const blocks = [...text.matchAll(/```[a-zA-Z]*\n([\s\S]*?)```/g)].map((m) => m[1]); | ||
| + if (blocks.length === 0) { | ||
| + return text.trim(); | ||
| + } | ||
| + return blocks.reduce((a, b) => (b.length >= a.length ? b : a)); | ||
| +}; | ||
Addedevals/package-lock.json+102−0
| @@ -0,0 +1,102 @@ | ||
| +{ | ||
| + "name": "tsus-evals", | ||
| + "lockfileVersion": 3, | ||
| + "requires": true, | ||
| + "packages": { | ||
| + "": { | ||
| + "name": "tsus-evals", | ||
| + "optionalDependencies": { | ||
| + "@anthropic-ai/sdk": "^0.135.0" | ||
| + } | ||
| + }, | ||
| + "node_modules/@anthropic-ai/sdk": { | ||
| + "version": "0.135.0", | ||
| + "resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.135.0.tgz", | ||
| + "integrity": "sha512-WWrdL57JhU5KoynpTjt9GXK6k7ud9nPw2uw/Trn+oBXh6s0KpjhAccWZm2Qxw08ARYG1UDQj5iC+JO42j1rt/g==", | ||
| + "license": "MIT", | ||
| + "optional": true, | ||
| + "dependencies": { | ||
| + "@opentelemetry/api": "^1.4.1", | ||
| + "json-schema-to-ts": "^3.1.1", | ||
| + "standardwebhooks": "^1.0.0" | ||
| + }, | ||
| + "bin": { | ||
| + "anthropic-ai-sdk": "bin/cli" | ||
| + }, | ||
| + "peerDependencies": { | ||
| + "zod": "^3.25.0 || ^4.0.0" | ||
| + }, | ||
| + "peerDependenciesMeta": { | ||
| + "zod": { | ||
| + "optional": true | ||
| + } | ||
| + } | ||
| + }, | ||
| + "node_modules/@babel/runtime": { | ||
| + "version": "7.29.10", | ||
| + "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.10.tgz", | ||
| + "integrity": "sha512-NpugTzbKgGrNe45Ui32cN8vCFFMxVtP/9rtfwv0nP5FRDAjvYBxJcLVpYA9mr1jxP6cnIT7x9LUzuDTbWYYNRA==", | ||
| + "license": "MIT", | ||
| + "optional": true, | ||
| + "engines": { | ||
| + "node": ">=6.9.0" | ||
| + } | ||
| + }, | ||
| + "node_modules/@opentelemetry/api": { | ||
| + "version": "1.9.1", | ||
| + "resolved": "https://registry.npmjs.org/@opentelemetry/api/-/api-1.9.1.tgz", | ||
| + "integrity": "sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==", | ||
| + "license": "Apache-2.0", | ||
| + "optional": true, | ||
| + "engines": { | ||
| + "node": ">=8.0.0" | ||
| + } | ||
| + }, | ||
| + "node_modules/@stablelib/base64": { | ||
| + "version": "1.0.1", | ||
| + "resolved": "https://registry.npmjs.org/@stablelib/base64/-/base64-1.0.1.tgz", | ||
| + "integrity": "sha512-1bnPQqSxSuc3Ii6MhBysoWCg58j97aUjuCSZrGSmDxNqtytIi0k8utUenAwTZN4V5mXXYGsVUI9zeBqy+jBOSQ==", | ||
| + "license": "MIT", | ||
| + "optional": true | ||
| + }, | ||
| + "node_modules/fast-sha256": { | ||
| + "version": "1.3.0", | ||
| + "resolved": "https://registry.npmjs.org/fast-sha256/-/fast-sha256-1.3.0.tgz", | ||
| + "integrity": "sha512-n11RGP/lrWEFI/bWdygLxhI+pVeo1ZYIVwvvPkW7azl/rOy+F3HYRZ2K5zeE9mmkhQppyv9sQFx0JM9UabnpPQ==", | ||
| + "license": "Unlicense", | ||
| + "optional": true | ||
| + }, | ||
| + "node_modules/json-schema-to-ts": { | ||
| + "version": "3.1.1", | ||
| + "resolved": "https://registry.npmjs.org/json-schema-to-ts/-/json-schema-to-ts-3.1.1.tgz", | ||
| + "integrity": "sha512-+DWg8jCJG2TEnpy7kOm/7/AxaYoaRbjVB4LFZLySZlWn8exGs3A4OLJR966cVvU26N7X9TWxl+Jsw7dzAqKT6g==", | ||
| + "license": "MIT", | ||
| + "optional": true, | ||
| + "dependencies": { | ||
| + "@babel/runtime": "^7.18.3", | ||
| + "ts-algebra": "^2.0.0" | ||
| + }, | ||
| + "engines": { | ||
| + "node": ">=16" | ||
| + } | ||
| + }, | ||
| + "node_modules/standardwebhooks": { | ||
| + "version": "1.1.1", | ||
| + "resolved": "https://registry.npmjs.org/standardwebhooks/-/standardwebhooks-1.1.1.tgz", | ||
| + "integrity": "sha512-bCbX9ZEyFkWPsRz7Bl3NuQUJohmwGSev/yhr7vhaGPlc4AfIrspIRa6cPTBuI1ItmrTDJ4d/S2hCsfe4+vQGnQ==", | ||
| + "license": "MIT", | ||
| + "optional": true, | ||
| + "dependencies": { | ||
| + "@stablelib/base64": "^1.0.0", | ||
| + "fast-sha256": "^1.3.0" | ||
| + } | ||
| + }, | ||
| + "node_modules/ts-algebra": { | ||
| + "version": "2.0.0", | ||
| + "resolved": "https://registry.npmjs.org/ts-algebra/-/ts-algebra-2.0.0.tgz", | ||
| + "integrity": "sha512-FPAhNPFMrkwz76P7cdjdmiShwMynZYN6SgOujD1urY4oNm80Ou9oMdmbR45LotcKOXoy7wSmHkRFE6Mxbrhefw==", | ||
| + "license": "MIT", | ||
| + "optional": true | ||
| + } | ||
| + } | ||
| +} | ||
Addedevals/package.json+15−0
| @@ -0,0 +1,15 @@ | ||
| +{ | ||
| + "name": "tsus-evals", | ||
| + "private": true, | ||
| + "type": "module", | ||
| + "description": "Eval harness: is AI-written code more correct in .tsus than in .ts?", | ||
| + "scripts": { | ||
| + "validate": "node validate.mjs", | ||
| + "selftest": "node run.mjs --provider selftest", | ||
| + "pilot": "node run.mjs --pilot", | ||
| + "report": "node report.mjs" | ||
| + }, | ||
| + "optionalDependencies": { | ||
| + "@anthropic-ai/sdk": "^0.135.0" | ||
| + } | ||
| +} | ||
Addedevals/prompts/system-ts.md+7−0
| @@ -0,0 +1,7 @@ | ||
| +You are writing one TypeScript module for an automated evaluation. Your code is compiled with `tsc` in strict mode (target ES2022, ESM, no DOM or Node type definitions) and then run against hidden tests. | ||
| + | ||
| +Rules: | ||
| +- Reply with exactly one fenced code block containing the complete contents of `solution.ts`, and nothing after it. | ||
| +- Keep every exported name, type, and function signature from the provided signature file exactly as given. You may add private helpers. | ||
| +- Do not import anything; the module must be self-contained. | ||
| +- Follow the error contract in the task: when the task says a function throws a typed error, throw an instance of the error class from the signature file with the right `kind`. | ||
Addedevals/prompts/system-tsus.md+9−0
| @@ -0,0 +1,9 @@ | ||
| +You are writing one TypeScript-Ultra-Strict (`.tsus`) module for an automated evaluation. Your code is checked with `tsus check`, compiled with `tsus build` to ESM, and then run against hidden tests. The language reference follows. | ||
| + | ||
| +Rules: | ||
| +- Reply with exactly one fenced code block containing the complete contents of `solution.tsus`, and nothing after it. | ||
| +- Keep every exported name, type, and function signature from the provided signature file exactly as given. You may add private helpers. | ||
| +- The only imports allowed are `@tsus/runtime` and `@tsus/runtime/boundary`. | ||
| +- Follow the error contract in the task: when the task says a function fails with a typed error, return `Err({ kind: ..., ... })` with the right `kind`. | ||
| + | ||
| +{{LANGUAGE}} | ||
Addedevals/report.mjs+204−0
| @@ -0,0 +1,204 @@ | ||
| +#!/usr/bin/env node | ||
| +// Writes evals/RESULTS.md from one or more run directories. | ||
| +// | ||
| +// node evals/report.mjs # every run under evals/results | ||
| +// node evals/report.mjs evals/results/<run> # just these runs | ||
| +// node evals/report.mjs --stdout <run> # print instead of writing | ||
| +import fs from "node:fs"; | ||
| +import path from "node:path"; | ||
| +import { evalsDir, repoRoot } from "./lib/compile.mjs"; | ||
| + | ||
| +const args = process.argv.slice(2); | ||
| +const toStdout = args.includes("--stdout"); | ||
| +const resultsRoot = path.join(evalsDir, "results"); | ||
| +let runDirs = args.filter((a) => !a.startsWith("--")).map((a) => path.resolve(a)); | ||
| +if (runDirs.length === 0) { | ||
| + runDirs = fs.existsSync(resultsRoot) | ||
| + ? fs | ||
| + .readdirSync(resultsRoot) | ||
| + .map((n) => path.join(resultsRoot, n)) | ||
| + .filter((d) => fs.existsSync(path.join(d, "meta.json"))) | ||
| + : []; | ||
| +} | ||
| + | ||
| +const loadRun = (dir) => { | ||
| + const meta = JSON.parse(fs.readFileSync(path.join(dir, "meta.json"), "utf8")); | ||
| + const file = path.join(dir, "results.jsonl"); | ||
| + const records = fs.existsSync(file) | ||
| + ? fs | ||
| + .readFileSync(file, "utf8") | ||
| + .split("\n") | ||
| + .filter(Boolean) | ||
| + .map((l) => JSON.parse(l)) | ||
| + : []; | ||
| + records.sort((a, b) => a.task.localeCompare(b.task) || a.lang.localeCompare(b.lang)); | ||
| + return { dir, meta, records }; | ||
| +}; | ||
| + | ||
| +// Runs where every job failed at the provider (no model access) carry no | ||
| +// information; leave them out. | ||
| +const runs = runDirs | ||
| + .map(loadRun) | ||
| + .filter((r) => r.records.some((rec) => rec.status !== "provider_error" && rec.status !== "aborted")) | ||
| + .sort((a, b) => b.meta.startedAt.localeCompare(a.meta.startedAt)); | ||
| + | ||
| +const pct = (n, d) => (d === 0 ? "n/a" : `${Math.round((100 * n) / d)}%`); | ||
| +const frac = (n, d) => `${n}/${d} (${pct(n, d)})`; | ||
| +const avg = (xs) => (xs.length === 0 ? 0 : xs.reduce((a, b) => a + b, 0) / xs.length); | ||
| +const money = (x) => `$${x.toFixed(x < 1 ? 4 : 2)}`; | ||
| + | ||
| +const isModelRun = (meta) => meta.provider !== "reference" && meta.provider !== "selftest"; | ||
| + | ||
| +const summarize = (records) => { | ||
| + const scored = records.filter((r) => r.status !== "provider_error" && r.status !== "aborted"); | ||
| + const withTests = scored.filter((r) => r.tests); | ||
| + return { | ||
| + jobs: records.length, | ||
| + scored: scored.length, | ||
| + providerErrors: records.length - scored.length, | ||
| + pass1: scored.filter((r) => r.passFirstTry).length, | ||
| + passFinal: scored.filter((r) => r.passFinal).length, | ||
| + compiledFirst: scored.filter((r) => r.compiledFirstTry).length, | ||
| + neverCompiled: scored.filter((r) => r.status === "compile_failed").length, | ||
| + tasksCaughtAtCompile: scored.filter((r) => r.compileRejections > 0).length, | ||
| + compileRejections: scored.reduce((n, r) => n + r.compileRejections, 0), | ||
| + compileDiagnostics: scored.reduce((n, r) => n + r.compileDiagnostics, 0), | ||
| + rescuedByRepair: scored.filter((r) => r.compileRejections > 0 && r.passFinal).length, | ||
| + avgRepairs: avg(scored.map((r) => r.repairRounds)), | ||
| + tasksFailingAtRuntime: withTests.filter((r) => !r.tests.allPassed).length, | ||
| + failingTestCases: withTests.reduce((n, r) => n + r.tests.fail, 0), | ||
| + totalTestCases: withTests.reduce((n, r) => n + r.tests.total, 0), | ||
| + cost: records.reduce((n, r) => n + r.costUsd, 0), | ||
| + outputTokens: records.reduce((n, r) => n + r.usage.outputTokens, 0), | ||
| + avgSeconds: avg(scored.map((r) => r.ms / 1000)), | ||
| + }; | ||
| +}; | ||
| + | ||
| +const codeCounts = (records) => { | ||
| + const counts = new Map(); | ||
| + for (const r of records) { | ||
| + for (const a of r.attempts) { | ||
| + for (const c of a.codes) { | ||
| + counts.set(c, (counts.get(c) ?? 0) + 1); | ||
| + } | ||
| + } | ||
| + } | ||
| + return [...counts.entries()].sort((a, b) => b[1] - a[1]); | ||
| +}; | ||
| + | ||
| +const cell = (r) => { | ||
| + if (!r) { | ||
| + return "not run"; | ||
| + } | ||
| + if (r.status === "provider_error" || r.status === "aborted") { | ||
| + return `${r.status}`; | ||
| + } | ||
| + const tests = r.tests ? `, tests ${r.tests.pass}/${r.tests.total}` : ""; | ||
| + const word = r.passFirstTry ? "pass@1" : r.passFinal ? "pass after repair" : r.status === "compile_failed" ? "never compiled" : "fails tests"; | ||
| + return `${word} (${r.attempts.length} attempt${r.attempts.length === 1 ? "" : "s"}${tests})`; | ||
| +}; | ||
| + | ||
| +const renderRun = ({ dir, meta, records }) => { | ||
| + const langs = meta.langs; | ||
| + const byLang = Object.fromEntries(langs.map((l) => [l, summarize(records.filter((r) => r.lang === l))])); | ||
| + const lines = []; | ||
| + const date = meta.startedAt.slice(0, 10); | ||
| + lines.push(`## ${meta.label[0].toUpperCase()}${meta.label.slice(1)}: ${date}, ${meta.provider === "claude-cli" || meta.provider === "anthropic" ? meta.model : meta.provider}`); | ||
| + lines.push(""); | ||
| + if (!isModelRun(meta)) { | ||
| + lines.push( | ||
| + `**No model was called in this run.** The \`${meta.provider}\` provider feeds the hand-written reference solutions${meta.provider === "selftest" ? " (with a planted compile error on the first attempt, to exercise the repair loop)" : ""} through the full pipeline. It shows the harness and the hidden tests work end to end. It says nothing about the thesis.`, | ||
| + ); | ||
| + lines.push(""); | ||
| + } | ||
| + lines.push( | ||
| + `Run \`${meta.runId}\` (${path.relative(repoRoot, dir)}): provider \`${meta.provider}\`, model \`${meta.model}\`, effort ${meta.effort}, up to ${meta.maxRepairs} repair rounds, ${meta.tasks.length} task(s), repo commit \`${meta.repoCommit}\`${meta.tsusCliDirty ? " (compiler had uncommitted changes)" : ""}, Node ${meta.node}. Total cost ${money(meta.totalCostUsd ?? 0)}.${meta.aborted ? ` Aborted: ${meta.aborted}.` : ""}`, | ||
| + ); | ||
| + lines.push(""); | ||
| + lines.push(`| Metric | ${langs.map((l) => `.${l}`).join(" | ")} |`); | ||
| + lines.push(`|---|${langs.map(() => "---").join("|")}|`); | ||
| + const row = (label, f) => lines.push(`| ${label} | ${langs.map((l) => f(byLang[l])).join(" | ")} |`); | ||
| + row("Tasks scored", (s) => `${s.scored}${s.providerErrors ? ` (+${s.providerErrors} provider errors)` : ""}`); | ||
| + row("pass@1 (first attempt compiles and passes every hidden test)", (s) => frac(s.pass1, s.scored)); | ||
| + row("Pass after compiler-guided repair", (s) => frac(s.passFinal, s.scored)); | ||
| + row("Compiled on first attempt", (s) => frac(s.compiledFirst, s.scored)); | ||
| + row("Tasks where the compiler caught something", (s) => `${s.tasksCaughtAtCompile}`); | ||
| + row("Compile-time catches (rejected attempts / diagnostics)", (s) => `${s.compileRejections} / ${s.compileDiagnostics}`); | ||
| + row("Tasks rescued by repair (rejected, then passed)", (s) => `${s.rescuedByRepair}`); | ||
| + row("Never compiled within the repair budget", (s) => `${s.neverCompiled}`); | ||
| + row("Average repair rounds", (s) => s.avgRepairs.toFixed(2)); | ||
| + row("Tasks failing at runtime (hidden tests)", (s) => `${s.tasksFailingAtRuntime}`); | ||
| + row("Failing hidden test cases", (s) => `${s.failingTestCases} of ${s.totalTestCases}`); | ||
| + row("Cost", (s) => money(s.cost)); | ||
| + row("Output tokens", (s) => `${s.outputTokens}`); | ||
| + row("Average wall time per task (model calls only)", (s) => `${s.avgSeconds.toFixed(1)} s`); | ||
| + lines.push(""); | ||
| + for (const l of langs) { | ||
| + const codes = codeCounts(records.filter((r) => r.lang === l)); | ||
| + if (codes.length > 0) { | ||
| + lines.push(`Diagnostic codes seen in .${l} attempts: ${codes.map(([c, n]) => `${c} x${n}`).join(", ")}.`); | ||
| + lines.push(""); | ||
| + } | ||
| + } | ||
| + lines.push(`<details><summary>Per-task detail</summary>`); | ||
| + lines.push(""); | ||
| + lines.push(`| Task | ${langs.map((l) => `.${l}`).join(" | ")} |`); | ||
| + lines.push(`|---|${langs.map(() => "---").join("|")}|`); | ||
| + const taskIds = [...new Set(records.map((r) => r.task))].sort(); | ||
| + for (const id of taskIds) { | ||
| + lines.push(`| ${id} | ${langs.map((l) => cell(records.find((r) => r.task === id && r.lang === l))).join(" | ")} |`); | ||
| + } | ||
| + lines.push(""); | ||
| + const failing = records.filter((r) => r.tests && !r.tests.allPassed); | ||
| + if (failing.length > 0) { | ||
| + lines.push("Failing hidden tests:"); | ||
| + lines.push(""); | ||
| + for (const r of failing) { | ||
| + lines.push(`- ${r.task} (.${r.lang}): ${r.tests.failures.join("; ") || "see attempts/*.tests.txt"}`); | ||
| + } | ||
| + lines.push(""); | ||
| + } | ||
| + const errors = records.filter((r) => r.error); | ||
| + if (errors.length > 0) { | ||
| + lines.push("Provider errors:"); | ||
| + lines.push(""); | ||
| + for (const r of errors) { | ||
| + lines.push(`- ${r.task} (.${r.lang}): ${r.error}`); | ||
| + } | ||
| + lines.push(""); | ||
| + } | ||
| + lines.push(`</details>`); | ||
| + lines.push(""); | ||
| + return lines.join("\n"); | ||
| +}; | ||
| + | ||
| +const taskCount = fs.readdirSync(path.join(evalsDir, "tasks")).filter((n) => fs.existsSync(path.join(evalsDir, "tasks", n, "task.json"))).length; | ||
| +const modelRuns = runs.filter((r) => isModelRun(r.meta)); | ||
| + | ||
| +const header = `# tsus eval results | ||
| + | ||
| +Generated by \`node evals/report.mjs\` from the runs in \`evals/results/\`. Method and caveats are in [evals/README.md](README.md). | ||
| + | ||
| +The question: when a model writes the same small program in plain TypeScript (\`tsc --strict\`) and in TypeScript-Ultra-Strict, which version is correct more often? Each of the ${taskCount} tasks has a prompt, a signature per language, and hidden \`node:test\` tests the model never sees. The compiler's errors are fed back for up to N repair rounds; test results never are. | ||
| + | ||
| +How to read the metrics: | ||
| + | ||
| +- **pass@1**: the first attempt compiled and passed every hidden test. | ||
| +- **Pass after repair**: some attempt within the repair budget compiled, and that version passed every hidden test. | ||
| +- **Compile-time catches**: attempts the compiler rejected. In .tsus these include the ban rules (TSUS001 to TSUS005) and macro errors. A catch is only good news if the repaired version then passes; "tasks rescued by repair" counts those. | ||
| +- **Failing at runtime**: the solution compiled but at least one hidden test failed. These are the bugs that would have shipped. | ||
| + | ||
| +${ | ||
| + modelRuns.length === 0 | ||
| + ? "**No model run has been recorded yet.** The sections below come from the harness self-test, which uses the reference solutions instead of a model. See evals/README.md for how to run the pilot.\n" | ||
| + : `**Sample size warning.** The largest model run below covers ${Math.max(...modelRuns.map((r) => r.meta.tasks.length))} task(s), one sample each, with one model. Differences of a few tasks between languages are within noise. Treat these numbers as a pilot, not as evidence.\n` | ||
| +}`; | ||
| + | ||
| +const body = runs.length === 0 ? "No runs found.\n" : runs.map(renderRun).join("\n"); | ||
| +const out = `${header}\n${body}`; | ||
| +if (toStdout) { | ||
| + process.stdout.write(out); | ||
| +} else { | ||
| + fs.writeFileSync(path.join(evalsDir, "RESULTS.md"), out); | ||
| + console.log(`wrote ${path.relative(repoRoot, path.join(evalsDir, "RESULTS.md"))} from ${runs.length} run(s)`); | ||
| +} | ||
Addedevals/results/2026-10-17-05-27-00-selftest/meta.json+66−0
| @@ -0,0 +1,66 @@ | ||
| +{ | ||
| + "runId": "2026-10-17-05-27-00-selftest", | ||
| + "label": "harness self-test", | ||
| + "startedAt": "2026-10-17T05:27:00.367Z", | ||
| + "provider": "selftest", | ||
| + "model": "none", | ||
| + "effort": "default", | ||
| + "maxRepairs": 3, | ||
| + "langs": [ | ||
| + "ts", | ||
| + "tsus" | ||
| + ], | ||
| + "tasks": [ | ||
| + "bank-transfer", | ||
| + "cart-total", | ||
| + "csv-rows", | ||
| + "date-range-merge", | ||
| + "deep-get", | ||
| + "feature-flag", | ||
| + "free-slot", | ||
| + "inventory-ledger", | ||
| + "json-api-handler", | ||
| + "lru-cache", | ||
| + "manager-chain", | ||
| + "merge-profile", | ||
| + "moving-average", | ||
| + "order-state-machine", | ||
| + "paginate-cursor", | ||
| + "parse-amount", | ||
| + "parse-config", | ||
| + "parse-duration", | ||
| + "parse-log-line", | ||
| + "query-params", | ||
| + "resolve-path", | ||
| + "retry-fetch", | ||
| + "route-match", | ||
| + "semver-compare", | ||
| + "shortest-path", | ||
| + "signup-validation", | ||
| + "stats-summary", | ||
| + "token-bucket", | ||
| + "topo-sort", | ||
| + "ttl-cache" | ||
| + ], | ||
| + "repoCommit": "6dae872", | ||
| + "tsusCliDirty": false, | ||
| + "tsusBin": "packages/cli/bin/tsus.js", | ||
| + "tsCompilerOptions": { | ||
| + "strict": true, | ||
| + "target": "ES2022", | ||
| + "module": "ESNext", | ||
| + "moduleResolution": "Bundler", | ||
| + "lib": [ | ||
| + "ES2022" | ||
| + ], | ||
| + "types": [], | ||
| + "rootDir": "src", | ||
| + "outDir": "dist", | ||
| + "skipLibCheck": true, | ||
| + "noEmitOnError": true | ||
| + }, | ||
| + "node": "v24.17.0", | ||
| + "finishedAt": "2026-10-17T05:27:27.279Z", | ||
| + "totalCostUsd": 0, | ||
| + "aborted": false | ||
| +} | ||
Addedevals/results/2026-10-17-05-27-00-selftest/results.jsonl+60−0
| @@ -0,0 +1,60 @@ | ||
| +{"task":"bank-transfer","lang":"ts","tags":["errors","validation","state","falsy"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":141,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":154,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"bank-transfer","lang":"tsus","tags":["errors","validation","state","falsy"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":149,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":315,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"cart-total","lang":"ts","tags":["validation","falsy","absence","errors"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":125,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":152,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"cart-total","lang":"tsus","tags":["validation","falsy","absence","errors"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":157,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":330,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"csv-rows","lang":"ts","tags":["parsing","errors","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":143,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":152,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"csv-rows","lang":"tsus","tags":["parsing","errors","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":166,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":323,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"date-range-merge","lang":"ts","tags":["dates","validation","parsing","data-structures"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":151,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":161,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"date-range-merge","lang":"tsus","tags":["dates","validation","parsing","data-structures"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":143,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":322,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"deep-get","lang":"ts","tags":["absence","falsy","parsing","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":152,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":140,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":11,"fail":0,"total":11,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"deep-get","lang":"tsus","tags":["absence","falsy","parsing","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":151,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":325,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":11,"fail":0,"total":11,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"feature-flag","lang":"ts","tags":["absence","falsy","validation","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":126,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":133,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"feature-flag","lang":"tsus","tags":["absence","falsy","validation","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":144,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":321,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"free-slot","lang":"ts","tags":["absence","falsy","boundary","validation","data-structures"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":125,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":150,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":11,"fail":0,"total":11,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"free-slot","lang":"tsus","tags":["absence","falsy","boundary","validation","data-structures"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":152,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":321,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":11,"fail":0,"total":11,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"inventory-ledger","lang":"ts","tags":["validation","absence","numeric","state","errors"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":138,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":139,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"inventory-ledger","lang":"tsus","tags":["validation","absence","numeric","state","errors"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":144,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":317,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"json-api-handler","lang":"ts","tags":["parsing","validation","boundary","falsy","errors"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":127,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":152,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"json-api-handler","lang":"tsus","tags":["parsing","validation","boundary","falsy","errors"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":164,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":347,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"lru-cache","lang":"ts","tags":["absence","falsy","state","data-structures"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":116,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":128,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"lru-cache","lang":"tsus","tags":["absence","falsy","state","data-structures"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":138,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":321,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"manager-chain","lang":"ts","tags":["absence","data-structures","errors","falsy"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":130,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":145,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"manager-chain","lang":"tsus","tags":["absence","data-structures","errors","falsy"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":140,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":353,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"merge-profile","lang":"ts","tags":["absence","falsy","validation"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":126,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":133,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"merge-profile","lang":"tsus","tags":["absence","falsy","validation"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":151,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":318,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"moving-average","lang":"ts","tags":["absence","numeric","falsy","validation"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":135,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":220,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"moving-average","lang":"tsus","tags":["absence","numeric","falsy","validation"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":145,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":288,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"order-state-machine","lang":"ts","tags":["state","validation","errors","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":146,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":147,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":14,"fail":0,"total":14,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"order-state-machine","lang":"tsus","tags":["state","validation","errors","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":149,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":309,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":14,"fail":0,"total":14,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"paginate-cursor","lang":"ts","tags":["absence","falsy","validation","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":125,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":128,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"paginate-cursor","lang":"tsus","tags":["absence","falsy","validation","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":144,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":323,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"parse-amount","lang":"ts","tags":["parsing","numeric","validation","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":153,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":148,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":10,"fail":0,"total":10,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"parse-amount","lang":"tsus","tags":["parsing","numeric","validation","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":234,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":320,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":10,"fail":0,"total":10,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"parse-config","lang":"ts","tags":["absence","validation"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":121,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":127,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":11,"fail":0,"total":11,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"parse-config","lang":"tsus","tags":["absence","validation"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":142,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":330,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":11,"fail":0,"total":11,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"parse-duration","lang":"ts","tags":["parsing","falsy","errors"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":127,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":135,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":11,"fail":0,"total":11,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"parse-duration","lang":"tsus","tags":["parsing","falsy","errors"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":158,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":306,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":11,"fail":0,"total":11,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"parse-log-line","lang":"ts","tags":["parsing","absence","errors","dates","falsy"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":147,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":139,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"parse-log-line","lang":"tsus","tags":["parsing","absence","errors","dates","falsy"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":144,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":316,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"query-params","lang":"ts","tags":["parsing","absence","falsy","errors"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":126,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":118,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"query-params","lang":"tsus","tags":["parsing","absence","falsy","errors"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":145,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":289,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"resolve-path","lang":"ts","tags":["parsing","errors","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":149,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":121,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"resolve-path","lang":"tsus","tags":["parsing","errors","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":151,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":289,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"retry-fetch","lang":"ts","tags":["async","errors","absence","falsy"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":109,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":119,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"retry-fetch","lang":"tsus","tags":["async","errors","absence","falsy"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":135,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":288,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"route-match","lang":"ts","tags":["parsing","absence","errors","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":122,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":128,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"route-match","lang":"tsus","tags":["parsing","absence","errors","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":136,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":300,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"semver-compare","lang":"ts","tags":["parsing","validation","errors"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":121,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":121,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"semver-compare","lang":"tsus","tags":["parsing","validation","errors"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":141,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":297,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"shortest-path","lang":"ts","tags":["absence","data-structures"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":116,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":119,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"shortest-path","lang":"tsus","tags":["absence","data-structures"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":134,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":291,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"signup-validation","lang":"ts","tags":["validation","absence","falsy","dates"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":117,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":125,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":14,"fail":0,"total":14,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"signup-validation","lang":"tsus","tags":["validation","absence","falsy","dates"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":140,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":300,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":14,"fail":0,"total":14,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"stats-summary","lang":"ts","tags":["numeric","errors","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":113,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":117,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"stats-summary","lang":"tsus","tags":["numeric","errors","boundary"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":135,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":288,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"token-bucket","lang":"ts","tags":["state","numeric","falsy","absence"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":117,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":193,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"token-bucket","lang":"tsus","tags":["state","numeric","falsy","absence"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":137,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":289,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"topo-sort","lang":"ts","tags":["absence","errors","data-structures"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":118,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":123,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"topo-sort","lang":"tsus","tags":["absence","errors","data-structures"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":136,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":290,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":13,"fail":0,"total":13,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"ttl-cache","lang":"ts","tags":["async","state","errors","falsy"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TS2322"],"modelMs":0,"compileMs":116,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":120,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
| +{"task":"ttl-cache","lang":"tsus","tags":["async","state","errors","falsy"],"attempts":[{"round":0,"compiled":false,"diagnosticCount":1,"codes":["TSUS002"],"modelMs":0,"compileMs":138,"costUsd":0,"outputTokens":0},{"round":1,"compiled":true,"diagnosticCount":0,"codes":[],"modelMs":0,"compileMs":289,"costUsd":0,"outputTokens":0}],"status":"passed","costUsd":0,"ms":0,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0},"tests":{"pass":12,"fail":0,"total":12,"timedOut":false,"failures":[],"allPassed":true},"compiledFirstTry":false,"passFirstTry":false,"passFinal":true,"repairRounds":1,"compileRejections":1,"compileDiagnostics":1,"runtimeFailures":0} | ||
Addedevals/run.mjs+297−0
| @@ -0,0 +1,297 @@ | ||
| +#!/usr/bin/env node | ||
| +// The tsus eval runner. | ||
| +// | ||
| +// For each task and each language: ask the model for a solution, compile it, | ||
| +// feed compiler errors back for up to --repairs rounds, then run the hidden | ||
| +// tests once on the first version that compiles. Test results are never shown | ||
| +// to the model; only the compiler talks back. | ||
| +// | ||
| +// node evals/run.mjs --pilot # 10 pilot tasks, ts + tsus | ||
| +// node evals/run.mjs --tasks lru-cache,parse-config --langs tsus | ||
| +// node evals/run.mjs --provider reference # harness self-test, no model | ||
| +// | ||
| +// Writes evals/results/<run-id>/{meta.json,results.jsonl,attempts/*}. | ||
| +import fs from "node:fs"; | ||
| +import path from "node:path"; | ||
| +import { execSync } from "node:child_process"; | ||
| +import { parseArgs } from "node:util"; | ||
| +import { compile, formatDiagnostics, runHiddenTests, evalsDir, repoRoot, defaultTsusBin, TS_COMPILER_OPTIONS } from "./lib/compile.mjs"; | ||
| +import { makeProvider, extractCode, DEFAULT_MODEL } from "./lib/providers.mjs"; | ||
| + | ||
| +const { values: opts } = parseArgs({ | ||
| + options: { | ||
| + tasks: { type: "string" }, | ||
| + pilot: { type: "boolean", default: false }, | ||
| + limit: { type: "string" }, | ||
| + langs: { type: "string", default: "ts,tsus" }, | ||
| + model: { type: "string", default: process.env.TSUS_EVAL_MODEL ?? DEFAULT_MODEL }, | ||
| + provider: { type: "string" }, | ||
| + effort: { type: "string" }, | ||
| + repairs: { type: "string", default: "3" }, | ||
| + concurrency: { type: "string", default: "4" }, | ||
| + label: { type: "string" }, | ||
| + "tsus-bin": { type: "string", default: defaultTsusBin }, | ||
| + "max-cost": { type: "string" }, | ||
| + help: { type: "boolean", default: false }, | ||
| + }, | ||
| +}); | ||
| + | ||
| +if (opts.help) { | ||
| + console.log(fs.readFileSync(new URL(import.meta.url), "utf8").split("\n").slice(1, 14).join("\n").replace(/^\/\/ ?/gm, "")); | ||
| + process.exit(0); | ||
| +} | ||
| + | ||
| +const tasksDir = path.join(evalsDir, "tasks"); | ||
| +const loadTasks = () => | ||
| + fs | ||
| + .readdirSync(tasksDir) | ||
| + .filter((name) => fs.existsSync(path.join(tasksDir, name, "task.json"))) | ||
| + .sort() | ||
| + .map((name) => { | ||
| + const dir = path.join(tasksDir, name); | ||
| + const meta = JSON.parse(fs.readFileSync(path.join(dir, "task.json"), "utf8")); | ||
| + return { ...meta, id: name, dir }; | ||
| + }); | ||
| + | ||
| +let tasks = loadTasks(); | ||
| +if (opts.tasks) { | ||
| + const wanted = opts.tasks.split(",").map((s) => s.trim()); | ||
| + const missing = wanted.filter((w) => !tasks.some((t) => t.id === w)); | ||
| + if (missing.length > 0) { | ||
| + console.error(`unknown task(s): ${missing.join(", ")}`); | ||
| + process.exit(2); | ||
| + } | ||
| + tasks = tasks.filter((t) => wanted.includes(t.id)); | ||
| +} else if (opts.pilot) { | ||
| + tasks = tasks.filter((t) => t.pilot === true); | ||
| +} | ||
| +if (opts.limit) { | ||
| + tasks = tasks.slice(0, Number(opts.limit)); | ||
| +} | ||
| +const langs = opts.langs.split(",").map((s) => s.trim()); | ||
| +const maxRepairs = Number(opts.repairs); | ||
| +const concurrency = Math.max(1, Number(opts.concurrency)); | ||
| +const maxCost = opts["max-cost"] ? Number(opts["max-cost"]) : Infinity; | ||
| +const provider = makeProvider({ provider: opts.provider, model: opts.model, effort: opts.effort }); | ||
| + | ||
| +const language = fs.readFileSync(path.join(evalsDir, "LANGUAGE.md"), "utf8"); | ||
| +const systemPrompts = { | ||
| + ts: fs.readFileSync(path.join(evalsDir, "prompts/system-ts.md"), "utf8"), | ||
| + tsus: fs.readFileSync(path.join(evalsDir, "prompts/system-tsus.md"), "utf8").replace("{{LANGUAGE}}", language), | ||
| +}; | ||
| + | ||
| +const taskPrompt = (task, lang) => { | ||
| + const prompt = fs.readFileSync(path.join(task.dir, "prompt.md"), "utf8").trim(); | ||
| + const signature = fs.readFileSync(path.join(task.dir, `signature.${lang}`), "utf8").trim(); | ||
| + return [ | ||
| + `# Task: ${task.title}`, | ||
| + "", | ||
| + prompt, | ||
| + "", | ||
| + `## Signature file (solution.${lang})`, | ||
| + "", | ||
| + "```" + lang, | ||
| + signature, | ||
| + "```", | ||
| + "", | ||
| + `Write the complete solution.${lang}.`, | ||
| + ].join("\n"); | ||
| +}; | ||
| + | ||
| +const repairPrompt = (task, lang, previous, diagnostics) => | ||
| + [ | ||
| + taskPrompt(task, lang), | ||
| + "", | ||
| + "## Your previous attempt", | ||
| + "", | ||
| + "```" + lang, | ||
| + previous, | ||
| + "```", | ||
| + "", | ||
| + "## The compiler rejected it", | ||
| + "", | ||
| + "```", | ||
| + formatDiagnostics(lang, diagnostics), | ||
| + "```", | ||
| + "", | ||
| + `Fix every error and reply with the complete corrected solution.${lang}.`, | ||
| + ].join("\n"); | ||
| + | ||
| +const gitSha = (dir) => { | ||
| + try { | ||
| + return execSync("git rev-parse --short HEAD", { cwd: dir, encoding: "utf8" }).trim(); | ||
| + } catch { | ||
| + return "unknown"; | ||
| + } | ||
| +}; | ||
| +const tsusCliDirty = (() => { | ||
| + try { | ||
| + return execSync("git status --porcelain packages/cli packages/runtime", { cwd: repoRoot, encoding: "utf8" }).trim() !== ""; | ||
| + } catch { | ||
| + return false; | ||
| + } | ||
| +})(); | ||
| + | ||
| +const started = new Date(); | ||
| +const runId = `${started.toISOString().slice(0, 19).replace(/[:T]/g, "-")}-${provider.name}`; | ||
| +const runDir = path.join(evalsDir, "results", runId); | ||
| +const attemptsDir = path.join(runDir, "attempts"); | ||
| +const workRoot = path.join(evalsDir, ".work", runId); | ||
| +fs.mkdirSync(attemptsDir, { recursive: true }); | ||
| + | ||
| +const meta = { | ||
| + runId, | ||
| + label: opts.label ?? (opts.pilot ? "pilot" : provider.name === "reference" || provider.name === "selftest" ? "harness self-test" : "run"), | ||
| + startedAt: started.toISOString(), | ||
| + provider: provider.name, | ||
| + model: provider.model, | ||
| + effort: opts.effort ?? "default", | ||
| + maxRepairs, | ||
| + langs, | ||
| + tasks: tasks.map((t) => t.id), | ||
| + repoCommit: gitSha(repoRoot), | ||
| + tsusCliDirty, | ||
| + tsusBin: path.relative(repoRoot, opts["tsus-bin"]), | ||
| + tsCompilerOptions: TS_COMPILER_OPTIONS, | ||
| + node: process.version, | ||
| +}; | ||
| +fs.writeFileSync(path.join(runDir, "meta.json"), JSON.stringify(meta, null, 2) + "\n"); | ||
| + | ||
| +const resultsFile = path.join(runDir, "results.jsonl"); | ||
| +let spent = 0; | ||
| +let consecutiveProviderErrors = 0; | ||
| +let aborted = ""; | ||
| + | ||
| +const runOne = async (task, lang) => { | ||
| + const record = { | ||
| + task: task.id, | ||
| + lang, | ||
| + tags: task.tags ?? [], | ||
| + attempts: [], | ||
| + status: "pending", | ||
| + costUsd: 0, | ||
| + ms: 0, | ||
| + usage: { inputTokens: 0, outputTokens: 0, cacheReadTokens: 0, cacheWriteTokens: 0 }, | ||
| + }; | ||
| + const workDir = path.join(workRoot, task.id, lang); | ||
| + let code = ""; | ||
| + let diagnostics = []; | ||
| + for (let round = 0; round <= maxRepairs; round += 1) { | ||
| + if (aborted) { | ||
| + record.status = "aborted"; | ||
| + return record; | ||
| + } | ||
| + const prompt = round === 0 ? taskPrompt(task, lang) : repairPrompt(task, lang, code, diagnostics); | ||
| + let reply; | ||
| + try { | ||
| + reply = await provider.generate({ system: systemPrompts[lang], prompt, task, lang, round }); | ||
| + consecutiveProviderErrors = 0; | ||
| + } catch (error) { | ||
| + consecutiveProviderErrors += 1; | ||
| + record.status = "provider_error"; | ||
| + record.error = String(error.message ?? error).slice(0, 500); | ||
| + if (consecutiveProviderErrors >= 3) { | ||
| + aborted = `3 provider errors in a row, last: ${record.error}`; | ||
| + } | ||
| + return record; | ||
| + } | ||
| + spent += reply.costUsd; | ||
| + record.costUsd += reply.costUsd; | ||
| + record.ms += reply.ms; | ||
| + for (const k of Object.keys(record.usage)) { | ||
| + record.usage[k] += reply.usage[k] ?? 0; | ||
| + } | ||
| + code = extractCode(reply.text); | ||
| + const base = path.join(attemptsDir, `${task.id}.${lang}.r${round}`); | ||
| + fs.writeFileSync(`${base}.${lang}`, code); | ||
| + const compileStarted = Date.now(); | ||
| + const result = compile(lang, workDir, code, { tsusBin: opts["tsus-bin"] }); | ||
| + diagnostics = result.diagnostics; | ||
| + record.attempts.push({ | ||
| + round, | ||
| + compiled: result.ok, | ||
| + diagnosticCount: diagnostics.length, | ||
| + codes: [...new Set(diagnostics.map((d) => d.code))], | ||
| + modelMs: reply.ms, | ||
| + compileMs: Date.now() - compileStarted, | ||
| + costUsd: reply.costUsd, | ||
| + outputTokens: reply.usage.outputTokens, | ||
| + }); | ||
| + if (diagnostics.length > 0) { | ||
| + fs.writeFileSync(`${base}.diagnostics.txt`, formatDiagnostics(lang, diagnostics) + "\n"); | ||
| + } | ||
| + if (result.ok) { | ||
| + const tests = runHiddenTests(lang, path.join(task.dir, "test.mjs"), result.emitted); | ||
| + record.tests = tests; | ||
| + record.status = tests.allPassed ? "passed" : "failed_tests"; | ||
| + if (!tests.allPassed && tests.rawTail) { | ||
| + fs.writeFileSync(`${base}.tests.txt`, tests.rawTail); | ||
| + } | ||
| + delete record.tests.rawTail; | ||
| + break; | ||
| + } | ||
| + record.status = "compile_failed"; | ||
| + if (spent >= maxCost) { | ||
| + aborted = `cost cap $${maxCost} reached`; | ||
| + } | ||
| + } | ||
| + const first = record.attempts[0]; | ||
| + record.compiledFirstTry = first?.compiled === true; | ||
| + record.passFirstTry = record.compiledFirstTry && record.status === "passed"; | ||
| + record.passFinal = record.status === "passed"; | ||
| + record.repairRounds = Math.max(0, record.attempts.length - 1); | ||
| + record.compileRejections = record.attempts.filter((a) => !a.compiled).length; | ||
| + record.compileDiagnostics = record.attempts.reduce((n, a) => n + a.diagnosticCount, 0); | ||
| + record.runtimeFailures = record.tests?.fail ?? 0; | ||
| + if (spent >= maxCost && !aborted) { | ||
| + aborted = `cost cap $${maxCost} reached`; | ||
| + } | ||
| + return record; | ||
| +}; | ||
| + | ||
| +const jobs = []; | ||
| +for (const task of tasks) { | ||
| + for (const lang of langs) { | ||
| + if (fs.existsSync(path.join(task.dir, `signature.${lang}`))) { | ||
| + jobs.push({ task, lang }); | ||
| + } | ||
| + } | ||
| +} | ||
| + | ||
| +console.log(`run ${runId}: ${jobs.length} job(s), provider=${provider.name}, model=${provider.model}, repairs=${maxRepairs}`); | ||
| +let next = 0; | ||
| +let done = 0; | ||
| +const worker = async () => { | ||
| + while (next < jobs.length) { | ||
| + const { task, lang } = jobs[next]; | ||
| + next += 1; | ||
| + const record = await runOne(task, lang); | ||
| + if (record.status === "provider_error") { | ||
| + providerErrors += 1; | ||
| + } | ||
| + fs.appendFileSync(resultsFile, JSON.stringify(record) + "\n"); | ||
| + done += 1; | ||
| + const t = record.tests ? ` tests ${record.tests.pass}/${record.tests.total}` : ""; | ||
| + console.log( | ||
| + `[${done}/${jobs.length}] ${task.id} (${lang}): ${record.status}, ${record.attempts.length} attempt(s)${t}, $${record.costUsd.toFixed(4)}${record.error ? ` (${record.error})` : ""}`, | ||
| + ); | ||
| + } | ||
| +}; | ||
| +let providerErrors = 0; | ||
| +await Promise.all(Array.from({ length: Math.min(concurrency, jobs.length) }, worker)); | ||
| + | ||
| +meta.finishedAt = new Date().toISOString(); | ||
| +meta.totalCostUsd = Number(spent.toFixed(4)); | ||
| +meta.aborted = aborted || false; | ||
| +fs.writeFileSync(path.join(runDir, "meta.json"), JSON.stringify(meta, null, 2) + "\n"); | ||
| +fs.rmSync(workRoot, { recursive: true, force: true }); | ||
| +console.log(`done: total cost $${spent.toFixed(4)}${aborted ? `, aborted: ${aborted}` : ""}`); | ||
| +console.log(`results: ${path.relative(repoRoot, runDir)}`); | ||
| +console.log(`report: node evals/report.mjs ${path.relative(repoRoot, runDir)}`); | ||
| +if (providerErrors === jobs.length && jobs.length > 0) { | ||
| + console.error("every job failed at the provider; check model access (see evals/README.md)"); | ||
| + process.exit(1); | ||
| +} | ||
| +if (aborted) { | ||
| + process.exit(1); | ||
| +} | ||
Addedevals/tasks/bank-transfer/mutant.ts+42−0
| @@ -0,0 +1,42 @@ | ||
| +// Bug: truthiness checks (`!amountCents`, `!accounts[id]`) and in-place mutation of the input accounts. | ||
| +export type Account = { balanceCents: number; frozen: boolean }; | ||
| + | ||
| +export type TransferFailure = | ||
| + | { kind: "invalid_amount" } | ||
| + | { kind: "same_account" } | ||
| + | { kind: "unknown_account"; id: string } | ||
| + | { kind: "frozen"; id: string } | ||
| + | { kind: "insufficient_funds"; id: string; balanceCents: number; amountCents: number }; | ||
| + | ||
| +export class TransferError extends Error { | ||
| + declare readonly kind: TransferFailure["kind"]; | ||
| + declare readonly id?: string; | ||
| + declare readonly balanceCents?: number; | ||
| + declare readonly amountCents?: number; | ||
| + constructor(failure: TransferFailure) { | ||
| + super(failure.kind); | ||
| + Object.assign(this, failure); | ||
| + } | ||
| +} | ||
| + | ||
| +export function transfer( | ||
| + accounts: Record<string, Account>, | ||
| + from: string, | ||
| + to: string, | ||
| + amountCents: number, | ||
| +): Record<string, Account> { | ||
| + if (!amountCents || amountCents < 0) throw new TransferError({ kind: "invalid_amount" }); | ||
| + if (from === to) throw new TransferError({ kind: "same_account" }); | ||
| + const src = accounts[from]; | ||
| + if (!src) throw new TransferError({ kind: "unknown_account", id: from }); | ||
| + const dst = accounts[to]; | ||
| + if (!dst) throw new TransferError({ kind: "unknown_account", id: to }); | ||
| + if (src.frozen) throw new TransferError({ kind: "frozen", id: from }); | ||
| + if (dst.frozen) throw new TransferError({ kind: "frozen", id: to }); | ||
| + if (src.balanceCents < amountCents) { | ||
| + throw new TransferError({ kind: "insufficient_funds", id: from, balanceCents: src.balanceCents, amountCents }); | ||
| + } | ||
| + src.balanceCents -= amountCents; | ||
| + dst.balanceCents += amountCents; | ||
| + return { ...accounts }; | ||
| +} | ||
Addedevals/tasks/bank-transfer/prompt.md+17−0
| @@ -0,0 +1,17 @@ | ||
| +Write `transfer`, which moves money from one account to another and returns the updated accounts. | ||
| + | ||
| +Input: | ||
| + | ||
| +- `accounts`: a plain object mapping account id to `{ balanceCents, frozen }`. Only own keys of `accounts` are accounts; inherited properties of plain objects are not. | ||
| +- `from`, `to`: account ids. | ||
| +- `amountCents`: the amount to move, in integer cents. | ||
| + | ||
| +On success, return a new accounts object with the same keys, where `from`'s balance went down by `amountCents` and `to`'s went up by the same amount. A balance may reach exactly zero. The input object and the account objects inside it must never be modified (accounts not involved in the transfer may be shared with the input). | ||
| + | ||
| +Failures, checked in exactly this order (report the first that applies): | ||
| + | ||
| +1. `{ kind: "invalid_amount" }` when `amountCents` is not an integer or is not greater than zero. | ||
| +2. `{ kind: "same_account" }` when `from` and `to` are the same id. | ||
| +3. `{ kind: "unknown_account", id }` when `from` is not an account, then the same check for `to`. | ||
| +4. `{ kind: "frozen", id }` when `from` is frozen, then the same check for `to`. | ||
| +5. `{ kind: "insufficient_funds", id, balanceCents, amountCents }` when `from`'s balance is less than `amountCents` (`id` is `from`, `balanceCents` its current balance). | ||
Addedevals/tasks/bank-transfer/reference.ts+47−0
| @@ -0,0 +1,47 @@ | ||
| +export type Account = { balanceCents: number; frozen: boolean }; | ||
| + | ||
| +export type TransferFailure = | ||
| + | { kind: "invalid_amount" } | ||
| + | { kind: "same_account" } | ||
| + | { kind: "unknown_account"; id: string } | ||
| + | { kind: "frozen"; id: string } | ||
| + | { kind: "insufficient_funds"; id: string; balanceCents: number; amountCents: number }; | ||
| + | ||
| +export class TransferError extends Error { | ||
| + declare readonly kind: TransferFailure["kind"]; | ||
| + declare readonly id?: string; | ||
| + declare readonly balanceCents?: number; | ||
| + declare readonly amountCents?: number; | ||
| + constructor(failure: TransferFailure) { | ||
| + super(failure.kind); | ||
| + Object.assign(this, failure); | ||
| + } | ||
| +} | ||
| + | ||
| +const lookup = (accounts: Record<string, Account>, id: string): Account => { | ||
| + if (!Object.hasOwn(accounts, id)) throw new TransferError({ kind: "unknown_account", id }); | ||
| + return accounts[id]; | ||
| +}; | ||
| + | ||
| +/** Throws TransferError when the transfer is not allowed. */ | ||
| +export function transfer( | ||
| + accounts: Record<string, Account>, | ||
| + from: string, | ||
| + to: string, | ||
| + amountCents: number, | ||
| +): Record<string, Account> { | ||
| + if (!Number.isInteger(amountCents) || amountCents <= 0) throw new TransferError({ kind: "invalid_amount" }); | ||
| + if (from === to) throw new TransferError({ kind: "same_account" }); | ||
| + const src = lookup(accounts, from); | ||
| + const dst = lookup(accounts, to); | ||
| + if (src.frozen) throw new TransferError({ kind: "frozen", id: from }); | ||
| + if (dst.frozen) throw new TransferError({ kind: "frozen", id: to }); | ||
| + if (src.balanceCents < amountCents) { | ||
| + throw new TransferError({ kind: "insufficient_funds", id: from, balanceCents: src.balanceCents, amountCents }); | ||
| + } | ||
| + return { | ||
| + ...accounts, | ||
| + [from]: { ...src, balanceCents: src.balanceCents - amountCents }, | ||
| + [to]: { ...dst, balanceCents: dst.balanceCents + amountCents }, | ||
| + }; | ||
| +} | ||
Addedevals/tasks/bank-transfer/reference.tsus+45−0
| @@ -0,0 +1,45 @@ | ||
| +import { Ok, Err, type Result } from "@tsus/runtime"; | ||
| + | ||
| +export type Account = { balanceCents: number; frozen: boolean }; | ||
| + | ||
| +export type TransferError = | ||
| + | { kind: "invalid_amount" } | ||
| + | { kind: "same_account" } | ||
| + | { kind: "unknown_account"; id: string } | ||
| + | { kind: "frozen"; id: string } | ||
| + | { kind: "insufficient_funds"; id: string; balanceCents: number; amountCents: number }; | ||
| + | ||
| +const lookup = (accounts: Record<string, Account>, id: string): Result<Account, TransferError> => | ||
| + Object.hasOwn(accounts, id) ? Ok(accounts[id]) : Err({ kind: "unknown_account", id }); | ||
| + | ||
| +const notFrozen = (account: Account, id: string): Result<Account, TransferError> => | ||
| + account.frozen ? Err({ kind: "frozen", id }) : Ok(account); | ||
| + | ||
| +const checkAmount = (amountCents: number): Result<number, TransferError> => | ||
| + Number.isInteger(amountCents) && amountCents > 0 ? Ok(amountCents) : Err({ kind: "invalid_amount" }); | ||
| + | ||
| +const distinct = (from: string, to: string): Result<string, TransferError> => | ||
| + from === to ? Err({ kind: "same_account" }) : Ok(to); | ||
| + | ||
| +export const transfer = ( | ||
| + accounts: Record<string, Account>, | ||
| + from: string, | ||
| + to: string, | ||
| + amountCents: number, | ||
| +): Result<Record<string, Account>, TransferError> => | ||
| + checkAmount(amountCents) | ||
| + .andThen(() => distinct(from, to)) | ||
| + .andThen(() => lookup(accounts, from)) | ||
| + .andThen((src) => | ||
| + lookup(accounts, to) | ||
| + .andThen((dst) => notFrozen(src, from).andThen(() => notFrozen(dst, to))) | ||
| + .andThen((dst): Result<Record<string, Account>, TransferError> => | ||
| + src.balanceCents < amountCents | ||
| + ? Err({ kind: "insufficient_funds", id: from, balanceCents: src.balanceCents, amountCents }) | ||
| + : Ok({ | ||
| + ...accounts, | ||
| + [from]: { ...src, balanceCents: src.balanceCents - amountCents }, | ||
| + [to]: { ...dst, balanceCents: dst.balanceCents + amountCents }, | ||
| + }), | ||
| + ), | ||
| + ); | ||
Addedevals/tasks/bank-transfer/signature.ts+29−0
| @@ -0,0 +1,29 @@ | ||
| +export type Account = { balanceCents: number; frozen: boolean }; | ||
| + | ||
| +export type TransferFailure = | ||
| + | { kind: "invalid_amount" } | ||
| + | { kind: "same_account" } | ||
| + | { kind: "unknown_account"; id: string } | ||
| + | { kind: "frozen"; id: string } | ||
| + | { kind: "insufficient_funds"; id: string; balanceCents: number; amountCents: number }; | ||
| + | ||
| +export class TransferError extends Error { | ||
| + declare readonly kind: TransferFailure["kind"]; | ||
| + declare readonly id?: string; | ||
| + declare readonly balanceCents?: number; | ||
| + declare readonly amountCents?: number; | ||
| + constructor(failure: TransferFailure) { | ||
| + super(failure.kind); | ||
| + Object.assign(this, failure); | ||
| + } | ||
| +} | ||
| + | ||
| +/** Throws TransferError when the transfer is not allowed. */ | ||
| +export function transfer( | ||
| + accounts: Record<string, Account>, | ||
| + from: string, | ||
| + to: string, | ||
| + amountCents: number, | ||
| +): Record<string, Account> { | ||
| + // TODO | ||
| +} | ||
Addedevals/tasks/bank-transfer/signature.tsus+19−0
| @@ -0,0 +1,19 @@ | ||
| +import { type Result } from "@tsus/runtime"; | ||
| + | ||
| +export type Account = { balanceCents: number; frozen: boolean }; | ||
| + | ||
| +export type TransferError = | ||
| + | { kind: "invalid_amount" } | ||
| + | { kind: "same_account" } | ||
| + | { kind: "unknown_account"; id: string } | ||
| + | { kind: "frozen"; id: string } | ||
| + | { kind: "insufficient_funds"; id: string; balanceCents: number; amountCents: number }; | ||
| + | ||
| +export const transfer = ( | ||
| + accounts: Record<string, Account>, | ||
| + from: string, | ||
| + to: string, | ||
| + amountCents: number, | ||
| +): Result<Record<string, Account>, TransferError> => { | ||
| + // TODO | ||
| +}; | ||
Addedevals/tasks/bank-transfer/task.json+1−0
| @@ -0,0 +1 @@ | ||
| +{ "title": "Move money between accounts without mutating state", "tags": ["errors", "validation", "state", "falsy"], "pilot": false } | ||
Addedevals/tasks/bank-transfer/test.mjs+89−0
| @@ -0,0 +1,89 @@ | ||
| +import { test } from "node:test"; | ||
| +import assert from "node:assert/strict"; | ||
| +import { h, load } from "../../lib/harness.mjs"; | ||
| + | ||
| +const { transfer } = await load(); | ||
| +const run = (accounts, from, to, amount) => h.res(() => transfer(accounts, from, to, amount)); | ||
| +const fail = (error) => ({ ok: false, error }); | ||
| +const acct = (balanceCents, frozen = false) => ({ balanceCents, frozen }); | ||
| +const book = () => ({ alice: acct(1000), bob: acct(50), carol: acct(0), ice: acct(500, true) }); | ||
| + | ||
| +test("moves money", () => { | ||
| + assert.deepEqual(run(book(), "alice", "bob", 300), { | ||
| + ok: true, | ||
| + value: { alice: acct(700), bob: acct(350), carol: acct(0), ice: acct(500, true) }, | ||
| + }); | ||
| +}); | ||
| + | ||
| +test("balance may reach exactly zero", () => { | ||
| + const r = h.okOf(run(book(), "bob", "carol", 50)); | ||
| + assert.deepEqual(r.bob, acct(0)); | ||
| + assert.deepEqual(r.carol, acct(50)); | ||
| +}); | ||
| + | ||
| +test("amount must be a positive integer", () => { | ||
| + for (const amount of [0, -1, 0.1, 12.5, Number.NaN, Number.POSITIVE_INFINITY]) { | ||
| + assert.deepEqual(run(book(), "alice", "bob", amount), fail({ kind: "invalid_amount" }), `amount ${amount}`); | ||
| + } | ||
| + assert.equal(h.okOf(run(book(), "alice", "bob", 1)).alice.balanceCents, 999); | ||
| +}); | ||
| + | ||
| +test("amount is checked before anything else", () => { | ||
| + assert.deepEqual(run(book(), "ghost", "ghost", 0), fail({ kind: "invalid_amount" })); | ||
| + assert.deepEqual(run(book(), "ghost", "bob", -3), fail({ kind: "invalid_amount" })); | ||
| +}); | ||
| + | ||
| +test("same account", () => { | ||
| + assert.deepEqual(run(book(), "alice", "alice", 10), fail({ kind: "same_account" })); | ||
| + assert.deepEqual(run(book(), "ghost", "ghost", 10), fail({ kind: "same_account" })); | ||
| +}); | ||
| + | ||
| +test("unknown accounts, from first", () => { | ||
| + assert.deepEqual(run(book(), "ghost", "bob", 10), fail({ kind: "unknown_account", id: "ghost" })); | ||
| + assert.deepEqual(run(book(), "alice", "ghost", 10), fail({ kind: "unknown_account", id: "ghost" })); | ||
| + assert.deepEqual(run(book(), "ghost1", "ghost2", 10), fail({ kind: "unknown_account", id: "ghost1" })); | ||
| + assert.deepEqual(run(book(), "alice", "ice", 999999), fail({ kind: "frozen", id: "ice" })); | ||
| +}); | ||
| + | ||
| +test("inherited object properties are not accounts", () => { | ||
| + assert.deepEqual(run(book(), "toString", "bob", 10), fail({ kind: "unknown_account", id: "toString" })); | ||
| + assert.deepEqual(run(book(), "alice", "constructor", 10), fail({ kind: "unknown_account", id: "constructor" })); | ||
| +}); | ||
| + | ||
| +test("frozen accounts, from first", () => { | ||
| + const accounts = { ...book(), ice2: acct(100, true) }; | ||
| + assert.deepEqual(run(accounts, "ice", "alice", 10), fail({ kind: "frozen", id: "ice" })); | ||
| + assert.deepEqual(run(accounts, "alice", "ice", 10), fail({ kind: "frozen", id: "ice" })); | ||
| + assert.deepEqual(run(accounts, "ice2", "ice", 10), fail({ kind: "frozen", id: "ice2" })); | ||
| +}); | ||
| + | ||
| +test("insufficient funds carries the details", () => { | ||
| + assert.deepEqual( | ||
| + run(book(), "bob", "alice", 51), | ||
| + fail({ kind: "insufficient_funds", id: "bob", balanceCents: 50, amountCents: 51 }), | ||
| + ); | ||
| + assert.deepEqual( | ||
| + run(book(), "carol", "alice", 1), | ||
| + fail({ kind: "insufficient_funds", id: "carol", balanceCents: 0, amountCents: 1 }), | ||
| + ); | ||
| +}); | ||
| + | ||
| +test("frozen is reported before insufficient funds", () => { | ||
| + assert.deepEqual(run(book(), "bob", "ice", 5000), fail({ kind: "frozen", id: "ice" })); | ||
| +}); | ||
| + | ||
| +test("input is never modified", () => { | ||
| + const accounts = book(); | ||
| + const snapshot = JSON.parse(JSON.stringify(accounts)); | ||
| + const r = h.okOf(run(accounts, "alice", "bob", 300)); | ||
| + assert.deepEqual(accounts, snapshot); | ||
| + assert.notEqual(r, accounts); | ||
| + run(accounts, "bob", "alice", 9999); | ||
| + assert.deepEqual(accounts, snapshot); | ||
| +}); | ||
| + | ||
| +test("chained transfers use the returned state", () => { | ||
| + const first = h.okOf(run(book(), "alice", "carol", 400)); | ||
| + const second = h.okOf(run(first, "carol", "bob", 400)); | ||
| + assert.deepEqual(second, { alice: acct(600), bob: acct(450), carol: acct(0), ice: acct(500, true) }); | ||
| +}); | ||
Addedevals/tasks/cart-total/mutant.ts+51−0
| @@ -0,0 +1,51 @@ | ||
| +// Bug: coupons[key] with a truthiness check (prototype keys like "toString" pass), and percent 0 / min 0 handled by truthiness. | ||
| +export type CartItem = { sku: string; unitPriceCents: number; quantity: number }; | ||
| + | ||
| +export type Coupon = | ||
| + | { kind: "percent"; percent: number } | ||
| + | { kind: "fixed"; amountCents: number; minSubtotalCents?: number }; | ||
| + | ||
| +export type CartTotal = { subtotalCents: number; discountCents: number; totalCents: number }; | ||
| + | ||
| +export type CartErrorKind = "invalid_item" | "unknown_coupon" | "coupon_not_applicable"; | ||
| + | ||
| +export class CartError extends Error { | ||
| + readonly index?: number; | ||
| + readonly code?: string; | ||
| + constructor( | ||
| + public readonly kind: CartErrorKind, | ||
| + detail: { index?: number; code?: string } = {}, | ||
| + ) { | ||
| + super(kind); | ||
| + Object.assign(this, detail); | ||
| + } | ||
| +} | ||
| + | ||
| +export function cartTotal( | ||
| + items: CartItem[], | ||
| + couponCode: string | undefined, | ||
| + coupons: Record<string, Coupon>, | ||
| +): CartTotal { | ||
| + let subtotalCents = 0; | ||
| + items.forEach((item, index) => { | ||
| + if (!Number.isInteger(item.unitPriceCents) || item.unitPriceCents < 0 || !Number.isInteger(item.quantity) || item.quantity < 0) { | ||
| + throw new CartError("invalid_item", { index }); | ||
| + } | ||
| + subtotalCents += item.unitPriceCents * item.quantity; | ||
| + }); | ||
| + let discountCents = 0; | ||
| + const code = couponCode?.trim().toLowerCase(); | ||
| + if (code) { | ||
| + const coupon = coupons[code]; | ||
| + if (!coupon) throw new CartError("unknown_coupon", { code: couponCode }); | ||
| + if (coupon.kind === "percent") { | ||
| + discountCents = Math.floor((subtotalCents * coupon.percent) / 100); | ||
| + } else { | ||
| + if (coupon.minSubtotalCents && subtotalCents < coupon.minSubtotalCents) { | ||
| + throw new CartError("coupon_not_applicable", { code: couponCode }); | ||
| + } | ||
| + discountCents = Math.min(coupon.amountCents, subtotalCents); | ||
| + } | ||
| + } | ||
| + return { subtotalCents, discountCents, totalCents: subtotalCents - discountCents }; | ||
| +} | ||
Addedevals/tasks/cart-total/prompt.md+18−0
| @@ -0,0 +1,18 @@ | ||
| +Write `cartTotal(items, couponCode, coupons)`, which prices a shopping cart. All amounts are integer cents. | ||
| + | ||
| +Inputs: | ||
| + | ||
| +- `items`: an array of `{ sku, unitPriceCents, quantity }`. Items come from an untrusted request: `unitPriceCents` and `quantity` must each be a non-negative integer of JavaScript type number (numeric strings, `null`, `NaN` and fractions are not). The first item that breaks this rule fails with `{ kind: "invalid_item", index }`. Items are validated before the coupon is looked at. | ||
| +- `couponCode`: optional string. When it is absent, or empty or blank after trimming, no coupon is applied. | ||
| +- `coupons`: the coupon table, keyed by lowercase code. It is trusted configuration. Each coupon is either `{ kind: "percent", percent }` with `percent` from 0 to 100, or `{ kind: "fixed", amountCents, minSubtotalCents }` where `minSubtotalCents` is optional. | ||
| + | ||
| +The code is looked up after trimming it and converting it to lowercase. Only codes defined in the coupons table itself are valid. A code that is not in the table fails with `{ kind: "unknown_coupon", code }`, where `code` is the couponCode string exactly as passed in. | ||
| + | ||
| +Pricing: | ||
| + | ||
| +- `subtotalCents` is the sum of `unitPriceCents * quantity`. | ||
| +- A percent coupon gives `discountCents = floor(subtotalCents * percent / 100)`. | ||
| +- A fixed coupon with a `minSubtotalCents` larger than the subtotal fails with `{ kind: "coupon_not_applicable", code }` (code as passed in). Otherwise it gives `discountCents = amountCents`, capped at the subtotal so the total is never negative. | ||
| +- With no coupon, `discountCents` is 0. `totalCents = subtotalCents - discountCents`. | ||
| + | ||
| +Return `{ subtotalCents, discountCents, totalCents }`. Do not modify the inputs. | ||
Addedevals/tasks/cart-total/reference.ts+54−0
| @@ -0,0 +1,54 @@ | ||
| +export type CartItem = { sku: string; unitPriceCents: number; quantity: number }; | ||
| + | ||
| +export type Coupon = | ||
| + | { kind: "percent"; percent: number } | ||
| + | { kind: "fixed"; amountCents: number; minSubtotalCents?: number }; | ||
| + | ||
| +export type CartTotal = { subtotalCents: number; discountCents: number; totalCents: number }; | ||
| + | ||
| +export type CartErrorKind = "invalid_item" | "unknown_coupon" | "coupon_not_applicable"; | ||
| + | ||
| +export class CartError extends Error { | ||
| + readonly index?: number; | ||
| + readonly code?: string; | ||
| + constructor( | ||
| + public readonly kind: CartErrorKind, | ||
| + detail: { index?: number; code?: string } = {}, | ||
| + ) { | ||
| + super(kind); | ||
| + Object.assign(this, detail); | ||
| + } | ||
| +} | ||
| + | ||
| +const isCount = (v: unknown): v is number => typeof v === "number" && Number.isInteger(v) && v >= 0; | ||
| + | ||
| +/** Throws CartError. */ | ||
| +export function cartTotal( | ||
| + items: CartItem[], | ||
| + couponCode: string | undefined, | ||
| + coupons: Record<string, Coupon>, | ||
| +): CartTotal { | ||
| + let subtotalCents = 0; | ||
| + items.forEach((item, index) => { | ||
| + if (!isCount(item.unitPriceCents) || !isCount(item.quantity)) { | ||
| + throw new CartError("invalid_item", { index }); | ||
| + } | ||
| + subtotalCents += item.unitPriceCents * item.quantity; | ||
| + }); | ||
| + let discountCents = 0; | ||
| + if (couponCode !== undefined && couponCode !== null && couponCode.trim() !== "") { | ||
| + const key = couponCode.trim().toLowerCase(); | ||
| + if (!Object.hasOwn(coupons, key)) throw new CartError("unknown_coupon", { code: couponCode }); | ||
| + const coupon = coupons[key]; | ||
| + if (coupon.kind === "percent") { | ||
| + discountCents = Math.floor((subtotalCents * coupon.percent) / 100); | ||
| + } else { | ||
| + const min = coupon.minSubtotalCents; | ||
| + if (min !== undefined && min !== null && subtotalCents < min) { | ||
| + throw new CartError("coupon_not_applicable", { code: couponCode }); | ||
| + } | ||
| + discountCents = Math.min(coupon.amountCents, subtotalCents); | ||
| + } | ||
| + } | ||
| + return { subtotalCents, discountCents, totalCents: subtotalCents - discountCents }; | ||
| +} | ||
Addedevals/tasks/cart-total/reference.tsus+55−0
| @@ -0,0 +1,55 @@ | ||
| +import { Ok, Err, type Option, type Result } from "@tsus/runtime"; | ||
| + | ||
| +export type CartItem = { sku: string; unitPriceCents: number; quantity: number }; | ||
| + | ||
| +export type Coupon = | ||
| + | { kind: "percent"; percent: number } | ||
| + | { kind: "fixed"; amountCents: number; minSubtotalCents: Option<number> }; | ||
| + | ||
| +export type CartTotal = { subtotalCents: number; discountCents: number; totalCents: number }; | ||
| + | ||
| +export type CartError = | ||
| + | { kind: "invalid_item"; index: number } | ||
| + | { kind: "unknown_coupon"; code: string } | ||
| + | { kind: "coupon_not_applicable"; code: string }; | ||
| + | ||
| +const isCount = (v: unknown): boolean => typeof v === "number" && Number.isInteger(v) && v >= 0; | ||
| + | ||
| +const subtotalOf = (items: CartItem[]): Result<number, CartError> => { | ||
| + let total = 0; | ||
| + for (let index = 0; index < items.length; index += 1) { | ||
| + const item = items[index]; | ||
| + if (!isCount(item.unitPriceCents) || !isCount(item.quantity)) { | ||
| + return Err({ kind: "invalid_item", index }); | ||
| + } | ||
| + total += item.unitPriceCents * item.quantity; | ||
| + } | ||
| + return Ok(total); | ||
| +}; | ||
| + | ||
| +const discountFor = (coupon: Coupon, subtotal: number, code: string): Result<number, CartError> => | ||
| + coupon.kind === "percent" | ||
| + ? Ok(Math.floor((subtotal * coupon.percent) / 100)) | ||
| + : coupon.minSubtotalCents.map((min) => subtotal < min).unwrapOr(false) | ||
| + ? Err({ kind: "coupon_not_applicable", code }) | ||
| + : Ok(Math.min(coupon.amountCents, subtotal)); | ||
| + | ||
| +const applyCode = (code: string, subtotal: number, coupons: Record<string, Coupon>): Result<number, CartError> => { | ||
| + const key = code.trim().toLowerCase(); | ||
| + if (key === "") { | ||
| + return Ok(0); | ||
| + } | ||
| + return Object.hasOwn(coupons, key) ? discountFor(coupons[key], subtotal, code) : Err({ kind: "unknown_coupon", code }); | ||
| +}; | ||
| + | ||
| +export const cartTotal = ( | ||
| + items: CartItem[], | ||
| + couponCode: Option<string>, | ||
| + coupons: Record<string, Coupon>, | ||
| +): Result<CartTotal, CartError> => | ||
| + subtotalOf(items).andThen((subtotalCents) => | ||
| + match (couponCode) { | ||
| + None => Ok(0), | ||
| + Some(code) => applyCode(code, subtotalCents, coupons), | ||
| + }.map((discountCents) => ({ subtotalCents, discountCents, totalCents: subtotalCents - discountCents })), | ||
| + ); | ||
Addedevals/tasks/cart-total/signature.ts+32−0
| @@ -0,0 +1,32 @@ | ||
| +export type CartItem = { sku: string; unitPriceCents: number; quantity: number }; | ||
| + | ||
| +export type Coupon = | ||
| + | { kind: "percent"; percent: number } | ||
| + | { kind: "fixed"; amountCents: number; minSubtotalCents?: number }; | ||
| + | ||
| +export type CartTotal = { subtotalCents: number; discountCents: number; totalCents: number }; | ||
| + | ||
| +export type CartErrorKind = "invalid_item" | "unknown_coupon" | "coupon_not_applicable"; | ||
| + | ||
| +export class CartError extends Error { | ||
| + /** Set for "invalid_item". */ | ||
| + readonly index?: number; | ||
| + /** Set for "unknown_coupon" and "coupon_not_applicable". */ | ||
| + readonly code?: string; | ||
| + constructor( | ||
| + public readonly kind: CartErrorKind, | ||
| + detail: { index?: number; code?: string } = {}, | ||
| + ) { | ||
| + super(kind); | ||
| + Object.assign(this, detail); | ||
| + } | ||
| +} | ||
| + | ||
| +/** Throws CartError. */ | ||
| +export function cartTotal( | ||
| + items: CartItem[], | ||
| + couponCode: string | undefined, | ||
| + coupons: Record<string, Coupon>, | ||
| +): CartTotal { | ||
| + // TODO | ||
| +} | ||
Addedevals/tasks/cart-total/signature.tsus+22−0
| @@ -0,0 +1,22 @@ | ||
| +import { type Option, type Result } from "@tsus/runtime"; | ||
| + | ||
| +export type CartItem = { sku: string; unitPriceCents: number; quantity: number }; | ||
| + | ||
| +export type Coupon = | ||
| + | { kind: "percent"; percent: number } | ||
| + | { kind: "fixed"; amountCents: number; minSubtotalCents: Option<number> }; | ||
| + | ||
| +export type CartTotal = { subtotalCents: number; discountCents: number; totalCents: number }; | ||
| + | ||
| +export type CartError = | ||
| + | { kind: "invalid_item"; index: number } | ||
| + | { kind: "unknown_coupon"; code: string } | ||
| + | { kind: "coupon_not_applicable"; code: string }; | ||
| + | ||
| +export const cartTotal = ( | ||
| + items: CartItem[], | ||
| + couponCode: Option<string>, | ||
| + coupons: Record<string, Coupon>, | ||
| +): Result<CartTotal, CartError> => { | ||
| + // TODO | ||
| +}; | ||
Addedevals/tasks/cart-total/task.json+1−0
| @@ -0,0 +1 @@ | ||
| +{ "title": "Cart total with coupons", "tags": ["validation", "falsy", "absence", "errors"], "pilot": true } | ||
Addedevals/tasks/cart-total/test.mjs+100−0
| @@ -0,0 +1,100 @@ | ||
| +import { test } from "node:test"; | ||
| +import assert from "node:assert/strict"; | ||
| +import { h, load } from "../../lib/harness.mjs"; | ||
| + | ||
| +const { cartTotal } = await load(); | ||
| + | ||
| +// Build a coupon table in the solution's language (minSubtotalCents is optional). | ||
| +const fixed = (amountCents, minSubtotalCents) => ({ kind: "fixed", amountCents, minSubtotalCents: h.maybe(minSubtotalCents) }); | ||
| +const percent = (p) => ({ kind: "percent", percent: p }); | ||
| +const coupons = () => ({ | ||
| + save10: percent(10), | ||
| + free: percent(100), | ||
| + zero: percent(0), | ||
| + five: fixed(500), | ||
| + big: fixed(1000, 5000), | ||
| + minzero: fixed(200, 0), | ||
| +}); | ||
| +const total = (items, code, table = coupons()) => h.res(() => cartTotal(items, h.maybe(code), table)); | ||
| +const ok = (subtotalCents, discountCents) => ({ ok: true, value: { subtotalCents, discountCents, totalCents: subtotalCents - discountCents } }); | ||
| +const errOf = (r, keys) => { | ||
| + if (r.ok !== false || !r.error) return r; | ||
| + const out = { kind: r.error.kind }; | ||
| + for (const k of keys) out[k] = r.error[k]; | ||
| + return out; | ||
| +}; | ||
| +const item = (unitPriceCents, quantity, sku = "x") => ({ sku, unitPriceCents, quantity }); | ||
| + | ||
| +test("no coupon", () => { | ||
| + assert.deepEqual(total([item(250, 2), item(100, 3)], undefined), ok(800, 0)); | ||
| +}); | ||
| + | ||
| +test("percent coupon rounds down", () => { | ||
| + assert.deepEqual(total([item(1999, 1)], "save10"), ok(1999, 199)); | ||
| + assert.deepEqual(total([item(5, 1)], "save10"), ok(5, 0)); | ||
| + assert.deepEqual(total([item(1234, 2)], "free"), ok(2468, 2468)); | ||
| +}); | ||
| + | ||
| +test("a 0 percent coupon is a valid coupon", () => { | ||
| + assert.deepEqual(total([item(1000, 1)], "zero"), ok(1000, 0)); | ||
| +}); | ||
| + | ||
| +test("fixed coupon is capped at the subtotal", () => { | ||
| + assert.deepEqual(total([item(300, 1)], "five"), ok(300, 300)); | ||
| + assert.deepEqual(total([item(300, 3)], "five"), ok(900, 500)); | ||
| +}); | ||
| + | ||
| +test("fixed coupon minimum subtotal", () => { | ||
| + assert.deepEqual(errOf(total([item(4999, 1)], "big"), ["code"]), { kind: "coupon_not_applicable", code: "big" }); | ||
| + assert.deepEqual(total([item(5000, 1)], "big"), ok(5000, 1000)); | ||
| + assert.deepEqual(total([item(0, 1)], "minzero"), ok(0, 0)); | ||
| + assert.deepEqual(errOf(total([], "big"), ["code"]), { kind: "coupon_not_applicable", code: "big" }); | ||
| +}); | ||
| + | ||
| +test("code lookup trims and ignores case; errors echo the code as passed", () => { | ||
| + assert.deepEqual(total([item(1000, 1)], " SAVE10 "), ok(1000, 100)); | ||
| + assert.deepEqual(total([item(1000, 1)], "Five"), ok(1000, 500)); | ||
| + assert.deepEqual(errOf(total([item(10, 1)], " Big "), ["code"]), { kind: "coupon_not_applicable", code: " Big " }); | ||
| + assert.deepEqual(errOf(total([item(10, 1)], " Nope"), ["code"]), { kind: "unknown_coupon", code: " Nope" }); | ||
| +}); | ||
| + | ||
| +test("empty or blank code means no coupon", () => { | ||
| + assert.deepEqual(total([item(1000, 1)], ""), ok(1000, 0)); | ||
| + assert.deepEqual(total([item(1000, 1)], " "), ok(1000, 0)); | ||
| + assert.deepEqual(total([item(1000, 1)], "\t"), ok(1000, 0)); | ||
| +}); | ||
| + | ||
| +test("only codes defined in the table are valid", () => { | ||
| + for (const code of ["toString", "__proto__", "constructor", "hasOwnProperty", "valueOf"]) { | ||
| + assert.deepEqual(errOf(total([item(1000, 1)], code), ["code"]), { kind: "unknown_coupon", code }, code); | ||
| + } | ||
| +}); | ||
| + | ||
| +test("invalid items report the first bad index", () => { | ||
| + const bad = [-1, 1.5, "2", NaN, null, undefined, Infinity]; | ||
| + for (const q of bad) { | ||
| + assert.deepEqual(errOf(total([item(100, 1), item(100, q)], undefined), ["index"]), { kind: "invalid_item", index: 1 }, String(q)); | ||
| + assert.deepEqual(errOf(total([item(q, 1), item(100, 1)], undefined), ["index"]), { kind: "invalid_item", index: 0 }, String(q)); | ||
| + } | ||
| + assert.deepEqual(errOf(total([item(1, 1), item(-5, 1), item(1, -1)], undefined), ["index"]), { kind: "invalid_item", index: 1 }); | ||
| +}); | ||
| + | ||
| +test("zero prices and quantities are valid", () => { | ||
| + assert.deepEqual(total([item(0, 5), item(250, 0)], undefined), ok(0, 0)); | ||
| + assert.deepEqual(total([item(0, 5), item(250, 0)], "save10"), ok(0, 0)); | ||
| + assert.deepEqual(total([], undefined), ok(0, 0)); | ||
| + assert.deepEqual(total([], "five"), ok(0, 0)); | ||
| +}); | ||
| + | ||
| +test("items are validated before the coupon", () => { | ||
| + assert.deepEqual(errOf(total([item(100, -1)], "nope"), ["index"]), { kind: "invalid_item", index: 0 }); | ||
| +}); | ||
| + | ||
| +test("inputs are not modified", () => { | ||
| + const items = [item(250, 2, "a"), item(100, 1, "b")]; | ||
| + const table = coupons(); | ||
| + const before = JSON.stringify([items, Object.keys(table)]); | ||
| + total(items, " SAVE10 ", table); | ||
| + total(items, "five", table); | ||
| + assert.equal(JSON.stringify([items, Object.keys(table)]), before); | ||
| +}); | ||
Addedevals/tasks/csv-rows/mutant.ts+52−0
| @@ -0,0 +1,52 @@ | ||
| +// Bug: reports the row's position among non-blank records instead of its physical line (ignores skipped blank lines and newlines inside quotes). | ||
| +export type CsvErrorKind = "missing_header" | "missing_column" | "column_count" | "unterminated_quote"; | ||
| + | ||
| +export class CsvError extends Error { | ||
| + readonly column?: string; | ||
| + readonly line?: number; | ||
| + constructor( | ||
| + public readonly kind: CsvErrorKind, | ||
| + detail: { column?: string; line?: number } = {}, | ||
| + ) { | ||
| + super(kind); | ||
| + Object.assign(this, detail); | ||
| + } | ||
| +} | ||
| + | ||
| +export function parseCsv(text: string, requiredColumns: string[]): Record<string, string>[] { | ||
| + const rows: string[][] = []; | ||
| + let field = ""; | ||
| + let row: string[] = []; | ||
| + let inQuotes = false; | ||
| + let i = 0; | ||
| + while (i < text.length) { | ||
| + const c = text[i]; | ||
| + if (inQuotes) { | ||
| + if (c === '"' && text[i + 1] === '"') { field += '"'; i += 2; continue; } | ||
| + if (c === '"') { inQuotes = false; i++; continue; } | ||
| + field += c; i++; continue; | ||
| + } | ||
| + if (c === '"' && field === "") { inQuotes = true; i++; continue; } | ||
| + if (c === ",") { row.push(field); field = ""; i++; continue; } | ||
| + if (c === "\r" && text[i + 1] === "\n") { i++; continue; } | ||
| + if (c === "\n") { | ||
| + row.push(field); | ||
| + if (!(row.length === 1 && row[0] === "")) rows.push(row); | ||
| + row = []; field = ""; i++; continue; | ||
| + } | ||
| + field += c; i++; | ||
| + } | ||
| + if (inQuotes) throw new CsvError("unterminated_quote", { line: rows.length + 1 }); | ||
| + if (field !== "" || row.length > 0) { row.push(field); rows.push(row); } | ||
| + if (rows.length === 0) throw new CsvError("missing_header"); | ||
| + const header = rows[0]; | ||
| + for (const column of requiredColumns) { | ||
| + if (!header.includes(column)) throw new CsvError("missing_column", { column }); | ||
| + } | ||
| + return rows.slice(1).map((r, k) => { | ||
| + if (r.length !== header.length) throw new CsvError("column_count", { line: k + 2 }); | ||
| + const rec: Record<string, string> = {}; | ||
| + header.forEach((name, j) => (rec[name] = r[j])); | ||
| + return rec; | ||
| + }); | ||
| +} | ||
Addedevals/tasks/csv-rows/prompt.md+16−0
| @@ -0,0 +1,16 @@ | ||
| +Write `parseCsv(text, requiredColumns)`, which parses CSV text into an array of records keyed by the header row. | ||
| + | ||
| +Format: | ||
| + | ||
| +- Fields are separated by commas. Line breaks are `\n` or `\r\n`. | ||
| +- A field that starts with a double quote is quoted: it runs to the next unpaired `"`, may contain commas and line breaks (kept verbatim), and `""` inside it stands for one `"`. A quote character inside an unquoted field is an ordinary character. | ||
| +- The first row is the header. Every later row becomes one record whose keys are the header fields and whose values are that row's fields, as strings. | ||
| +- A completely empty line (nothing between two line breaks, or between a line break and the end of the text) is skipped and is not a row. A trailing line break does not create a row. | ||
| + | ||
| +Line numbers are 1-based physical lines of the text. Every line counts, including skipped blank lines and lines inside quoted fields. A row's line is the line on which it starts. | ||
| + | ||
| +Errors, reported in this order: | ||
| + | ||
| +- `{ kind: "missing_header" }` when the text has no rows at all (it is empty or contains only blank lines). | ||
| +- `{ kind: "missing_column", column }` when a name in `requiredColumns` is not a header field; `column` is the first such name in `requiredColumns` order. | ||
| +- Then the first bad row, in order of appearance: `{ kind: "column_count", line }` when a row has a different number of fields than the header, or `{ kind: "unterminated_quote", line }` when a quoted field is never closed. | ||
Addedevals/tasks/csv-rows/reference.ts+99−0
| @@ -0,0 +1,99 @@ | ||
| +export type CsvErrorKind = "missing_header" | "missing_column" | "column_count" | "unterminated_quote"; | ||
| + | ||
| +export class CsvError extends Error { | ||
| + readonly column?: string; | ||
| + readonly line?: number; | ||
| + constructor( | ||
| + public readonly kind: CsvErrorKind, | ||
| + detail: { column?: string; line?: number } = {}, | ||
| + ) { | ||
| + super(kind); | ||
| + Object.assign(this, detail); | ||
| + } | ||
| +} | ||
| + | ||
| +type Row = { line: number; fields: string[] } | { line: number; unterminated: true }; | ||
| + | ||
| +const readRows = (text: string): Row[] => { | ||
| + const rows: Row[] = []; | ||
| + const n = text.length; | ||
| + let i = 0; | ||
| + let line = 1; | ||
| + const atBreak = () => text[i] === "\n" || text.startsWith("\r\n", i); | ||
| + const skipBreak = () => { | ||
| + i += text[i] === "\n" ? 1 : 2; | ||
| + line += 1; | ||
| + }; | ||
| + while (i < n) { | ||
| + if (atBreak()) { | ||
| + skipBreak(); | ||
| + continue; | ||
| + } | ||
| + const startLine = line; | ||
| + const fields: string[] = []; | ||
| + for (;;) { | ||
| + let value = ""; | ||
| + if (text[i] === '"') { | ||
| + i += 1; | ||
| + let closed = false; | ||
| + while (i < n) { | ||
| + const c = text[i]; | ||
| + if (c === '"') { | ||
| + if (text[i + 1] === '"') { | ||
| + value += '"'; | ||
| + i += 2; | ||
| + } else { | ||
| + i += 1; | ||
| + closed = true; | ||
| + break; | ||
| + } | ||
| + } else { | ||
| + if (c === "\n") line += 1; | ||
| + value += c; | ||
| + i += 1; | ||
| + } | ||
| + } | ||
| + if (!closed) { | ||
| + rows.push({ line: startLine, unterminated: true }); | ||
| + return rows; | ||
| + } | ||
| + } | ||
| + while (i < n && text[i] !== "," && !atBreak()) { | ||
| + value += text[i]; | ||
| + i += 1; | ||
| + } | ||
| + fields.push(value); | ||
| + if (i < n && text[i] === ",") { | ||
| + i += 1; | ||
| + continue; | ||
| + } | ||
| + if (i < n) skipBreak(); | ||
| + break; | ||
| + } | ||
| + rows.push({ line: startLine, fields }); | ||
| + } | ||
| + return rows; | ||
| +}; | ||
| + | ||
| +/** Throws CsvError. */ | ||
| +export function parseCsv(text: string, requiredColumns: string[]): Record<string, string>[] { | ||
| + const rows = readRows(text); | ||
| + if (rows.length === 0) throw new CsvError("missing_header"); | ||
| + const head = rows[0]; | ||
| + if (!("fields" in head)) throw new CsvError("unterminated_quote", { line: head.line }); | ||
| + const header = head.fields; | ||
| + for (const column of requiredColumns) { | ||
| + if (!header.includes(column)) throw new CsvError("missing_column", { column }); | ||
| + } | ||
| + const out: Record<string, string>[] = []; | ||
| + for (const row of rows.slice(1)) { | ||
| + if (!("fields" in row)) throw new CsvError("unterminated_quote", { line: row.line }); | ||
| + if (row.fields.length !== header.length) throw new CsvError("column_count", { line: row.line }); | ||
| + const record: Record<string, string> = {}; | ||
| + header.forEach((name, k) => { | ||
| + record[name] = row.fields[k]; | ||
| + }); | ||
| + out.push(record); | ||
| + } | ||
| + return out; | ||
| +} | ||
Addedevals/tasks/csv-rows/reference.tsus+108−0
| @@ -0,0 +1,108 @@ | ||
| +import { Ok, Err, type Result } from "@tsus/runtime"; | ||
| + | ||
| +export type CsvError = | ||
| + | { kind: "missing_header" } | ||
| + | { kind: "missing_column"; column: string } | ||
| + | { kind: "column_count"; line: number } | ||
| + | { kind: "unterminated_quote"; line: number }; | ||
| + | ||
| +type Row = { line: number; fields: string[]; unterminated: boolean }; | ||
| + | ||
| +const isBreak = (text: string, i: number): boolean => text.charAt(i) === "\n" || text.startsWith("\r\n", i); | ||
| +const breakLen = (text: string, i: number): number => (text.charAt(i) === "\n" ? 1 : 2); | ||
| + | ||
| +const readRows = (text: string): Row[] => { | ||
| + const rows: Row[] = []; | ||
| + const n = text.length; | ||
| + let i = 0; | ||
| + let line = 1; | ||
| + while (i < n) { | ||
| + if (isBreak(text, i)) { | ||
| + i += breakLen(text, i); | ||
| + line += 1; | ||
| + continue; | ||
| + } | ||
| + const startLine = line; | ||
| + const fields: string[] = []; | ||
| + let rowDone = false; | ||
| + while (!rowDone) { | ||
| + let value = ""; | ||
| + if (text.charAt(i) === '"') { | ||
| + i += 1; | ||
| + let closed = false; | ||
| + while (i < n && !closed) { | ||
| + const c = text.charAt(i); | ||
| + if (c === '"' && text.charAt(i + 1) === '"') { | ||
| + value += '"'; | ||
| + i += 2; | ||
| + } else if (c === '"') { | ||
| + i += 1; | ||
| + closed = true; | ||
| + } else { | ||
| + line += c === "\n" ? 1 : 0; | ||
| + value += c; | ||
| + i += 1; | ||
| + } | ||
| + } | ||
| + if (!closed) { | ||
| + rows.push({ line: startLine, fields: [], unterminated: true }); | ||
| + return rows; | ||
| + } | ||
| + } | ||
| + while (i < n && text.charAt(i) !== "," && !isBreak(text, i)) { | ||
| + value += text.charAt(i); | ||
| + i += 1; | ||
| + } | ||
| + fields.push(value); | ||
| + if (i < n && text.charAt(i) === ",") { | ||
| + i += 1; | ||
| + } else { | ||
| + if (i < n) { | ||
| + i += breakLen(text, i); | ||
| + line += 1; | ||
| + } | ||
| + rowDone = true; | ||
| + } | ||
| + } | ||
| + rows.push({ line: startLine, fields, unterminated: false }); | ||
| + } | ||
| + return rows; | ||
| +}; | ||
| + | ||
| +const toRecord = (header: string[], fields: string[]): Record<string, string> => { | ||
| + const record: Record<string, string> = {}; | ||
| + header.forEach((name, k) => { | ||
| + record[name] = fields[k]; | ||
| + }); | ||
| + return record; | ||
| +}; | ||
| + | ||
| +const buildRecords = (header: string[], rows: Row[]): Result<Record<string, string>[], CsvError> => { | ||
| + const out: Record<string, string>[] = []; | ||
| + for (const row of rows) { | ||
| + if (row.unterminated) { | ||
| + return Err({ kind: "unterminated_quote", line: row.line }); | ||
| + } | ||
| + if (row.fields.length !== header.length) { | ||
| + return Err({ kind: "column_count", line: row.line }); | ||
| + } | ||
| + out.push(toRecord(header, row.fields)); | ||
| + } | ||
| + return Ok(out); | ||
| +}; | ||
| + | ||
| +export const parseCsv = (text: string, requiredColumns: string[]): Result<Record<string, string>[], CsvError> => { | ||
| + const rows = readRows(text); | ||
| + if (rows.length === 0) { | ||
| + return Err({ kind: "missing_header" }); | ||
| + } | ||
| + const head = rows[0]; | ||
| + if (head.unterminated) { | ||
| + return Err({ kind: "unterminated_quote", line: head.line }); | ||
| + } | ||
| + const missing = requiredColumns.filter((c) => !head.fields.includes(c)); | ||
| + if (missing.length > 0) { | ||
| + return Err({ kind: "missing_column", column: missing[0] }); | ||
| + } | ||
| + return buildRecords(head.fields, rows.slice(1)); | ||
| +}; | ||
Addedevals/tasks/csv-rows/signature.ts+20−0
| @@ -0,0 +1,20 @@ | ||
| +export type CsvErrorKind = "missing_header" | "missing_column" | "column_count" | "unterminated_quote"; | ||
| + | ||
| +export class CsvError extends Error { | ||
| + /** Set for "missing_column". */ | ||
| + readonly column?: string; | ||
| + /** Set for "column_count" and "unterminated_quote": 1-based line where the row starts. */ | ||
| + readonly line?: number; | ||
| + constructor( | ||
| + public readonly kind: CsvErrorKind, | ||
| + detail: { column?: string; line?: number } = {}, | ||
| + ) { | ||
| + super(kind); | ||
| + Object.assign(this, detail); | ||
| + } | ||
| +} | ||
| + | ||
| +/** Throws CsvError. */ | ||
| +export function parseCsv(text: string, requiredColumns: string[]): Record<string, string>[] { | ||
| + // TODO | ||
| +} | ||
Addedevals/tasks/csv-rows/signature.tsus+11−0
| @@ -0,0 +1,11 @@ | ||
| +import { type Result } from "@tsus/runtime"; | ||
| + | ||
| +export type CsvError = | ||
| + | { kind: "missing_header" } | ||
| + | { kind: "missing_column"; column: string } | ||
| + | { kind: "column_count"; line: number } | ||
| + | { kind: "unterminated_quote"; line: number }; | ||
| + | ||
| +export const parseCsv = (text: string, requiredColumns: string[]): Result<Record<string, string>[], CsvError> => { | ||
| + // TODO | ||
| +}; | ||
Addedevals/tasks/csv-rows/task.json+1−0
| @@ -0,0 +1 @@ | ||
| +{ "title": "Parse CSV rows with quoting and line numbers", "tags": ["parsing", "errors", "boundary"], "pilot": false } | ||
Addedevals/tasks/csv-rows/test.mjs+88−0
| @@ -0,0 +1,88 @@ | ||
| +import { test } from "node:test"; | ||
| +import assert from "node:assert/strict"; | ||
| +import { h, load } from "../../lib/harness.mjs"; | ||
| + | ||
| +const { parseCsv } = await load(); | ||
| +const parse = (text, required = []) => h.res(() => parseCsv(text, required)); | ||
| +const ok = (value) => ({ ok: true, value }); | ||
| +const errOf = (r, keys = []) => { | ||
| + if (r.ok !== false || !r.error) return r; | ||
| + const out = { kind: r.error.kind }; | ||
| + for (const k of keys) out[k] = r.error[k]; | ||
| + return out; | ||
| +}; | ||
| + | ||
| +test("simple rows", () => { | ||
| + assert.deepEqual( | ||
| + parse("name,age\nann,30\nbob,41", ["name"]), | ||
| + ok([{ name: "ann", age: "30" }, { name: "bob", age: "41" }]), | ||
| + ); | ||
| +}); | ||
| + | ||
| +test("trailing line break and CRLF", () => { | ||
| + assert.deepEqual(parse("a,b\n1,2\n"), ok([{ a: "1", b: "2" }])); | ||
| + assert.deepEqual(parse("a,b\r\n1,2\r\n3,4\r\n"), ok([{ a: "1", b: "2" }, { a: "3", b: "4" }])); | ||
| +}); | ||
| + | ||
| +test("quoted fields with commas, escaped quotes and line breaks", () => { | ||
| + assert.deepEqual( | ||
| + parse('id,note\n1,"hello, world"\n2,"say ""hi"""\n3,"two\nlines"\n'), | ||
| + ok([ | ||
| + { id: "1", note: "hello, world" }, | ||
| + { id: "2", note: 'say "hi"' }, | ||
| + { id: "3", note: "two\nlines" }, | ||
| + ]), | ||
| + ); | ||
| + assert.deepEqual(parse('a\r\n"x\r\ny"\r\n'), ok([{ a: "x\r\ny" }])); | ||
| +}); | ||
| + | ||
| +test("quoted header fields", () => { | ||
| + assert.deepEqual(parse('"first name","a,b"\nann,x\n', ["first name", "a,b"]), ok([{ "first name": "ann", "a,b": "x" }])); | ||
| +}); | ||
| + | ||
| +test("blank lines are skipped", () => { | ||
| + assert.deepEqual(parse("a,b\n\n1,2\n\n\n3,4\n\n"), ok([{ a: "1", b: "2" }, { a: "3", b: "4" }])); | ||
| + assert.deepEqual(parse("\n\na\n1\n"), ok([{ a: "1" }])); | ||
| +}); | ||
| + | ||
| +test("empty fields are kept as empty strings", () => { | ||
| + assert.deepEqual(parse("a,b,c\n,,\n1,,3\n"), ok([{ a: "", b: "", c: "" }, { a: "1", b: "", c: "3" }])); | ||
| + assert.deepEqual(parse('a\n""\n'), ok([{ a: "" }])); | ||
| + assert.deepEqual(parse("a,b\n0,\n"), ok([{ a: "0", b: "" }])); | ||
| +}); | ||
| + | ||
| +test("header only gives no records", () => { | ||
| + assert.deepEqual(parse("a,b"), ok([])); | ||
| + assert.deepEqual(parse("a,b\n\n"), ok([])); | ||
| +}); | ||
| + | ||
| +test("missing header", () => { | ||
| + assert.deepEqual(errOf(parse("")), { kind: "missing_header" }); | ||
| + assert.deepEqual(errOf(parse("\n\n")), { kind: "missing_header" }); | ||
| + assert.deepEqual(errOf(parse("\r\n", ["a"])), { kind: "missing_header" }); | ||
| +}); | ||
| + | ||
| +test("missing required column, first in requiredColumns order", () => { | ||
| + assert.deepEqual(errOf(parse("id,name\n1,x\n", ["id", "email", "phone"]), ["column"]), { kind: "missing_column", column: "email" }); | ||
| + assert.deepEqual(errOf(parse("id,name\n1,x\n", ["Name"]), ["column"]), { kind: "missing_column", column: "Name" }); | ||
| +}); | ||
| + | ||
| +test("column count error uses the physical line", () => { | ||
| + assert.deepEqual(errOf(parse("a,b\n1,2\n1,2,3\n"), ["line"]), { kind: "column_count", line: 3 }); | ||
| + assert.deepEqual(errOf(parse("a,b\n\n\n1\n"), ["line"]), { kind: "column_count", line: 4 }); | ||
| + assert.deepEqual(errOf(parse('a,b\n1,"x\ny\nz"\n5\n'), ["line"]), { kind: "column_count", line: 5 }); | ||
| + assert.deepEqual(errOf(parse("a,b\r\n1,2\r\n\r\n9\r\n"), ["line"]), { kind: "column_count", line: 4 }); | ||
| +}); | ||
| + | ||
| +test("unterminated quote reports the line where its row starts", () => { | ||
| + assert.deepEqual(errOf(parse('a,b\n1,"oops\n2,3\n'), ["line"]), { kind: "unterminated_quote", line: 2 }); | ||
| + assert.deepEqual(errOf(parse('a,b\n\n1,2\n\n3,"x'), ["line"]), { kind: "unterminated_quote", line: 5 }); | ||
| +}); | ||
| + | ||
| +test("missing column is reported before row errors", () => { | ||
| + assert.deepEqual(errOf(parse("a,b\n1,2,3\n", ["c"]), ["column"]), { kind: "missing_column", column: "c" }); | ||
| +}); | ||
| + | ||
| +test("the first bad row wins", () => { | ||
| + assert.deepEqual(errOf(parse("a,b\n1\n1,2,3\n"), ["line"]), { kind: "column_count", line: 2 }); | ||
| +}); | ||
Addedevals/tasks/date-range-merge/mutant.ts+37−0
| @@ -0,0 +1,37 @@ | ||
| +// Plausible bug: validates with `new Date(s)` (which silently rolls 2023-02-29 over and accepts loose formats) and sorts the input in place. | ||
| +export type DateRange = { start: string; end: string }; | ||
| + | ||
| +export type RangeErrorDetails = | ||
| + | { kind: "invalid_date"; index: number; value: string } | ||
| + | { kind: "inverted_range"; index: number }; | ||
| + | ||
| +export class DateRangeError extends Error { | ||
| + public readonly kind: RangeErrorDetails["kind"]; | ||
| + constructor(details: RangeErrorDetails) { | ||
| + super(details.kind); | ||
| + this.kind = details.kind; | ||
| + Object.assign(this, details); | ||
| + } | ||
| +} | ||
| + | ||
| +const DAY = 86_400_000; | ||
| +const ms = (s: string): number => new Date(s).getTime(); | ||
| + | ||
| +export function mergeRanges(ranges: DateRange[]): DateRange[] { | ||
| + ranges.forEach((r, index) => { | ||
| + if (Number.isNaN(ms(r.start))) throw new DateRangeError({ kind: "invalid_date", index, value: r.start }); | ||
| + if (Number.isNaN(ms(r.end))) throw new DateRangeError({ kind: "invalid_date", index, value: r.end }); | ||
| + if (ms(r.start) > ms(r.end)) throw new DateRangeError({ kind: "inverted_range", index }); | ||
| + }); | ||
| + ranges.sort((a, b) => ms(a.start) - ms(b.start)); | ||
| + const out: DateRange[] = []; | ||
| + for (const r of ranges) { | ||
| + const last = out[out.length - 1]; | ||
| + if (last && ms(r.start) <= ms(last.end) + DAY) { | ||
| + if (ms(r.end) > ms(last.end)) last.end = r.end; | ||
| + } else { | ||
| + out.push({ ...r }); | ||
| + } | ||
| + } | ||
| + return out; | ||
| +} | ||
Addedevals/tasks/date-range-merge/prompt.md+12−0
| @@ -0,0 +1,12 @@ | ||
| +Write `mergeRanges(ranges)`, which merges a list of inclusive calendar-date ranges. | ||
| + | ||
| +Each range is `{ start, end }` where both are dates written `YYYY-MM-DD`: exactly a four-digit year, a two-digit month, and a two-digit day separated by `-`, with nothing else in the string. Dates are plain Gregorian calendar days (no time zones): the month must be 01 to 12 and the day must exist in that month, using the Gregorian leap-year rule (divisible by 4, except centuries not divisible by 400). Both ends are inclusive, so `{ start: "2024-01-01", end: "2024-01-01" }` is a one-day range. | ||
| + | ||
| +Validation runs over the input in order, and the first problem found is returned. For each range at position `index`, check `start`, then `end`, then their order: | ||
| + | ||
| +- A date that is not a valid date in this format fails with `{ kind: "invalid_date", index, value }` where `value` is the offending string. | ||
| +- A range whose `start` is after its `end` fails with `{ kind: "inverted_range", index }`. | ||
| + | ||
| +If every range is valid, return the merged ranges sorted by start. Two ranges merge when they overlap or when they are adjacent, meaning the day after one range's end is the other range's start. Merging is transitive: keep merging until no two output ranges overlap or touch. Output dates use the same `YYYY-MM-DD` format. | ||
| + | ||
| +An empty input returns an empty list. The input array and its range objects must not be modified; return new objects. | ||
Addedevals/tasks/date-range-merge/reference.ts+59−0
| @@ -0,0 +1,59 @@ | ||
| +export type DateRange = { start: string; end: string }; | ||
| + | ||
| +export type RangeErrorDetails = | ||
| + | { kind: "invalid_date"; index: number; value: string } | ||
| + | { kind: "inverted_range"; index: number }; | ||
| + | ||
| +export class DateRangeError extends Error { | ||
| + public readonly kind: RangeErrorDetails["kind"]; | ||
| + constructor(details: RangeErrorDetails) { | ||
| + super(details.kind); | ||
| + this.kind = details.kind; | ||
| + Object.assign(this, details); | ||
| + } | ||
| +} | ||
| + | ||
| +const isLeap = (y: number): boolean => (y % 4 === 0 && y % 100 !== 0) || y % 400 === 0; | ||
| +const daysIn = (y: number, m: number): number => (m === 2 ? (isLeap(y) ? 29 : 28) : [4, 6, 9, 11].includes(m) ? 30 : 31); | ||
| + | ||
| +type Ymd = { y: number; m: number; d: number }; | ||
| + | ||
| +const parse = (s: unknown): Ymd | undefined => { | ||
| + if (typeof s !== "string") return undefined; | ||
| + const match = /^(\d{4})-(\d{2})-(\d{2})$/.exec(s); | ||
| + if (!match) return undefined; | ||
| + const y = Number(match[1]); | ||
| + const m = Number(match[2]); | ||
| + const d = Number(match[3]); | ||
| + if (m < 1 || m > 12 || d < 1 || d > daysIn(y, m)) return undefined; | ||
| + return { y, m, d }; | ||
| +}; | ||
| + | ||
| +const pad = (n: number, w: number): string => String(n).padStart(w, "0"); | ||
| + | ||
| +const nextDay = ({ y, m, d }: Ymd): string => { | ||
| + if (d < daysIn(y, m)) return `${pad(y, 4)}-${pad(m, 2)}-${pad(d + 1, 2)}`; | ||
| + if (m < 12) return `${pad(y, 4)}-${pad(m + 1, 2)}-01`; | ||
| + return `${pad(y + 1, 4)}-01-01`; | ||
| +}; | ||
| + | ||
| +/** Throws DateRangeError (constructed with the matching RangeErrorDetails). */ | ||
| +export function mergeRanges(ranges: readonly DateRange[]): DateRange[] { | ||
| + ranges.forEach((r, index) => { | ||
| + if (!parse(r.start)) throw new DateRangeError({ kind: "invalid_date", index, value: r.start }); | ||
| + if (!parse(r.end)) throw new DateRangeError({ kind: "invalid_date", index, value: r.end }); | ||
| + if (r.start > r.end) throw new DateRangeError({ kind: "inverted_range", index }); | ||
| + }); | ||
| + // YYYY-MM-DD strings sort lexicographically in date order. | ||
| + const sorted = ranges.map((r) => ({ start: r.start, end: r.end })).sort((a, b) => (a.start < b.start ? -1 : a.start > b.start ? 1 : 0)); | ||
| + const out: DateRange[] = []; | ||
| + for (const r of sorted) { | ||
| + const last = out[out.length - 1]; | ||
| + if (last && r.start <= nextDay(parse(last.end) as Ymd)) { | ||
| + if (r.end > last.end) last.end = r.end; | ||
| + } else { | ||
| + out.push(r); | ||
| + } | ||
| + } | ||
| + return out; | ||
| +} | ||
Addedevals/tasks/date-range-merge/reference.tsus+75−0
| @@ -0,0 +1,75 @@ | ||
| +import { Some, None, Ok, Err, type Option, type Result } from "@tsus/runtime"; | ||
| +import { fromNullable } from "@tsus/runtime/boundary"; | ||
| + | ||
| +export type DateRange = { start: string; end: string }; | ||
| + | ||
| +export type DateRangeError = | ||
| + | { kind: "invalid_date"; index: number; value: string } | ||
| + | { kind: "inverted_range"; index: number }; | ||
| + | ||
| +type Ymd = { y: number; m: number; d: number }; | ||
| + | ||
| +const isLeap = (y: number): boolean => (y % 4 === 0 && y % 100 !== 0) || y % 400 === 0; | ||
| +const daysIn = (y: number, m: number): number => (m === 2 ? (isLeap(y) ? 29 : 28) : [4, 6, 9, 11].includes(m) ? 30 : 31); | ||
| + | ||
| +const toYmd = (parts: RegExpExecArray): Option<Ymd> => { | ||
| + const y = Number(parts[1]); | ||
| + const m = Number(parts[2]); | ||
| + const d = Number(parts[3]); | ||
| + return m >= 1 && m <= 12 && d >= 1 && d <= daysIn(y, m) ? Some({ y, m, d }) : None; | ||
| +}; | ||
| + | ||
| +const parse = (s: unknown): Option<Ymd> => | ||
| + typeof s === "string" ? fromNullable(/^(\d{4})-(\d{2})-(\d{2})$/.exec(s)).andThen(toYmd) : None; | ||
| + | ||
| +const pad = (n: number, w: number): string => String(n).padStart(w, "0"); | ||
| + | ||
| +const nextDay = (p: Ymd): string => | ||
| + p.d < daysIn(p.y, p.m) | ||
| + ? `${pad(p.y, 4)}-${pad(p.m, 2)}-${pad(p.d + 1, 2)}` | ||
| + : p.m < 12 | ||
| + ? `${pad(p.y, 4)}-${pad(p.m + 1, 2)}-01` | ||
| + : `${pad(p.y + 1, 4)}-01-01`; | ||
| + | ||
| +const checkOne = (r: DateRange, index: number): Option<DateRangeError> => | ||
| + parse(r.start).isNone() | ||
| + ? Some({ kind: "invalid_date", index, value: r.start }) | ||
| + : parse(r.end).isNone() | ||
| + ? Some({ kind: "invalid_date", index, value: r.end }) | ||
| + : r.start > r.end | ||
| + ? Some({ kind: "inverted_range", index }) | ||
| + : None; | ||
| + | ||
| +const firstProblem = (ranges: readonly DateRange[]): Option<DateRangeError> => { | ||
| + for (let i = 0; i < ranges.length; i += 1) { | ||
| + const problem = checkOne(ranges[i], i); | ||
| + if (problem.isSome()) { | ||
| + return problem; | ||
| + } | ||
| + } | ||
| + return None; | ||
| +}; | ||
| + | ||
| +// Is `next` (sorted after `last`) overlapping or adjacent to `last`? | ||
| +const touches = (last: DateRange, next: DateRange): boolean => | ||
| + match (parse(last.end)) { | ||
| + Some(p) => next.start <= nextDay(p), | ||
| + None => false, | ||
| + }; | ||
| + | ||
| +const merge = (ranges: readonly DateRange[]): DateRange[] => { | ||
| + const sorted = ranges.map((r) => ({ start: r.start, end: r.end })).sort((a, b) => (a.start < b.start ? -1 : a.start > b.start ? 1 : 0)); | ||
| + return sorted.reduce((out: DateRange[], r) => { | ||
| + const last = out.length > 0 ? Some(out[out.length - 1]) : None; | ||
| + return match (last) { | ||
| + Some(l) => touches(l, r) ? [...out.slice(0, -1), { start: l.start, end: r.end > l.end ? r.end : l.end }] : [...out, r], | ||
| + None => [r], | ||
| + }; | ||
| + }, []); | ||
| +}; | ||
| + | ||
| +export const mergeRanges = (ranges: readonly DateRange[]): Result<DateRange[], DateRangeError> => | ||
| + match (firstProblem(ranges)) { | ||
| + Some(e) => Err(e), | ||
| + None => Ok(merge(ranges)), | ||
| + }; | ||
Addedevals/tasks/date-range-merge/signature.ts+19−0
| @@ -0,0 +1,19 @@ | ||
| +export type DateRange = { start: string; end: string }; | ||
| + | ||
| +export type RangeErrorDetails = | ||
| + | { kind: "invalid_date"; index: number; value: string } | ||
| + | { kind: "inverted_range"; index: number }; | ||
| + | ||
| +export class DateRangeError extends Error { | ||
| + public readonly kind: RangeErrorDetails["kind"]; | ||
| + constructor(details: RangeErrorDetails) { | ||
| + super(details.kind); | ||
| + this.kind = details.kind; | ||
| + Object.assign(this, details); | ||
| + } | ||
| +} | ||
| + | ||
| +/** Throws DateRangeError (constructed with the matching RangeErrorDetails). */ | ||
| +export function mergeRanges(ranges: readonly DateRange[]): DateRange[] { | ||
| + // TODO | ||
| +} | ||
Addedevals/tasks/date-range-merge/signature.tsus+11−0
| @@ -0,0 +1,11 @@ | ||
| +import { type Result } from "@tsus/runtime"; | ||
| + | ||
| +export type DateRange = { start: string; end: string }; | ||
| + | ||
| +export type DateRangeError = | ||
| + | { kind: "invalid_date"; index: number; value: string } | ||
| + | { kind: "inverted_range"; index: number }; | ||
| + | ||
| +export const mergeRanges = (ranges: readonly DateRange[]): Result<DateRange[], DateRangeError> => { | ||
| + // TODO | ||
| +}; | ||
Addedevals/tasks/date-range-merge/task.json+1−0
| @@ -0,0 +1 @@ | ||
| +{ "title": "Merge inclusive calendar date ranges", "tags": ["dates", "validation", "parsing", "data-structures"], "pilot": true } | ||
Addedevals/tasks/date-range-merge/test.mjs+104−0
| @@ -0,0 +1,104 @@ | ||
| +import { test } from "node:test"; | ||
| +import assert from "node:assert/strict"; | ||
| +import { h, load } from "../../lib/harness.mjs"; | ||
| + | ||
| +const { mergeRanges } = await load(); | ||
| +const merge = (ranges) => h.res(() => mergeRanges(ranges)); | ||
| +const ok = (value) => ({ ok: true, value }); | ||
| +const r = (start, end) => ({ start, end }); | ||
| +const invalidDate = (index, value) => ({ ok: false, error: { kind: "invalid_date", index, value } }); | ||
| +const inverted = (index) => ({ ok: false, error: { kind: "inverted_range", index } }); | ||
| + | ||
| +test("empty input", () => { | ||
| + assert.deepEqual(merge([]), ok([])); | ||
| +}); | ||
| + | ||
| +test("single range is returned as a new object", () => { | ||
| + const input = [r("2024-03-01", "2024-03-05")]; | ||
| + const out = merge(input); | ||
| + assert.deepEqual(out, ok([r("2024-03-01", "2024-03-05")])); | ||
| + assert.notEqual(out.value[0], input[0]); | ||
| +}); | ||
| + | ||
| +test("overlapping ranges merge and output is sorted", () => { | ||
| + assert.deepEqual( | ||
| + merge([r("2024-05-10", "2024-05-20"), r("2024-01-01", "2024-01-10"), r("2024-05-15", "2024-06-01"), r("2024-01-05", "2024-01-07")]), | ||
| + ok([r("2024-01-01", "2024-01-10"), r("2024-05-10", "2024-06-01")]), | ||
| + ); | ||
| +}); | ||
| + | ||
| +test("adjacent ranges merge, a one-day gap does not", () => { | ||
| + assert.deepEqual(merge([r("2024-01-01", "2024-01-05"), r("2024-01-06", "2024-01-09")]), ok([r("2024-01-01", "2024-01-09")])); | ||
| + assert.deepEqual( | ||
| + merge([r("2024-01-01", "2024-01-05"), r("2024-01-07", "2024-01-09")]), | ||
| + ok([r("2024-01-01", "2024-01-05"), r("2024-01-07", "2024-01-09")]), | ||
| + ); | ||
| +}); | ||
| + | ||
| +test("adjacency across month ends", () => { | ||
| + assert.deepEqual(merge([r("2024-01-20", "2024-01-31"), r("2024-02-01", "2024-02-03")]), ok([r("2024-01-20", "2024-02-03")])); | ||
| + assert.deepEqual(merge([r("2024-05-01", "2024-05-02"), r("2024-04-29", "2024-04-30")]), ok([r("2024-04-29", "2024-05-02")])); | ||
| + assert.deepEqual( | ||
| + merge([r("2024-04-29", "2024-04-30"), r("2024-05-02", "2024-05-03")]), | ||
| + ok([r("2024-04-29", "2024-04-30"), r("2024-05-02", "2024-05-03")]), | ||
| + ); | ||
| +}); | ||
| + | ||
| +test("adjacency across a year end", () => { | ||
| + assert.deepEqual(merge([r("2024-01-01", "2024-01-02"), r("2023-12-30", "2023-12-31")]), ok([r("2023-12-30", "2024-01-02")])); | ||
| +}); | ||
| + | ||
| +test("leap years", () => { | ||
| + // 2024 has Feb 29, so Feb 28 and Mar 1 are not adjacent. | ||
| + assert.deepEqual( | ||
| + merge([r("2024-02-27", "2024-02-28"), r("2024-03-01", "2024-03-02")]), | ||
| + ok([r("2024-02-27", "2024-02-28"), r("2024-03-01", "2024-03-02")]), | ||
| + ); | ||
| + assert.deepEqual(merge([r("2024-02-27", "2024-02-29"), r("2024-03-01", "2024-03-02")]), ok([r("2024-02-27", "2024-03-02")])); | ||
| + assert.deepEqual(merge([r("2023-02-27", "2023-02-28"), r("2023-03-01", "2023-03-02")]), ok([r("2023-02-27", "2023-03-02")])); | ||
| + assert.deepEqual(merge([r("2000-02-29", "2000-02-29")]), ok([r("2000-02-29", "2000-02-29")])); | ||
| +}); | ||
| + | ||
| +test("invalid dates", () => { | ||
| + for (const bad of ["2023-02-29", "1900-02-29", "2024-02-30", "2024-04-31", "2024-13-01", "2024-00-10", "2024-01-00", "2024-01-32", "2024-1-05", "2024-01-5", "20240105", "2024-01-05T00:00:00Z", " 2024-01-05", "2024/01/05", ""]) { | ||
| + assert.deepEqual(merge([r(bad, "2025-01-01")]), invalidDate(0, bad), `start ${JSON.stringify(bad)}`); | ||
| + assert.deepEqual(merge([r("2020-01-01", bad)]), invalidDate(0, bad), `end ${JSON.stringify(bad)}`); | ||
| + } | ||
| +}); | ||
| + | ||
| +test("single-day ranges and duplicates", () => { | ||
| + assert.deepEqual( | ||
| + merge([r("2024-06-01", "2024-06-01"), r("2024-06-01", "2024-06-01"), r("2024-06-02", "2024-06-02")]), | ||
| + ok([r("2024-06-01", "2024-06-02")]), | ||
| + ); | ||
| +}); | ||
| + | ||
| +test("contained and bridging ranges merge transitively", () => { | ||
| + assert.deepEqual( | ||
| + merge([r("2024-01-01", "2024-01-31"), r("2024-01-10", "2024-01-12"), r("2024-03-01", "2024-03-10"), r("2024-02-01", "2024-02-29")]), | ||
| + ok([r("2024-01-01", "2024-03-10")]), | ||
| + ); | ||
| +}); | ||
| + | ||
| +test("inverted range", () => { | ||
| + assert.deepEqual(merge([r("2024-01-01", "2024-01-02"), r("2024-03-02", "2024-03-01")]), inverted(1)); | ||
| + assert.deepEqual(merge([r("2025-01-01", "2024-12-31")]), inverted(0)); | ||
| +}); | ||
| + | ||
| +test("first problem in input order wins, start before end", () => { | ||
| + assert.deepEqual(merge([r("2024-01-01", "2024-01-02"), r("2024-02-30", "2024-13-01")]), invalidDate(1, "2024-02-30")); | ||
| + assert.deepEqual(merge([r("2024-05-01", "2024-04-01"), r("nope", "2024-01-01")]), inverted(0)); | ||
| + assert.deepEqual(merge([r("2024-01-01", "2023-02-29"), r("2024-05-01", "2024-04-01")]), invalidDate(0, "2023-02-29")); | ||
| +}); | ||
| + | ||
| +test("input is not modified", () => { | ||
| + const input = [r("2024-02-01", "2024-02-10"), r("2024-01-01", "2024-02-05")]; | ||
| + const snapshot = JSON.parse(JSON.stringify(input)); | ||
| + const first = input[0]; | ||
| + Object.freeze(input[0]); | ||
| + Object.freeze(input[1]); | ||
| + const out = merge(input); | ||
| + assert.deepEqual(out, ok([r("2024-01-01", "2024-02-10")])); | ||
| + assert.deepEqual(input, snapshot); | ||
| + assert.equal(input[0], first); | ||
| +}); | ||
Addedevals/tasks/deep-get/mutant.ts+20−0
| @@ -0,0 +1,20 @@ | ||
| +// Bug: splits the path with a permissive regex (invalid paths accepted) and walks with plain indexing, so inherited props and string "length" leak through. | ||
| +export class PathError extends Error { | ||
| + constructor( | ||
| + public readonly kind: "invalid_path", | ||
| + public readonly path: string, | ||
| + ) { | ||
| + super(`${kind}: ${path}`); | ||
| + } | ||
| +} | ||
| + | ||
| +export function getPath(obj: unknown, path: string): unknown { | ||
| + if (!path) throw new PathError("invalid_path", path); | ||
| + const parts = path.split(/[.[\]]+/).filter(Boolean); | ||
| + let cur: any = obj; | ||
| + for (const part of parts) { | ||
| + if (cur == null) return undefined; | ||
| + cur = cur[part]; | ||
| + } | ||
| + return cur ?? undefined; | ||
| +} | ||
Addedevals/tasks/deep-get/prompt.md+15−0
This diff has 16 lines, more than this page shows. View the file or the raw file.
Addedevals/tasks/deep-get/reference.ts+62−0
This diff has 63 lines, more than this page shows. View the file or the raw file.
Addedevals/tasks/deep-get/reference.tsus+73−0
This diff has 74 lines, more than this page shows. View the file or the raw file.
Addedevals/tasks/deep-get/signature.ts+13−0
| @@ -0,0 +1,13 @@ | ||
| +export class PathError extends Error { | ||
| + constructor( | ||
| + public readonly kind: "invalid_path", | ||
| + public readonly path: string, | ||
| + ) { | ||
| + super(`${kind}: ${path}`); | ||
| + } | ||
| +} | ||
| + | ||
| +/** Returns undefined when the value is absent. Throws PathError for an invalid path. */ | ||
| +export function getPath(obj: unknown, path: string): unknown { | ||
| + // TODO | ||
| +} | ||
Addedevals/tasks/deep-get/signature.tsus+7−0
This diff has 8 lines, more than this page shows. View the file or the raw file.
Addedevals/tasks/deep-get/task.json+1−0
This diff has 2 lines, more than this page shows. View the file or the raw file.
Addedevals/tasks/deep-get/test.mjs+106−0
This diff has 107 lines, more than this page shows. View the file or the raw file.
Addedevals/tasks/feature-flag/mutant.ts+27−0
This diff has 28 lines, more than this page shows. View the file or the raw file.
Addedevals/tasks/feature-flag/prompt.md+20−0
This diff has 21 lines, more than this page shows. View the file or the raw file.
Addedevals/tasks/feature-flag/reference.ts+57−0
This diff has 58 lines, more than this page shows. View the file or the raw file.
Addedevals/tasks/feature-flag/reference.tsus+54−0
This diff has 55 lines, more than this page shows. View the file or the raw file.
Addedevals/tasks/feature-flag/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/feature-flag/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/feature-flag/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/feature-flag/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/free-slot/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/free-slot/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/free-slot/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/free-slot/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/free-slot/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/free-slot/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/free-slot/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/free-slot/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/inventory-ledger/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/inventory-ledger/mutant.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/inventory-ledger/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/inventory-ledger/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/inventory-ledger/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/inventory-ledger/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/inventory-ledger/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/inventory-ledger/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/inventory-ledger/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/json-api-handler/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/json-api-handler/mutant.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/json-api-handler/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/json-api-handler/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/json-api-handler/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/json-api-handler/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/json-api-handler/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/json-api-handler/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/json-api-handler/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/lru-cache/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/lru-cache/mutant.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/lru-cache/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/lru-cache/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/lru-cache/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/lru-cache/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/lru-cache/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/lru-cache/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/lru-cache/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/manager-chain/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/manager-chain/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/manager-chain/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/manager-chain/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/manager-chain/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/manager-chain/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/manager-chain/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/manager-chain/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/merge-profile/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/merge-profile/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/merge-profile/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/merge-profile/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/merge-profile/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/merge-profile/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/merge-profile/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/merge-profile/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/moving-average/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/moving-average/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/moving-average/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/moving-average/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/moving-average/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/moving-average/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/moving-average/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/moving-average/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/order-state-machine/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/order-state-machine/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/order-state-machine/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/order-state-machine/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/order-state-machine/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/order-state-machine/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/order-state-machine/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/order-state-machine/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/paginate-cursor/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/paginate-cursor/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/paginate-cursor/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/paginate-cursor/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/paginate-cursor/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/paginate-cursor/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/paginate-cursor/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/paginate-cursor/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-amount/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-amount/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-amount/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-amount/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-amount/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-amount/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-amount/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-amount/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-config/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-config/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-config/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-config/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-config/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-config/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-config/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-config/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-duration/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-duration/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-duration/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-duration/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-duration/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-duration/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-duration/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-duration/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-log-line/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-log-line/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-log-line/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-log-line/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-log-line/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-log-line/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-log-line/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/parse-log-line/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/query-params/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/query-params/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/query-params/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/query-params/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/query-params/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/query-params/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/query-params/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/query-params/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/resolve-path/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/resolve-path/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/resolve-path/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/resolve-path/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/resolve-path/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/resolve-path/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/resolve-path/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/resolve-path/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/retry-fetch/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/retry-fetch/mutant.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/retry-fetch/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/retry-fetch/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/retry-fetch/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/retry-fetch/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/retry-fetch/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/retry-fetch/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/retry-fetch/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/route-match/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/route-match/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/route-match/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/route-match/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/route-match/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/route-match/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/route-match/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/route-match/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/semver-compare/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/semver-compare/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/semver-compare/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/semver-compare/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/semver-compare/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/semver-compare/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/semver-compare/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/semver-compare/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/shortest-path/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/shortest-path/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/shortest-path/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/shortest-path/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/shortest-path/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/shortest-path/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/shortest-path/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/shortest-path/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/signup-validation/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/signup-validation/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/signup-validation/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/signup-validation/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/signup-validation/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/signup-validation/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/signup-validation/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/signup-validation/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/stats-summary/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/stats-summary/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/stats-summary/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/stats-summary/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/stats-summary/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/stats-summary/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/stats-summary/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/stats-summary/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/token-bucket/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/token-bucket/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/token-bucket/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/token-bucket/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/token-bucket/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/token-bucket/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/token-bucket/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/token-bucket/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/topo-sort/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/topo-sort/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/topo-sort/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/topo-sort/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/topo-sort/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/topo-sort/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/topo-sort/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/topo-sort/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/ttl-cache/mutant.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/ttl-cache/prompt.md
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/ttl-cache/reference.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/ttl-cache/reference.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/ttl-cache/signature.ts
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/ttl-cache/signature.tsus
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/ttl-cache/task.json
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/tasks/ttl-cache/test.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Addedevals/validate.mjs
Not shown: this page diffs the first 60 files. View the file or the raw file.
Checks
No checks reported for this commit.
Runs
No runs for this commit.