Architecture

How it works, where the boundaries are, and how to change it.

docs/architecture.md Edit this page on GitHub

How Toolbench is put together, why the boundaries sit where they do, and what you need to know to change it.

Audience. Anyone working on the packages, reviewing a change, or deciding whether to adopt this. It assumes you can read TypeScript. It does not assume you have written a custom element, used a Web Worker, or thought about bundler chunking before; §2 covers the vocabulary.

1. Scope

Toolbench does three things:

1. Defines what a tool is: a function from named inputs to one of a fixed set of result shapes, plus a JSON manifest describing both.
2. Renders a tool in a web page, in one of three presentations, and runs it.
3. Keeps old tools working as the contract grows.

Everything else is out of scope on purpose. There is no editor, no sandbox, no server, no plugin marketplace, and no dataflow between tools. §15 lists what that costs.

2. Terminology

Skip this if the words are familiar.

3. System overview

Three phases, and the boundary between them is the thing to hold onto: the manifest is data and can move at any time; the tool's code must be present at build time.

The diagram shows the three phases, build, host and browser, and what crosses between them.

Reading the diagram:

• A manifest is validated and migrated before anything sees it, so every component downstream deals with current, well-formed data only.
• A tool's module reaches the browser through the host's bundler. Nothing fetches code at runtime, because doing that on a static site requires eval, which costs bundling, tree shaking, and any Content-Security-Policy worth having.
• The worker is optional and per tool. The runner's interface is identical either way.
• Results are migrated on the way out, so a tool written against an older contract renders through today's renderers without knowing.

4. Repository layout

toolbench/├── packages/│   ├── sdk/                        @toolbench/sdk   no dependencies, no DOM│   │   └── src/│   │       ├── types.ts            the contract: Output, InputSpec, Manifest, Ctx, Tool│   │       ├── validate.ts         manifest validation and the cross-field invariants│   │       ├── inputs.ts           inputProblem, the checks before a run, apart so a page can have them without the validator│   │       ├── build.ts            a tool.json checked at build time and written as a frozen, marked module. Exported at ./build│   │       ├── cli.ts, bin.ts      `toolbench-manifests`, build.ts for any bundler│   │       ├── brand.ts            the mark's key, apart so boot can read it without build.ts│   │       ├── migrate.ts          version migration chain, and the output half of it│   │       ├── upgrade.ts          upgradeManifest, the manifest half, apart so boot does not carry it│   │       ├── seed.ts             build-time seeding, and safe serialisation for a script tag│   │       ├── fixtures.ts         the tool-directory walk. Node-only, exported at ./fixtures│   │       ├── testing.ts          fixture runner and result comparison│   │       ├── graph.ts            graphProblem, the graph kind's structural rules│   │       ├── version.ts          SDK_VERSION, supported versions, changelog│   │       ├── features.ts         the version each key and output field arrived in, and the check against `sdk`│   │       └── index.ts            public surface│   └── runtime/                    @toolbench/runtime   no dependencies, no framework│       ├── test/graph/             the graph families, seeded graphs and geometric checks the tests use│       └── src/│           ├── element.ts          <tool-host>: modes, form, states, accessibility│           ├── runner.ts           execution: sequencing, abort, timeout, progress│           ├── worker.ts           createToolWorker, the worker side of the protocol│           ├── protocol.ts         message types and error classes│           ├── sources.ts          ToolSource interface, RegistrySource│           ├── validation.ts       manifest validation for manifests the build did not check, a chunk fetched by list()│           ├── checks.ts           the checks before a run, a chunk fetched with a tool's code│           ├── vite.ts             @toolbench/runtime/vite, build.ts as a Vite plugin│           ├── styles.ts           the shadow stylesheet and the theming surface│           ├── dom.ts              three element helpers, so no framework is needed│           ├── controls.ts         the newer inputs' controls, a chunk fetched only by a tool with one│           ├── choices.ts          a multiselect's checkboxes and chips, a chunk fetched only by a tool with one│           ├── rows.ts             a rows input's list, a chunk fetched only by a tool with one│           ├── render/index.ts     renderer dispatch, fields, table, text, code, error, group│           ├── render/chart.ts     the series renderer, SVG plus a data table│           ├── render/bytes.ts     the bytes renderer and its rules, a chunk fetched when a dump is drawn│           ├── graph/              the graph kind: layout, routing, drawing, and its chunks│           ├── register.ts         side-effecting entry for pages without a bundler│           └── index.ts            public surface├── tools/│   ├── cases.test.ts               runs every tool's manifest and fixtures│   ├── tsconfig.json               tools compile with no DOM and no path aliases│   ├── percentiles/                pure, main thread, group of fields and table│   ├── queue-explorer/             pure, worker, group of series and fields, plus convergence tests│   ├── utf8-bytes/                 pure, main thread, group of fields and bytes│   ├── regex-explainer/            pure, worker with a 2 s timeout, text and code│   ├── histogram/                  pure, main thread, bare series, bar shape, no primary input│   └── build-order/                pure, main thread, group of fields and a graph├── bench/│   ├── index.html                  card mode, theming, lifecycle status│   ├── failure.html                the failing tool, off the landing page│   ├── tool.html                   page mode│   ├── article.html                embed mode, two tools in prose│   ├── graphs.html                 the eleven graph families, one figure each│   ├── failure.html                stress fixture, kept off the front page│   ├── bench.test.ts               the runtime driven by Chrome against the built bench│   ├── fixtures/stress/            a tool that misbehaves on purpose│   ├── fixtures/retired/           a tool that must not run│   ├── fixtures/deprecated/        a tool that still runs, marked│   ├── host/                       the smallest host page, measured as the published runtime size│   └── src/                        registry, boot, worker entry, page scripts├── scripts/│   ├── new-tool.mjs                scaffolds the four files under tools/<id>│   └── size-check.mjs              transfer budgets for the built bench and the host page├── docs/                           this file, authoring, versioning└── .github/workflows/             tests.yml, build.yml, pages.yml, release.yml

5. The contract

packages/sdk/src/types.ts is the whole agreement between a tool author and a host. It is worth reading in full; this is the shape of it.

5.1 A tool

export interface Tool<I extends InputValues = InputValues> {  run(input: I, ctx: Ctx): Output | Promise<Output>;} export interface Ctx {  signal: AbortSignal;  progress(fraction: number, partial?: Output): void;}

Three rules follow, and they are not stylistic:

1. run must not touch the DOM. It has to work in Node (fixtures), in a worker (no DOM exists), and during a host's build (seeding). Any one of those would force it.
2. run must be a function of its inputs. No clock, no unseeded randomness, no I/O. That is what makes a fixture meaningful.
3. A loop whose length depends on an input must check ctx.signal, and the tool should declare thread: "worker" so a timeout can stop it.

5.2 Results

export type Output = (  | { kind: "fields";   fields: Field[] }  | { kind: "text";     text: string; mono?: boolean }  | { kind: "code";     lang: string; source: string }  | { kind: "table";    columns: Column[]; rows: Cell[][]; caption?: string; link?: { chart: string; keys: (number | string | null)[] } }  | { kind: "series";   chart: Chart }  | { kind: "heatmap";  heatmap: Heatmap }  | { kind: "pie";      pie: Pie }  | { kind: "graph";    graph: Graph }          // contract v7: nodes and edges the runtime lays out  | { kind: "group";    parts: Output[] }  | { kind: "bytes";    bytes: number[]; offset?: number; highlight?: ByteRange[]; caption?: string }  | { kind: "error";    message: string; input?: string; at?: number; len?: number }  ) & { role?: string; id?: string };   // any part may be named, contract v5

The union is closed. A tool cannot invent a shape nobody can draw, and the runtime can promise to draw anything a tool returns. Two members carry more weight than the rest:

• group is how one tool answers in several shapes. A decoder returning header fields and a table of records is the normal case, not an exception. Without it, tools would either lose half their output or the contract would need a second result channel.
• error means the input was wrong and the tool behaved correctly. input names the offending control, which is what lets a generated form mark it invalid and point a screen reader at the message. Without that field, an error can only float unattached. A tool with a genuine bug should throw instead; §7.5 shows how the two render differently.

5.3 Inputs

export type InputSpec =  | (InputBase & { type: "text";     default: string; maxLength?: number })  | (InputBase & { type: "textarea"; default: string; rows?: number; maxLength?: number; lang?: string })  | (InputBase & { type: "number";   default: number; min: number; max: number; step?: number })  | (InputBase & { type: "select";   default: string; options: SelectOption[] })  | (InputBase & { type: "toggle";   default: boolean }); interface InputBase {  id: string;  label: string;  description?: string;   // becomes aria-describedby, not a placeholder  unit?: string;          // "ms", "req/s". A number without one is a riddle  primary?: boolean;      // show this input on a compact card; mark as many as the question needs  dir?: "ltr" | "auto";   // bytes and code stay left to right inside right-to-left prose} interface SelectOption {  value: string;  label: string;  sets?: InputValues;     // a preset: choosing this option fills those inputs (contract v6)}

A preset is derived state, not remembered state. After any write (typing, a sample, a host values), a preset select shows the option whose sets the form matches, else the first option with no sets, else it stays put. Remembering the choice and switching to "Custom" on a keystroke was mocked up first, and a sample walked straight past it: it filled a read timeout of 120 while the select still said "Library A", whose read timeout is 30. A select the write names is taken literally, which is how a host prefills one without expanding it.

min and max are required on a number input rather than optional. They are the only thing standing between a bounded computation and an unbounded one, and the element refuses a value outside them before the tool sees it: a typed or linked value is marked under its box, and a host's write is clamped.

5.4 The manifest

The fields, and who reads each one:

6. Components

6.1 @toolbench/sdk

No dependencies, no DOM, no framework. Four modules that matter.

types.ts is documentation as much as code; §5 is a summary of it.

validate.ts turns unknown JSON into a Manifest or throws a ManifestError naming the field. Hand written rather than schema-library driven, for one reason: the error messages are the interface a first-time tool author meets. Compare what a generic validator says with what this says:

inputs[0].options: a select needs at least two options; with one, use a fixed value timeoutMs: is only meaningful with runtime.thread = "worker". On the main thread there           is nothing to terminate, so a timeout here would be a field that lies.           Either move the tool to a worker or bound its input.

The cross-field invariants at the bottom of the file are the interesting part; §8 lists them.

migrate.ts holds the migration chain, and upgradeOutput; upgradeManifest is beside it in upgrade.ts, because the runtime upgrades outputs in the chunk every page loads and validates manifests in a chunk of its own, and a bundler splits by module. The chain: two steps today, 1 → 2 and 2 → 3, with every half the identity function. The machinery was built before anything needed it, and tested with synthetic versions, which is why the first real entry was a five-line change rather than a design exercise under pressure. It repaid that twice over: while still synthetic, the tests caught a bug where the chain silently did nothing, because a helper used the module constant instead of the injected current version.

testing.ts runs fixtures and compares results. Comparison is exact by default. Subset matching is opt-in and has to justify itself:

6.2 @toolbench/runtime

The element is the hub: it asks the source for manifests and code, the runner for results, and the renderers for markup, while a worker is reached only through the runner.

element.ts is <tool-host>, and it is the only public surface most hosts touch. It owns:

• reading tool, mode, parts and more attributes, and the inline seed;
• capping a grouped result at parts in any mode, and, when more="expand", rendering the hidden parts up front behind a disclosure button instead of a line of text. The cap is the host's, not the tool's: it is a question about the room on this page. Expanding redraws from the output already held, never re-runs, which is what makes it usable for a worker-mode tool, and it takes over the inner field and byte notices so a reader is never shown a truncation they cannot act on;
• a values setter and run() method, so a host can prefill the form and opt into running without reaching into the shadow root;
• the tool-input and tool-run events (#212), composed and bubbling, so a page can follow the form and each settled run without listening for the controls' own events inside the shadow root. tool-input is compared with the last values reported, so a write that changes nothing is silent, and neither fires while the tool opens with its starting values, an autoRun tool's first run included;
• the share button (#227) on page and embed: copies this page's address with the inputs as <tool id>.<input id> query parameters, and reads them back as the tool opens, as its starting values; never an input marked share: false. The button, the link writer and the clipboard are a chunk of their own, fetched with the tool's code; only the reading is in boot, because it has to happen before anything awaits. share="none", share-label, share-position and their defineToolHost options, including a host's own { url, read } scheme;
• the links attribute and property (#216): all, the manifest's links and the default outside a card; page, only "Open the full tool"; none, nothing. A card reads anything but none as page, and a retired tool keeps its links whatever it says, since its notice points at them;
• a Reset button beside Run (#208), back to the values the tool first opened with, a host prefill included, with the result cleared, the seed redrawn and an autoRun tool's default answer computed again. Built once and hidden while the values are the starting ones, rather than inserted on the first edit, because Chrome ends an undo group at any node insertion. reset="none" leaves it out;
• reveal(id), so a page can link to one part (#147): it activates a closed card, runs a tool with nothing on screen, opens a disclosure or lifts a plain parts cap when the part is behind one, then scrolls to the part and focuses it. Only for an id the result has, so a wrong link changes nothing;
• deciding when to activate (click for a card, intersection for a page or embed);
• honouring status: retired never activates and renders links; deprecated is marked and is not a live card;
• building the form from manifest.inputs, including labels, descriptions, bounds and text direction. A segmented select (#203) is native radios in a <fieldset> captioned by its <legend>, from controls.ts, a chunk fetched with the tool's code only by a tool that has one, with its own rules; the row stacks into plain radios while a segment would cut its label, measured as the group's width changes. Without the chunk it is the dropdown, which is what an older runtime draws. A colour's labelled control is its hex box, with the system's picker in a swatch beside it from the same chunk, kept in step both ways; without the chunk, the box alone. A secret is a password field with an eye button inside its end edge, also from the chunk, aria-pressed and named "Show" or "Hide" with the input's label. A multiselect (#204) is checkboxes, or chips past six options, in the same kind of fieldset, from choices.ts, fetched the same way; without it, a native multiple select. Its count line is part of the group's description, and Run is refused below min. An error naming any input is shown under it, the field's name first, in the slot that was a screen reader's alone. A rows input (#204) is a list from rows.ts, fetched the same way: Add, a cross per row, and a handle moved by the arrow keys or a drag, focus kept on what moved and put on the next row after a removal, each change announced in the unseen status region; its cells stack beside the handle in a tool narrower than 26rem. An error with a row is shown under that row;
• the state machine: facade, loading, idle, running, result, stale, error;
• accessibility: label association, aria-describedby, aria-invalid driven by error.input, a role="status" region carrying a short summary, focus moved only on an explicit run;
• teardown, which is what stops a worker leaking on client-side navigation;
• an optional highlight hook on defineToolHost, so a host can paint code results without the runtime bundling a highlighter. The hook returns a Node, not a string. The same hook paints a textarea that declares lang, from paint.ts, a chunk of its own: a <pre> with the textarea's box and font sits behind it, aria-hidden, and the textarea's text goes transparent while the paint is current, so caret, selection, undo and the accessible name stay the real textarea's. A tool's own grammar for that lang is tokenised in the same chunk, with no regular expression, and paints when there is no hook or the hook hands back plain text.

runner.ts executes. Its whole job is the difference between "call a function" and "call a function that might not come back":

worker.ts is the worker side. It caches loaded modules per id, keeps one AbortController per in-flight sequence, converts a thrown error into a crash message, and swallows its own aborts because the page already knows about those.

protocol.ts defines the messages and three error types (ToolTimeoutError, ToolCrashError, WorkerUnavailableError). It is a separate file because the message shapes are the actual contract between two threads, and burying them inside the runner made that easy to forget.

sources.ts defines ToolSource:

export interface ToolSource {  list(): Promise<Manifest[]>;  load(id: string): Promise<LoadedTool>;}

RegistrySource is the implementation hosts use. Every manifest it hands out has been migrated and validated, so a malformed tool rejects list() and load() with a field name rather than becoming a mystery at render time, and it refuses two tools claiming the same id. There are two paths to that (#259):

• Checked at build time. @toolbench/runtime/vite and the toolbench-manifests command run the validator in Node, through one implementation, @toolbench/sdk/build, and fail the build with its message and the file. They emit the validated manifest frozen at every level and carrying a non-enumerable Symbol.for("toolbench.validated"). When every entry is such a manifest, the constructor indexes them, refusing a repeated id, which no single file can show, and the validator is never downloaded.
• Checked in the browser. Any other manifest, from a server, a CMS or a host without the build step, sends the source to the validation chunk on its first list() or load(). Marked manifests pass through it as they are. A host whose manifests come from a server preloads that chunk (preloadValidation in the plugin, or @toolbench/runtime/validation), so it is fetched beside the runtime.

isMarked accepts the mark only on a frozen object. The mark is the build's word that the object has not changed since it was checked, and an unfrozen object could have been. JSON cannot carry a symbol, so data can never arrive marked; page code can forge one, but page code already runs with the page's authority (SECURITY.md).

⚠️ The constructor never throws, on either path, which it did until #259: the browser path cannot await the chunk there. Every <tool-host> on the page shows the rejection instead. A manifest reaches the element only through list(), so nothing is ever drawn from one that has not been validated, which is the property the link-scheme defect in §14 depends on.

The checks before a run (inputProblem) are needed whichever path a manifest took, because readers type values, so they are a chunk of their own, checks, fetched with a tool's code when it activates: 722 bytes, which in the runtime chunk measured 462 bytes more, on every page, opened or not. 869 with the number rule (#262).

render/ holds one function per result kind plus the dispatcher. The dispatcher ends with:

default: {  const _exhaustive: never = output;  return unknownOutput((_exhaustive as { kind: string }).kind);}

Adding a kind to the SDK without adding a renderer is therefore a compile error in this package. The unknownOutput fallback covers the case a compiler cannot: an old runtime meeting a newer tool at runtime. It renders a message saying the page needs a newer runtime, never a blank space.

render/chart.ts draws SVG. It has no charting dependency, which is a deliberate trade: a library would be 40 to 200 KB and would decide how every tool looks. It also emits the chart's numbers as a <table> inside a <details>, because a chart is an image and a screen reader gets nothing from it.

graph/ is the graph kind (#152), and it is the one renderer that decides where things go rather than only drawing where the tool put them. Its base chunk, graph/graph.ts and what it imports, validates the graph against the SDK's graphProblem, lays it out and draws it, synchronously, as every renderer does. The layout is a pure function from the graph to plain coordinates and SVG path strings (graph/model.ts), with no DOM, which is what lets a worker run it and Node test it:

1. Place. Layered placement in layered.ts: cycles broken in input order, ranks by longest path with pinned ranks honoured, ranks doubled so a label gets a layer of its own, long edges as chains of dummies, then barycentre and median sweeps with transposition for the order, and pool-adjacent-violators for the coordinates. Tree, radial, force and manual placements are chunks.
2. Attach. ports.ts puts each end on the side facing the other end, one end at the side's midpoint, several packed symmetrically 16px apart in the order of what they reach, never at a corner, and a shape's own outline limits a side: a diamond's left side has room only at its vertex.
3. Route. router.ts finds each edge on a grid of pitch 8 between its two fixed ports, by A* with libavoid's penalties: a crossing 1000, a bend 35, a segment running backwards 60. The grid makes "no overlap" structural: a grid edge has one owner, a vertex a route turns at is that route's, and node interiors, rings, labels, titles and every port's own lane are blocked to everyone else.
4. Repair, search, finish. A route that could only be found by sharing goes to the repair chunk; a dense graph's order to the search chunk; then corners are rounded and the drawing is made.

Everything optional is a chunk of its own, fetched for the first graph that needs it (graph/chunks.ts): graph-tree, graph-force and what they share in graph-free, graph-lines for straight lines and curves, graph-shapes, graph-er for records and crow's feet, graph-groups, graph-interact for path highlighting, graph-search, graph-repair and graph-large. A renderer that finds one missing throws ChartRendererMissing carrying the load, and the element fetches it and draws again, the pattern the chart kinds use. Whether the search or the repair is needed is only known after the base layout, so the layout itself asks for them and the drawing starts again with them.

⚠️ Determinism is a property of the code, not of the engine. Math.sin, Math.cos, Math.atan2, Math.hypot and ** are not correctly rounded, so engines differ in the last bit, and a layout is a long chain of comparisons in which a last bit decides an order. The layout uses only + - * / and Math.sqrt, which IEEE 754 fixes: an ellipse by its rational parametrisation, the radial placement with its own sine series, and a seeded integer generator for force layouts. The budgets that bound the search are counted in A* states, never in milliseconds, so a slow machine draws the same picture as a fast one. A browser test compares every family's drawing, the large one from the worker included, with the layout Node computes.

A graph past a hundred nodes is the graph-large chunk's: the figure comes back at once with the edge table open and a line saying the picture is being laid out, and the layout runs in one worker for the page, made with the host's workerFactory, which imports graph/graph-worker.ts through createToolWorker's layout message. Without a factory it runs on the main thread after a frame.

Accessibility is the drawing's, not a later pass. The picture is one tab stop, a role="group" with the graph's summary as its name; the arrow along the flow follows the selected edge out of the focused node, the arrow against it goes back, the arrows across choose among the outgoing edges, and a live region says where focus is and what is selected. Every edge is a row of a table in a <details>, and a node with no edges a row saying so. A redraw under the reader, when a group is opened or closed, puts focus back on the new picture, so it never falls to the shadow root.

styles.ts is one template string adopted into every shadow root via adoptedStyleSheets, so one parsed stylesheet serves every instance on the page. All of it, and every chart sheet, is inside @layer toolbench, so a host's unlayered rule wins whatever its specificity. It also orders the sheets in a root: the runtime's first, a chart's after it, then whatever a host adopted by hand, then the sheets from the host's styles option, which setHostStyles can replace on every live root. STYLING.md is the contract a host sheet is written against: the classes, data-kind on every part, data-role, data-label on table cells, and what counts as a breaking change to them.

6.3 What the host owns

Deliberately small, and deliberately not zero:

7. Control flow

7.1 Registration and first paint

The sequence shows registration, from the host's script to a painted facade. Nothing has been downloaded but the manifest data, which was already in the page's bundle, and, only for manifests the build did not check, the validation chunk, once per page.

7.2 Activation

The sequence shows activation, from a click or a scroll to a drawn form. Two decisions are visible here. The form is not painted until the module has arrived, because a Run button that exists and does nothing is worse than a spinner. And activation does not run the tool: see §7.3 for why.

7.3 A run on the main thread

The sequence shows one run on the main thread, from Run to a drawn result. The reader triggers every run. An earlier version ran on activation and again on every keystroke, and the result was a Run button that appeared broken: the answer was already on screen before you looked at it, and pressing the button changed nothing visible. It also did work nobody asked for. A tool that is genuinely instant can opt back in with autoRun.

7.4 A run in a worker

The sequence shows one run in a worker, including the timeout that is armed beside it. Ctx is built on the side that runs the tool, never sent across the boundary. An AbortSignal is not structured-cloneable, progress is a function, and a Response is neither. Once that is true, an RPC library adds a wrapper and removes the one thing needed here, which is a timeout that can stop a running tool, so the protocol is hand written in about forty lines.

7.5 Failure paths

The sequence shows the four ways a run can fail, and what each one puts on screen. Four distinct outcomes, four distinct messages. A reader who cannot tell them apart cannot do anything useful about any of them, and the failure paths are the part a demo never exercises.

8. Invariants, and where they are enforced

Rules the rest of the system relies on. Each is checked once, in one place, and each has a test.

9. Lifecycle and teardown

connectedCallback  └─ read seed, list manifests, paint       ├─ (retired) stop: explain, render links. Never activate.       ├─ (card, not deprecated) click ──┐       └─ (page/embed) observe ──────────┴─ activate                                            └─ load module, create Runner, paint form                                                 └─ run, run, run ...disconnectedCallback  └─ runner.dispose()   cancel in flight, terminate the worker  └─ observer.disconnect()  └─ clear the debounce and slow-run timers

Teardown matters more than it looks. On a site with client-side navigation the element is removed rather than the page being reloaded, so a worker with no owner survives. Each one holds a few megabytes. Nothing else in the system will clean it up, which is why disconnectedCallback is not optional and why a host should never create a worker outside the element's lifetime.

10. How to change it

10.1 Add a result kind

This is a contract change, so it is also a version bump. Read versioning.md first.

1. packages/sdk/src/types.ts: add the member to Output and to OUTPUT_KINDS. Define any new supporting interfaces in the same file.
2. packages/sdk/src/version.ts: bump SDK_VERSION, append to SUPPORTED_SDK_VERSIONS, add a SDK_CHANGELOG line.
3. packages/sdk/src/migrate.ts: add the Migration for the boundary you just created. Both halves are normally the identity function; if they are not, the change was not additive.
4. packages/runtime/src/render/: add the renderer. The exhaustive switch will not compile until you do, which is the intended pressure.
5. packages/runtime/src/styles.ts: styles for it, using the existing custom properties.
6. A tool or a fixture that exercises it, and pnpm check. Every existing tool's fixtures must pass without being edited.

10.2 Add an input type

1. types.ts: add the variant to InputSpec and to INPUT_TYPES.
2. validate.ts: validate its fields, including whatever bound keeps it from being unbounded.
3. element.ts: add a case to #control. Everything there is per type: the control, its aria-describedby, the keyboard behaviour, and what a change does. values.ts: what coerce does with a value a host or a link writes. If the browser cannot hold a reader to the type's rules, inputProblem in validate.ts says what is wrong, and the element refuses the value before run, naming the input.
4. styles.ts, then a fixture and a bench entry.

Also a version bump.

10.3 Add a capability

Contract version 1 has only pure. A capability is a promise about what a tool will try to do, so adding one means deciding three things before writing code:

• what the runtime injects for it (ctx.fetch, ctx.asset, a file handle);
• what the runtime refuses without it, and where that refusal is enforced;
• whether a tool holding it may still be a live card. For anything touching files or the network the answer is no, and validate.ts is where that is written down.

10.4 Add a ToolSource

Implement two methods. The only rule is the one in §3: list() may do anything, including a network request, but load() must resolve to a module the bundler has already seen. A source backed by a remote repository works by generating a registry at build time, not by fetching JavaScript in the reader's browser.

10.5 Add a display mode

Mode in element.ts, plus the branch in #paint and the activation policy in #prepare. Before adding one, check whether an existing mode plus host CSS gets there; three modes already cover compact, full and in-prose, and a fourth needs a reason beyond looking different.

11. Build pipeline

11.1 What runs

The flow shows the pipeline from install to the published artifacts. Each step is also a script you can run alone. pnpm check is steps 2 and 3, which is what to run while working.

11.2 Compiler settings that are load bearing

From tsconfig.base.json:

tools/tsconfig.json is separate on purpose: lib is ES2023 with no DOM, so a tool cannot reach for document by accident, and there are no path aliases, because Node's type stripping does not resolve them.

11.3 Bundling, and two traps

The bench uses Vite. Two settings are not optional for anyone doing the same thing:

// bench/vite.config.tsworker: { format: "es" },              // the default, "iife", cannot code-splitbuild: {  rollupOptions: {    output: {      manualChunks(id) {        const m = /\/(?:tools|fixtures)\/([^/]+)\/index\.ts$/.exec(id);        return m ? `tool-${m[1]}` : null;      },    },  },},

• worker.format defaults to "iife". A worker that dynamically imports anything then builds fine in development and fails the production build with "UMD and IIFE output formats are not supported for code-splitting builds".
• manualChunks exists because every tool's entry file is named index.ts, so without it every chunk is called index-<hash>.js. That makes the network panel useless for the one thing the bench demonstrates, and it made a test count the page's own entry as a tool chunk.
• worker.rollupOptions needs the same treatment, separately, because none of build.rollupOptions applies to the worker's build. Skipping it leaves the worker's copies anonymous, which is how a duplicate copy of every tool went unnoticed for a while.
• base must be /toolbench/ on the Pages build. A project Pages site is not at the origin root, and without the prefix the HTML loads while every /assets/... request 404s. The published build is the project site, with SITE_BASE=/toolbench/; the bench keeps / unless BENCH_BASE is set. In-page nav is relative (./tool.html) in both, for the same reason.

11.4 Artifacts

11.5 CI

Four workflows, split so each badge answers one question:

• tests.yml: typecheck, every Node-runnable test, and the coverage report.
• build.yml: packages, published contents, the bench, transfer budgets, and the browser suite on Chromium, Firefox and WebKit.
• pages.yml: builds the project site in site/ with SITE_BASE=/toolbench/, checks its links and examples, and, on main, publishes it to GitHub Pages. The bench is not published. Enabling Pages, under Settings then Pages then Source: GitHub Actions, is a maintainer click; the workflow is already in the tree.
• release.yml: version pull requests and the npm publish, after a human at both gates.

⚠️ types and unit tests and packages, bench, browser are required status checks on main, so renaming either job silently stops it gating anything. The second is deliberately an unmatrixed job that waits on the three browser legs, because GitHub appends matrix values to a job's name and the required context would otherwise stop reporting. See the comment on that job.

The step that matters most is pnpm test, because it runs every tool's fixtures against the current SDK and runtime. That is the mechanism behind the compatibility promise, not a nicety.

tests.yml also runs pnpm coverage after pnpm test and writes the uncovered list to the job summary. It reports; it does not gate on a percentage, and the step is advisory so a report cannot block a merge. The globs come from the test script rather than a copy of it. The browser suite is not in those numbers, and the report says which files it never loaded for that reason.

12. Testing strategy

Eight layers. Each catches something the others structurally cannot.

The Count column is measured, not maintained: pnpm test:counts runs each layer and reports what the table says beside what it found, and --update rewrites the cells. It exists because these numbers changed on almost every pull request and collided in four consecutive rebases, five pull requests in one batch each correcting the browser count to a different value that was right against the main it was written on. It also sums the Node rows and compares them against pnpm test: if tests live somewhere no row covers, the table is describing a subset and the script says so rather than letting a number be corrected. That is how the missing scripts/*.test.ts row was found. CI runs it on the Chromium leg of build.yml, so a count that drifts fails the required check rather than waiting for someone to notice.

pnpm coverage runs the Node rows of that table with Node's built-in test coverage (the same collector as --experimental-test-coverage) and prints the uncovered lines and branches. It does not fail on a percentage, and the CI step is advisory, so a report cannot hold a merge shut. It takes the globs from the test script rather than keeping a copy: a copy had already drifted by a whole test directory within a day of being written, and a coverage figure over a smaller suite than you think you are measuring reads as good news.

The browser row is not in those numbers: Playwright coverage is a different collection, and files the Node process never loads (packages/runtime, which needs a DOM) do not appear as 0%. That is why the report ends with the list of files it never loaded, which is the honest caveat on a figure covering 10 of 22 source files. Read that list once and file what it reveals.

The browser layer is weighted towards things that only exist in a browser or only appear in a production build:

• a card fetches no tool chunk until it is opened, and exactly one when it is;
• a seeded card shows a real result before anything runs;
• activation does not run the tool, and the status says "press Run";
• changing an input marks the result stale without re-running, and Run clears it;
• a host values write fills the form without running, clamps like typing, works before a card opens, and run() after it produces a result;
• tool-input fires once per keystroke, select change, sample and changing values write, before the setter returns and not for a write that changes nothing; tool-run fires once per Run or run(), after the result is drawn, with ok false and the message for an error result and a crash, once for a debounced burst on an autoRun tool, and neither fires while the tool opens;
• Reset is hidden at first and after the reader puts a value back by hand, sits after Run, restores the first render's values after edits, a run or a sample, including a host's prefill, clears the result, moves focus to the first input, is reached by Tab and pressed by Enter, dispatches tool-reset once, redraws an autoRun tool's default answer, goes with reset="none", and leaves a typed burst one undo;
• links draws the manifest's links by default and for a value it does not know, only "Open the full tool" for page, no row for none, set through the property as well, keeps the reader's inputs and result when it changes, leaves a retired tool's links, and on a card removes only with none;
• the share button sits at the title's end on a page and the Run row's end in an embed, is absent on a card, moves and goes with its attributes and options, copies the address with the inputs under the tool's id and keeps the page's own query, shows and announces "Link copied" for two seconds, dispatches tool-share, is reached by Tab and pressed by Enter, opens a tool with a link's values without an event, reads a toggle's true and false, never writes or reads a share: false input, uses a host's own scheme both ways, copies with execCommand when the clipboard refuses or is missing, shows the link selected when that fails too and says copying needs HTTPS on an insecure page, and puts the link on the real clipboard in Chromium, with the Clipboard API and without it;
• a sample fills the form, sets several inputs at once, leaves the ones it does not name alone, keeps focus on the button that was pressed, and draws no row on a card;
• a select preset fills the inputs it names and leaves the rest, goes stale without running, and passes its own value to run; the select then follows typing, a sample and a host write, announces the change in the status region, and is left alone when the host names it;
• a textarea declaring lang is painted through the host hook behind the real textarea, whose label, description target and accessibility tree are unchanged; the layer has the textarea's box and font, a click on a painted glyph puts the caret on it, scroll follows, typing shows plain text until a pause and keeps one undo for the burst, a sample repaints it, 20,001 characters stays plain, and a hook that throws, returns a string or returns different text leaves the plain textarea; the paint chunk is fetched only when a tool and its host can use it;
• a tool's grammar paints its exact tokens with no hook or under a hook that does not know its lang, a hook that does know it wins, the accessibility tree equals the unpainted textarea's, a click puts the caret on the glyph, every token keeps 4.5:1 in both themes, and the paint runs no regular expression;
• Run attention is a fill change, not a ring, so it cannot be mistaken for :focus-visible;
• all three modes, including two tools in one article with different threading;
• a tool that spins forever is killed by the timeout, the page stays responsive, and the next run gets a fresh worker;
• a crash reads differently from bad input, and bad input marks the right control invalid;
• a retired tool never fetches its code, explains, and renders its links; a deprecated tool is marked, still runs on its page, and is not a live card;
• the landing page has no stress host, in-page nav stays relative, and each tool links at tools/<id>/;
• labels, aria-describedby targets that exist, the status region, bounded number inputs;
• theming through custom properties, and a host's own sheets: kept after the runtime's in every root, winning over it through the runtime's layer, replaced everywhere by setHostStyles, and each of STYLING.md's four examples producing the computed style it exists for;
• every graph family drawn through the element is the drawing Node lays out, in all three engines, the one past 100 nodes laid out in the worker; every part carries data-kind and its data-role; the edge table is a closed <details> with a row per edge and one for a node with none; the arrow keys follow the edges and the live region says so; focus stays on the picture when a group closes under it; and a plain layered graph fetches the graph's base chunk and nothing else of it;
• the suite refuses to start when its port is already serving, so it cannot test another build;
• a host highlight hook replaces the code text node with the returned Node, and omitting it keeps readable plain text;
• Copy on a code result writes its source exactly and on a table writes tab-separated text, from the keyboard, announced; a refused or missing clipboard copies the same text with execCommand, and where that fails too says so and selects the text instead; a closed card has no Copy.

Two habits worth keeping. Test against the built artifact, because a minifier deleted a loop that made a timeout test pass for the wrong reason (§14). And assert an allowlist rather than a denylist when checking what loaded, because a denylist only catches the leaks you predicted.

13. Performance budget

Measured with gzip, not estimated. Reproduce with pnpm bench:build && node scripts/size-check.mjs, which is also a CI step: every figure below has a budget, so a number in this table cannot quietly stop being true.

The runtime's row is a host's page, not a bench chunk. bench/host/ is the smallest honest host, built with Vite's defaults: an HTML page, defineToolHost and RegistrySource, and one trivial tool whose manifest it imports. The row is everything that page downloads before its card paints, and the browser suite fails if the page fetches anything to paint that the row does not count. Every other row is a chunk of bench/dist.

Where the budget is spent: the host's chunk is the runtime (element, runner, every renderer not split out, validation and styles.ts) plus its one manifest. The bench's boot is that plus the bench's page wiring, and the paragraphs below that follow boot describe the bench's chunk, which was the published figure when they were written.

The chart renderer is split out, which issue #12 asked about and #76 measured before it was taken. The shape is the one that keeps render synchronous, and that property is what made it worth doing: <tool-host> awaits the chunk inside #prepare when the manifest declares series in kinds, so a seeded card can paint a chart with no chance to await and every later draw is synchronous too. A tool that returns a series without declaring it gets a redraw one frame later rather than a wrong answer, because render throws a specific error that #draw catches. Measured: boot 20,360 to 18,988, a saving of 1,372 bytes, against a 1,970 byte chart chunk that only a page with a chart tool downloads. The ceiling came down from 20,500 to 19,500 with it, because a ceiling that only ever rises stops being a constraint.

That chunk now measures 19,637 bytes against a 20,000 byte ceiling, leaving 363 bytes. So the next thing that costs real bytes buys them explicitly, by raising the budget in the commit that spends it and moving this table with it. The chart split was the obvious lever and it has been taken, so the next one will have to be argued rather than reached for.

Worth following the last two changes together, because in isolation each looks like the figure going the wrong way. Splitting the chart renderer took boot from 20,360 to 18,988 and the ceiling from 20,500 to 19,500. Honouring status then cost 649 bytes, 137 over that new ceiling, so the ceiling went to 20,000. Net: the figure a consumer reads is lower than before either change, 19,637 against 20,360, while the runtime gained a feature. That is the shape to aim for, and it only worked because the saving was measured before it was spent.

Measured on this tree rather than remembered, since several of these figures have been wrong at some point: the host values and run() API cost 567 gzipped bytes, the highlight hook 131, and honouring status 649. The demo highlighter is not in boot at all: it is a bench-highlight chunk of 668 bytes, alongside bench-fixtures at 1,267 and the chart renderer at 1,970. The first two are deployOnly because a consumer downloads neither; the chart chunk is not, because a reader of a chart tool does.

The host values and run() API cost 567 of those bytes and the highlight hook 131, measured after the bench fixtures were split out, which leaves 140 bytes free. That is not headroom, and the levers left are narrow: the fixture split is already spent, and §13's chart-renderer split was measured and argued against separately. The next change that costs real bytes should expect to raise the ceiling and say what bought them.

What the figure deliberately excludes. A test instrument that no consumer ships does not belong in a number a consumer reads, so two of them are split into their own chunks with their own budget lines: the bench's theme switch, and the fixture manifests under bench/fixtures. The second was not obvious. Manifests are collected with an eager glob, which inlines each tool.json into whichever chunk imports it, and registry.ts is reachable from every entry, so a fixture whose whole purpose is to fail on purpose was compiling into both boot and the worker bundle: 340 and 334 bytes for one fixture, at a point when the ceiling had 498 bytes left and three more fixtures were queued in open pull requests. Splitting them did not make the bench download less. It made the published figure answer the question it claims to answer, because a consumer running the tool-directory harness over their own tools has no bench/fixtures at all.

One duplication to know about. Vite builds a worker in a separate Rollup pass, so every tool reachable from the worker is emitted twice: tool-<id> for the main thread and worker-tool-<id> for the worker. A reader downloads one of the two, because a tool declares one thread, so the cost is deploy bytes rather than transfer bytes. It is named rather than hidden: before those names existed both copies were called index-*.js, and the second copy was invisible.

14. Defect register

Every entry is a real defect found while building this, and the guard that now stops it recurring. It is here because the guards look arbitrary without it.

15. Limitations

Known and accepted, with what each costs.

• The contract is pure functions only, at version 3. No file input, no network. A converter or an API client needs version 4 or 5. Both real version bumps so far have held the policy: every migration half is the identity function, and no existing tool's fixtures changed for either.
• No sandbox. A tool runs with the page's privileges. Worker mode isolates the thread, not the origin: it shares cookies and does not inherit the page's Content-Security-Policy. Fine for code you wrote; not fine for code you did not.
• No bundled syntax highlighter. code results stay readable as preformatted text with data-lang set. A host that already has a highlighter passes highlight to defineToolHost. The hook returns a Node, not a string, so the runtime never assigns innerHTML. Bundling one would roughly double the runtime. The same hook paints a textarea that declares lang. A tool cannot ship its own grammar yet, so a tool's own language is painted only on a host whose highlighter knows it.
• A painted textarea lags the typing by a pause. It shows plain text from the first key until 250 ms after the last, because repainting on every key cost Chrome its undo grouping. Past 20,000 characters it is not painted at all.
• The chart is deliberately simple. No tooltips, no zoom, no time axis. It draws what Chart describes.
• RegistrySource validates every manifest at once. All of them, on the first list(), so a malformed tool rejects that call naming the field. Measured 2026-09-19 with pnpm measure:registry on Node v24.21.0 (linux x64, Intel Xeon): 500 synthetic current-contract manifests, two inputs each. Three consecutive process runs, 21 timed constructions after 5 warmups, had medians of 0.71 ms, 0.98 ms and 0.77 ms (lowest sample 0.47 ms, highest 1.96 ms). Re-measured 2026-10-26 on Node v26.7.0, Apple M3 Pro, when validation moved into list() (#259): the constructor before the move had medians of 2.31, 2.30 and 2.15 ms in three runs, and the constructor plus the first list() after it 2.06, 2.05 and 2.01 ms (lowest sample 1.82 ms, highest 2.75 ms). A review on the same chip had measured 0.33 ms; what grew it since was not measured. Still a few milliseconds at several hundred tools, so validation stays all-at-once. Revisit if a host is in the thousands. The script times construction and the first list(), after a warmup that has already fetched the validation module. Generation sits outside the timed region, and so does JSON.parse: a host already has objects when it constructs the source.
• A page whose tool draws a chart downloads one extra chunk. The series renderer is no longer in boot: it is about 2.0 KB fetched only when a manifest declares series. render stays synchronous, because <tool-host> awaits the chunk during #prepare rather than at first use. The cost is one request for chart tools; the saving is 1,372 bytes for every page without one. See §13.
• Node 24 or newer for development. Tools and tests run as TypeScript with no build step, which is worth the floor.

16. Contributing

16.1 Getting set up

git clone <this repository>cd toolbenchpnpm installpnpm check      # should be green before you change anythingpnpm bench      # http://localhost:5180

16.2 The loop

1. pnpm check while working. It is typecheck plus every Node test, and it takes about a second.
2. pnpm bench and look at the thing. Every visual defect in §14 was found by looking, not by reasoning.
3. pnpm bench:build && pnpm test:bench before opening a pull request. The browser tests run against the built bundle, and that difference has caught real bugs.

16.3 Standards

• Every behaviour change needs a test in the layer that can see it. A rendering change belongs in the browser suite; a contract change belongs in the SDK suite.
• Comments explain why, not what. If a line looks arbitrary, say what went wrong without it. The ⚠️ marker is for a trap someone could reasonably fall into again.
• No new dependencies in packages/ without saying in the pull request what it buys and what it costs in transfer size. Both packages have zero dependencies today, and that is a feature.
• Accessibility is part of the change, not a follow-up. A new input type needs its label, its description target and its keyboard behaviour in the same commit.
• Additive changes only, for anything in the contract. See versioning.md.

16.4 Pull request checklist

• pnpm check passes
• pnpm bench:build && pnpm test:bench passes
• Every existing tool's fixtures pass without being edited. If one needed editing, explain why the change is not a breaking one
• New behaviour has a test in the right layer
• Contract change: version bumped, migration added, changelog line written
• Public API change: README and the relevant doc updated in the same commit
• Transfer size checked if packages/runtime grew: pnpm bench:build && pnpm size, which measures the host page in bench/host/ and the bench's boot chunk

16.5 Commit messages

A subject line that says what changed, then a body that says why, in prose. If the change came from a defect, describe the defect concretely: what was on screen, what should have been. Those descriptions are what §14 is built from, and they are the most useful thing in the history six months later.

16.6 Releasing

Both packages are versioned together, because the runtime's renderers and the SDK's types are two halves of one contract. Publishing is pnpm build then pnpm publish per package. The contract version in version.ts moves independently, and much more slowly, than the package version.

17. Appendix

17.1 Worker message reference

Page to worker:

Worker to page:

Aborts are not acknowledged. The page already knows, and the worker's own abort is swallowed rather than reported as a crash.

17.2 Theming reference

Set any of these on tool-host. Every one has a light fallback followed by a light-dark() declaration, so browsers without that function get the light palette rather than nothing.

17.3 Public API index

@toolbench/sdk

@toolbench/runtime