11 KiB

Raw Blame History Unescape Escape

Permutation runner

Automated CI tool that cycles soma components through their declared state space and validates morfo compliance at every step.

Why this exists. morfo:check validates the DOM at page-load time, in a single state. smoke catches pageerror during mount. Neither ejercices the transitions between states — and that's where most of the reactivity bugs we've caught this quarter actually live (toolbar A35, form A36, slider RTL transform). The permutation runner is the third validation layer that closes that gap.

What it validates

For every demo page, for every permutation the page declares:

No reactivity loops. effect_update_depth_exceeded AND microtask-mediated loops (A35 / A36) both count as failures. A long microtask starvation (>2 s) is the proxy for A36.
No runtime errors. pageerror or console.error during a state transition fails the permutation.
Morfo contract holds. After the transition settles, the emitted DOM still conforms to the component's morfo (same rules as morfo:check — declared attrs present, values in enum, no undeclared namespaced attrs).
ARIA relationships remain valid. If a part emits aria-labelledby="X", the element with id X must exist in the DOM at that state.

What it does not validate (future work):

Visual correctness (that's eidos' problem).
Animation timing or data-last-action → data-state ordering (dedicated MutationObserver test).
Cross-component interactions (it runs one demo at a time).

Convention: `data-perm-*` attributes

Demo pages opt in by tagging interactive controls with data-perm-* attrs. The runner discovers these, clicks them in sequence, and validates after each click.

`data-perm-step="N"`

Declares the click order. Starts at 0. The runner clicks elements in ascending order.

<button data-perm-step="0" onclick={() => dialogOpen = true}>Open</button>
<button data-perm-step="1" onclick={() => dialogOpen = false}>Close</button>

`data-perm-label="..."`

Human-readable label for the log. Defaults to the element's text content.

<button data-perm-step="0" data-perm-label="open dialog via trigger">
  Open
</button>

`data-perm-settle="ms"`

Override the default 500ms settle time before morfo re-validation. Use for animations, async validation, etc.

<button data-perm-step="0" data-perm-settle="800">
  Submit (async)
</button>

`data-perm-skip-validate`

Click the element but don't validate afterwards. For intermediate actions (e.g. typing a value before submit). The next step validates.

`data-perm-mode`

Overrides the default click interaction:

click (default) — await page.click(el).
focus — await el.focus().
type="abc" — type the string.
key="Escape" — dispatch a keydown with the given key.

<input data-perm-step="0" data-perm-mode='type="ada@example.com"' />
<div data-perm-step="1" data-perm-mode='key="Tab"' />

Example: Dialog demo

Minimal instrumentation that cycles open → close → open via Escape:

<Dialog.Trigger data-perm-step="0" data-perm-label="open via trigger">
  Open
</Dialog.Trigger>

{#if dialogOpen}
  <Dialog.Content data-perm-step="1" data-perm-mode='key="Escape"' data-perm-label="close via Escape">
    …
  </Dialog.Content>
{/if}

<button data-perm-step="2" data-perm-label="re-open">Open</button>

Runner output:

PASS  dialog   3 permutations
  ·  step 0 (open via trigger)          morfo OK, 0 errors
  ·  step 1 (close via Escape)          morfo OK, 0 errors
  ·  step 2 (re-open)                   morfo OK, 0 errors

Failure format

FAIL  toolbar  3 permutations, 1 failed
  ·  step 0 (focus first button)        morfo OK
  ·  step 1 (type "b")                  PAGEERR: effect_update_depth_exceeded
  ·  step 2 (click toggle)              skipped (previous step failed)

CLI

# Requires `npm run dev` running (auto-detects port 5173–5180).
npm run perm:check

The current docs-shell route map is /uix/components/{kebab}. Components without a routed page are reported as SKIP no route; pages that exist but cannot be loaded are reported as SKIP no demo load; routed pages without data-perm-step or perm-axes opt-in are reported as SKIP no perm instrumentation. Override the prefix with PERM_ROUTE_PREFIX=/some/path when testing another shell.

Exit codes:

0 — every instrumented routed demo passes all permutations. If no routed demo is instrumented yet, the runner exits cleanly and says so.
1 — at least one permutation failed.
2 — no dev server / no morfos found.

Demos without data-perm-step / perm-axes are skipped with a SKIP line (they haven't opted in yet).

v2 — URL-driven state injection

v1 covers "click through a state sequence and validate after each step." v2 adds URL-seeded initial state: the demo reads ?perm.X=Y params from the URL and uses them as initial $state values. Every instrumented demo runs at least once on the baseline URL. Extra URL variants run only when the page declares the axis with <meta name="perm-axes">; this keeps coverage honest while the new docs shell is being instrumented.

Layout plumbing

The old /test/soma shell used a shared _perms.svelte.ts context. That shell was retired; the current runner only provides the URL variants and expects each routed page under web/routes to read ?perm.* params directly or through a new shell-level helper when we decide to add one. Until a page does that and declares perm-axes / data-perm-step, it is skipped rather than counted as coverage.

Demo opt-in

Read the perms and use them as initial state (not live-bound — the user can still click through transitions after landing):

<script lang="ts">
  import { page } from '$app/state';
  const params = page.url.searchParams;

  // Perm-runner v2 seed — `?perm.open=true` / `?perm.variant=alertdialog`
  // land directly in that state without walking the click sequence.
  let open = $state(params.get('perm.open') === 'true');
  let variant = $state((params.get('perm.variant') as 'default' | 'alertdialog') ?? 'default');
</script>

Type coercion is naive: 'true' / 'false' → boolean, bare digits → number, everything else stays a string. Cast to the component's actual union at the read site.

URL variants (v2 baseline + v3 axis-at-a-time)

The runner composes each demo's variant list from two sources:

Baseline — every instrumented demo runs once on its plain route.
Per-demo axes (v3) — the demo declares which axes it honours via <meta name="perm-axes" content="...">. The runner adds ONE variant per declared axis, flipping that axis from baseline while keeping everything else at default (linear, not cartesian — the matrix stays bounded).

Supported axes (v3 initial set):

Axis	Baseline	Flipped variant	Typical use
`dir`	route default	`rtl`	Components with direction-sensitive keyboard/layout behavior.
`orientation`	`horizontal`	`vertical`	Toolbars, Tabs, Splitters, RadioGroup, Slider.
`disabled`	unset	`true`	Any component with a `disabled` prop.
`readonly`	unset	`true`	Inputs, Editable, Calendar.
`invalid`	unset	`true`	Form-participating components.
`loading`	unset	`true`	Async components with pending states.

Demos opt in by listing the axes they honour. Meta-less demos still run the baseline transition cycle if they have data-perm-step annotations.

v3 opt-in — 3 steps

Add the meta tag (list only the axes your demo actually wires):

<svelte:head>
  <title>Tabs · Soma</title>
  <meta name="perm-axes" content="dir,orientation,disabled" />
</svelte:head>

Seed the $state from perms for each declared axis:

<script lang="ts">
  import { page } from '$app/state';
  const params = page.url.searchParams;

  let orientation = $state<'horizontal' | 'vertical'>(
    (params.get('perm.orientation') as 'horizontal' | 'vertical') ?? 'horizontal'
  );
  let disabled = $state(params.get('perm.disabled') === 'true');
</script>

Wire the state to the Provider:

<Tabs.Provider bind:value {orientation} {disabled}>
  ...
</Tabs.Provider>

The runner now visits /tabs?perm.orientation=vertical and /tabs?perm.disabled=true in addition to ltr / rtl, and re-runs the full data-perm-step click cycle at each URL.

Per-component value seeds (?perm.value=tab-2, ?perm.open=true, ?perm.format=bold,italic) are NOT enumerated by the runner — they're just defaults the demo honours when testing manually or via direct URL navigation. Adding them to the enumerated matrix is a v4 task (the runner would need to read the morfo's states + data.values to know which values are meaningful).

`[initial seed]` step

At each URL variant, the runner validates morfo BEFORE running any click step. That validation appears in the log as [initial seed] and catches morfo drift that exists at the URL-seeded state — exactly the gap a click-only cycle can't reach.

Roadmap

v1 (shipped 2026-04-22) — click-based state transitions, morfo re-validation after each step.
v2 (shipped 2026-04-22) — URL-seeded initial state, universal axes (dir, locale), [initial seed] validation step before clicks. Demo layout reads ?perm.* params, demos seed initial $state from them.
v3 (shipped 2026-04-22) — per-demo axis matrix via <meta name="perm-axes">. Runner adds one variant per declared axis (axis-at-a-time, not cartesian). Supported axes: dir (universal), orientation, disabled, readonly, invalid, loading.
v4 — morfo-inferred value cycling. Reads states[] + data.values from each morfo and enumerates meaningful value combinations automatically (would cover ?perm.value=tab-2 / ?perm.variant=alertdialog / ?perm.open=true without per-demo declaration).
v5 — integrate with MutationObserver to validate data-last-action / data-state ordering for Sema.

The convention (data-perm-step / data-perm-mode / perm.X=Y / perm-axes meta) is forward-compatible: v4+ infrastructure adds automation without invalidating earlier annotations.

morfo README — authoring reference for the morfos this runner validates against.
scripts/permutation-check.ts — the runner itself.
scripts/morfo-check.ts — single-state DOM validator that shares the per-morfo validation logic.

11 KiB Raw Blame History Unescape Escape

Permutation runner

What it validates

Convention: data-perm-* attributes

data-perm-step="N"

data-perm-label="..."

data-perm-settle="ms"

data-perm-skip-validate

data-perm-mode