The Phase 0 benchmark

The measurement the whole thesis rests on: what it measures, why one input to it is held out, and the dependency rule that keeps the public project forkable. It has not run.

The benchmark has not run. No arm of it — frontier or small-model — has produced a number. The harness mechanics are unspecified past the decisions on this page. Everything below is design, and the kill criterion has therefore never had the chance to fire.

What it measures

An agent authors N equivalent components in Mzizi’s syntax, and the same components in raw Dioxus and Leptos under the same harness. Three metrics:

Metric	Which failure modes it tests
Tokens consumed	FM-6 (ceremony) and FM-8 (formatting entropy)
Iterations to a clean compile	FM-2 (anonymous delimiters) and FM-5 (round-trip starvation)
Defect rate — compiles but wrong	FM-1, FM-3, FM-7, and contracts being in-language

The stopping rule, from RFC-0001 §6:

If Mzizi doesn’t beat both on at least two of three metrics, the thesis is wrong and Phase 1 does not start.

And from RFC-0002 §5, an addition that matters because the whole design target changed: the benchmark must include a small open-weight model arm. The frontier arm alone cannot validate the thesis, because the frontier model is the one the design helps least.

The corpus, and what counts as a defect

Both resolved in the charter (§6, dated 2026-08-23).

Task set: Mzizi’s own components. The registry’s 571-plus components, partially ported to Rust, with the existing .tsx and .rs implementations as ground truth. Explicitly not a port of an external library — not shadcn, not a generic primitive set. The nine primitives and the one worked example in the repository are hand ports from exactly this corpus.

A defect is code that compiles cleanly but is behaviourally wrong. It passes the compiler and fails a contract or behaviour test against the reference implementation. This mirrors the contract-test pattern already used for the .tsx → .rs ports: the reference is read from disk, and disagreement is the new code’s fault unless it is a documented, deliberate divergence.

Note what is not a defect: a syntax or compile error. That is the normal friction the compile-error-density design goal is trying to minimise. The defect rate measures what gets past the compiler wrong.

Why one input is held out

This is RFC-0004, and it opens by correcting its own premise rather than building on it.

The prompt for that RFC was SQLite — public-domain code, proprietary TH3 test harness, and the intuition that the split makes a project harder to attack. Two corrections:

SQLite’s split is commercial, not defensive. TH3 is proprietary because it is sold and carries DO-178B avionics certification evidence with its own licensing constraints. SQLite simultaneously ships a very large public test suite.

Private tests are close to worthless as a security control. An attacker has the source, a fuzzer and symbolic execution. A test suite mostly documents what a project already handles correctly — one of the least useful artifacts an attacker could be handed. Meanwhile hiding it costs real things: contributors cannot verify their own work, and coverage gaps become invisible to the people best placed to point them out. “Security that depends on the mechanism being secret is the failure mode Kerckhoffs named in 1883.”

So no test in Mzizi is private because privacy makes it stronger. One reason is decisive:

Benchmark contamination

If the task set and its expected outputs are public, they get scraped into training data. After that the benchmark measures memorisation, not the language, and reports a flattering number for exactly the wrong reason. This is not hypothetical — it is what happened to GSM8K, HumanEval and most public LLM benchmarks.

It is also the one failure that cannot be detected from inside: a contaminated benchmark looks like a successful one. The charter gives Phase 0 a kill criterion, and a kill criterion that cannot fire is not a criterion. A held-out set is therefore a correctness requirement, not a secrecy preference.

Three narrower reasons also qualify: fuzzing seed corpora (a curated set of inputs that once crashed the compiler is a head start against unpatched forks — but never the fix or the regression test, both of which stay public); embargo windows, which are temporary by design; and certification evidence, which is not applicable today and is named only so the boundary is already drawn.

The split

Public — everything below

The language, compiler, primitives and RFCs. The entire correctness suite — unit tests, contract tests, parse gates, measured IR properties — permanently. The benchmark harness: runner, metric definitions, scoring code. A published fixture format, so anyone can write their own task set and run it.

Private — a small annex

The held-out task set and its expected outputs. Fuzzing seed corpora. Embargoed security tests, for the length of the embargo only.

RFC-0004 states the ratio plainly: “this is a small private annex to a large public suite, not a public shell around private testing. If someone cannot verify Mzizi’s correctness from the public repository alone, the split has been drawn wrongly.”

The dependency rule

Private consumes public. Public never consumes private.

Public CI must be completely self-contained: a fork with no secrets, no access and no relationship to the private repository must be able to run the full public suite and get a green result. The moment public CI needs a private token, every outside contributor’s CI fails and the project is open source in name only.

Three consequences the RFC calls non-negotiable:

1. A missing private result is neutral, never failure. On a fork PR, a Dependabot PR, whenever the secret is absent, the check reports “skipped”. “A red X that an outside contributor is structurally unable to turn green is a wall, not a gate.”
2. The private result is advisory to outsiders, blocking only for maintainers. Branch protection may require it on main; it must not be required to propose a change.
3. A private failure must be reportable in public without leaking the test. The check reports the shape of the failure — which metric regressed, by how much, against which component — not the task input.

The mechanism, and its current state

The public half exists and is inert. .github/workflows/mzizi-lang-benchmark-dispatch.yml POSTs a repository_dispatch to the private repository carrying four public facts: commit SHA, ref, source repository, run id.

It is inert unless two things are configured — the MZIZI_HELDOUT_REPO repository variable and the MZIZI_DISPATCH_TOKEN secret (contents:write on the private repository, nothing else). Neither exists, so today the job runs and reports “not configured”. That is the intended steady state, not a failure. It is additionally guarded on github.repository so a fork skips it, and it never fails the build — a dispatch error is a warning, because a missed notification does not mean the commit is bad.

The companion rule: the private runner must also poll for public commits it has not measured. The dispatch is a latency optimisation; the poll is the correctness guarantee.

The private repository does not exist, deliberately

RFC-0004 settles where it will live — mzizi-dev, so read access to the held-out set is governed by one org’s membership, which is the Mzizi maintainer set and nothing wider — and then argues it should not be created yet, for a design reason rather than a scheduling one:

Its whole job is to run the public harness against a private input, so if it exists before the harness does, the harness ends up shaped around the private runner — the exact inversion §3 forbids.

Plus an argument about incentives worth quoting, because it is the kind of thing usually left unsaid:

An empty private repository is an attractive nuisance. Today every test in this project is public, which is correct. The moment the repository exists, the marginal cost of filing a test there drops to zero; each individual “this one is easier to keep private” is defensible, and the aggregate is the public-shell-around-private-testing outcome §2 rules out.

Trigger: the first held-out task. Not before.

What this deliberately does not claim

• It does not make Mzizi harder to attack. Security comes from the public suite, the fuzzing, the -D warnings gate and review — all public.
• It does not hide the compiler’s behaviour. Every assertion about what Mzizi does is public; only a held-out measurement of how well an agent uses it is not.
• It is not permanent for security tests. Embargoed tests move to public at disclosure, and a test still private after its embargo has expired is a bug in the process.

Still unsolved

Held-out set rotation. A held-out set leaks slowly through published results and needs a refresh policy — probably a fraction rotated per reported run.

Third-party verification. If an outside party needs to reproduce a benchmark claim there has to be a path, most likely a time-limited grant under an agreement not to publish. “Worth solving before any number is published.”