Why the compatibility kit holds no code

The goal decides the form

The goal for implementations is a plain one. Anyone builds a ubilang equivalent in their own language from the specification, runs the tests, and demonstrates that the result is specification compliant. That goal names two parties who share nothing except the specification. Both parties must read whatever decides conformance between them.

The reference implementation uses Python, and its tests are Python unittest cases, one per obligation of the specification. A second implementation in Rust or in Kotlin cannot run them. If those tests stay the only suite, conformance rests on the ability to run Python. That is a fact about tooling and not about the specification.

Why a second suite is worse than no suite

The obvious answer is that each implementation writes its own suite from the specification. That answer has a failure that only appears later. Two suites written from one specification can disagree, and when they disagree there is no way to tell which one read the specification correctly.

At that point the specification is no longer the authority. The suites become the authority, and there are two of them. Every implementation that joins afterwards inherits the disagreement and picks a side.

The kit is input and an expected outcome

The kit is a set of case directories. Each case holds the documents that go in and a file that states what comes out. That file carries the verdict for the input and the findings a validator reports against it. There is no test framework in the kit, no runner, and no file that belongs to any one programming language.

The identifier that both sides share is the obligation name. An invariant carries the name the specification gives it, such as AHiddenLabelCarriesAReason. Every other obligation carries the identifier the specification’s own planning output assigns, such as entity-fields.Concept. Two implementations are comparable because they report the same name for the same departure. The kit compares no message wording, so an implementation keeps its own voice.

The three options the project rejected

The Python unittest cases as the only tests were the first option. The project rejected them because another language cannot run them, which is the whole problem.

Gherkin feature files with per-language step definitions were the second option. Gherkin looks language-neutral, and the step definitions are not: each implementation writes its own, and those step definitions re-implement the kit. The disagreement between two suites returns under a different name.

A shared HTTP test service was the third option. A service judges every implementation identically. It also makes the validation of one local file depend on a service and a network. An implementation that needs a network for a local document is harder to adopt than one that does not.

What the kit cannot cover

Two obligations of the specification have no case directory, and the kit’s manifest names both of them.

The first is the reader-facing surface’s actor rule. The specification declares one actor, a Reader, and declares no rule that changes a concept. The obligation is that an implementation offers no way to write: no create, no update, no delete. No input document witnesses the absence of a write operation, because absence is not a property a document has. An implementation demonstrates that obligation against its own interface.

The second is the atlas search root. The tool walks up from the working directory until it finds an atlas file. The walk stops at the home directory or at the root of the filesystem, and the nearest atlas wins. All of that happens before the tool reads any document. A case directory holds one atlas and states nothing about the walk, and a case with two atlas files tests the runner’s own directory layout instead of the specification.

The manifest names these two rather than fakes them, which is not an admission of a gap. A kit that fakes them with a case directory reports a pass for something it never examined. Such a pass is false in exactly the way a conformance suite exists to prevent.

Both directions are tested

Every obligation carries at least one input that satisfies it and at least one that does not. An implementation that rejects every document it reads passes a kit that holds only violations.

Several conformant cases carry the harder claim. The specification allows the same preferred label Mehrwertsteuer in a Sales glossary and in a Billing glossary at once, because across schemes the collision is expected. An implementation that flags it misreads the boundary, and only a conformant case catches that mistake.

The procedure for running the kit against an implementation is in Run the compatibility kit against your implementation.