Run the compatibility kit against your implementation

Before you start

Build a validator that reads a glossary document and an atlas document.

Give the validator a way to report each finding as data. Include an obligation name, a document name, and a path in that data.

Keep your message wording outside that data. The kit compares no wording.

Run the cases

1. Copy the tck/ directory from the specification repository into your project.
2. Read tck/manifest.yaml.
3. Take the case list from the obligations block of the manifest.
4. Open the directory tck/<case> for the first case.
5. Load every document the case lists under inputs.
6. Load those documents together, as one input set.
7. If the case directory holds config.yaml, read the config values from that file.
8. If the case directory holds no config.yaml, use the defaults the specification declares.
9. Validate the input set.
10. Produce one verdict and one set of findings.
11. Read expected.yaml from the case directory.
12. Compare your verdict against the conformant value.
13. Compare your findings against the findings list as a set.
14. Ignore order. Order is not part of the expectation.
15. Compare the obligation, the document, and the path of each finding.
16. Compare nothing else.
17. If the case carries a navigation block, answer every question in that block.
18. If the case carries an exposure block, read each named item of each named concept.
19. Record the case as passed when the verdict, the finding set, and the answers all agree.
20. Repeat steps 4 to 19 for every case the manifest names.

The kit passes when every case in the manifest passes.

Use the obligation name the kit gives

Report the obligation name exactly as the specification and the manifest write it.

Do not invent your own identifier for an obligation.

Name an invariant as the specification names it, such as AConceptHasOnePreferredLabelPerLanguage.

Name every other obligation as the manifest writes it, such as entity-optional.Mapping.note.

Do not assume the obligation of a finding matches the directory name of the case. A case under config-default.minimum_definitions_per_concept raises the minimum. It then expects the invariant AConceptCarriesADefinition to fire.

Match the path conventions

Write each path from the root of the document it belongs to.

Write $ for the document itself. Write $.concepts[2].prefLabels for a field of the third concept, counting from zero.

Report an invariant over a pair of concepts against the later of the two, in document order.

Report an invariant over a pair of glossaries against the later of the two, in the load order of the manifest.

Report a missing, wrongly typed, or undeclared field under entity-fields.<TypeName>.

Report a wrongly shaped value inside an optional field under entity-optional.<Entity>.<field>.

Report a value outside a declared enum under enum-comparable.<EnumName>. Point the path at the field and not at the object.

Run both directions

Run the conformant cases and the violating cases.

Treat a case marked conformant: true as passed only when your validator reports no finding.

An implementation that rejects every document passes the violating half of the kit. It fails the conformant half.

Expect the same preferred label in two different glossaries to pass. Expect the same concept id in two different glossaries to pass.

Demonstrate the two obligations the kit omits

The manifest lists two obligations under notExpressibleAsDocuments. Answer both outside the case directories.

Demonstrate surface-actor.Glossary against your own interface. Show that your public surface offers the reader no create operation, no update operation, and no delete operation.

Demonstrate TheAtlasMarksTheSearchRoot against a filesystem you build yourself. Walk up from a working directory and stop at the nearest atlas. Raise an error on two atlas files in one directory.

Related

For the reasoning behind a kit that holds no code, see Why the compatibility kit holds no code.