Skip to content

The Conformance Suite

kBars ships a language-agnostic conformance suite in the conformance/ directory of the repository. Each fixture file is a versioned JSON envelope wrapping an array of test cases:

{ "schema": "https://bars.commonsware.com/conformance/schema/render-fixture.schema.json", "version": "0.1.0", "cases": [ … ] }

A conformance runner in any language can deserialize these fixtures and check its own implementation without reading a line of Kotlin — they describe what the parser and renderer must do, not how the reference implementation does it.

The reference implementation dogfoods the suite: the kbars CLI's conformance subcommand runs every fixture directory against the live rendering engine, and :kbars / :kbars-conformance drive their own test suites from the same fixtures.

Three artifacts together describe Bars behavior, and each has a distinct role: the Bars specification is normative — it defines what a conforming implementation must do; this conformance suite is the executable test of conformance — passing it is the operational definition of "conforms"; the reference ANTLR grammar is informative — it is what the reference implementation happens to use. Where the specification and the suite disagree, that is a defect to be reported and fixed, not a case where one silently overrides the other.

If you are writing a workalike in another language, read the Bars specification for the normative rules and use these fixtures as your acceptance test. This page describes the fixtures' structure; the workalike guide covers the Kotlin-specific idioms you will need to map, and Fixture Schemas is the authoritative per-directory field reference. For a browsable, human-readable view of the fixtures' successful cases, see the Conformance Examples gallery — it is generated directly from the fixtures, so the fixtures themselves remain the normative source of truth.

Directory layout

conformance/
  ERROR-CODES.md                      Registry of all KB-NNNN error codes
  error-codes.json                    Machine-readable companion to ERROR-CODES.md
  manifest.json                       Generated inventory of every fixture file below — do not hand-edit
  schema/
    grammar-fixture.schema.json       JSON Schema for grammar fixture files
    prepass-fixture.schema.json       JSON Schema for pre-pass fixture files
    render-fixture.schema.json        JSON Schema for render fixture files
    partials-fixture.schema.json      JSON Schema for partials fixture files
    env-property-fixture.schema.json  JSON Schema for environment-property fixture files
    error-codes.schema.json           JSON Schema for error-codes.json
    manifest.schema.json              JSON Schema for manifest.json
    render-report.schema.json         JSON Schema for `kbars render --json` output
    full-report.schema.json           JSON Schema for `kbars conformance --full-report` output
  grammar/                            Grammar parse fixtures (one file per feature group)
  prepass/                            Pre-pass normalization fixtures (one file per scenario group)
  render/                             Render output fixtures (one file per segment/behaviour group)
  partials/                           Partials fixtures ({{> key }} rendering, failures, template stacks)
  transforms/                         Render fixtures specifically exercising built-in transforms
  env-properties/                     Render fixtures exercising the @env namespace (clock + properties)

Files in render/ and transforms/ share the render-fixture.schema.json schema because they use the same fixture shape. transforms/ is kept as its own directory rather than folded into render/ because the transform roster is the standard library — see Transforms and @env are the standard library.

The manifest

conformance/manifest.json is a generated, out-of-band integrity index of the suite — a suite version plus, per fixture file (including error-codes.json), its version, case count, and conformance tier:

{
  "manifestFormat": "1.1",
  "suiteVersion": "0.1.0",
  "files": [
    { "path": "error-codes.json", "version": "0.1.0", "caseCount": 19, "tier": "core" },
    { "path": "render/with-block.json", "version": "0.1.0", "caseCount": 6, "tier": "core" },
    { "path": "transforms/upcase.json", "version": "0.1.0", "caseCount": 3, "tier": "full" }
  ]
}

path is relative to conformance/ and includes the subdirectory; files is sorted by path. tier is "core" for the template-system fixtures and "full" for the standard-library fixtures (transforms/ and env-properties/), so a runner targeting only core-tier conformance can skip the "full" entries. It exists to guard against a runner's silent green on nothing failure mode: a runner that globs fixture files and loads fewer than expected cannot detect the shortfall on its own, because a file it never opened cannot assert anything about itself. The manifest lives outside the glob and answers "did I see everything I was supposed to?"

Do not hand-edit manifest.json. It is generated from the fixture files themselves:

./gradlew generateConformanceManifest   # regenerate conformance/manifest.json
./gradlew checkConformanceManifest      # verify it matches the fixture files (wired into `check`)

checkConformanceManifest fails the build with a "run generateConformanceManifest and commit" message when the committed manifest is stale — for example after adding, removing, or editing a fixture file without regenerating it. A companion manifest.schema.json documents the envelope shape for schema-driven deserialization, alongside the fixture schemas.

If you are writing a runner, treat the manifest as a read-only guard: after discovering fixture files under a directory, compare your own { path → caseCount } inventory against manifest.json's files and treat any missing file, unexpected file, or case-count mismatch as a suite-integrity failure distinct from an ordinary case failure.

Reporting the targeted suite version

A build of the kbars library publicly reports the conformance-suite version it targets via com.commonsware.kbars.api.KbConformance.SUITE_VERSION, a public const val String (for example, "0.1.0"). It is generated from the same conformanceSuite catalog entry that produces this manifest's suiteVersion, and a parity test in kbars-conformance guards the two against drift — so a downstream consumer can ask the published artifact which suite version it was built against without inspecting manifest.json directly. A workalike implementation should expose an equivalent publicly-readable constant, string, or function reporting the suite version it targets.

Running the suite

A black-box runner ships with the kBars CLI. It dispatches each fixture file by its envelope's schema field, so all six fixture directories — render/, transforms/, partials/, env-properties/, grammar/, and prepass/ — run through the same command:

kbars conformance              # defaults to ./conformance/render/
kbars conformance some/file.json some/other/dir
kbars conformance conformance/ # runs the whole suite: all six shapes

A file whose schema does not match any known fixture schema (for example manifest.json or error-codes.json) contributes zero cases and is skipped rather than causing an error.

Failures print the fixture name, source file, expected output, and actual output to stderr; a --full-report[=PATH] mode emits a stable JSON report instead. See the CLI Reference for both output modes and the exit-code contract, and Claiming Conformance for how that report doubles as the verifiable artifact behind a public conformance claim.

Writing a runner

A minimal runner, per fixture category:

  1. Deserialize the fixture JSON into typed objects using the fixture schemas.
  2. Build the input (parse the template source, build the context from the JSON, …).
  3. Execute the operation under test (parse, pre-pass normalize, or render).
  4. Compare against the expect field; report pass/fail using the fixture name as the test-case id.
  5. When the fixture carries expectedSoftFails, also compare the set of soft-fail codes your implementation reported during the render — see Error Codes.

The name field is a stable, kebab-case identifier suitable as a test name in any framework.

Next: Writing a Workalike covers the Kotlin idioms you will meet in the reference and how to map them into your target language. When a question comes up mid-port that no fixture answers, Using the Reference CLI as an Oracle covers asking the reference implementation directly instead of guessing. Once your port is green against the fixtures, see Claiming Conformance for how to publish a verifiable proof of that.

The suite is also published as a versioned, downloadable ZIP — see Distribution — with its own Changelog tracking suite-level changes independently of the library.