Skip to content

String Semantics

The normative rule is in the Bars specification; this page covers how it appears in fixtures and in the reference implementation's transforms.

Four string transforms count string length, and one list transform orders strings for sort. All five measure and compare in Unicode code points, matching the code-point rule for Source Locations — not UTF-16 code units, and not raw bytes.

Counting transforms

size (on a string input), slice, truncate, and split with an empty delimiter ("") all count in code points:

  • size returns the number of code points in the string, not the number of UTF-16 Chars.
  • slice takes its offset and length in code points, so a surrogate pair is always sliced out whole — it can never be split into a lone half.
  • truncate takes its length (and its ellipsis's length) in code points, for the same reason.
  • split: "" explodes the string into one list element per code point, so a surrogate pair becomes a single element, not two.

A character outside the Basic Multilingual Plane (for example, an emoji) is one code point even though it occupies two UTF-16 Chars in a Kotlin String. A port whose native string type is UTF-16-backed (Java, C#, JavaScript) or UTF-8-backed (Rust, Go, Swift's String internals) must count in code points explicitly for these four transforms — the host string type's native "length" or index operations typically count code units or bytes, not code points.

The lone-surrogate rule

An unpaired (lone) surrogate — a high surrogate with no following low surrogate, or a low surrogate with no preceding high surrogate — counts as its own single code-point-sized unit and passes through unchanged. It is never treated as zero-width, never merged with a neighboring Char, and never normalized to U+FFFD (the Unicode replacement character). Only a well-formed high+low surrogate pair is treated as one two-Char unit; everything else, including a lone surrogate, is one unit per Char.

What is unaffected

remove, remove_first, remove_last, replace, replace_first, and replace_last all locate a whole target substring (via indexOf/lastIndexOf) and act on the substring's real length, so their boundaries always align to a real character regardless of whether the counting unit is a code point or a UTF-16 code unit. These six transforms are unaffected by the code-point rule and require no special handling in a port.

Sort collation

sort's string branch orders by Unicode code-point value, not by String.compareTo's raw UTF-16 code-unit comparison. This matters because UTF-16 surrogate pairs for supplementary-plane characters (U+10000 and above) start at 0xD800, which is numerically less than Basic Multilingual Plane characters in the U+E000–U+FFFF range. A naive UTF-16 comparison therefore sorts a supplementary- plane character (e.g. an emoji, U+1F600) before a private-use-area BMP character (e.g. U+E000) — the opposite of true code-point order, where U+E000 sorts first. A port must decode surrogate pairs to their scalar value before comparing, not compare the raw UTF-16 units.

Case mapping

upcase and downcase use Unicode default case mapping (locale-invariant), which is what Kotlin's String.uppercase()/String.lowercase() (with no locale argument) already provide — no code change was needed to make these two transforms conform. Default case mapping is not always length-preserving: the German sharp s (ß, one code point) upcases to "SS" (two code points). A port that assumes uppercasing/downcasing never changes a string's code-point count will miscount or mis-slice a case-mapped result derived from upcase/downcase output.

Fixtures

  • conformance/transforms/transform-size.json — size-astral-string-counts-code-points, size-lone-surrogate-counts-as-one.
  • conformance/transforms/transform-slice.json — slice-astral-does-not-split-surrogate.
  • conformance/transforms/transform-truncate.json — truncate-astral-counts-code-points.
  • conformance/transforms/transform-split.json — split-empty-delimiter-astral.
  • conformance/transforms/transform-sort.json — sort-strings-code-point-order.
  • conformance/transforms/transform-upcase.json — upcase-sharp-s-expands.
  • conformance/transforms/transform-downcase.json — downcase-greek-capital.