The string has more than one ruler

The string has brought several rulers to the measurement. Each ruler produces a perfectly reasonable answer to a slightly different question.

Ask which length you mean

An ASCII letter can make the distinction invisible. A Greek letter is a more helpful test subject:

import pooka/bit_array
import pooka/string

main() {
  let text = "α"
  assert string.length(text) == 1
  assert bit_array.byte_size(bit_array.from_string(text)) == 2
}

The UTF-8 representation needs two bytes. That does not mean the text contains two letters.

For network packets and byte offsets, the byte count matters. For a string operation describing text, treating each byte as an independent character would manufacture pieces that were never valid text in the first place.

A larger symbol is still text

Use more than ASCII when testing the boundary:

import pooka/bit_array
import pooka/string

main() {
  let text = "α🐇"
  assert string.length(text) == 2
  let bytes = bit_array.from_string(text)
  assert bit_array.byte_size(bytes) == 6
  assert bit_array.to_string(bytes) == Ok(text)
}

The rabbit occupies four UTF-8 bytes. The round trip checks that converting to bytes and back preserves the original text rather than merely producing something printable.

Do not turn a small example into a universal rule

These examples avoid another distinction: a visible character can be made from several Unicode code points. Combining marks and emoji sequences mean that counting code points is not always counting the units a reader perceives on screen.

In this library, string.length counts Unicode code points. A combining mark therefore counts separately:

import pooka/string

main() {
  assert string.length("e\u{301}") == 2
  assert string.length("é") == 1
}

The two spellings can look like the same accented letter while having different code-point sequences. A byte-oriented slice also needs to consider what happens when its boundary lands inside an encoded character.

This is relevant to the compiler itself. Source spans can use byte offsets internally, while diagnostics need to place underlines in displayed source. Confusing the measurements points the complaint at an innocent neighbour.

The rulers are useful. Trouble begins when one is handed to every caller without asking what they were trying to measure.

— Peregrine Finch