Gleam gives every token a proper name

The compiler’s desk is accumulating little scraps of paper. Some say “integer”. Some contain an integer. These are surprisingly different scraps.

Give the alternatives a shape

In Gleam, a custom type can say exactly which token shapes exist. Here is a deliberately small illustration, rather than Pooka’s real token definition:

pub type Token {
  Integer(Int)
  Plus
  LeftParen
  RightParen
}

pub fn kind(token: Token) -> String {
  case token {
    Integer(_) -> "integer"
    Plus -> "addition"
    LeftParen -> "opening bracket"
    RightParen -> "closing bracket"
  }
}

pub fn main() {
  let assert "integer" = kind(Integer(42))
  let assert "addition" = kind(Plus)
}

Integer carries a value. Plus does not need one. The reader of a token can inspect the alternatives without asking whether some unrelated string field happens to contain a number today.

Let the implementation language help

Suppose another token shape is added. Readers with exhaustive matches must account for it. This does not finish the lexer for us, but it makes unfinished work easier to locate.

source characters
    -> a known token constructor
        -> a parser decision about that constructor

A spelling mistake in a constructor is an implementation error the Gleam compiler can reject. A spelling mistake in an arbitrary string may remain a perfectly legal string while making the parser quietly stop recognising anything.

That distinction saves a good deal of searching underneath the desk.

Keep the boundary honest

An algebraic data type describes the token’s shape. It does not prove the lexer chose the right token or recorded the right location. Those still need tests against source text.

Nor does putting the entire compiler into one large type make it clearer. Tokens, syntax expressions and inferred types answer different questions. Giving each phase its own useful vocabulary keeps an accidental shortcut from becoming everybody’s permanent responsibility.

I like this arrangement because the compiler implementation can ask for help before it has learned to help its own users.

The scraps now have proper names. We are still responsible for putting the correct things on them.

— Peregrine Finch