A comma arrives without an appointment

The lexer has a modest desk. Characters arrive on one side. Tokens leave on the other. Occasionally a comma turns up and behaves as though everyone was expecting it.

What the lexer sees

Consider this small Pooka expression:

main() {
  let total = 1 + 2 * 3
  assert total == 7
}

At this stage, multiplication does not outrank addition. The lexer has no business deciding that. It recognises names, integers, operators and delimiters, and records where each one appeared.

The parser will arrange the arithmetic later. Sharing responsibilities sounds friendly until two compiler phases both believe they own the same decision.

Read far enough

input:   total == 7
tokens:  Name("total"), EqualEqual, Integer(7)

input:   total = 7
tokens:  Name("total"), Equal, Integer(7)

These token names illustrate the distinction; they are not a dump of the implementation’s constructors.

One decision the lexer really does own is how much text belongs to a token. = and == begin the same way. Reading the shorter one too eagerly turns equality into two unrelated surprises. A keyword has a similar boundary: let is special, but a longer identifier beginning with those letters must remain an identifier.

Strings are a less tidy delivery. Their quotes mark the boundary, escapes have meaning, and a missing closing quote should produce an error near the opening one. It should not turn the remaining file into a very large, very unfortunate name.

Ask the awkward questions

Tests help most when they ask impolite questions. What happens at the end of an empty file? What if a comment ends there? What if an unfamiliar character appears between two perfectly reasonable tokens?

I like testing both the token sequence and its locations. Getting the right words with the wrong addresses will still make the diagnostics wander into somebody else’s sentence.

It is not glamorous work. The lexer does not know what the program means. It makes the next person’s work possible by refusing to be vague about what it saw.

The comma can stay. We have found it a chair.

— Peregrine Finch