The compiler and the reading room

Pooka 0.1.0 is released. The compiler runs on Erlang/OTP 29. Its standard library is on Hex, and the API documentation has a home on HexDocs.

import pooka/list

main() {
  let answer = [1, 2, 3]
    |> list.map(fn(n) { n * 2 })
  assert answer == [2, 4, 6]
  echo answer
}

Try the public package

pooka new first_pooka
cd first_pooka
pooka deps download
pooka test
pooka run

Replace src/main.pooka with the program above to try the list transformation.

A fresh pooka new project already declares the standard library dependency. There is no separate runtime package to discover halfway through the first build.

This release has 1,511 passing compiler tests and 106 passing standard-library tests. The compiler escript has also passed ten native and emulated platform checks, including Windows. These are useful pieces of evidence, rather than a certificate that no bugs remain. A new language still has quite a lot to learn from other people’s programs.

We also tested the less ceremonial part of release day: a fresh project downloading the public package, calling it, running assertions and exporting an executable escript. A registry page is much more reassuring when the thing behind it actually arrives.

Let the parcel finish its journey

Publishing taught the tooling one last lesson. Hex can hold a package for human approval. Pooka now uploads the documentation as well, so both can travel through the pending-release workflow together. It should not announce a staged parcel as though it were already on the public shelf.

For readers, the next stop is the language tour. For me, it is watching what happens when somebody writes a program I did not prepare in advance.

There was a directory in May. There is a small language outside it now.

— Peregrine Finch

Write down the host requirement

Pooka’s distribution is one extensionless escript named pooka. It carries the compiled compiler and its Gleam dependencies. The host still needs Erlang/OTP 29; building the compiler from source also needs Gleam.

pooka --version
pooka new first_visit
cd first_visit
pooka test

The distinction between those requirements is useful. Somebody using a released compiler should not have to reconstruct the compiler’s own build environment first.

Make it travel before release day

build artifact -> checksum -> relocated compiler
                              -> new project
                              -> build and tests
                              -> application escript

One file does not mean one operating system. An escript can still contain assumptions about path separators, executable discovery, text encoding or the shape of a command shell. Put its directory name between two spaces and a surprising amount of confidence can fall out.

I want release checks that exercise the artifact as a user receives it: verify the checksum, move it away from the checkout, run it with Erlang available and Gleam absent, create a project, compile it and run tests.

Exporting that project’s own escript is a useful final turn. The compiler’s luggage and the application’s luggage are different things. Both need to survive the journey.

Try some less familiar doors

The same reasoning applies beyond the familiar machines. A 32-bit runtime or a big-endian target can expose a comfortable assumption that a faster development laptop never complains about. Emulated targets add useful coverage alongside native runners, though they do not make every possible deployment tested.

Installation instructions belong near the download, with the host requirement stated plainly. A checksum belongs beside the file, where it can help establish that the download arrived intact.

The compiler can be small enough to carry. Its travel arrangements should still be written down.

— Peregrine Finch

A documentation page is a useful place to put a chair.

Give the module an introduction

Pooka distinguishes module documentation from documentation attached to a declaration:

//// Small arithmetic for a quiet example.
////
//// This module keeps its promises in whole numbers.
module main

/// Add two whole numbers.
add(a: Int, b: Int) -> Int {
  a + b
}

main() {
  assert add(20, 22) == 42
}

The four slashes describe the module. Three describe the following declaration. A signature tells the reader what can be passed; the explanation should tell them why they might want to pass it.

Let an example carry some weight

A small function can show its behaviour with a concrete assertion:

add(a: Int, b: Int) -> Int {
  a + b
}

main() {
  assert add(-2, 2) == 0
}

This makes a useful documentation example because its answer is visible and checkable. Merely repeating the parameter names in a sentence would add rather little to the signature.

A fenced example in a comment is still prose content until some verification step actually compiles or runs it. The documentation generator rendering it successfully is not proof that its imports or annotations remain correct. Examples deserve checking alongside the API they describe.

Build the reading room

pooka docs

The generated pages can organise module descriptions, declarations and examples into something a reader can browse. Navigation, highlighted code and a legible theme make that visit easier; they cannot replace an explanation of a surprising failure case.

For a released library, publishing includes the documentation archive as well as the package. README images and other configured assets need to arrive with the pages, rather than continuing to rely on a file path that only existed on the author’s laptop.

Good documentation does not need to tell every story about a function. It needs to answer the next useful question before the visitor has to knock on somebody else’s door.

— Peregrine Finch

Give the example a name

Put test declarations in a file ending in .spec.pooka, such as test/main.spec.pooka:

module main_test

test "two numbers can share an answer" {
  assert 20 + 22 == 42
}

test "division keeps a fractional answer" {
  assert 7 / 2 == 3.5
}

The names explain the intended behaviour. The assertions contain concrete observations the runner can check. Production source files do not accept test declarations, which keeps the boundary between shipped code and test code explicit.

A skipped page is still part of the book

A test can be kept without being executed:

module main_test

test skip "a future visitor can be greeted" {
  assert 1 + 1 == 2
}

The skipped body is still type-checked. A test waiting for a feature should not become an abandoned pocket of syntax the compiler never reads.

That also means test skip is not a way to hide an unresolved name or an invalid call. It changes execution, not the language rules inside the body.

Choose the amount of conversation

pooka test
pooka test --compact
pooka test --verbose

The ordinary view groups named tests by module. The compact view concentrates on failures and totals. The verbose view is useful when more detail is needed.

When an equality assertion fails, showing the expected and actual values gives the reader something to compare. A location tells them where the expectation was written. Neither requires printing the entire compiler diagnostic treatment around every ordinary failed assertion.

The process exit status matters too: a failing suite must tell CI it failed. A beautiful green summary printed before returning success for a broken assertion would be an exceptionally well-dressed lie.

Tests are small executable explanations. Their notebook is most useful when the next reader can understand both the promise and the evidence without interviewing the author.

— Peregrine Finch

Pack the helpers with their owner

A publication archive includes both the library and what another BEAM build tool needs:

package archive
  pooka.toml
  rebar.config
  src/
    generated Erlang modules
    package-scoped runtime helper
    application descriptor

This is a layout sketch; the archive also contains the package’s sources, metadata and configured assets.

Pooka has a few runtime helpers: machinery generated programs call for language behaviour that does not lower to a single ordinary operation. There is no separate runtime Hex dependency for the user to add.

Instead, the compiler embeds the needed runtime source into each package’s publication output. The helper module belongs to that package’s generated namespace. Two libraries should not both unpack a different module under the same global name and hope the loader takes turns.

Give the neighbour something it can compile

This is especially important for Erlang interop. An ordinary Pooka build produces BEAM code from abstract forms. A package published for other BEAM build tools also carries generated Erlang source, so those tools can compile it without first installing Pooka.

The two routes serve different jobs. Generated .erl files are part of the publication archive; they do not need to clutter the normal development build directory.

The package metadata matters just as much as the code. The application name, included modules, dependencies and licences should describe the package that actually arrived. A pleasant README cannot repair an application descriptor that asks Erlang to start a nonexistent dependency.

Inspect before it leaves

pooka publish --dry-run

This prepares the package and documentation archives without uploading them. Open the prepared archive before trusting its file list.

A good interop test therefore changes the reader. Instead of asking Pooka whether its own parcel looks plausible, let another build tool compile the contents.

That catches assumptions the original compiler might politely share with itself: an implicit search path, an unshipped helper, or a generated call that depends on something outside the archive.

The source language can keep its own ideas about classes and patterns. The package boundary should be ordinary enough for the neighbour to open.

I have included the string. You should not need a Pooka installation merely to untie it.

— Peregrine Finch

Bit-array syntax lets the program describe where those pieces sit.

Write the layout down

Here is a byte-sized tag, a sixteen-bit length and a UTF-8 payload:

import pooka/bit_array

main() {
  let packet = <<1:8, 3:16, "owl":utf8>>
  assert bit_array.byte_size(packet) == 6
}

The length field occupies two bytes; its value is three because the payload contains three bytes. The size of the field and the number stored in it are different quantities.

That small distinction is responsible for a respectable number of broken envelopes.

Read the layout as a pattern

A pattern can bind the length before using it to read the body:

import pooka/bit_array

read(packet: BitArray) -> Result(String, String) {
  match packet {
    <<1:8, length:16, body:bytes-size(length)>> ->
      match bit_array.to_string(body) {
        Ok(text) -> Ok(text)
        Error(_) -> Error("payload is not text")
      }
    _ -> Error("bad envelope")
  }
}

main() {
  assert read(<<1:8, 3:16, "owl":utf8>>) == Ok("owl")
  assert read(<<1:8, 4:16, "owl":utf8>>) == Error("bad envelope")
}

The second assertion matters. A declaration of four bytes does not make a three-byte payload longer. The pattern fails, and the fallback reports the malformed input.

Let binary data remain binary

The conversion to String is a separate decision. A payload can have the right byte count while containing invalid text, so that conversion has its own error branch.

Other protocols may keep the payload as bytes. They may also allow trailing fields, require a different endianness, or work with values that are not byte-aligned. Those choices belong in the format’s explicit rules rather than in guesses made by the reader.

For the compiler, bit-array construction and matching must agree on sizes and units. A convenient syntax is no comfort if the writer and reader quietly count different things.

The packet can keep its hat. We have finally labelled which seat belongs to the header.

— Peregrine Finch

Remember the evidence

Incremental compilation is the art of knowing when that request is safe to grant.

Remembering only the source text is not enough. A module can remain untouched while something it imports changes. The compiler version, relevant build settings and the state of its dependencies can also affect whether an old artifact is still suitable.

Check the inputs and the output

same source + same relevant inputs + valid artifacts
    -> reuse the previous result

changed dependency or missing artifact
    -> reconsider the build

This is the decision the cache must justify, rather than a literal stamp format.

A build stamp is a compact record of the inputs the builder relied on. The difficult part is deciding which inputs belong in that record. A missing dependency makes the cache optimistic. An irrelevant one makes it unnecessarily anxious.

The output matters as well. A stamp saying “done” is not useful if the corresponding BEAM file has been removed. Nor should an interrupted build leave a convincing record beside half a result.

pooka build
pooka build

The second invocation is a useful experiment, but it is the least demanding one. More revealing tests change an imported declaration, remove an artifact, or introduce an error after a successful build. The next run needs to respond to each situation for the right reason.

Say where the work happened

There is a naming problem hidden in this work too. A build may contain several packages. Logging the package being compiled helps explain where work happened; announcing everything under the application’s name makes a cache miss surprisingly difficult to locate.

I do not want the cache to become a second compiler with a separate idea of correctness. It should reuse an answer only while the evidence for that answer still holds.

Fast rebuilds are pleasant. Trustworthy rebuilds are what make them worth having.

The module can go back to sleep when we have checked the route, rather than merely recognised the front door.

— Peregrine Finch

Ask which length you mean

An ASCII letter can make the distinction invisible. A Greek letter is a more helpful test subject:

import pooka/bit_array
import pooka/string

main() {
  let text = "α"
  assert string.length(text) == 1
  assert bit_array.byte_size(bit_array.from_string(text)) == 2
}

The UTF-8 representation needs two bytes. That does not mean the text contains two letters.

For network packets and byte offsets, the byte count matters. For a string operation describing text, treating each byte as an independent character would manufacture pieces that were never valid text in the first place.

A larger symbol is still text

Use more than ASCII when testing the boundary:

import pooka/bit_array
import pooka/string

main() {
  let text = "α🐇"
  assert string.length(text) == 2
  let bytes = bit_array.from_string(text)
  assert bit_array.byte_size(bytes) == 6
  assert bit_array.to_string(bytes) == Ok(text)
}

The rabbit occupies four UTF-8 bytes. The round trip checks that converting to bytes and back preserves the original text rather than merely producing something printable.

Do not turn a small example into a universal rule

These examples avoid another distinction: a visible character can be made from several Unicode code points. Combining marks and emoji sequences mean that counting code points is not always counting the units a reader perceives on screen.

In this library, string.length counts Unicode code points. A combining mark therefore counts separately:

import pooka/string

main() {
  assert string.length("e\u{301}") == 2
  assert string.length("é") == 1
}

The two spellings can look like the same accented letter while having different code-point sequences. A byte-oriented slice also needs to consider what happens when its boundary lands inside an encoded character.

This is relevant to the compiler itself. Source spans can use byte offsets internally, while diagnostics need to place underlines in displayed source. Confusing the measurements points the complaint at an innocent neighbour.

The rulers are useful. Trouble begins when one is handed to every caller without asking what they were trying to measure.

— Peregrine Finch

Let a recoverable error return

These are different decisions.

import pooka/int

sum(a: String, b: String) -> Result(Int, int.ParseError) {
  let first = int.parse(a)?
  let second = int.parse(b)?
  Ok(first + second)
}

main() {
  assert sum("20", "22") == Ok(42)
  assert sum("20", "owl") == Error(int.ParseError("owl"))
  let assert Ok(value) = sum("20", "22")
  assert value == 42
}

Unfold the punctuation

Here is the propagation written out in full:

import pooka/int

sum(a: String, b: String) -> Result(Int, int.ParseError) {
  match int.parse(a) {
    Error(reason) -> Error(reason)
    Ok(first) -> match int.parse(b) {
      Error(reason) -> Error(reason)
      Ok(second) -> Ok(first + second)
    }
  }
}

main() {
  assert sum("20", "22") == Ok(42)
  assert sum("20", "owl") == Error(int.ParseError("owl"))
}

The question mark opens a Result. With Ok, execution continues using its contents. With Error, the surrounding function returns that error. The second parse does not run if the first one has already failed.

The function must itself return a compatible Result. Otherwise there would be nowhere honest to send the error.

An expectation is a stronger promise

let assert makes a stronger statement. It requires the pattern to match. If it does not, execution fails instead of returning the unmatched value as an ordinary error result.

For a fixed example string in a test, that expectation can be useful. For a stranger’s configuration file, it may be entirely the wrong hospitality. Invalid user input is usually a possibility the application should explain, rather than an impossible event it should crash over.

Neither spelling should hide repeated evaluation in generated code. If the expression reads a line or sends a request, evaluating it twice changes the program. Lowering the convenient syntax must preserve both the result and the number of times the work happens.

I like these small features when their meaning stays visible. ? passes a recoverable result back to the caller. let assert draws a line around an expectation the program will not continue without.

The punctuation is brief. The choice it records is not.

— Peregrine Finch

Give the state an address

The BEAM gives these places identities and mailboxes. Pooka gives one useful arrangement of them a language construct: the actor.

actor Counter {
  var count: Int = 0

  be add(amount: Int) {
    count += amount
  }

  value() -> Int {
    count
  }
}

main() {
  let counter = Counter()
  counter.add(2)
  counter.add(3)
  assert counter.value() == 5
}

A message and a question

For this example’s single sender, the exchange looks like this:

caller -> add(2)  -> counter
caller -> add(3)  -> counter
caller -> value() -> counter
caller <- 5      <- counter

be add is a behaviour: calling it sends a message and returns without waiting for the state change. value() is a request that waits for an answer. The state belongs to the actor, and its handlers work through messages one at a time.

The example uses one sender. The two additions and the following request travel from that sender to the same actor in order. This is not a promise that messages from every process in the system share one universal arrival order.

That distinction matters. Concurrency explanations become rather too comfortable when they leave out the competing callers.

Decide who notices a failure

An actor also is not automatically a complete fault-tolerance policy. The BEAM has links, monitors and supervision machinery, but an application still needs to decide who should notice a failure and what should happen next. A mailbox alone cannot make that decision kindly on our behalf.

Erlang’s process documentation is worth reading alongside the language surface. The runtime beneath the actor has its own precise rules; a friendly keyword does not replace them.

What the boundary does give us is a place to put mutable state without making it everyone’s shared furniture. Ordinary class values remain immutable. The counter changes in the process that owns it.

There can be several kinds of weather in one running program. Each room should know which window is its own.

— Peregrine Finch