Inside the Lateralus Compiler: From Source to Bytecode

August 2025 · 12 min read

I've spent the last year building the Lateralus compiler, and I want to pull back the curtain on how it works — every phase, every intermediate representation, every design decision. Whether you're contributing to the compiler or just curious about language implementation, this post walks through the full pipeline from source text to executable output.

Compiler Pipeline Overview

Phase 1: Lexing

The lexer (or tokenizer) is the compiler's front door. It reads raw source text character by character and groups them into tokens — the atoms of the language grammar. The Lateralus lexer is hand-written (not generated) for maximum control over error messages and performance.

Let's trace a simple expression through the lexer:

// Source input:
let total = numbers |> filter(x => x > 0) |> sum()

// Token stream output:
// KW_LET      "let"
// IDENT       "total"
// EQUALS      "="
// IDENT       "numbers"
// PIPE        "|>"
// IDENT       "filter"
// LPAREN      "("
// IDENT       "x"
// ARROW       "=>"
// IDENT       "x"
// GT          ">"
// INT_LIT     "0"
// RPAREN      ")"
// PIPE        "|>"
// IDENT       "sum"
// LPAREN      "("
// RPAREN      ")"
// EOF

The trickiest part of the Lateralus lexer is the |> pipeline operator. The | character is also used for pattern match arms and bitwise OR, so the lexer must look ahead one character to distinguish them. We handle this with a simple peek:

// In the lexer's main switch
match current_char
    | '|' => if peek() == '>'
        then { advance(); emit(PIPE) }
        else emit(BAR)
    | '=' => if peek() == '>'
        then { advance(); emit(ARROW) }
        else if peek() == '='
        then { advance(); emit(EQ_EQ) }
        else emit(EQUALS)

Every token carries its source span (file, line, column, byte offset) so that later compiler phases can produce precise error messages that point to the exact location in the original source.

Phase 2: Parsing

The parser consumes the token stream and produces an Abstract Syntax Tree (AST). Lateralus uses a recursive descent parser with Pratt parsing for operator precedence. This combination handles both statements (let, match, type declarations) and complex nested expressions (pipelines, lambdas, binary ops) cleanly.

AST Node Types

The AST is defined as a sum type — naturally expressed in Lateralus itself:

type Expr =
    | IntLit(Int)
    | StrLit(String)
    | Ident(String)
    | Lambda(List<String>, Expr)
    | Call(Expr, List<Expr>)
    | BinOp(Op, Expr, Expr)
    | Pipeline(Expr, Expr)
    | Let(String, Expr)
    | Match(Expr, List<Arm>)
    | Block(List<Expr>)

For our earlier example, numbers |> filter(x => x > 0) |> sum(), the parser produces:

// AST output (simplified):
Let("total",
  Pipeline(
    Pipeline(
      Ident("numbers"),
      Call(Ident("filter"), [
        Lambda(["x"],
          BinOp(Gt, Ident("x"), IntLit(0)))
      ])
    ),
    Call(Ident("sum"), [])
  )
)

Note how pipelines are left-associative: a |> b |> c becomes Pipeline(Pipeline(a, b), c). This matches the left-to-right data flow semantics.

Phase 3: Semantic Analysis

After parsing, the compiler performs semantic analysis on the AST. This phase handles name resolution (linking each identifier to its declaration), scope checking (ensuring variables are used where they're visible), and desugaring (expanding syntactic sugar into core forms).

One important desugaring: the pipeline operator. Internally, x |> f(y) is rewritten to f(x, y) — the left-hand value becomes the first argument of the right-hand call. This simplification means later phases don't need to know about pipelines at all.

Phase 4: Type Checking

This is where the Hindley-Milner inference engine runs. I covered this in detail in the type inference blog post, but briefly: the checker walks the AST, generates type constraints, and solves them via unification. After this phase, every AST node is annotated with its concrete type.

Phase 5: IR Generation

The typed AST is lowered to a flat intermediate representation — a sequence of instructions in three-address code form. This IR is target-agnostic: the same IR feeds into both the Python transpiler and the C99 transpiler.

// Source:
let total = [1, 2, 3] |> map(x => x * 2) |> sum()

// IR output:
//   t0 = MAKE_LIST [1, 2, 3]
//   t1 = MAKE_CLOSURE λ(x) { MUL x 2 }
//   t2 = CALL map(t0, t1)
//   t3 = CALL sum(t2)
//   STORE total = t3

Phase 6: Optimization

The optimizer runs several passes over the IR:

Pipeline fusion is particularly impactful. Without it, list |> map(f) |> map(g) would traverse the list twice. After fusion, it traverses once applying the composed function x => g(f(x)).

Phase 7: Code Generation

The final phase emits target code. Lateralus currently supports two backends:

Python Backend (default)

The Python backend transpiles IR to clean, readable Python 3. This is the default for rapid development and scripting use cases:

# Generated Python output:
total = sum(map(lambda x: x * 2, [1, 2, 3]))

C99 Backend (systems)

The C99 backend produces portable C for performance-critical and systems programming targets. It handles memory management, struct layout, and function pointers:

// Generated C99 output:
int total;
{
    int arr[] = {1, 2, 3};
    int mapped[3];
    for (int i = 0; i < 3; i++) {
        mapped[i] = arr[i] * 2;
    }
    total = 0;
    for (int i = 0; i < 3; i++) {
        total += mapped[i];
    }
}

One language, multiple targets. Write your logic once in Lateralus, then compile to Python for development or C99 for production. The compiler handles the translation.

Performance

The compiler itself is fast. On a modern machine, it processes roughly 50,000 lines per second through all seven phases. The lexer and parser account for about 15% of the time, type checking about 35%, and code generation the remaining 50%. We're actively working on incremental compilation to make the edit-compile-run cycle even faster.

🧪 See the compiler in action ⭐ Read the compiler source

Compilation by the numbers

Here's what the compiler does on a typical 5,000-line Lateralus project:

The C compilation step dominates. With -O0 (debug mode), total time drops to 300ms. With incremental compilation (only recompiling changed modules), most rebuilds are under 500ms.

Lateralus is built by bad-antics. Follow development on GitHub or try the playground.

Compilation by the numbers

Here's what the compiler does on a typical 5,000-line Lateralus project:

The C compilation step dominates. With -O0 (debug mode), total time drops to 300ms. With incremental compilation (only recompiling changed modules), most rebuilds are under 500ms.

Lateralus is built by bad-antics. Follow development on GitHub or try the playground.