What if you could design your own programming language? In this post, I walk through building TinyLang — a complete interpreter from scratch in Python with lexer, parser, AST, closures, first-class functions, and more.

🔗 GitHub · 🌐 Live Demo

Why Build a Language Interpreter?

Building a programming language interpreter is one of the best exercises in computer science. It forces you to understand:

• How code is read — lexical analysis and tokenization
• How structure is extracted — parsing and abstract syntax trees
• How meaning is given — evaluation and runtime semantics
• How state is managed — scoping, closures, and environments

TinyLang is a dynamically-typed language with Python-like syntax and curly-brace blocks. It supports integers, floats, strings, booleans, arrays, dictionaries, first-class functions, closures, arrow functions, a pipeline operator, structured error handling, and 60+ built-in functions — all in zero-dependency Python.

The Compiler Pipeline

Source Code → Lexer → Tokens → Parser → AST → Interpreter → Result

Every interpreter follows this pipeline. Let's walk through each stage.

Stage 1: The Lexer (Tokenizer)

The lexer breaks raw source code into a stream of tokens — the smallest meaningful units of the language.

from dataclasses import dataclass
from enum import Enum, auto

class TokenType(Enum):
 INTEGER = auto()
 FLOAT = auto()
 STRING = auto()
 IDENTIFIER = auto()
 PLUS = auto()
 MINUS = auto()
 FN = auto()
 LET = auto()
 IF = auto()
 # ... 50+ token types

@dataclass
class Token:
 type: TokenType
 value: any
 line: int
 column: int

The lexer processes characters one at a time, handling multi-character operators (==, !=, |>, =>), string escape sequences (\n, \t, \uXXXX), numeric literals with underscores (1_000_000), and comments (single-line // and multi-line /* */).

Key insight: Handle operator ambiguity by checking the next character. When you see =, peek ahead — is it == (equality) or => (arrow)?

def _read_operator(self):
 ch = self.current
 if ch == '=' and self.peek() == '>':
 self.advance()
 self.advance()
 return Token(TokenType.ARROW, "=>", self.line, start)
 elif ch == '=' and self.peek() == '=':
 self.advance()
 self.advance()
 return Token(TokenType.EQ, "==", self.line, start)
 elif ch == '=':
 self.advance()
 return Token(TokenType.ASSIGN, "=", self.line, start)

Stage 2: The Parser (Recursive Descent)

The parser takes tokens and builds an Abstract Syntax Tree (AST). I chose recursive descent because it's intuitive, debuggable, and maps cleanly to the grammar.

Operator Precedence

The key challenge is getting operator precedence right. Each precedence level gets its own parsing function:

def parse_expression(self):
 return self.parse_assignment()

def parse_assignment(self):
 left = self.parse_pipe() # |>
 # ... handle = and +=/-=/*=

def parse_pipe(self):
 left = self.parse_logical_or() # or
 # ... handle |>

def parse_logical_or(self):
 left = self.parse_logical_and() # and
 # ...

def parse_addition(self):
 left = self.parse_multiplication() # * / %
 # ...

def parse_power(self):
 base = self.parse_unary() # - not
 # Right-associative: 2 ** 3 ** 2 = 2 ** 9

Each level calls the next-higher precedence level, creating a natural hierarchy where multiplication binds tighter than addition, which binds tighter than comparison, and so on.

AST Nodes

Every language construct becomes a dataclass node:

@dataclass
class BinaryOp:
 op: str
 left: any # AST node
 right: any
 line: int

@dataclass
class FunctionDeclaration:
 name: str
 params: list[str]
 defaults: list # Default parameter values
 body: list # List of AST statements

@dataclass
class IfStatement:
 condition: any
 then_body: list
 else_body: list

Stage 3: The Interpreter (Tree-Walking)

The interpreter visits each AST node and evaluates it. I used the visitor pattern — a dictionary mapping node types to handler methods:

class Interpreter:
 def __init__(self):
 self.visitors = {
 IntegerLiteral: self.visit_IntegerLiteral,
 BinaryOp: self.visit_BinaryOp,
 FunctionDeclaration: self.visit_FunctionDeclaration,
 CallExpression: self.visit_CallExpression,
 # ... 30+ visitors
 }

 def execute(self, node, env):
 visitor = self.visitors.get(type(node))
 return visitor(node, env)

Closures: The Hardest Part

Closures are functions that "remember" their defining environment. This is the single most important concept in the interpreter:

class UserFunction:
 def __init__(self, declaration, closure_env):
 self.declaration = declaration
 self.closure_env = closure_env # ← captures the defining scope

def visit_FunctionDeclaration(self, node, env):
 # Capture the current environment as the closure
 func = UserFunction(node, env)
 env.set(node.name, func)
 return func

def _call_user_function(self, func, args, node):
 # Create new scope INSIDE the closure's environment
 call_env = Environment(parent=func.closure_env)
 for param, arg in zip(func.declaration.params, args):
 call_env.set(param, arg)
 # Execute function body
 for stmt in func.declaration.body:
 self.execute(stmt, call_env)

The key: when calling a function, the new scope's parent is the closure environment (where the function was defined), NOT the calling environment. This is what makes closures work:

fn make_counter() {
 let count = 0 // lives in make_counter's scope
 fn increment() {
 count = count + 1 // accesses parent scope
 return count
 }
 return increment // carries make_counter's scope with it
}

let c = make_counter()
print(c()) // 1
print(c()) // 2 — count persists!

The Environment (Scope Chain)

The Environment class implements lexical scoping:

class Environment:
 def __init__(self, parent=None):
 self.variables = {}
 self.constants = set()
 self.parent = parent

 def get(self, name):
 if name in self.variables:
 return self.variables[name]
 if self.parent:
 return self.parent.get(name)
 raise NameError(f"Undefined variable: {name}")

 def set(self, name, value):
 if name in self.constants:
 raise RuntimeError(f"Cannot reassign constant: {name}")
 if name in self.variables:
 self.variables[name] = value
 elif self.parent:
 self.parent.set(name, value)

Built-in Functions

TinyLang ships with 60+ built-in functions wrapped in a TinyLangCallable class:

class TinyLangCallable:
 def __init__(self, name, func, min_args=0, max_args=None):
 self.name = name
 self.func = func
 self.min_args = min_args
 self.max_args = max_args

 def __call__(self, *args):
 # Validate argument count
 return self.func(*args)

This includes map, filter, reduce, sorted, reversed, enumerate, zip, math functions, string operations, and type conversion. The interpreter wraps user functions in Python callables before passing them to builtins like map — so map((x) => x * 2, [1, 2, 3]) works seamlessly.

The Pipeline Operator

One of TinyLang's nicest features — the pipeline operator |> passes the left side as the first argument to the right side:

[1, 2, 3, 4, 5] |> sum |> str
// Equivalent to: str(sum([1, 2, 3, 4, 5]))
// Result: "15"

Implementation is straightforward:

def visit_PipeExpression(self, node, env):
 value = self.execute(node.left, env)
 func = self.execute(node.right, env)
 # Call func with value as first argument
 return self._call_function(func, [value], node)

Error Handling

TinyLang supports try/catch/throw with any value type:

try {
 throw {"code": 404, "message": "Not found"}
} catch err {
 print(err["code"]) // 404
 print(err["message"]) // Not found
}

Under the hood, throw raises a Python ThrowSignal exception that carries the thrown value. try/catch catches it and binds the value to the catch variable.

Testing: 248 Tests

The test suite covers every language feature across four modules:

tests/
├── test_lexer.py # Token-level tests
├── test_parser.py # AST structure tests
├── test_interpreter.py # Execution tests
└── test_programs.py # Full program integration tests

Tests include edge cases like recursive closures, max recursion depth, division by zero, index out of bounds, and multi-line expressions.

What I Learned

1. Recursive descent is elegant. Each grammar rule maps to a function. The precedence hierarchy emerges naturally.

2. Closures are surprisingly simple once you understand that functions capture their defining environment, not the calling environment.

3. Error messages matter. Every error carries line and column numbers. Users need to know where things went wrong.

4. The visitor pattern scales. Adding new node types is just adding a new visitor method — the interpreter stays clean.

5. Test-driven development works. Writing tests first caught bugs in operator precedence, closure semantics, and edge cases before they became hard to debug.

Try It Yourself

pip install git+https://github.com/hajirufai/tinylang.git
tinylang # Start the REPL

Or run an example:

tinylang examples/closures.tl
tinylang examples/fibonacci.tl
tinylang -e 'print(map((x) => x ** 2, range(10)))'

📦 GitHub Repository — 5,300+ lines, 248 tests, zero dependencies.

This is project #11 in my portfolio series. Previously: RaftKV, QueryForge, and more on my dev.to profile.