Zen Compiler Architecture

The Zen compiler is a sophisticated, multi-stage pipeline designed to translate Zen source code into executable binaries. Built in Rust, it adheres to the language's core principles, from its "zero keywords" philosophy to allocator-driven concurrency. The compiler's architecture is modular, with distinct components handling lexical analysis, syntactic parsing, semantic validation, compile-time execution, and efficient code generation via LLVM.

High-Level Compilation Flow

The compilation process generally follows these steps:

1. Source Code: The .zen source file is fed into the compiler.
2. Lexical Analysis (Lexer): The raw text is converted into a stream of meaningful tokens.
3. Syntactic Analysis (Parser): The token stream is transformed into an Abstract Syntax Tree (AST), representing the program's structure.
4. Semantic Analysis (Type Checker): The AST is validated for type correctness, variable scope, function signatures, and behavior adherence. This phase often involves multiple passes for complete type resolution.
5. Compile-Time Execution (Comptime Interpreter): Special @meta blocks and comptime expressions are evaluated, potentially modifying the AST or generating new code before runtime compilation.
6. Monomorphization: Generic types and functions are instantiated into concrete versions based on their usage, eliminating generics before low-level code generation.
7. Foreign Function Interface (FFI): Manages interactions with external C libraries, including function signatures and type mappings.
8. Code Generation (LLVM): The validated and monomorphized AST is translated into LLVM Intermediate Representation (IR), which can then be compiled to machine code for various targets.

Main Components and Their Roles

1. Lexer (src/lexer.rs)

The Lexer is the first stage of the compiler. Its primary responsibility is to read the Zen source code character by character and convert it into a sequence of tokens. These tokens are the fundamental building blocks for the parser.

• Role: Converts raw text into a stream of Tokens (identifiers, literals, operators, symbols, and special @ keywords).
• Input: Zen source code as a string.
• Output: A Vec<TokenWithSpan>, where each token includes its location in the source code for error reporting.
• Key Design: Includes special handling for Zen's unique syntax, such as ? for pattern matching, ::= for mutable assignment, range operators (.., ..=), and string interpolation ("${expr}"). It also handles pub, @std, @this, @meta, and @export special identifiers.

2. Parser (src/parser/ module)

The Parser consumes the token stream produced by the lexer and constructs the Abstract Syntax Tree (AST). The AST is a hierarchical representation of the program's structure, abstracting away the concrete syntax.

• Role: Builds the Program AST from tokens. This involves recognizing declarations (functions, structs, enums, traits, imports), statements, and expressions.
• Input: Stream of TokenWithSpan from the Lexer.
• Output: An ast::Program structure, containing a list of ast::Declarations and top-level statements.
• Key Design: Zen's "zero keywords" philosophy significantly influences the parser. It relies heavily on operator precedence, symbol recognition (?, |, :=, ::=), and contextual lookaheads to infer intent where keywords would traditionally guide parsing (e.g., value ? | pattern { ... } instead of match). It also includes dedicated logic for parsing complex constructs like structs, enums, closures, pattern matches, and method calls.

3. Module System (src/module_system/ module)

The Module System handles the organization, loading, and resolution of Zen code across multiple files, including the standard library.

• Role: Manages module imports ({ io, maths } = @std), resolves module paths, loads .zen files, extracts exported symbols, and merges program components.
• Input: ast::Program potentially containing ModuleImport declarations, and filesystem paths.
• Output: A unified ast::Program with all imported declarations integrated and references resolved.
• Key Design: Supports @std and custom module imports. It performs a recursive loading process to handle nested module dependencies. It's crucial for providing context for semantic analysis and code generation.

4. Type Checker (src/typechecker/ module)

The Type Checker is the semantic analysis phase, validating the AST for type correctness, variable usage, and adherence to language rules. It ensures the program is semantically sound before code generation.

• Role: Performs static analysis to verify type compatibility, variable declarations (mutability, initialization), function call arguments, method resolution, and behavior implementations. It maintains symbol tables (scopes) to track variable and function information.
• Input: The raw ast::Program from the parser, including all declarations.
• Output: Error diagnostics (error::CompileError) if semantic issues are found. It also enriches the AST with inferred type information where necessary (e.g., Void return types).
• Key Design:
  • Two-Pass Approach: It uses multiple passes to resolve dependencies:
    1. First Pass (Collection): Gathers all struct, enum, and function signatures without checking their bodies. This allows for forward declarations and basic recursive type definitions.
    2. Second Pass (Resolution & Inference): Resolves generic type placeholders in struct fields (e.g., Generic { name: "MyStruct" } to Struct { name: "MyStruct", fields: [...] }). This is crucial for correctly modeling complex data structures and resolving self-referential types.
    3. Third Pass (Function Body & Full Check): Analyzes function bodies, statements, and expressions, inferring types, checking compatibility, and validating logic.
  • Scope Management: Uses a stack of hashmaps (scopes) to manage local variable declarations and their visibility.
  • Behavior Resolution (src/typechecker/behaviors.rs): Manages trait definitions and implements/requires blocks, ensuring that types correctly implement required behaviors. It handles replacing Self types with concrete types in trait implementations.
  • Type Inference (src/typechecker/inference.rs): Automatically deduces the types of expressions and variables when not explicitly provided, adhering to Zen's type inference rules.

5. Comptime Interpreter (src/comptime/mod.rs)

The Comptime Interpreter is a unique feature of Zen, allowing a subset of the language to be executed during compilation. This enables powerful metaprogramming and compile-time code generation.

• Role: Executes comptime { ... } blocks and comptime expr expressions within the AST. It can generate new AST nodes, evaluate constants, and perform complex compile-time computations.
• Input: ast::Program containing ComptimeBlock or Comptime expressions.
• Output: A transformed ast::Program where compile-time logic has been evaluated and its effects integrated into the AST.
• Key Design: Operates on a specialized ComptimeValue and Environment to ensure deterministic, side-effect-free execution. It allows for advanced scenarios like generating data structures, validating configurations, and optimizing code based on compile-time knowledge.

6. FFI (Foreign Function Interface) (src/ffi/mod.rs)

The FFI module provides a safe and structured way to interact with external C libraries, aligning with Zen's philosophy of explicit control.

• Role: Manages the definition of external functions and types, library loading, symbol lookup, and provides a builder pattern for configuring FFI interactions.
• Input: ast::ExternalFunction declarations and runtime library paths.
• Output: At compile time, it primarily registers the signatures of external functions. At runtime (for the JIT or when linking), it interfaces with libloading to load dynamic libraries and resolve symbols.
• Key Design: Emphasizes explicit RawPtr<T> for unsafe operations, FnSignature for function types, and robust error handling. It allows specifying calling conventions, safety checks, and platform-specific configurations.

7. Monomorphization (src/type_system/monomorphization.rs)

Monomorphization is the process of converting generic code (e.g., List<T>) into concrete code for each specific type it's used with (e.g., List_i32, List_String). This happens before LLVM code generation.

• Role: Identifies all used generic types and functions, creates specialized versions for each unique set of type arguments, and replaces generic references in the AST with these concrete versions.
• Input: The semantically checked ast::Program which might still contain generic constructs.
• Output: An ast::Program where all generic types and functions have been replaced by concrete, specialized implementations, ready for LLVM code generation.
• Key Design: Employs a TypeEnvironment and TypeInstantiator to manage generic definitions and their instantiated versions. It uses a "pending instantiations" queue to iteratively discover and specialize generics used within other specialized generics until all are resolved.

8. LLVM Code Generation (src/codegen/llvm/ module)

The LLVM Code Generator is the backend of the Zen compiler, translating the monomorphized AST into LLVM Intermediate Representation (IR).

• Role: Converts Zen's AST into platform-agnostic LLVM IR. This involves mapping Zen types to LLVM types, translating control flow (? operator, loops) into LLVM basic blocks and branches, and generating LLVM instructions for expressions and declarations.
• Input: The fully validated and monomorphized ast::Program.
• Output: An inkwell::module::Module object containing the LLVM IR, which can then be optimized, linked, and compiled into an executable binary or run via a JIT engine.
• Key Design:
  • Context, Module, Builder: Uses inkwell (Rust bindings for LLVM) to manage the LLVM Context, Module, and Builder objects for IR construction.
  • Unified Function Call (UFC): Dynamically resolves object.method() calls by looking up implementations in impl blocks or by checking if method(object, args) exists.
  • Allocator-Driven Concurrency: Generates calls to specific allocator functions (e.g., GPA.init(), AsyncPool.init()) which determine the synchronous or asynchronous nature of memory operations, eliminating the need for explicit async/await keywords in Zen.
  • Behavior/Trait Dispatch (src/codegen/llvm/behaviors.rs): Currently implements static dispatch for traits. A future goal might include dynamic dispatch via vtables for runtime polymorphism.
  • Explicit Pointers: Directly maps Ptr<T>, MutPtr<T>, and RawPtr<T> to LLVM pointer types, with safety semantics handled at the language level rather than by the backend.
  • String/Collection Support: Provides LLVM IR for Zen's built-in string and collection types, often relying on calls to C standard library functions (e.g., malloc, free, memcpy, strlen, sprintf).

Interactions and Design Synergies

The Zen compiler's components are highly interconnected:

• The Lexer and Parser form the frontend, establishing the syntactic structure.
• The Module System provides the context for symbol resolution throughout the AST.
• The Type Checker performs deep semantic validation, influencing later stages by inferring types and enforcing correctness. Its two-pass design (collect-then-check) handles complex type interdependencies.
• The Comptime Interpreter acts as a pre-processor, allowing compile-time transformations that simplify the AST before subsequent semantic checks and code generation.
• Monomorphization ensures that the backend only deals with concrete types, simplifying LLVM IR generation. It relies on the Type Checker's context to infer type arguments for generic instantiations.
• The FFI definitions inform both the Type Checker (for external function signatures) and the LLVM Code Generation (for actual calls to external libraries).
• The LLVM Code Generator translates the final, validated, and monomorphized AST into executable code, reflecting all earlier design decisions, such as explicit memory management, allocator-driven concurrency, and Zen's unique control flow mechanisms.

This modular yet integrated architecture allows Zen to achieve its minimalist design goals while providing a robust and extensible compilation pipeline.