intermediate representation

.gitignore

@@ -1,2 +1,5 @@
 node_modules
 .vscode
+*.ll
+*.s
+/output

README.md

@@ -2,15 +2,15 @@
 
 <img src="https://i.imgur.com/NLVzU2tm.png" alt="Complect Logo"/>
 
-Complect is a toy compiler developed in Node.js. It operates as an async generator-based compiler, processing code incrementally with push-like data flow for efficiency and modularity. Currently, Complect functions primarily as a transpiler, converting Complect code into JavaScript using the Babel Abstract Syntax Tree (AST). Future enhancements include support for switchable AST backends, enabling compilation to WebAssembly (WASM), custom interpreters, and other targets.
+Complect is a toy compiler developed in Node.js. It operates as an async generator-based compiler, processing code incrementally with push-like data flow for efficiency and modularity. Complect supports multiple backends: transpilation to JavaScript using Babel AST, and compilation to LLVM IR for native code generation. The pluggable backend architecture enables easy addition of WebAssembly, custom interpreters, and other targets.
 
 The initial implementation of this compiler was created to support a talk I presented at **OpenJS World 2022**. You can find the contents of this talk here. [Slides](https://static.sched.com/hosted_files/openjsworld2022/78/OpenJSW%20World%202022.pdf) [Video](https://youtu.be/aPHf_-N2yTU)
 
 ## Stages
 - Preprocessor: Transforms an input stream into a sequence of preprocessing tokens yielded incrementally.
 - Tokenizer: Converts the sequence of preprocessing tokens into a sequence of tokens yielded incrementally.
-- Abstract Syntax Tree (AST): Generates an AST from the sequence of tokens, currently utilizing Babel.
-- Output: The Babel AST outputs code in JavaScript.
+- Abstract Syntax Tree (AST): Generates an intermediate representation (IR) from the sequence of tokens.
+- Output: Pluggable backends convert IR to JavaScript (via Babel AST) or LLVM IR for native compilation.
 
 ## Design
 Complect utilizes a handcrafted parser and lexer to give developers fine-grained control over the compilation process. The parser in Complect is a top-down parser of the LL(1) type, chosen for its simplicity and efficiency in parsing.
@@ -22,18 +22,55 @@ Complect utilizes a handcrafted parser and lexer to give developers fine-grained
 
 - Async Generator-Based Compilation: The compiler processes code using async generators, yielding tokens and AST nodes incrementally for efficiency and modularity. This design supports asynchronous processing and easier debugging, with stages pushing data to the next via iteration (e.g., preprocessor yields tokens to tokenizer). While not a pure stream, it balances performance for large inputs with simplicity for a toy compiler.
 
-- Babel AST Integration: Complect currently generates an Abstract Syntax Tree (AST) using Babel, which outputs JavaScript code. This integration with Babel allows leveraging its robust ecosystem to further process and transform the generated JavaScript code.
+- LLVM IR Generation: Complect can generate LLVM Intermediate Representation for native code compilation using clang or llc.
 
-- Modular AST Generation: The architecture supports pluggable backends, enabling output to JavaScript, WebAssembly, or custom interpreters.
+- Modular AST Generation: The architecture supports pluggable backends via an intermediate representation (IR), enabling output to JavaScript, LLVM IR, WebAssembly, or custom interpreters.
+
+- Full Language Support: Both backends support the complete Complect language including integers, strings, arithmetic, concatenation, comparisons, and control flow.
 
 ### Future
-- Support for switchable AST backends, including Binaryen for WebAssembly output and a custom AST for interpretation
-- Modular AST generation via an intermediate representation (IR) to enable pluggable backends for JavaScript, WebAssembly, or custom interpreters
-- Explore compiler optimization passes
+- Additional backends: WebAssembly (Binaryen), custom interpreters
+- Compiler optimization passes
+- Enhanced language features
 
 ## Usage
 ### CLI
-You can use Complect as a command-line tool. The entry point of the application is `cli.js`. It reads from the standard input, compiles the input, and writes the output to the standard output.
+You can use Complect as a command-line tool. The entry point of the application is `bin/cli.js`.
+
+```bash
+# Show help
+complect --help
+
+# Compile from stdin to stdout (default: babel backend)
+cat program | complect
+
+# Compile from file to stdout
+complect --file program
+
+# Compile to specific backend
+complect --file program --backend llvm
+
+# Compile to file
+complect --file program --output program.js
+complect --file program --backend llvm --output program.ll
+```
+
+#### LLVM IR Usage
+When using the LLVM backend, Complect generates LLVM Intermediate Representation (.ll) files that can be compiled to native binaries:
+
+```bash
+# Generate LLVM IR
+complect --file fib --backend llvm --output fib.ll
+
+# Compile to assembly (for inspection)
+llc fib.ll -o fib.s
+
+# Compile to executable binary
+clang fib.ll -o fib
+
+# Run the binary
+./fib
+```
 
 ### Testing
 Tests are written using Node's built-in test module.
@@ -80,4 +117,4 @@ Complect is created by Jarrod Connolly.
 ## License
 MIT License
 
-See [COPYING](COPYING) for the full license text.
+See [LICENSE](LICENSE) for the full license text.

bin/cli.js

@@ -6,17 +6,81 @@
 */
 import { compile } from '../lib/compiler.js';
 import { pretty } from '../lib/util/pretty-hrtime.js';
+import { parseArgs } from 'node:util';
+import { createReadStream, createWriteStream } from 'node:fs';
 
-const inputStream = process.stdin;
+const parsedArgs = parseArgs({
+  options: {
+    help: {
+      type: 'boolean',
+      short: 'h',
+      default: false,
+    },
+    backend: {
+      type: 'string',
+      short: 'b',
+      default: 'babel',
+    },
+    file: {
+      type: 'string',
+      short: 'f',
+      default: '',
+    },
+    output: {
+      type: 'string',
+      short: 'o',
+      default: '',
+    }
+  },
+});
+
+if (parsedArgs.values.help) {
+  console.log(`Complect Compiler CLI
+
+Usage: complect [options]
+
+Options:
+  -h, --help          Show help information
+  -b, --backend      Specify the backend to use (default: babel)
+  -f, --file         Input source file (default: stdin)
+  -o, --output       Output file (default: stdout)
+`);
+  process.exit(0);
+}
+
+const backend = parsedArgs.values.backend;
+const file = parsedArgs.values.file;
+const output = parsedArgs.values.output;
+
+console.log(`Backend: ${backend}`);
+if (file) {
+  console.log(`Compiling: ${file}`);
+}
+if (output) {
+  console.log(`Output: ${output}`);
+}
+
+const inputStream = file ? createReadStream(file) : process.stdin;
+const outputStream = output ? createWriteStream(output) : process.stdout;
 
 const start = process.hrtime.bigint();
-compile(inputStream)
+compile(inputStream, backend)
   .then((results) => {
     const end = process.hrtime.bigint();
     const totalTime = pretty(end - start);
     const message = `Total Time: ${totalTime} PreTokens: ${results.preprocessorTokenCount} Tokens: ${results.tokenCount} AST Nodes: ${results.astNodeCount} `;
-    console.log(`// ${message}`);
-    console.log(`\n${results.code}`);
+
+
+    if (backend === 'llvm') {
+      outputStream.write(`; ${message}\n\n`);
+      outputStream.write(results.code);
+    } else {
+      outputStream.write(`// ${message}\n\n`);
+      outputStream.write(results.code);
+    }
+
+    outputStream.end();
+
   })
   .catch((err) => {
     console.error(err);

exec.sh

@@ -0,0 +1,11 @@
+#! /bin/bash
+if [ -z "$1" ]; then
+  echo "Usage: $0 <test-name>"
+  exit 1
+fi
+complect -b llvm -f fixtures/${1} -o output/${1}.ll
+llc output/${1}.ll -o output/${1}.s
+clang output/${1}.ll -s -o output/${1}
+./output/${1}
+valgrind --tool=memcheck --leak-check=full ./output/${1} 2>&1 | grep -E "(allocs| frees| allocated| lost)"
+

fixtures/fib

@@ -1,7 +1,7 @@
 make a 0
 make b 1
 make t 0
-make n 12
+make n 32
 as n > 0
   n = n - 1
   assign t a

fixtures/fizzbuzz

@@ -3,10 +3,10 @@ make f 0
 make b 0
 make output ''
 
-as i <= 16
+as i <= 1024
+  assign output ''
   f = i % 3
   b = i % 5
-  assign output ''
   if f == 0
     output = output + 'Fizz'
   endif

lib/ast/ast-builder.js

@@ -0,0 +1,113 @@
+/* Complect - Compiler for the Complect programming language
+ *
+ * Copyright © 2024 Jarrod Connolly
+ * MIT License
+*/
+
+import { TokenType } from '../tokenizer/token-type.js';
+import { Program, VariableDeclaration, AssignmentExpression, BinaryExpression, IfStatement, WhileStatement, PrintStatement, FreeStatement, Identifier, NumericLiteral, StringLiteral } from './ir-nodes.js';
+
+// ASTBuilder builds an IR AST from tokens using a state machine
+export class ASTBuilder {
+  constructor() {
+    this.tokens = null;
+    this.index = 0;
+  }
+
+  // Main entry point: build AST from async generator of tokens
+  async build(tokens) {
+    const allTokens = [];
+    for await (const token of tokens) {
+      allTokens.push(token);
+    }
+    this.tokens = allTokens;
+    this.index = 0;
+    this.parse();
+    return this.program;
+  }
+
+  parse() {
+    this.program = new Program(this.parseBlock(null), null);
+  }
+
+  parseBlock(endKeyword) {
+    const statements = [];
+    while (this.index < this.tokens.length) {
+      if (endKeyword && this.tokens[this.index].type === TokenType.keyword && this.tokens[this.index].value === endKeyword) {
+        this.index++;
+        return statements;
+      }
+      statements.push(this.parseStatement());
+    }
+    if (endKeyword) throw new Error(`Expected '${endKeyword}'`);
+    return statements;
+  }
+
+  parseStatement() {
+    const token = this.tokens[this.index++];
+    if (token.type === TokenType.keyword) {
+      if (token.value === 'make') {
+        const idToken = this.tokens[this.index++];
+        if (idToken.type !== TokenType.identifier) throw new Error(`Expected identifier after 'make'`);
+        const expr = this.parseFullExpression();
+        return new VariableDeclaration(idToken.value, expr, { start: { line: token.line, column: token.column }, end: { line: token.line, column: token.column } });
+      } else if (token.value === 'assign') {
+        const idToken = this.tokens[this.index++];
+        if (idToken.type !== TokenType.identifier) throw new Error(`Expected identifier after 'assign'`);
+        const expr = this.parseFullExpression();
+        return new AssignmentExpression(idToken.value, expr, { start: { line: token.line, column: token.column }, end: { line: token.line, column: token.column } });
+      } else if (token.value === 'print') {
+        const expr = this.parseFullExpression();
+        return new PrintStatement(expr, { start: { line: token.line, column: token.column }, end: { line: token.line, column: token.column } });
+      } else if (token.value === 'free') {
+        const idToken = this.tokens[this.index++];
+        if (idToken.type !== TokenType.identifier) throw new Error(`Expected identifier after 'free'`);
+        return new FreeStatement(idToken.value, { start: { line: token.line, column: token.column }, end: { line: token.line, column: token.column } });
+      } else if (token.value === 'if') {
+        const test = this.parseFullExpression();
+        const consequent = this.parseBlock('endif');
+        return new IfStatement(test, consequent, { start: { line: token.line, column: token.column }, end: { line: token.line, column: token.column } });
+      } else if (token.value === 'as') {
+        const test = this.parseFullExpression();
+        const body = this.parseBlock('repeat');
+        return new WhileStatement(test, body, { start: { line: token.line, column: token.column }, end: { line: token.line, column: token.column } });
+      } else {
+        throw new Error(`Unexpected keyword: ${token.value}`);
+      }
+    } else if (token.type === TokenType.identifier) {
+      // assignment
+      const opToken = this.tokens[this.index++];
+      if (opToken.type !== TokenType.operator || opToken.value !== '=') throw new Error(`Expected '=' after identifier`);
+      const expr = this.parseFullExpression();
+      return new AssignmentExpression(token.value, expr, { start: { line: token.line, column: token.column }, end: { line: token.line, column: token.column } });
+    } else {
+      throw new Error(`Unexpected token: ${token.type} ${token.value}`);
+    }
+  }
+
+  // Parse a full expression, including binary operations
+  parseFullExpression() {
+    const token = this.tokens[this.index++];
+    let expr = this.parseExpression(token);
+    while (this.index < this.tokens.length && this.tokens[this.index].type === TokenType.operator) {
+      const opToken = this.tokens[this.index++];
+      const rightToken = this.tokens[this.index++];
+      const right = this.parseExpression(rightToken);
+      expr = new BinaryExpression(expr, opToken.value, right, { start: expr.loc.start, end: right.loc.end });
+    }
+    return expr;
+  }
+
+  // Parse a single expression from a token
+  parseExpression(token) {
+    if (token.type === TokenType.identifier) {
+      return new Identifier(token.value, { start: { line: token.line, column: token.column }, end: { line: token.line, column: token.column } });
+    } else if (token.type === TokenType.number) {
+      return new NumericLiteral(Number(token.value), { start: { line: token.line, column: token.column }, end: { line: token.line, column: token.column } });
+    } else if (token.type === TokenType.string) {
+      return new StringLiteral(token.value, { start: { line: token.line, column: token.column }, end: { line: token.line, column: token.column } }); // Already without quotes from preprocessor
+    } else {
+      throw new Error(`Unexpected expression token: ${token.type} ${token.value}`);
+    }
+  }
+}