Syntax Highlighting Engine

Relevant source files

The following files were used as context for generating this wiki page:

• js/highlight/prettify.js
• batch-prettify-node.js
• html/css/style.css
• README.md
• batch-prettify.js
• create_metafiles.rb

The Syntax Highlighting Engine in the SICP-Rust project is a multi-stage system designed to provide modern, visual code formatting for the "Structure and Interpretation of Computer Programs" (SICP) digital editions. It primarily targets Scheme (Lisp) and Rust code blocks, which are central to this version of the book. The engine utilizes a core JavaScript lexer for tokenization and is integrated into the project's build pipeline to enable server-side or offline processing.

The engine's scope covers the identification of language-specific keywords, comments, strings, and literals, followed by the application of CSS classes to these tokens. While it can run in a browser environment, the project predominantly uses it via automation scripts during the HTML generation process to ensure that the final output is pre-rendered with highlighting.

Sources: README.md:15-20, batch-prettify-node.js:6-9

Core Lexer Architecture

The heart of the system is a regular expression-based lexer implemented in prettify.js. This engine decomposes source code into a flat list of "decorations," which consist of character indices and their corresponding style markers.

Tokenization Logic

The engine uses a sophisticated pattern combination strategy. Multiple regular expressions representing different token types (keywords, strings, comments) are merged into a single global regex to ensure linear-time scanning.

The diagram shows the data flow from raw input code to the final HTML output with embedded styling spans.
Sources: js/highlight/prettify.js:274-280, js/highlight/prettify.js:564-580

Style Definitions

Tokens are mapped to specific CSS classes defined as constants within the JavaScript engine.

Style Constant	CSS Class	Description
PR_STRING	str	String literals
PR_KEYWORD	kwd	Language keywords
PR_COMMENT	com	Source comments
PR_TYPE	typ	Type names
PR_LITERAL	lit	Numeric, boolean, or null literals
PR_PUNCTUATION	pun	Symbols and punctuation
PR_PLAIN	pln	Standard text

Sources: js/highlight/prettify.js:167-210

Language Support and Registration

The engine supports multiple languages by registering specific keyword sets and pattern handlers. While prettify.js contains defaults, the project extends this for Rust and Lisp/Scheme.

Rust Implementation

The engine includes a specific keyword list for Rust, including flow control and structural keywords.

var RUST_KEYWORDS = [
  FLOW_CONTROL_KEYWORDS,
  "as,assert,const,copy,drop,"
  + "enum,extern,fail,false,fn,impl,let,log,loop,match,mod,move,mut,priv,"
  + "pub,pure,ref,self,static,struct,true,trait,type,unsafe,use",
];

Sources: js/highlight/prettify.js:146-150

Extension Registry

Handlers are registered via the registerLangHandler function, which maps file extensions (e.g., rs, rust, scm, lisp) to specific lexing functions.

registerLangHandler(
  sourceDecorator({
    "keywords": RUST_KEYWORDS,
    "cStyleComments": true,
    "multilineStrings": true,
  }),
  ["rc", "rs", "rust"]
);

In the project's implementation, the langExtension is often bluntly set to "scm" for Scheme processing during batch jobs.
Sources: js/highlight/prettify.js:846-855, js/highlight/prettify.js:952

Batch Processing and Build Integration

The project employs a "batch-prettify" workflow using Node.js and JSDOM to process HTML files offline. This eliminates the need for the client's browser to execute the heavy lexing logic on every page load.

Build Workflow

The build script (batch-prettify-node.js) automates the highlighting process:

1. Environment Setup: Loads HTML files and initializes a JSDOM virtual environment.
2. Script Injection: Injects prettify.js, lang-lisp.js, and lang-rust.js into the virtual document.
3. Target Selection: Identifies <pre> blocks with classes lisp, rust, or example and adds the prettyprint class.
4. Execution: Calls window.prettyPrint() synchronously within the JSDOM context.
5. Cleanup: Removes the script tags before saving the processed HTML back to disk.

The sequence shows how the project performs server-side syntax highlighting using a headless browser environment.
Sources: batch-prettify-node.js:45-55, batch-prettify-node.js:77-96, batch-prettify.js:46-60

Visual Styling

The visual representation of highlighted code is controlled via style.css. The engine wraps tokens in <span> tags with classes like kwd or str, which the CSS then colors.

CSS Integration

The project uses specific font families for code blocks, including "Inconsolata LGC" and "DejaVu Sans Mono".

pre, code, tt, samp {
  font-family: "Inconsolata LGC", "DejaVu Sans Mono", monospace;
  font-size: 0.8rem;
}

pre, pre code, pre var {
  font-size: 0.77rem;
  line-height: 150%;
}

Specific styling for Lisp blocks ensures that mathematical symbols rendered via MathJax inside code maintain visual consistency with the surrounding text.
Sources: html/css/style.css:168-176, html/css/style.css:178-180

Summary

The Syntax Highlighting Engine provides a robust framework for displaying technical code within the SICP-Rust project. By combining a flexible, regex-based lexer with a Node.js-based build step, the project achieves high-quality syntax highlighting that is performance-optimized for the end-user. This architecture supports the project's goal of creating a modern, readable version of SICP that bridges the gap between classic Scheme examples and contemporary Rust implementations.