The Rendering Pipeline

Relevant source files

The following files were used as context for generating this wiki page:

• src/renderer.ts
• src/rendertron.ts
• README.md
• src/test/app-test.ts
• middleware/src/middleware.ts
• package.json

Rendertron is a headless Chrome rendering solution built with Puppeteer, designed to render and serialize web pages on the fly. The core of this system is its rendering pipeline, which processes incoming requests for a specific URL, uses a headless browser to load and render the page content, and then returns either the serialized static HTML or a screenshot. This capability is primarily aimed at serving fully-rendered content to bots and web crawlers that do not execute JavaScript, thereby improving the Search Engine Optimization (SEO) of Progressive Web Apps (PWAs) and other modern web applications.

The pipeline is exposed via a standalone HTTP server, which offers distinct endpoints for rendering and for taking screenshots. It is designed to be used with a middleware layer in the client application, which intelligently proxies requests from bots to the Rendertron service. The pipeline includes features to handle mobile viewports, web components, and custom status codes, and has a configurable timeout to manage rendering budgets.

Sources: README.md:4-14, src/rendertron.ts:13-16, middleware/README.md:6-10

Architecture Overview

The Rendertron rendering pipeline consists of a Koa.js HTTP server that orchestrates the rendering process. The server exposes API endpoints that accept a URL to be rendered. When a request is received, it is passed to the Renderer class, which uses a Puppeteer-managed browser instance to perform the actual rendering. Optional caching can be enabled to store and serve rendered results from Google Cloud Datastore.

The diagram below illustrates the high-level flow from a client request to a rendered response.

Sources: src/rendertron.ts, src/renderer.ts, middleware/src/middleware.ts, README.md:10-14

Request Handling and API

The entry point to the rendering pipeline is the Rendertron class, which initializes and configures a Koa server. This server is responsible for routing incoming HTTP requests to the appropriate handler functions.

Sources: src/rendertron.ts:17-20

API Endpoints

The service exposes two primary endpoints for rendering and screenshotting, along with a health check.

Method	Endpoint	Description
GET	/render/<url>	Renders the specified URL and returns the serialized HTML content.
GET	/screenshot/<url>	Takes a screenshot of the specified URL and returns a JPEG image.
POST	/screenshot/<url>	Takes a screenshot with advanced options provided in the JSON request body.
GET	/_ah/health	A health check endpoint that responds with "OK" and a 200 status.

Sources: src/rendertron.ts:46-56, README.md:27-40

URL Validation

Before processing any request, the pipeline validates the requested URL to prevent security vulnerabilities. The restricted method ensures that only http and https protocols are allowed. This blocks attempts to access local files (e.g., file:///) or other schemes. If an invalid URL is detected, the server responds with a 403 Forbidden status.

Sources: src/rendertron.ts:74-84, src/test/app-test.ts:121-132

The Core Renderer

The Renderer class is the heart of the pipeline, encapsulating the logic for interacting with the headless browser via Puppeteer. It is instantiated with a browser instance and the application configuration.

Sources: src/renderer.ts:20-25, src/rendertron.ts:30-31

Page Serialization (serialize method)

The serialize method handles the process of rendering a page and returning its static HTML. It is a multi-step process that ensures content is fully loaded and scripts are handled correctly before serialization.

The sequence diagram below details the steps involved in the serialization process.

Sources: src/renderer.ts:27-133

Page Screenshot (screenshot method)

The screenshot method is similar to serialize but captures a JPEG image of the rendered page instead of its HTML content. It also supports custom dimensions and other Puppeteer screenshot options.

Key steps include:

1. Creating a new page and setting the viewport.
2. Navigating to the URL and waiting for network activity to cease (networkidle0).
3. Checking for forbidden headers (metadata-flavor: Google) to prevent access to cloud metadata services.
4. Taking the screenshot with specified options. The format is always jpeg and encoding is binary.

Sources: src/renderer.ts:135-172, README.md:36-49

Page Post-Processing

After the page is loaded in the headless browser but before the content is serialized, the renderer executes several client-side scripts to clean up the DOM and ensure correctness.

Script and Import Stripping

The stripPage function is executed on the page to remove most <script> tags and all <link rel="import"> tags. This prevents the browser from executing additional JavaScript or loading more resources when a user or bot views the serialized HTML. However, it specifically preserves script tags with type="application/ld+json", which are used for structured data and are important for SEO.

Sources: src/renderer.ts:33-40, src/test/app-test.ts:80-92

Web Component Polyfills

To ensure Web Components render correctly, especially those using Shadow DOM, Rendertron injects polyfills before the page loads. It forces ShadyDOM and ShadyCSS to be active, which are shims that provide cross-browser compatibility for Shadow DOM. This is controlled by the wc-inject-shadydom query parameter.

Sources: src/renderer.ts:82-84, README.md:72-81, src/test/app-test.ts:58-79

Status Code Handling

The final HTTP status code of the response is determined by the following logic:

1. The status code from the initial response of the target URL is used.
2. A 304 Not Modified status is treated as 200 OK.
3. If the original status code was 200, the renderer looks for a meta tag in the page's head: <meta name="render:status_code" content="XXX" />. If found, the content value overrides the status code. If multiple such tags exist, the first one is used.
4. If the original status code was not 200 (e.g., 404, 500), it is preserved, and the meta tag is ignored.

This flow is confirmed by tests that check for forwarded status codes and meta tag overrides.

Sources: src/renderer.ts:111-125, src/test/app-test.ts:94-110, test-resources/http-meta-status-code.html:14

Configuration

The rendering pipeline's behavior can be customized via a config.json file or environment variables.

Option	Default	Description
datastoreCache	false	Enables caching of rendered pages in Google Cloud Datastore.
timeout	10000	The timeout in milliseconds for rendering a page.
port	3000	The port for the Rendertron service. Overridden by process.env.PORT.
width	1000	The default viewport width for rendering.
height	1000	The default viewport height for rendering.

Sources: README.md:99-105, src/rendertron.ts:26, src/renderer.ts:24

Caching

Rendertron supports an optional caching layer using Google Cloud Datastore. When datastoreCache is enabled in the configuration, the DatastoreCache class provides a Koa middleware that intercepts requests. If a request is found in the cache, the stored response (including content, status, and headers) is served directly, bypassing the entire rendering process. Only successful responses (status 200) are cached.

Sources: src/rendertron.ts:49-52, src/test/datastore-cache-test.ts