Integrating with prerender.io Middleware

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• middleware/README.md
• middleware/src/middleware.ts
• src/rendertron.ts
• src/renderer.ts
• middleware/src/test/middleware-test.ts
• src/test/app-test.ts

Rendertron is a headless Chrome rendering solution designed to serve pre-rendered HTML content to bots that do not execute JavaScript, thereby improving SEO for Progressive Web Apps (PWAs). A key aspect of using Rendertron is implementing a differential serving layer, typically as middleware within an application's web server. This layer is responsible for identifying bot traffic and proxying those requests to a running Rendertron service instance.

While Rendertron provides its own Express.js middleware, it is also designed to be compatible with existing solutions like the prerender.io middleware. The integration pattern remains the same: configure the middleware to proxy requests from specific user-agents to the Rendertron service's /render endpoint. This document details the architecture and request flow, using the native rendertron-middleware as a reference implementation.

Sources: README.md:11-16, README.md:40-42

Architecture Overview

The integration involves three main components: the end-user's client (browser or bot), the web application server with the middleware, and the standalone Rendertron service. The middleware acts as a gatekeeper, deciding whether to serve the standard application response or to fetch a pre-rendered version from Rendertron.

The following diagram illustrates the high-level architecture for differential serving with Rendertron.

This diagram shows how the middleware intercepts all requests, forwarding bot requests to Rendertron and serving standard responses to human users.

Sources: middleware/README.md:8-13, middleware/src/middleware.ts:80-87

The Rendertron Middleware

The rendertron-middleware package provides a reference implementation for Express.js applications. It inspects the User-Agent header of incoming requests and proxies matching requests to the Rendertron service.

Sources: middleware/README.md, middleware/package.json

Configuration

The middleware is initialized by calling makeMiddleware with a configuration object. The key properties are detailed below.

Property	Default	Description
proxyUrl	Required	Base URL of the running Rendertron service instance (e.g., http://my-rendertron/render).
userAgentPattern	A RegExp matching a list of common bots like Baiduspider, bingbot, facebookexternalhit, Slackbot, Twitterbot, etc.	RegExp for matching requests by the User-Agent header. If a request matches, it will be proxied.
excludeUrlPattern	A RegExp matching common static file extensions (.css, .js, .jpg, .png, etc.).	RegExp for excluding requests by URL path. Matching requests are never proxied, even if the user agent matches.
injectShadyDom	false	If true, appends ?wc-inject-shadydom=true to the render URL to force web components polyfills to be loaded.
timeout	11000 (ms)	Timeout for the proxy request to Rendertron. If exceeded, the middleware calls next() and serves the standard, non-rendered response. This is slightly higher than the default Rendertron 10s render timeout.

Sources: middleware/README.md:23-31, middleware/src/middleware.ts:60-78

Request Processing Flow

When a request is received, the middleware executes the following logic. This flow is what any compatible middleware (including prerender.io's) would need to replicate.

This sequence diagram shows the decision-making process within the middleware, including user-agent checks, URL exclusion, and the final proxy to the Rendertron service.

Sources: middleware/src/middleware.ts:80-96, middleware/src/test/middleware-test.ts:79-84, middleware/src/test/middleware-test.ts:121-127

Rendertron Service API

The middleware communicates with the Rendertron service via its HTTP API. For prerendering, the /render endpoint is used.

GET /render/:url

This is the primary endpoint for rendering a web page. The middleware constructs the request to this endpoint.

• URL Parameter: :url - The full, URI-encoded URL of the page to be rendered.
• Query Parameters:
  • mobile: If present (e.g., ?mobile), the page is rendered with a mobile user-agent and viewport.
  • wc-inject-shadydom: If true, forces Shady DOM polyfills for web components. The rendertron-middleware can add this automatically.

The Rendertron service performs several checks before rendering:

1. URL Restriction: It validates that the protocol is http or https to prevent requests for local files (file:///). If the check fails, it returns a 403 Forbidden status.
2. Rendering: It uses Puppeteer to navigate to the URL, waits for network activity to cease (networkidle0), and serializes the resulting DOM.
3. Status Code: The response status code from the original page request is preserved. This can be overridden by a <meta name="render:status_code" content="404" /> tag in the rendered page's HTML.

Sources: README.md:46-51, src/rendertron.ts:74-87, src/renderer.ts:114-126, src/test/app-test.ts:100-104

Using with prerender.io Middleware

Rendertron is compatible with prerender.io middleware because it exposes a similar API endpoint. To use it, you would install the appropriate prerender.io middleware for your framework and configure it to point to your Rendertron instance instead of the Prerender.io service.

For example, you would set the prerender service URL to http://<your-rendertron-host>:<port>/render. The prerender.io middleware would then append the target URL, effectively making the same request as rendertron-middleware. Note that the list of user agents used by the prerender.io middleware may differ from the default list in rendertron-middleware.

Sources: README.md:40-42, middleware/src/middleware.ts:20-36

Summary

Integrating a rendering service like Rendertron requires a middleware component to differentiate between bot and human traffic. While Rendertron provides a native Express.js middleware, its simple and robust /render API endpoint ensures compatibility with other solutions like the prerender.io middleware. The core task is to configure the middleware to identify bot user-agents, avoid proxying static assets, and direct rendering requests to the correct Rendertron service URL. This setup enables PWAs and other JavaScript-heavy sites to become fully indexable by search engines and crawlers.