Using the Express Middleware

The rendertron-middleware is an Express middleware designed to integrate the Rendertron service with a Node.js application. Its primary function is to enable server-side rendering for bots and web crawlers that do not execute JavaScript, thereby improving SEO for Progressive Web Apps (PWAs) and other modern web applications.

The middleware intercepts incoming HTTP requests, inspects the User-Agent header, and if it matches a known bot, it proxies the request to a specified Rendertron instance. The Rendertron service then renders the page using a headless Chrome browser and returns the serialized HTML. For all other requests (e.g., from regular users or for static assets), the middleware passes control to the next handler in the Express chain, allowing the standard application logic to proceed.

Sources: middleware/README.md, middleware/src/middleware.ts:100-117, README.md:25-30

Core Functionality

The middleware is initialized via the makeMiddleware function, which returns an Express request handler. This handler contains the logic for deciding whether to proxy a request.

Sources: middleware/src/middleware.ts:90-93

Request Interception Flow

The following diagram illustrates the decision-making process for each incoming request.

This diagram shows the request handling logic of the middleware.
Sources: middleware/src/middleware.ts:100-117, middleware/src/test/middleware-test.ts

User-Agent Filtering

By default, the middleware checks the request's User-Agent against a predefined list of bot and crawler user agents. If a match is found, the request is considered for proxying. This list can be customized with a regular expression.

The default list of bot user agents includes:

• Baiduspider
• bingbot
• Embedly
• facebookexternalhit
• LinkedInBot
• outbrain
• pinterest
• quora link preview
• rogerbot
• showyoubot
• Slackbot
• TelegramBot
• Twitterbot
• vkShare
• W3C_Validator
• WhatsApp

Sources: middleware/src/middleware.ts:16-33, middleware/src/middleware.ts:101-102

URL Path Exclusion

To prevent proxying requests for static assets, the middleware uses a regular expression to exclude certain URL paths based on file extensions. Like the user-agent pattern, this can also be customized.

The default excluded file extensions are:
ai, avi, css, dat, dmg, doc, exe, flv, gif, ico, iso, jpeg, jpg, js, less, m4a, m4v, mov, mp3, mp4, mpeg, mpg, pdf, png, ppt, psd, rar, rss, svg, swf, tif, torrent, ttf, txt, wav, wmv, woff, xls, xml, zip.

Sources: middleware/src/middleware.ts:38-44, middleware/src/middleware.ts:102-103, middleware/src/test/middleware-test.ts:106-114

Configuration

The makeMiddleware function accepts a configuration object to customize its behavior.

| Property | Type | Default - | Description - - |
| proxyUrl | string | Required - | Base URL of the Rendertron proxy service. - |
| userAgentPattern | RegExp | new RegExp(botUserAgents.join('|'), 'i') - | RegExp for matching requests by User-Agent header. - |
| excludeUrlPattern | RegExp | new RegExp('\\.(' + staticFileExtensions.join('|') + ')$', 'i') - | RegExp for excluding requests by the path component of the URL. - |
| injectShadyDom | boolean | false - | Force the web components polyfills to be loaded. - |
| timeout | number | 11000 - | Millisecond timeout for the proxy request to Rendertron. - |

Sources: middleware/README.md, middleware/src/middleware.ts:49-88

Basic Usage

To use the middleware, require it and add it to your Express application stack. It is crucial to place it before any static file handlers, like express.static.

const express = require('express');
const rendertron = require('rendertron-middleware');

const app = express();

app.use(rendertron.makeMiddleware({
  proxyUrl: 'http://my-rendertron-instance/render',
}));

app.use(express.static('files'));
app.listen(8080);

Sources: middleware/README.md

Proxying and Error Handling

Proxy Request Flow

This sequence diagram details the interactions between a bot client, the Express application using the middleware, and the Rendertron service.

This diagram shows both a successful proxy and a fallthrough case on error/timeout.
Sources: middleware/src/middleware.ts:100-117, middleware/src/test/middleware-test.ts:151-161, 163-172

Proxy URL Construction

The middleware constructs the final URL to send to the Rendertron service by combining the proxyUrl, the encoded incoming request URL, and optionally the wc-inject-shadydom query parameter.

1. The full incoming URL is determined from the request's protocol, host, and original URL.
2. This URL is then URI-encoded.
3. The final URL is [proxyUrl]/[encoded_incoming_url].
4. If injectShadyDom is true, ?wc-inject-shadydom=true is appended.

For example, a request to http://example.com/foo with proxyUrl: 'https://render-tron.io/render' becomes a request to https://render-tron.io/render/http%3A%2F%2Fexample.com%2Ffoo.

Sources: middleware/src/middleware.ts:106-110, middleware/src/test/middleware-test.ts:77-82, 93-99

Timeout and Error Handling

The middleware uses the request library to proxy to the Rendertron service. It implements a timeout to prevent requests from hanging.

• The default timeout is 11,000 milliseconds. This is slightly longer than the Rendertron service's own default 10-second render limit.
• If the request to the Rendertron service fails (e.g., connection error) or exceeds the timeout, the middleware will log an error to the console and call next(), falling through to the next handler. This ensures the client does not receive a hanging or broken response.
• If the Rendertron service returns an error status code (e.g., 500), that status code and the response body are forwarded directly to the original client.

Sources: middleware/src/middleware.ts:86, 111-117, middleware/src/test/middleware-test.ts:151-172

Summary

The rendertron-middleware provides a simple yet powerful way to add pre-rendering capabilities to an Express-based web application. By selectively proxying requests from search engine crawlers and social media bots to a Rendertron instance, it ensures that JavaScript-heavy applications are correctly indexed and displayed as rich previews, without affecting the experience for regular users. Its configurable nature allows developers to fine-tune which user agents and URL patterns trigger the rendering service, providing a flexible solution for SEO in the modern web ecosystem.