Using Community Middleware
Relevant source files
The following files were used as context for generating this wiki page:
Rendertron is a standalone HTTP server that renders pages using headless Chrome. To integrate it with a web application, a differential serving layer is required. This layer, typically implemented as middleware, intercepts incoming requests, identifies which ones are from bots (e.g., search engine crawlers, social media scrapers), and proxies them to the Rendertron service for pre-rendering. This ensures that bots receive fully-rendered HTML, which is crucial for SEO and link previews.
While the Rendertron project provides an official middleware for Express.js, there are also community-maintained middleware solutions for other platforms like Firebase Functions and ASP.NET Core. The system is also compatible with prerender.io middleware. This document focuses on the official Express.js middleware as a reference implementation and lists other available community options.
Sources: README.md:20-25, middleware/README.md:6-11
Express.js Middleware (rendertron-middleware)
The rendertron-middleware package provides a simple way to integrate Rendertron with an Express.js application. Its primary function is to inspect the User-Agent header of each request. If the user agent matches a configurable list of known bots, the middleware forwards the request to a specified Rendertron instance.
Sources: middleware/README.md:6-11, middleware/src/middleware.ts
How It Works
The middleware function, created by makeMiddleware, follows a specific logic to decide whether to proxy a request or pass it to the next handler in the Express chain.
The diagram below illustrates the decision flow for an incoming request.
This flow ensures that only requests from specified bots for non-static content are sent for rendering.
Sources: middleware/src/middleware.ts:102-106
Configuration
The middleware is configured by passing an options object to the makeMiddleware function. The proxyUrl is the only required property.
| Property | Default | Description |
|---|---|---|
proxyUrl | Required | Base URL of the running Rendertron proxy service. |
userAgentPattern | A RegExp of known bots like Baiduspider, bingbot, facebookexternalhit, Twitterbot, and more. | RegExp for matching requests by the User-Agent header. |
excludeUrlPattern | A RegExp matching common static file extensions like .css, .js, .jpg, .png, etc. | RegExp for excluding requests by the path component of the URL. |
injectShadyDom | false | If true, adds ?wc-inject-shadydom=true to the render URL to force web components polyfills to be loaded. |
timeout | 11000 | Millisecond timeout for the proxy request to Rendertron. If exceeded, the middleware calls next() and serves the standard response. This should be slightly higher than the Rendertron service timeout. |
Sources: middleware/README.md:21-32, middleware/src/middleware.ts:74-96
Request Flow and Proxying
When the middleware determines a request should be proxied, it constructs a new URL to call the Rendertron service's /render/ endpoint. The original request's full URL is URI-encoded and appended to the proxyUrl.
The following sequence diagram shows the interactions between a bot, an application using the middleware, and the Rendertron service.
If the request to the Rendertron service fails (e.g., due to a timeout or network error), the middleware will log the error and call next(), allowing the application to serve the non-prerendered version of the page as a fallback.
Sources: middleware/src/middleware.ts:107-116, middleware/src/test/middleware-test.ts:143-151
Identifying Bots and Static Files
The middleware comes with default regular expressions to identify bots and exclude common static file types. These can be overridden in the configuration.
Default Bot User Agents
The default userAgentPattern is built from a list of common crawler and bot user agents.
| Bot User Agent |
|---|
Baiduspider |
bingbot |
Embedly |
facebookexternalhit |
LinkedInBot |
outbrain |
pinterest |
quora link preview |
rogerbot |
showyoubot |
Slackbot |
TelegramBot |
Twitterbot |
vkShare |
W3C_Validator |
WhatsApp |
Sources: middleware/src/middleware.ts:24-41
Default Excluded File Extensions
The default excludeUrlPattern prevents requests for static assets from being sent to Rendertron.
| File Extension | File Extension | File Extension |
|---|---|---|
ai | jpeg | ppt |
avi | jpg | psd |
css | js | rar |
dat | less | rss |
dmg | m4a | svg |
doc | m4v | swf |
exe | mov | tif |
flv | mp3 | torrent |
gif | mp4 | ttf |
ico | mpeg | txt |
iso | mpg | wav |
wmv | woff | xls |
xml | zip |
Sources: middleware/src/middleware.ts:47-53
Other Community Middleware
The Rendertron architecture is flexible, and the community has created middleware for other popular frameworks.
- Firebase Functions: A community-maintained solution for serverless environments on Google Cloud.
- ASP.NET Core: A middleware implementation for the ASP.NET Core framework.
Additionally, Rendertron is compatible with middleware designed for the prerender.io service, offering even more integration options.
Sources: README.md:29-33
Summary
Using middleware is the standard and recommended way to integrate the Rendertron service into a web application. The middleware acts as a smart proxy, selectively forwarding requests from bots to be pre-rendered, which is essential for modern web applications that rely on client-side JavaScript to be indexed correctly by search engines and to display rich previews on social media platforms. The official Express.js middleware provides a robust, configurable, and easy-to-use solution, while community contributions extend this capability to other ecosystems.