Content Processing with Contentlayer

Relevant source files

The following files were used as context for generating this wiki page:

• contentlayer.config.ts
• src/pages/[...slug].tsx
• src/pages/dynamic/[...slug].tsx
• package.json
• next.config.js
• src/layouts/DocsLayout.tsx
• src/pages/api/llms-docs.md.ts

The Railway Documentation project utilizes Contentlayer to manage and process its documentation, which is written in MDX (Markdown with JSX). Contentlayer acts as a content SDK, transforming the raw .md files from the src/docs directory into structured, type-safe JSON data that can be easily consumed by the Next.js application. This system simplifies content handling by validating data schemas, processing markdown, and generating typed data artifacts during the build process.

The core of this system is the integration between Contentlayer and Next.js. A dedicated configuration defines the content schema, and the build process is augmented to first run Contentlayer before the standard Next.js build. The generated data is then used by Next.js pages to statically generate or server-render documentation pages, providing a streamlined workflow from content creation to final presentation.

Core Configuration

The heart of the content processing pipeline is defined in contentlayer.config.ts. This file configures how Contentlayer finds, parses, and validates the documentation files.

Sources: contentlayer.config.ts

Document Type Definition

A single document type, Page, is defined to represent each markdown file in the src/docs directory. This definition specifies the expected frontmatter fields and computes additional useful properties.

The diagram below illustrates the structure of the Page document type.

The schema for the Page document type is detailed in the table below:

Field	Type	Description	Required	Source
title	string	The title of the page	Yes	contentlayer.config.ts:11-15
description	string	The description of the page	Yes	contentlayer.config.ts:16-20
url	string	Computed URL path (e.g., /<path>)	N/A	contentlayer.config.ts:23-26

MDX Plugin Configuration

Contentlayer is configured to use several remark plugins to enhance the markdown processing. These plugins automatically add features like slugs for headings, anchor links, and support for GitHub Flavored Markdown.

Plugin	Description
remarkSlug	Adds id attributes to headings.
remarkAutoLinkHeadings	Wraps headings in anchor links.
remarkGfm	Adds support for GFM (tables, strikethrough, etc.).

Sources: contentlayer.config.ts:30

Build Process Integration

Contentlayer is tightly integrated into the project's build process, ensuring that content is processed before the Next.js application is built.

Build Script

The build script in package.json explicitly invokes Contentlayer before Next.js. This generates the necessary data in the .contentlayer directory.

"scripts": {
  "build": "contentlayer build && next build",
}

Sources: package.json:7

Next.js Configuration

The next.config.js file uses the withContentlayer higher-order function to wrap the Next.js configuration. This integrates Contentlayer's build-time data generation with Next.js's development and production builds.

// next.config.js
const { withContentlayer } = require("next-contentlayer");

/** @type {import('next').NextConfig} */
const nextConfig = withContentlayer({
  // ... next.js config
});

module.exports = nextConfig;

Sources: next.config.js:1-5, next.config.js:19

The overall build process is visualized below.

Content Rendering

The data generated by Contentlayer is consumed by Next.js pages to render the documentation. The project uses both Static Site Generation (SSG) and Server-Side Rendering (SSR) for different routes.

Static Page Generation ([...slug].tsx)

For most documentation pages, content is statically generated at build time.

1. getStaticPaths: This function reads all page URLs from allPages (an array of all processed Page documents) and provides them to Next.js to pre-render all documentation routes.
2. getStaticProps: For each path, this function finds the corresponding page object from allPages using the URL slug. It then passes the page object and its rawMarkdown content as props to the PostPage component.
3. PostPage Component: This component receives the page prop, extracts the compiled MDX code (page.body.code), and renders it using the useMDXComponent hook from next-contentlayer/hooks.

Sources: src/pages/[...slug].tsx:69-80, src/pages/[...slug].tsx:82-88, src/pages/[...slug].tsx:33-54

Dynamic Page Rendering (dynamic/[...slug].tsx)

Certain paths are rendered on the server at request time. The logic is similar to the static version but uses getServerSideProps.

1. Middleware: A middleware intercepts requests. If a path matches a known documentation page URL, it rewrites the request to /dynamic/.... If the request headers indicate a preference for markdown (text/markdown or ?format=md), it adds a query parameter.
2. getServerSideProps: This function finds the page from allPages. If the format=md query parameter is present, it returns the raw markdown content with frontmatter. Otherwise, it passes the page object and rawMarkdown to the PostPage component for rendering.

This flow allows serving both the rendered HTML page and the raw markdown source from the same URL.

Sources: src/middleware.ts, src/pages/dynamic/[...slug].tsx:60-86

The following sequence diagram shows the request flow for a statically generated page.

Layout and Data Consumption

The final rendered MDX content is wrapped by DocsLayout.tsx, which uses the frontmatter and raw markdown for additional functionality.

The PostPage component passes frontMatter (derived from the Contentlayer page object) and rawMarkdown to the Layout component.

// src/pages/[...slug].tsx
<Layout
  frontMatter={{
    title: page.title,
    description: page.description,
    url: page.url,
  }}
  rawMarkdown={rawMarkdown}
>
  <MDXContent components={componentsWithProps} />
</Layout>

Sources: src/pages/[...slug].tsx:56-65

The DocsLayout component uses these props for:

• SEO: The frontMatter object is passed to the SEO component to generate <title>, <meta>, and structured data tags.
• Page Title: frontMatter.title is used for the <h1> tag.
• Copy as Markdown: The rawMarkdown prop is used with reconstructMarkdownWithFrontmatter to provide a "Copy as Markdown" feature.
• Edit on GitHub Link: The page slug is used to construct a direct link to edit the source file on GitHub.

Sources: src/layouts/DocsLayout.tsx:28-31, src/layouts/DocsLayout.tsx:75-78, src/layouts/DocsLayout.tsx:117, src/layouts/DocsLayout.tsx:129-132

Alternative Content Processing for LLMs

While Contentlayer is the primary pipeline for rendering pages, a separate process exists at the /api/llms-docs.md.ts API route. This endpoint is designed to generate a single, concatenated text file of all documentation for consumption by Large Language Models (LLMs).

This process manually:

1. Reads the sidebar structure from src/data/sidebar.ts.
2. Iterates through each page slug.
3. Reads the corresponding markdown file from src/docs.
4. Uses the gray-matter library to parse frontmatter and content.
5. Strips all markdown formatting (code blocks, links, etc.) to create a plain text representation.
6. Concatenates all content into a single response.

This demonstrates an alternative, ad-hoc content processing pipeline that bypasses Contentlayer for a specific use case.