Content Data Flow

Relevant source files

The following files were used as context for generating this wiki page:

• contentlayer.config.ts
• src/pages/[...slug].tsx
• src/pages/dynamic/[...slug].tsx
• src/middleware.ts
• src/data/sidebar.ts
• src/layouts/DocsLayout.tsx
• src/pages/api/llms-docs.md.ts
• src/types.ts
• meilisearch-docs-scraper.config.json

The content data flow in the railways-docs project describes the entire lifecycle of documentation content, from raw Markdown files to the final rendered HTML pages served to users. The system is built on Next.js and leverages Contentlayer to process and manage .mdx files. It features both static site generation (SSG) for most pages and a dynamic, server-side rendering path for serving raw Markdown content on demand.

This architecture ensures fast page loads through static generation while providing flexibility for alternative content formats via middleware and dynamic routes. The content's structure is centrally managed, and specialized endpoints exist for search indexing and consumption by Large Language Models (LLMs).

Overall Architecture

The system ingests Markdown files from the src/docs directory, processes them at build time using Contentlayer, and generates static pages. A middleware layer intercepts incoming requests to route them to either the statically generated content or a dynamic renderer, which can also serve raw Markdown.

The following diagram illustrates the high-level data flow from content creation to user consumption.

Content Source and Structure

The documentation's content and navigational structure are defined in separate parts of the codebase.

Markdown Content

All documentation pages are written in Markdown (.md) and stored in the src/docs directory. Contentlayer is configured to treat these files as the Page document type.

Sources: contentlayer.config.ts:25

Sidebar and Navigation Structure

The navigation hierarchy, including sections, subsections, and page order, is explicitly defined in src/data/sidebar.ts. This file exports a sidebarContent array which structures the entire documentation. The data structures for this configuration are defined in src/types.ts.

The relationship between these types is shown below.

Sources: src/data/sidebar.ts:13-306, src/types.ts:5-22

The flattenSidebarContent utility function in DocsLayout.tsx is used to create a flat array of all pages (IPage[]), which is useful for implementing "Next" and "Previous" page navigation.

Sources: src/layouts/DocsLayout.tsx:40-63

Content Processing with Contentlayer

Contentlayer is the core component for processing Markdown files. It reads the .md files, parses their frontmatter and content, and generates typed data and JSON files that can be easily imported into the Next.js application.

Document Type Definition

The contentlayer.config.ts file defines a Page document type.

Field	Type	Required	Description
title	string	Yes	The title of the page.
description	string	Yes	The description of the page for SEO.
url	string	No (Computed)	The URL path, computed from the file path.

Sources: contentlayer.config.ts:7-24

The configuration also specifies the content directory and the MDX plugins to be used.

// contentlayer.config.ts
export default makeSource({
  contentDirPath: "src/docs",
  documentTypes: [Page],
  mdx: { remarkPlugins: [remarkSlug, remarkAutoLinkHeadings, remarkGfm] },
});

Sources: contentlayer.config.ts:25-29

This setup automatically adds IDs to headings (remark-slug), links them (remark-autolink-headings), and enables GitHub Flavored Markdown features (remark-gfm).

Page Rendering and Routing

The project uses a combination of static and dynamic rendering, with a middleware to route requests appropriately.

Static Page Rendering

Most documentation pages are statically generated at build time for optimal performance.

1. getStaticPaths: In src/pages/[...slug].tsx, this function reads all page URLs from the allPages object (generated by Contentlayer) and tells Next.js which paths to pre-render.
2. getStaticProps: For each path, this function finds the corresponding page data from allPages and passes it, along with the raw Markdown content, as props to the PostPage component.
3. PostPage Component: This component uses the useMDXComponent hook to render the MDX content. It also injects custom React components (e.g., CodeBlock, Banner) to be used within the Markdown files.
4. Layout: The rendered content is wrapped in a Layout component which provides the overall page structure, including the sidebar and SEO metadata.

Sources: src/pages/[...slug].tsx:58-81

Dynamic Rendering and Markdown-as-API

A special dynamic route, src/pages/dynamic/[...slug].tsx, exists to handle requests for raw Markdown content. This is controlled by a middleware.

The following sequence diagram shows how a request for a Markdown file (e.g., via curl or a ?format=md query) is handled.

Sources: src/middleware.ts:1-33, src/pages/dynamic/[...slug].tsx:59-89

The middleware.ts checks three conditions to determine if the user wants Markdown:

• The path ends with .md.
• The query string contains format=md.
• The Accept header prefers text/markdown over text/html.

If any of these are true, the request is rewritten to the /dynamic route with the format=md query parameter. The getServerSideProps function in dynamic/[...slug].tsx then detects this parameter and sends the raw Markdown as the response.

Sources: src/middleware.ts:7-11, 20-25, src/pages/dynamic/[...slug].tsx:70-77

Specialized Content Endpoints

Beyond standard page rendering, the system has endpoints for other forms of content consumption.

LLM Content Generation

The API route /api/llms-docs.md.ts is designed to generate a single, consolidated text file of the entire documentation for consumption by Large Language Models.

The process is as follows:

1. Get all pages by flattening the sidebarContent.
2. Group pages by their top-level section (e.g., "Overview", "Guides").
3. Iterate through each page, read the corresponding .md file from src/docs.
4. Use gray-matter to separate frontmatter from Markdown content.
5. Clean the Markdown by removing code blocks, inline code styling, and links to get plain text.
6. Append the cleaned title, description, and section content to a single string.
7. Code blocks are extracted and appended separately under a "Code Examples" heading.
8. The final concatenated string is sent as a text/plain response.

Sources: src/pages/api/llms-docs.md.ts

Search Indexing

The project uses Meilisearch for its search functionality. The configuration file meilisearch-docs-scraper.config.json defines how the documentation site is scraped to build the search index.

The scraper starts at the root URL and uses CSS selectors to extract content hierarchies and text.

Selector Key	CSS Selector	Description
lvl0	.sidebar .current-section	The main category of the page.
lvl1	.docs-content h1	The main page title.
lvl2	.docs-content h2	Section headings.
lvl3	.docs-content h3	Subsection headings.
lvl4	.docs-content h4	Sub-subsection headings.
text	.docs-content p, li, a, code, ...	The indexable text content of the page.

Sources: meilisearch-docs-scraper.config.json:8-19

This configuration allows Meilisearch to build a structured search index that understands the hierarchy of the documentation, providing more relevant search results.

Conclusion

The content data flow of the railways-docs project is a well-architected system that balances performance with flexibility. By using Contentlayer for build-time processing and static generation for most pages, it ensures a fast user experience. The addition of a middleware-driven dynamic rendering path provides a powerful "Markdown-as-API" feature. Finally, specialized endpoints for search indexing and LLM consumption demonstrate a thoughtful approach to making the documentation content accessible and useful in various contexts.