API: Dynamic robots.txt Generation

Relevant source files

The following files were used as context for generating this wiki page:

• src/pages/robots.txt.ts
• astro.config.ts
• package.json
• README.md
• src/pages/database/data.json.ts

The CyberPath project utilizes the Astro framework to dynamically generate the robots.txt file. This file serves as a crucial instruction set for web crawlers, guiding them on which parts of the site to crawl or ignore. By generating it dynamically, the robots.txt file can be programmatically updated, ensuring it always contains accurate information, most importantly the correct location of the site's sitemap.

This functionality is implemented as a server-side API route within the Astro project structure. It leverages the site's base URL configuration to construct an absolute path to the sitemap, making the setup robust and portable across different environments.

Sources: src/pages/robots.txt.ts, README.md:1-4

API Endpoint: GET /robots.txt

The dynamic generation is handled by a single API endpoint that responds to GET requests for the /robots.txt path.

Property	Description
Path	/robots.txt
Method	GET
Handler	src/pages/robots.txt.ts
Response	A text/plain response containing the contents of the robots.txt file.
Framework	This is an Astro API Route. The file-based routing convention places robots.txt.ts in the src/pages/ directory to handle requests for /robots.txt.

Sources: src/pages/robots.txt.ts:13-16, src/pages/database/data.json.ts:28-30

Core Logic and Data Flow

When a client (like a search engine crawler) requests /robots.txt, the Astro server executes the logic defined in src/pages/robots.txt.ts. The process involves retrieving the site's base URL, constructing the full sitemap URL, and generating the text content for the response.

The following sequence diagram illustrates the request-response flow.

This diagram shows how a request for robots.txt is handled by the API route, which uses the site configuration to build the final response.

Sources: src/pages/robots.txt.ts:13-16, astro.config.ts:11

Key Functions

GET Handler

This is the main export of the module and serves as the entry point for the API route. It receives the Astro API context, which includes the site object.

// src/pages/robots.txt.ts:13-16
export const GET: APIRoute = ({ site }): Response => {
  const sitemapURL = new URL("sitemap-index.xml", site);
  return new Response(getRobotsTxt(sitemapURL));
};

The handler's primary responsibilities are:

1. Constructing the full sitemapURL by appending sitemap-index.xml to the base site URL.
2. Calling getRobotsTxt() with the generated URL.
3. Returning a new Response object with the generated text.

Sources: src/pages/robots.txt.ts:13-16

getRobotsTxt()

This helper function takes a URL object and returns the formatted string content for the robots.txt file.

// src/pages/robots.txt.ts:3-7
const getRobotsTxt = (sitemapURL: URL) => `
User-agent: *
Allow: /

Sitemap: ${sitemapURL.href}
`;

Sources: src/pages/robots.txt.ts:3-7

Configuration

The dynamic nature of this API relies on configuration defined in other parts of the project.

Site URL

The base URL for the website is defined in astro.config.ts. This site property is passed to the GET handler and is essential for creating the absolute URL for the sitemap.

// astro.config.ts:10-11
export default defineConfig({
    site:         "https://cyberpath-hq.com",
    // ...
});

Sources: astro.config.ts:11

Sitemap Integration

The robots.txt file points to a sitemap located at /sitemap-index.xml. This sitemap is generated by the @astrojs/sitemap integration, which is configured in astro.config.ts and listed as a dependency in package.json.

// astro.config.ts:12-25
// ...
    integrations: [
        // ...
        sitemap(),
        // ...
    ],
// ...

The presence of this integration ensures that the sitemap-index.xml file referenced in robots.txt actually exists.

Sources: astro.config.ts:21, package.json:15

robots.txt Content

The generated robots.txt file contains three directives.

Directive	Value	Description
User-agent	*	The rules apply to all web crawlers.
Allow	/	Crawlers are permitted to access all content on the site starting from the root.
Sitemap	https://cyberpath-hq.com/...	Provides the absolute URL to the site's XML sitemap, helping crawlers discover all crawlable pages more efficiently. The URL is dynamically generated based on the site config.

This configuration is considered SEO-friendly as it allows full crawling access and provides a direct link to the sitemap.

Sources: src/pages/robots.txt.ts:3-7