AI Content Moderation Hooks

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• docs/PLAN.md
• src/hooks/use-moderation.ts
• src/app/api/moderation/route.ts
• src/lib/ai/moderation-client.ts

AI Content Moderation Hooks provide a standardized interface for auditing user-generated content across the Inkweave platform. These hooks interact with internal API routes to process content through AI safety models, allowing administrators to identify and flag policy violations programmatically.

The system operates within the inkweave-admin environment, which acts as a private administrative layer over the upstream/inkweave submodule. By centralizing moderation logic in React hooks, the application ensures consistent enforcement of safety guidelines across various administrative views.

Sources: README.md:1-5, docs/PLAN.md

System Architecture

The moderation system follows a tiered architecture involving the client-side UI, a centralized React hook, and a server-side API route that communicates with external AI services.

Data Flow

When an administrator initiates a moderation check, the data flows through the following stages:

1. Trigger: The UI component calls the hook with specific content.
2. Request: The hook dispatches an authorized POST request to the internal moderation API.
3. Analysis: The API route forwards the payload to the AI moderation client.
4. Response: The AI service returns safety scores and category flags.
5. Update: The hook updates the local state, triggering a UI re-render to display the results.

The diagram above illustrates the request-response cycle for content auditing.
Sources: src/hooks/use-moderation.ts, src/app/api/moderation/route.ts

Client-Side Implementation

The primary interface for moderation is the useModeration hook. This hook encapsulates the state management and network logic required to perform content checks.

The useModeration Hook

The useModeration hook manages three primary states: loading, error, and results. It exposes a checkContent function that administrators invoke to process text or image data.

• Input: A content string or object containing the metadata to be audited.
• Behavior: It sets the loading state to true, clears previous errors, and performs the fetch operation.
• Output: Returns a ModerationResult object containing boolean flags for categories like "harassment," "hate," or "self-harm."

Sources: src/hooks/use-moderation.ts:10-45

Server-Side Processing

The server-side components handle authentication and communication with the AI provider. This ensures that sensitive API keys for the AI services remain secure within the Vercel environment.

API Endpoint Configuration

The moderation API endpoint validates the incoming request before communicating with the AI model.

Parameter	Type	Description
content	string	The raw text or URL of the content to be analyzed.
context	string	Optional metadata about where the content appears.
userId	string	The identifier for the user who created the content.

Sources: src/app/api/moderation/route.ts:5-20

AI Client Logic

The ModerationClient in src/lib/ai/moderation-client.ts abstracts the specific implementation details of the AI provider. If the AI service detects a violation, the client maps the provider-specific error codes to internal Inkweave moderation categories.

// Example mapping logic in moderation-client.ts
const mapResults = (aiOutput) => {
  return {
    isFlagged: aiOutput.flagged,
    categories: aiOutput.categories,
    scores: aiOutput.category_scores
  };
};

Sources: src/lib/ai/moderation-client.ts:15-30

Implementation Procedure

To implement a moderation check within a new admin tool, follow these steps:

1. Import the useModeration hook from the hooks directory.
2. Initialize the hook within the functional component.
3. Pass the target content to the checkContent function inside an event handler.
4. Monitor the isFlagged property to update the UI status of the content.

Sources: docs/PLAN.md, src/hooks/use-moderation.ts

Summary

AI Content Moderation Hooks streamline the administrative workflow by providing real-time safety analysis. By leveraging hooks and a centralized API, inkweave-admin maintains a secure and scalable method for monitoring content across the Inkweave ecosystem. This architecture allows for rapid updates to moderation policies and AI model versions without disrupting the UI components.