Data Format (JSON)
Relevant source files
The following files were used as context for generating this wiki page:Introduction
Cyberpath Sentinel is a document-based DBMS where the filesystem serves as the database. The core data format for documents is JSON (JavaScript Object Notation). This choice underpins the project's primary goals of trust, transparency, and compliance. By storing data in a human-readable, plain-text format like JSON, every piece of data becomes directly inspectable, auditable, and easy to manage with standard command-line tools.
This approach ensures zero vendor lock-in, as the data is not stored in a proprietary binary format. It can be easily read, migrated, or exported to other systems like PostgreSQL, MongoDB, or formats such as CSV and XML. The use of JSON is a fundamental design decision that enables many of Sentinel's key features, including Git-based versioning, OS-level security, and straightforward compliance with regulations like GDPR.
Sources: README.md:1, 40-43, 13-15, IMPLEMENTATION_PLAN.md:14-18
Data Storage Model
Sentinel's storage model directly maps database concepts to filesystem structures. A "Store" is a root directory, a "Collection" is a sub-directory, and each "Document" is a file within its collection's directory. The primary key of a document is its filename.
Sources: README.md:9-11, 100-111
Filesystem Structure
The following diagram illustrates the planned data storage model on the filesystem.
This diagram shows how different collections are organized as folders, each containing JSON documents and special-purpose subdirectories.
Sources: README.md:100-111, IMPLEMENTATION_PLAN.md:46-65
Key Components of the Structure
| Component | Type | Description |
|---|---|---|
data/ | Directory | The root path for a Sentinel Store. |
users/ | Directory | Represents a "Collection" named users. |
user-123.json | File | A document with the primary key user-123. The content is JSON. |
.deleted/ | Directory | Contains documents that have been soft-deleted. |
.metadata.json | File | Planned to hold collection-specific metadata like index configurations. |
.index/ | Directory | Planned to store lazy-built indices for query acceleration. |
.wal/ | Directory | Planned to contain the Write-Ahead Log for durability. |
Sources: README.md:100-111, IMPLEMENTATION_PLAN.md:46-65
Document Structure
Each document is a JSON file. The implementation plan specifies a structure that includes both system-managed metadata fields (prefixed with an underscore) and user-defined data.
Sources: IMPLEMENTATION_PLAN.md:67-78
Example Document
{
"_id": "user-123",
"_ts": "2026-01-11T12:00:00Z",
"_v": 3,
"_hash": "sha256:abc123...",
"_sig": "rsa2048:def456...",
"name": "Alice",
"email": "alice@example.com",
"role": "admin",
"encrypted_fields": ["password_hash"]
}
Sources: IMPLEMENTATION_PLAN.md:67-78
Metadata Fields
The following metadata fields are planned for inclusion within each document to support versioning, integrity, and auditing.
| Field | Type | Description |
|---|---|---|
_id | String | The document's primary key, matching the filename. |
_ts | String | An ISO 8601 timestamp of the last update. |
_v | Integer | The version number of the document. |
_hash | String | A cryptographic hash (e.g., SHA-256) of the document's content for integrity verification. |
_sig | String | A digital signature (e.g., RSA-2048) to verify the author and prevent tampering. |
encrypted_fields | Array | A list of field names within the document that are encrypted. |
Sources: IMPLEMENTATION_PLAN.md:67-78, 110-115
Serialization and Data Flow
The process of writing a document to the store involves serializing a Rust data structure into a JSON string and writing it to a file. The serde and serde_json crates are specified for this purpose.
Sources: IMPLEMENTATION_PLAN.md:100, 421-423
Document Insertion Flow
The sequence diagram below shows the high-level steps for inserting a new document.
This flow demonstrates how a client request to insert data results in a JSON file being created on the filesystem.
Sources: README.md:80-85, IMPLEMENTATION_PLAN.md:100, 120
Supporting Libraries
The choice of JSON is supported by the planned use of high-performance Rust libraries for serialization.
| Crate | Purpose |
|---|---|
serde | A framework for serializing and deserializing Rust data structures efficiently. |
serde_json | Provides JSON serialization and deserialization for serde. |
Sources: IMPLEMENTATION_PLAN.md:100, 421-423, Cargo.toml:1
Alternative Formats and Interoperability
While JSON is the primary storage format, the documentation also mentions BSON (Binary JSON) as a possibility. The non-proprietary nature of the data storage allows for trivial exporting to other common formats.
- BSON: Mentioned as an alternative to pure JSON, likely for performance or type-fidelity reasons.
- Exportability: Data can be migrated to other databases or exported to formats like CSV and XML using standard tools, ensuring no vendor lock-in.
Sources: README.md:40-43
Summary
The use of JSON as the primary data format is a foundational architectural choice in Cyberpath Sentinel. It directly enables the project's core tenets of transparency, auditability, and operational simplicity. By representing documents as plain-text JSON files and collections as directories, Sentinel leverages the reliability and tooling of the underlying filesystem. This model, combined with a planned metadata structure for integrity and versioning, creates a database that is "compliant by design" and free from proprietary data lock-in.
Sources: README.md:9-11, 30-32, 40-43