CLI: Secure Data Export
Relevant source files
The following files were used as context for generating this wiki page:The Secure Data Export system is a command-line interface (CLI) tool designed for managing and exporting subscriber and analytics data from the newsletter program. It provides functionality for creating comprehensive data backups, generating GDPR-compliant reports for individual subscribers, and producing anonymized datasets for research purposes. The system emphasizes security and data privacy through features like data anonymization and management of export files.
This tool is an essential part of the newsletter's data management and compliance strategy, offering controlled access to sensitive user information. It is invoked via the npm run export script, which executes the newsletter-export binary defined in the project's package.json.
Sources: newsletter-program/README-SUBSCRIBER-MANAGEMENT.md, newsletter-program/src/cli/export-cli.js, newsletter-program/package.json:13, 19
Architecture and Components
The export feature is architecturally split into a CLI layer and a core logic layer. The export-cli.js file handles command parsing and user interaction, while the SecureExportManager class in secure-export.js contains the business logic for data aggregation, anonymization, and file generation.
The SecureExportManager collaborates with SubscriberFileManager and AnalyticsManager to fetch the necessary data for the exports.
Sources: newsletter-program/src/cli/export-cli.js:5-8, newsletter-program/src/core/secure-export.js:7-12
The following diagram illustrates the relationship between the main components of the export system.
CLI Commands
The export command, aliased from node src/cli/export-cli.js, provides several sub-commands for managing data exports.
Sources: newsletter-program/package.json:19, newsletter-program/src/cli/export-cli.js
| Command | Description | Options |
|---|---|---|
create | Create a secure export of all data. | --no-subscribers, --no-analytics, --anonymize, -o, --output <dir> |
gdpr | Create a GDPR-compliant data export for a specific subscriber. | -e, --email <email>, -o, --output <dir> |
anonymize | Create an anonymized dataset for research or analysis. | -o, --output <dir> |
list | List all available exports. | --format <format> (table/json) |
clean | Clean up old export files. | -d, --days <days>, --confirm |
verify | Verify the integrity of an export file. | -f, --file <file> |
Sources: newsletter-program/src/cli/export-cli.js:15-268, newsletter-program/README-SUBSCRIBER-MANAGEMENT.md:129-155
Export Types and Data Structures
The system can generate three distinct types of data exports, each with a specific purpose and data structure. All exports are generated as JSON files in the newsletter-program/src/exports directory by default.
Sources: newsletter-program/src/core/secure-export.js:10
1. Full Secure Export (create)
This command generates a comprehensive backup of subscriber and/or analytics data. It can be created in a standard or anonymized format.
- Standard Export: Includes all raw subscriber and analytics data.
- Anonymized Export: When the
--anonymizeflag is used, sensitive data is hashed or removed. Subscriber emails and analytics event emails are hashed using SHA256.
Sources: newsletter-program/src/core/secure-export.js:17-74, newsletter-program/src/cli/export-cli.js:15-46
The data flow for creating a secure export is as follows:
2. GDPR Export (gdpr)
This command generates a GDPR-compliant data portability report for a single subscriber, identified by their email address. The export contains all data related to that individual.
The process involves finding the subscriber, filtering all associated analytics events, and packaging the data with legal and contextual information.
Sources: newsletter-program/src/core/secure-export.js:79-136, newsletter-program/src/cli/export-cli.js:49-86
The sequence diagram below shows the process for generating a GDPR export.
The resulting JSON file includes the following key sections:
| Key | Description |
|---|---|
exportType | Always GDPR_DATA_EXPORT. |
createdAt | ISO timestamp of the export creation. |
subscriber | An object containing the subscriber's full profile. |
events | An array of all analytics events associated with the subscriber. |
dataProcessingPurposes | A list of reasons why the data is collected. |
legalBasis | The legal basis for data processing (e.g., Consent). |
retentionPeriod | The data retention policy. |
dataControllers | Contact information for the data controller. |
Sources: newsletter-program/src/core/secure-export.js:96-123
3. Anonymized Dataset (anonymize)
This command creates a dataset suitable for research and analysis by removing or hashing all personally identifiable information (PII).
Anonymization methods include:
- Hashing subscriber and event IDs with a truncated SHA256 hash.
- Removing email addresses entirely.
- Stripping personal metadata and preserving only aggregate patterns.
Sources: newsletter-program/src/core/secure-export.js:141-201, newsletter-program/src/cli/export-cli.js:89-120
The structure of the anonymized dataset is as follows:
| Key | Description |
|---|---|
datasetType | Always ANONYMIZED_RESEARCH_DATASET. |
createdAt | ISO timestamp of the dataset creation. |
anonymizationMethod | Description of the hashing method used (SHA256 hashing with truncation). |
totalSubscribers | The total number of records in the dataset. |
subscribers | An array of anonymized subscriber objects. |
analytics | An object containing anonymized newsletter and event data. |
Sources: newsletter-program/src/core/secure-export.js:174-190
Management and Verification
The CLI provides commands to manage the lifecycle of export files and ensure their integrity.
Listing and Cleaning Exports
export list: Scans the export directory, parses each JSON file, and displays a summary including file type, creation date, size, and anonymization status. The output can be a table or JSON.export clean: Deletes export files older than a specified number of days (default is 30). Requires a--confirmflag to prevent accidental deletion.
Sources: newsletter-program/src/cli/export-cli.js:123-202, newsletter-program/src/core/secure-export.js:206-252
Verifying Export Integrity
export verify: Performs a series of checks on a given export file to validate its structure and content. This is useful for confirming that an export file is not corrupted.
The verification process checks for:
- Valid Structure: Presence of
createdAtandexportType/datasetTypekeys. - Valid Timestamp: The
createdAtvalue is a valid date. - Data Presence: At least one data-related key (
data,subscribers,analytics) exists. - Anonymization Status: Correctly identifies if the file is an anonymized export.