Data Stores & API Communication
Relevant source files
The following files were used as context for generating this wiki page:
Data Stores and API Communication within the DXKB web application form the backbone of how biological data is retrieved, managed, and displayed. The system utilizes a combination of RESTful API calls to a central Data API and specialized client-side stores to handle complex datasets like genomic metadata, subsystems, and workspace objects.
The architecture relies heavily on asynchronous communication, primarily through Dojo's dojo/request (XHR) and specialized data stores that interface with Solr-backed services. This ensures that large-scale bioinformatics data can be fetched, filtered, and rendered efficiently in the user interface without blocking the main execution thread.
API Communication Architecture
The application communicates with several backend services, primarily the Data API for biological records and the Workspace Manager for user-generated content and job results.
Request Handling and Headers
API requests are standardized using dojo/request. Most requests toward the Data API include specific headers to facilitate proper content negotiation and security.
| Header | Purpose |
|---|---|
Accept | Typically set to application/json or application/solr+json for data retrieval. |
Content-Type | Set to application/rqlquery+x-www-form-urlencoded for complex queries or application/jsonpatch+json for metadata updates. |
Authorization | Passes the window.App.authorizationToken to authenticate the user. |
Sources: public/js/p3/widget/app/AppBase.js:14, public/js/p3/widget/MetaEditor.js:154, public/js/p3/widget/SubSystemsOverviewMemoryGrid.js:314
Data Flow Diagram
The following diagram illustrates the typical flow from a UI component request to the backend API and back through the data stores.
The diagram shows how UI widgets interact with stores or managers, which handle the underlying XHR communication with the DXKB APIs. Sources: public/js/p3/widget/WorkspaceBrowser.js:1030, public/js/p3/widget/TaxonomyOverview.js:65
Specialized Data Stores
Memory and REST Stores
The system distinguishes between data that remains on the server and data that is cached or managed locally.
- Memory Stores: Used for client-side filtering and rapid access to datasets like subsystems or library data for job submissions.
- JSON REST Stores: Interface directly with backend Solr cores via RQL (Resource Query Language) to fetch subsets of data.
Sources: public/js/p3/widget/SubSystemsOverviewMemoryGrid.js:370, public/js/p3/widget/app/AppBase.js:639
Subsystem Data Processing
For complex subsystems, the application uses a SubsystemServiceWorker to offload data parsing and facet counting to a background thread. This prevents UI stuttering when handling thousands of genomic roles.
The worker processes raw subsystems data to calculate gene counts and facet frequencies. Sources: public/worker/SubsystemServiceWorker.js:1, public/worker/SubsystemServiceWorker.js:93
Metadata Management and Updates
The application provides a MetaEditor for modifying biological metadata (e.g., genome information). This component uses JSON Patch (RFC 6902) to send incremental updates to the API.
Patch Operations
Updates are sent as a list of operations, typically using the add operator to set or replace field values.
// Example JSON Patch generated by MetaEditor
[{
"op": "add",
"path": "/comments",
"value": "Updated genomic annotation details"
}]
Sources: public/js/p3/widget/MetaEditor.js:203-228
Workspace Communication
The WorkspaceManager and WorkspaceBrowser handle interactions with the user's private and public workspace. Communication involves object retrieval, deletion, and permission management.
Permission Synchronization
Permissions for folders and genome groups are updated by sending a list of user IDs and their associated permission levels (n for none, r for read, w for write, a for admin) to the Workspace API.
Technical Summary
The DXKB data layer is characterized by its strict adherence to asynchronous patterns and the use of RQL for complex server-side querying. By offloading heavy processing to service workers and utilizing JSON Patch for updates, the system maintains high performance even when dealing with large-scale genomic data. Sources: public/js/p3/widget/SubSystemsOverviewMemoryGrid.js:314, public/js/p3/widget/MetaEditor.js:203, public/worker/SubsystemServiceWorker.js:106