Extracting Data

Relevant source files

The following files were used as context for generating this wiki page:

• src/api/extract.ts
• src/api/extract.spec.ts
• src/static.ts
• website/src/content/blog/2024-08-07-version-1.md
• src/cheerio.spec.ts

Extracting data in Cheerio allows developers to transform structured HTML or XML documents into plain JavaScript objects using a declarative syntax. This feature, primarily exposed through the extract method, simplifies the process of web scraping and data mining by mapping CSS selectors to specific object keys and values.

The extraction system supports simple string selectors, descriptor objects for fine-grained control, and nested extraction maps for complex document structures. It is available both as an instance method on Cheerio objects and as a static method on the Cheerio API.

Sources: src/api/extract.ts, website/src/content/blog/2024-08-07-version-1.md:57-89

Core Extraction Mechanism

The extraction process revolves around the ExtractMap, which defines the structure of the desired output. For each key in the map, Cheerio performs a selection and retrieves the data according to the specified descriptor.

Extraction Logic Flow

The following diagram illustrates how the extract method processes a map to produce an output object.

The system differentiates between single-element extraction and multiple-element extraction (arrays). When an array is provided in the map, Cheerio returns an array of values for all matching elements.

Sources: src/api/extract.ts:108-144, src/api/extract.spec.ts:58-76

Data Descriptors

Cheerio provides several ways to define what data should be extracted from a selected element.

Descriptor Type	Description	Example
String Selector	Short-hand for extracting the textContent of the first match.	title: 'h1'
Descriptor Object	Object with selector and value properties for detailed extraction.	{ selector: 'a', value: 'href' }
Nested Map	An object containing another ExtractMap for recursive extraction.	{ selector: 'div', value: { sub: 'p' } }
Custom Function	A function (el, key) => any for bespoke extraction logic.	(el) => $(el).attr('id')

Sources: src/api/extract.ts:16-55, src/api/extract.spec.ts:80-128

The value Property Options

Within a descriptor object, the value property determines the extraction target. If omitted, it defaults to the text content of the element.

• String Constants: innerHTML, outerHTML, textContent, innerText.
• Attribute Names: Any string that matches an attribute (e.g., href, src, data-id).
• Functions: Receives the element and the key name to return any value.
• Nested Maps: Allows for deep extraction within the context of the selected element.

Sources: src/api/extract.ts:24-43, src/api/extract.spec.ts:123-146

Implementation Details

API Signature

The extract method is available on the Cheerio prototype and as a static method.

// Prototype Method
extract<M extends ExtractMap>(map: M): ExtractedMap<M>;

// Static Method (src/static.ts)
export function extract<M extends ExtractMap>(
  this: CheerioAPI,
  map: M,
): ExtractedMap<M> {
  return this.root().extract(map);
}

Sources: src/api/extract.ts:108, src/static.ts:241-248

Contextual Selection

When performing nested extractions, the selectors within the nested map are executed within the context of the element matched by the parent selector. This enables the extraction of grouped data, such as rows in a table or items in a list.

Technical Constraints and Behavior

• Empty Selectors: If no selectors are provided in the map, the method returns an empty object.
• Non-Matches: If a selector does not match any element, the resulting value for that key is undefined (or an empty array for array descriptors).
• Property Access: When a string like href is used as a value, Cheerio first attempts to retrieve it via .prop(value). If that fails, it falls back to .attr(value).
• Type Safety: The extraction system is strongly typed, ensuring the returned object matches the structure defined in the ExtractMap.

Sources: src/api/extract.ts:122-140, src/api/extract.spec.ts:17-40, src/api/extract.spec.ts:223-228

Conclusion

Data extraction in Cheerio provides a powerful, declarative alternative to manual DOM traversal. By combining CSS selectors with structured descriptors and nested maps, developers can efficiently transform complex HTML documents into structured data formats suitable for application logic.