Popular Models Web Scraping

Relevant source files

The following files were used as context for generating this wiki page:

• lib/catalog.sh
• README.md
• lib/menu.sh
• lib/config.sh
• lib/auth.sh
• lib/flags.sh

Introduction

The Popular Models Web Scraping system in ollacc is a mechanism designed to dynamically retrieve and rank available AI models from the Ollama Cloud. Since no public API exists to provide the "Popular" sorting order of models, the project implements a scraping strategy that parses the HTML of the Ollama search page to provide users with a curated list of trending models.

This system ensures that the launcher's model menu tracks real-time rankings on ollama.com rather than relying on a static, hardcoded list. The scraped results are intersected with the user's authorized catalog to ensure only accessible models are presented in the interface.
Sources: README.md:143-149, lib/catalog.sh:22-24

Architecture and Logic

The scraping and resolution logic resides primarily within lib/catalog.sh. The process follows a multi-step workflow to ensure data integrity and provide fallbacks in case of network or parsing failures.

Data Flow Overview

The logic executes in five distinct stages:

1. Authenticated Catalog Fetch: Retrieves the full list of models the user is authorized to access via the Ollama API.
2. Popularity Scraping: Downloads the HTML content from the Ollama cloud search page.
3. Intersection and Extraction: Parses the HTML for model slugs and verifies their existence in the authenticated catalog.
4. First Fallback: If scraping fails, the system uses the raw order from the authenticated API.
5. Second Fallback: If all network requests fail, a hardcoded "last-resort" list is used.

The following diagram illustrates the sequence of operations within the catalog_resolve function:

Implementation Details

Model Identification and Parsing

The system uses curl to fetch data and standard Unix utilities like grep and Bash regex for parsing. Because model names may contain special characters, the system utilizes a bash_escape helper to ensure literal substring matches during the intersection phase.

Component	Target URL / Source	Method	Purpose
Auth API	https://ollama.com/api/tags	JSON / API	Verifies user access to models.
Scraper	https://ollama.com/search?c=cloud	HTML / Scrape	Determines current popularity ranking.
Parser	href=\"/library/([^\"/]+)\"	Regex	Extracts model slugs from HTML anchors.

Sources: lib/catalog.sh:37-64, lib/config.sh:16-32

Configuration Parameters

The scraping behavior is influenced by timeouts and environment variables managed by the configuration system.

Variable	Default Value	Description
ANTHROPIC_AUTH_TOKEN	(User Provided)	Used as the Bearer token for the API fetch.
--max-time	10	Maximum time in seconds for the curl operation.
--connect-timeout	5	Maximum time to wait for a connection.
MAPFILE	(Empty Array)	Global array storing the final 10 model slugs.

Sources: lib/catalog.sh:15-18, lib/catalog.sh:37-39, lib/auth.sh:10-15

Hardcoded Fallback List

If both the API and the web scraper are unreachable, the system populates the MAPFILE with a predefined list of models to maintain functionality:

• kimi-k2.7-code
• minimax-m3
• qwen3.5
• glm-5.1
• minimax-m2.7

Sources: lib/catalog.sh:86-89

Integration with User Interface

Once the scraping and resolution are complete, the MAPFILE is passed to lib/menu.sh for rendering. The menu provides a numbered list based on the scraped order, followed by a "[Custom]" option for manual entry.

# Example extraction logic from lib/catalog.sh
while [[ "$remaining_scrape" =~ href=\"/library/([^\"/]+)\" ]]; do
    local candidate="${BASH_REMATCH[1]}"
    if [[ ! " ${MAPFILE[*]} " =~ " ${candidate} " ]] \
       && printf '%s' "$authd_json" | grep -qF "\"name\":\"$candidate\""; then
        MAPFILE+=("$candidate")
    fi
    remaining_scrape="${remaining_scrape#*href=\"/library/}"
    [ ${#MAPFILE[@]} -eq 10 ] && break
done

Sources: lib/catalog.sh:58-67, lib/menu.sh:12-19

Conclusion

The Popular Models Web Scraping module provides a robust, fail-safe method for synchronizing the ollacc launcher with the latest offerings on Ollama Cloud. By combining live HTML scraping with API-based authorization checks, it ensures that users are always presented with a relevant and functional selection of models while maintaining a lightweight footprint.