External Data Fetchers (Task Helpers)

Relevant source files

The following files were used as context for generating this wiki page:

• lib/task_helpers/national_shapefile_request.rb
• lib/task_helpers/state_shapefile_request.rb
• lib/task_helpers/topojson_task_helper.rb
• lib/task_helpers/task_helper.rb
• lib/task_helpers/state_fips_task_helper.rb

Introduction

The External Data Fetchers, organized within the lib/task_helpers module, are a suite of utility classes designed to retrieve, process, and manage geographic and administrative data from external sources, primarily the U.S. Census Bureau. These helpers facilitate the acquisition of Shapefiles and Federal Information Processing Series (FIPS) data, which are essential for rendering maps and identifying political boundaries within the Actionmap application.

The system automates the workflow of downloading compressed geographic data, unzipping contents, and converting raw Shapefiles into optimized formats like GeoJSON and TopoJSON. This ensures that the application has access to lightweight, web-ready geographic data for both national and state-level visualizations.

Architecture and Data Flow

Geographic Data Processing Pipeline

The core of the data fetching logic resides in the interaction between request-specific classes (NationalShapefileRequest, StateShapefileRequest) and the TopojsonTaskHelper. The process utilizes external tools like mapshaper to simplify and convert geometry.

Key Components

Component	Responsibility	Source File
TaskHelper	Low-level filesystem operations (reinit directories, extract ZIPs, write JSON).	lib/task_helpers/task_helper.rb
TopojsonTaskHelper	High-level orchestration of fetching, unzipping, and converting shapefiles.	lib/task_helpers/topojson_task_helper.rb
NationalShapefileRequest	Configuration for fetching U.S. national state boundaries.	lib/task_helpers/national_shapefile_request.rb
StateShapefileRequest	Configuration for fetching specific state county/subdivision boundaries.	lib/task_helpers/state_shapefile_request.rb
StateFipsTaskHelper	Fetching and parsing FIPS code data for counties from census.gov.	lib/task_helpers/state_fips_task_helper.rb

Geographic Request Implementation

National and State Requests

Requests are encapsulated in classes that define the fetch_url and local file paths. Both NationalShapefileRequest and StateShapefileRequest implement a cmd method that generates the shell command for mapshaper.

• National Request: Targeted at cb_2018_us_state_500k.zip to provide a high-level view of all states.
• State Request: Targeted at cousub (county subdivision) files, which are then processed with a -dissolve COUNTYFP operation to merge subdivisions into recognizable counties.

Sources: lib/task_helpers/national_shapefile_request.rb:19-25, lib/task_helpers/state_shapefile_request.rb:23-31

Mapshaper Integration

The TopojsonTaskHelper relies on mapshaper (located at node_modules/.bin/mapshaper) for geometric processing.

• Simplification: Uses -simplify 10% or similar values to reduce file size while keeping shapes intact.
• Dissolving: Uses -dissolve COUNTYFP to aggregate smaller geometric units into single county entities.

Sources: lib/task_helpers/topojson_task_helper.rb:7, lib/task_helpers/state_shapefile_request.rb:38

Administrative Data (FIPS) Fetching

The StateFipsTaskHelper manages the retrieval of textual FIPS data. This data maps county names to their numerical codes, which is critical for linking geographic shapes to database records.

FIPS Processing Logic

The helper fetches .txt files from the Census Bureau and parses them as CSV. It includes specific overrides for historical or administrative changes in certain states:

• Alaska: Updates "Wade Hampton Census Area" to "Kusilvak Census Area".
• South Dakota: Updates "Shannon County" to "Oglala Lakota County".
• Virginia: Handles the removal of "Bedford City" as an independent entity.

Sources: lib/task_helpers/state_fips_task_helper.rb:39-86

Filesystem Operations

The TaskHelper module provides standardized methods for handling local assets:

• reinit_dir(dir_name): Ensures a clean workspace by deleting and recreating directories.
• extract_zip(zip_file, dest_folder): Uses the Zip library to unpack downloaded shapefiles.
• write_json(filename, content): Exports processed data to the filesystem with UTF-8 encoding and pretty formatting.

Sources: lib/task_helpers/task_helper.rb:7-35

Summary

The Task Helpers provide a robust pipeline for synchronizing the Actionmap application with official U.S. Census data. By combining geometric processing via mapshaper with administrative data parsing in StateFipsTaskHelper, the system ensures that the mapping interface is both geographically accurate and administratively current.