File Scanning

Relevant source files

The following files were used as context for generating this wiki page:

• iac-recert-engine/internal/scan/file_scanner.go
• iac-recert-engine/internal/scan/git_history.go
• iac-recert-engine/internal/scan/recert_checker.go
• iac-recert-engine/internal/engine/engine.go
• iac-recert-engine/internal/types/types.go
• iac-recert-engine/example.config.yaml
• iac-recert-engine/internal/config/types.go
• iac-recert-engine/internal/scan/recert_checker_test.go

The File Scanning module is the foundational component of the IAC Recertification Engine (ICE). Its primary responsibility is to identify files within a target Git repository that are subject to recertification policies. The process involves three main phases: discovering files based on configured patterns, enriching them with Git history metadata, and checking them against recertification thresholds.

This module is orchestrated by the main Engine and its output serves as the input for the subsequent Grouping and Pull Request Generation stages. The entire process is driven by the patterns defined in the application's configuration file.

Sources: iac-recert-engine/internal/engine/engine.go:81-110

Architecture and Data Flow

The scanning process is executed sequentially by the Engine. It involves three distinct components from the scan package: Scanner, HistoryAnalyzer, and Checker.

The diagram below illustrates the high-level data flow through the scanning, enrichment, and checking phases.

Configuration

File scanning is entirely configured via the patterns array in the main config.yaml file. Each element in the array defines a set of files, their recertification period, and other metadata.

Sources: iac-recert-engine/example.config.yaml:26-52, iac-recert-engine/internal/config/types.go:39-48

An example pattern configuration:

patterns:
  - name: "terraform-prod"
    description: "Production Terraform configurations"
    paths:
      - "terraform/prod/**/*.tf"
    exclude:
      - "**/*.test.tf"
    recertification_days: 180
    enabled: true
    decorator: "# Last Recertification: {timestamp}\n"

Sources: iac-recert-engine/example.config.yaml:26-38

Pattern Fields

Field	Type	Description	Required
name	string	A unique identifier for the pattern.	Yes
description	string	A human-readable description of the pattern.	No
paths	[]string	An array of doublestar glob patterns to find files.	Yes
exclude	[]string	An array of doublestar glob patterns to exclude matched files.	No
recertification_days	int	The number of days after which a file needs recertification.	Yes
enabled	bool	If false, this pattern is ignored during the scan.	No (defaults to false)
decorator	string	A string to prepend to the file content when a recertification PR is created. {timestamp} is a placeholder.	No

Sources: iac-recert-engine/internal/config/types.go:39-48

Phase 1: File Discovery

The first phase is handled by scan.Scanner. It is responsible for cloning the target repository and finding all files that match the configured patterns.

Scanner.Scan Logic

1. A temporary directory is created to clone the repository.
2. The repository specified by repository.url is cloned with --depth 1.
3. The scanner iterates through each enabled config.Pattern.
4. For each pattern, it uses doublestar.FilepathGlob to find all files matching the paths.
5. Each matched file is then checked against the exclude patterns.
6. Files that are not excluded and have not been seen before are added to the results.
7. The final output is a slice of types.FileInfo structs and the path to the temporary scan directory.

Sources: iac-recert-engine/internal/scan/file_scanner.go:28-121

Phase 2: History Enrichment

After files are discovered, scan.HistoryAnalyzer enriches the FileInfo objects with data from the Git history. This step is crucial for determining the last modification date.

HistoryAnalyzer.Enrich Logic

The HistoryAnalyzer iterates through the files provided by the Scanner. For each file, it calls the GetLastModificationDate method on the configured GitProvider (e.g., GitHubProvider). This provider API call retrieves the details of the last commit that affected the file. The FileInfo struct is then updated with the commit hash, author, email, message, and timestamp.

The sequence diagram below shows the interaction between the Engine, HistoryAnalyzer, and the GitProvider.

Data Structure: FileInfo

This struct holds the metadata for each scanned file. It is initially populated by the Scanner and later updated by the HistoryAnalyzer.

Field	Type	Description	Populated By
Path	string	The absolute path to the file in the temporary clone.	Scanner
Size	int64	The size of the file in bytes.	Scanner
LastModified	time.Time	The timestamp of the last commit affecting the file.	HistoryAnalyzer
CommitHash	string	The SHA of the last commit.	HistoryAnalyzer
CommitAuthor	string	The author name from the last commit.	HistoryAnalyzer
CommitEmail	string	The author email from the last commit.	HistoryAnalyzer
CommitMsg	string	The message of the last commit.	HistoryAnalyzer

Sources: iac-recert-engine/internal/types/types.go:7-15

Phase 3: Recertification Check

The final phase is performed by scan.Checker, which determines if a file needs recertification.

Checker.Check Logic

The Checker processes the enriched FileInfo list. For each file, it performs the following steps:

1. Finds the first enabled config.Pattern that matches the file's relative path.
2. Calculates the number of days since the LastModified date.
3. Compares this duration to the recertification_days threshold from the matched pattern. A file NeedsRecert if daysSince >= threshold.
4. Calculates a Priority based on how far past the threshold the file is.
5. Constructs a types.RecertCheckResult for each file and returns the complete list.

Sources: iac-recert-engine/internal/scan/recert_checker.go:21-118

Priority Calculation

The priority is determined by the ratio of daysSince to the threshold.

Condition	Priority
ratio > 1.5	Critical
ratio >= 1.0	High
ratio >= 0.8 (and NeedsRecert is false)	Medium
ratio >= 0.8 (and NeedsRecert is true)	High or Critical
All other cases	Low

Note: The code contains a slight ambiguity. If NeedsRecert is true, the ratio is always > 1.0, so the Medium priority is only assigned to files approaching their due date. The logic for NeedsRecert: true and ratio >= 0.8 will result in High or Critical.
Sources: iac-recert-engine/internal/scan/recert_checker.go:80-97, iac-recert-engine/internal/scan/recert_checker_test.go

Data Structure: RecertCheckResult

This struct represents the final output of the scanning and checking process for a single file.

Field	Type	Description
File	FileInfo	The enriched file information.
PatternName	string	The name of the configuration pattern that matched the file.
DaysSince	int	Days since the file was last modified.
Threshold	int	The recertification threshold in days from the pattern.
Priority	string	Calculated priority: Critical, High, Medium, or Low.
NeedsRecert	bool	true if the file is due for recertification.
NextDueDate	time.Time	The calculated next recertification due date.

Sources: iac-recert-engine/internal/types/types.go:17-25

Summary

The File Scanning module is a critical pipeline that transforms a repository URL and a set of rules into a concrete list of actionable items. It systematically discovers files, enriches them with essential historical context, and applies recertification logic to determine their status. The output, a collection of RecertCheckResult objects, provides all the necessary data for the engine to group files and initiate the automated pull request workflow for recertification.