CI Test Results Processing
Relevant source files
The following files were used as context for generating this wiki page:
CI Test Results Processing encompasses the automated systems and scripts used within the PowerShell repository to validate code changes, compare performance benchmarks, and summarize security analysis. This system ensures that every contribution adheres to quality standards by executing unit tests via Pester, checking for git merge conflicts, and performing regression analysis on performance metrics.
The processing pipeline is distributed across specialized tools that handle different types of output data, ranging from standard test results and markdown summaries to complex JSON-based performance comparisons and security snapshots.
Core Infrastructure and Validation
The infrastructure relies on a set of core utility functions designed to run within CI environments. These functions perform prerequisite tasks such as environment setup and sanity checks on the codebase before intensive testing begins.
Merge Conflict Detection
The Test-MergeConflictMarker function is a critical utility used to detect Git merge conflict markers (e.g., <<<<<<<, =======, >>>>>>>) that may have been accidentally committed. It scans provided files and generates both structured output for GitHub Actions and a Markdown summary for developer review.
The following table describes the outputs generated during the merge conflict check:
| Output Property | Description | Source |
|---|---|---|
files-checked | Total count of files scanned during the process. | test/infrastructure/ciModule.Tests.ps1:28 |
conflicts-found | Total count of files containing conflict markers. | test/infrastructure/ciModule.Tests.ps1:29 |
summary.md | A markdown report containing a table of line numbers and marker types. | test/infrastructure/ciModule.Tests.ps1:135 |
Sources: test/infrastructure/ciModule.Tests.ps1:10-40, test/infrastructure/ciModule.Tests.ps1:130-145
Performance Results Comparison
The performance testing suite utilizes a dedicated ResultsComparer tool to analyze benchmark outputs. This tool compares "Base" and "Diff" results to identify statistically significant regressions or improvements.
Statistical Analysis Logic
The system uses the MannWhitneyTest and CalculateTost helper to determine the equivalence of two sets of benchmark data. It utilizes two distinct thresholds: a Statistical Test Threshold (e.g., 5%) and a Noise Threshold (e.g., 0.3ns) to filter out insignificant fluctuations.
The comparison results are exported into multiple formats:
- CSV: For deep data analysis of original values.
- XML: Formatted for CI test reporters, marking regressions as "Fail" and improvements as "Skip".
- Markdown: Tables summarizing the ID, Median, and Modality of the benchmarks.
Sources: test/perf/dotnet-tools/ResultsComparer/Program.cs:40-75, test/perf/dotnet-tools/ResultsComparer/Program.cs:150-185
Security Analysis Summarization
The Attack Surface Analyzer (ASA) system processes security snapshots of the environment before and after PowerShell installation. The results are stored in an asa.sqlite database or a JSON file, which is then summarized for CI reports.
ASA Result Categories
The processing script Summarize-AsaResults.ps1 categorizes findings into levels and rule types, focusing specifically on file-related security issues.
| Category | Description | Source |
|---|---|---|
ERROR | Critical security issues requiring immediate attention. | tools/AttackSurfaceAnalyzer/Summarize-AsaResults.ps1:132 |
WARNING | Potential security risks or policy violations. | tools/AttackSurfaceAnalyzer/Summarize-AsaResults.ps1:133 |
INFORMATION | Informational events usually filtered out of CI summaries. | tools/AttackSurfaceAnalyzer/Summarize-AsaResults.ps1:105 |
The summarization logic includes specific detection for Binaries with expired signatures, grouping them by the issuing certificate authority and providing expiration dates.
Sources: tools/AttackSurfaceAnalyzer/Summarize-AsaResults.ps1:100-150, tools/AttackSurfaceAnalyzer/Summarize-AsaResults.ps1:340-360
Automated Test Suites
The CI pipeline executes various functional test suites, notably for the Help System and Formatting Data.
Help System Validation
Tests ensure that Get-Help returns valid HelpInfo objects and that the documentation exists in the correct culture-specific directories. The pipeline validates:
- HelpInfo Types: Ensures categories are strings.
- Contextual Help: Verifies provider-specific help for FileSystem, WSMan, and Cert providers.
- Updatable Help: Checks the ability to download and save help CAB/ZIP files from the web for various modules like
Microsoft.PowerShell.Core.
Sources: test/powershell/engine/Help/HelpSystem.Tests.ps1:40-60, test/powershell/engine/Help/UpdatableHelpSystem.Tests.ps1:150-200
Formatting Data Verification
The CI processes formatting XML files (.ps1xml) using Update-FormatData. It validates that new formatting rules are correctly prepended or appended to the session without errors. The system specifically tests for the resilience of the XML parser against invalid attributes on the Configuration node.
Sources: test/powershell/Modules/Microsoft.PowerShell.Utility/Update-FormatData.Tests.ps1:10-45
Summary
CI Test Results Processing in this repository serves as a multi-layered validation framework. It combines binary signature verification, statistical performance modeling, and standard Pester unit testing to maintain a high bar for code quality. By automating the summarization of these disparate data sources into Markdown and XML, the system provides actionable feedback to developers and maintainers directly within the pull request workflow.