actions/_tests/README.md

GitHub Actions Testing Framework

A comprehensive testing framework for validating GitHub Actions in this monorepo. This guide covers everything from basic usage to advanced testing patterns.

🚀 Quick Start

# Run all tests
make test

# Run only unit tests
make test-unit

# Run tests for specific action
make test-action ACTION=node-setup

# Run with coverage reporting
make test-coverage

Prerequisites

# Install ShellSpec (testing framework)
curl -fsSL https://github.com/shellspec/shellspec/releases/latest/download/shellspec-dist.tar.gz | tar -xz
sudo make -C shellspec-* install

# Install nektos/act (optional, for integration tests)
brew install act  # macOS
# or: curl https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash

📁 Framework Overview

Architecture

The testing framework uses a multi-level testing strategy:

1. Unit Tests - Fast validation of action logic, inputs, and outputs
2. Integration Tests - Test actions in realistic workflow environments
3. External Usage Tests - Validate actions work as ivuorinen/actions/action-name@main

Technology Stack

• Primary Framework: ShellSpec - BDD testing for shell scripts
• Local Execution: nektos/act - Run GitHub Actions locally
• Coverage: kcov integration for shell script coverage
• Mocking: Custom GitHub API and service mocks
• CI Integration: GitHub Actions workflows

Directory Structure

_tests/
├── README.md                    # This documentation
├── run-tests.sh                 # Main test runner script
├── framework/                   # Core testing utilities
│   ├── setup.sh                # Test environment setup
│   ├── utils.sh                # Common testing functions
│   ├── validation_helpers.sh   # Validation helper functions
│   ├── validation.py           # Python validation utilities
│   └── mocks/                  # Mock services (GitHub API, etc.)
├── unit/                       # Unit tests by action
│   ├── version-file-parser/    # Example unit tests
│   ├── node-setup/            # Example unit tests
│   └── ...                    # One directory per action
├── integration/               # Integration tests
│   ├── workflows/             # Test workflows for nektos/act
│   └── external-usage/        # External reference tests
├── coverage/                  # Coverage reports
└── reports/                   # Test execution reports

✍️ Writing Tests

Basic Unit Test Structure

#!/usr/bin/env shellspec
# _tests/unit/my-action/validation.spec.sh

Include _tests/framework/utils.sh

Describe "my-action validation"
  ACTION_DIR="my-action"
  ACTION_FILE="$ACTION_DIR/action.yml"

  BeforeAll "init_testing_framework"

  Context "input validation"
    It "validates all inputs comprehensively"
      # Use validation helpers for comprehensive testing
      test_boolean_input "verbose"
      test_boolean_input "dry-run"

      # Numeric range validations (use test_input_validation helper)
      test_input_validation "$ACTION_DIR" "max-retries" "1" "success"
      test_input_validation "$ACTION_DIR" "max-retries" "10" "success"
      test_input_validation "$ACTION_DIR" "timeout" "3600" "success"

      # Enum validations (use test_input_validation helper)
      test_input_validation "$ACTION_DIR" "strategy" "fast" "success"
      test_input_validation "$ACTION_DIR" "format" "json" "success"

      # Version validations (use test_input_validation helper)
      test_input_validation "$ACTION_DIR" "tool-version" "1.0.0" "success"

      # Security and path validations (use test_input_validation helper)
      test_input_validation "$ACTION_DIR" "command" "echo test" "success"
      test_input_validation "$ACTION_DIR" "working-directory" "." "success"
    End
  End

  Context "action structure"
    It "has valid structure and metadata"
      test_standard_action_structure "$ACTION_FILE" "Expected Action Name"
    End
  End
End

Integration Test Example

# _tests/integration/workflows/my-action-test.yml
name: Test my-action Integration
on: workflow_dispatch

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Test action locally
        id: test-local
        uses: ./my-action
        with:
          required-input: 'test-value'

      - name: Validate outputs
        run: |
          echo "Output: ${{ steps.test-local.outputs.result }}"
          [[ -n "${{ steps.test-local.outputs.result }}" ]] || exit 1

      - name: Test external reference
        uses: ivuorinen/actions/my-action@main
        with:
          required-input: 'test-value'

🛠️ Testing Helpers

Available Validation Helpers

The framework provides comprehensive validation helpers that handle common testing patterns:

Boolean Input Testing

test_boolean_input "verbose"          # Tests: true, false, rejects invalid
test_boolean_input "enable-cache"
test_boolean_input "dry-run"

Numeric Range Testing

# Note: test_numeric_range_input helper is not yet implemented.
# Use test_input_validation with appropriate test values instead:
test_input_validation "$ACTION_DIR" "max-retries" "1" "success"    # min value
test_input_validation "$ACTION_DIR" "max-retries" "10" "success"   # max value
test_input_validation "$ACTION_DIR" "max-retries" "0" "failure"    # below min
test_input_validation "$ACTION_DIR" "timeout" "3600" "success"
test_input_validation "$ACTION_DIR" "parallel-jobs" "8" "success"

Version Testing

# Note: test_version_input helper is not yet implemented.
# Use test_input_validation with appropriate test values instead:
test_input_validation "$ACTION_DIR" "version" "1.0.0" "success"           # semver
test_input_validation "$ACTION_DIR" "version" "v1.0.0" "success"          # v-prefix
test_input_validation "$ACTION_DIR" "version" "1.0.0-rc.1" "success"      # pre-release
test_input_validation "$ACTION_DIR" "tool-version" "2.3.4" "success"

Enum Testing

# Note: test_enum_input helper is not yet implemented.
# Use test_input_validation with appropriate test values instead:
test_input_validation "$ACTION_DIR" "strategy" "linear" "success"
test_input_validation "$ACTION_DIR" "strategy" "exponential" "success"
test_input_validation "$ACTION_DIR" "strategy" "invalid" "failure"
test_input_validation "$ACTION_DIR" "format" "json" "success"
test_input_validation "$ACTION_DIR" "format" "yaml" "success"

Docker-Specific Testing

# Available framework helpers:
test_input_validation "$action_dir" "$input_name" "$test_value" "$expected_result"
test_action_outputs "$action_dir"
test_external_usage "$action_dir"

# Note: Docker-specific helpers (test_docker_image_input, test_docker_tag_input,
# test_docker_platforms_input) are referenced in examples but not yet implemented.
# Use test_input_validation with appropriate test values instead.

Complete Action Validation Example

Describe "comprehensive-action validation"
  ACTION_DIR="comprehensive-action"
  ACTION_FILE="$ACTION_DIR/action.yml"

  Context "complete input validation"
    It "validates all input types systematically"
      # Boolean inputs
      test_boolean_input "verbose"
      test_boolean_input "enable-cache"
      test_boolean_input "dry-run"

      # Numeric ranges (use test_input_validation helper)
      test_input_validation "$ACTION_DIR" "max-retries" "1" "success"
      test_input_validation "$ACTION_DIR" "max-retries" "10" "success"
      test_input_validation "$ACTION_DIR" "timeout" "3600" "success"
      test_input_validation "$ACTION_DIR" "parallel-jobs" "8" "success"

      # Enums (use test_input_validation helper)
      test_input_validation "$ACTION_DIR" "strategy" "fast" "success"
      test_input_validation "$ACTION_DIR" "format" "json" "success"

      # Docker-specific (use test_input_validation helper)
      test_input_validation "$ACTION_DIR" "image-name" "myapp:latest" "success"
      test_input_validation "$ACTION_DIR" "tag" "1.0.0" "success"
      test_input_validation "$ACTION_DIR" "platforms" "linux/amd64,linux/arm64" "success"

      # Security validation (use test_input_validation helper)
      test_input_validation "$ACTION_DIR" "command" "echo test" "success"
      test_input_validation "$ACTION_DIR" "build-args" "ARG1=value" "success"

      # Paths (use test_input_validation helper)
      test_input_validation "$ACTION_DIR" "working-directory" "." "success"
      test_input_validation "$ACTION_DIR" "output-directory" "./output" "success"

      # Versions (use test_input_validation helper)
      test_input_validation "$ACTION_DIR" "tool-version" "1.0.0" "success"

      # Action structure
      test_standard_action_structure "$ACTION_FILE" "Comprehensive Action"
    End
  End
End

🎯 Testing Patterns by Action Type

Setup Actions (node-setup, php-version-detect, etc.)

Focus on version detection and environment setup:

Context "version detection"
  It "detects version from config files"
    create_mock_node_repo  # or appropriate repo type

    # Test version detection logic
    export INPUT_LANGUAGE="node"
    echo "detected-version=18.0.0" >> "$GITHUB_OUTPUT"

    When call validate_action_output "detected-version" "18.0.0"
    The status should be success
  End

  It "falls back to default when no version found"
    # Use test_input_validation helper for version validation
    test_input_validation "$ACTION_DIR" "default-version" "1.0.0" "success"
  End
End

Linting Actions (eslint-fix, prettier-fix, etc.)

Focus on file processing and fix capabilities:

Context "file processing"
  BeforeEach "setup_test_env 'lint-test'"
  AfterEach "cleanup_test_env 'lint-test'"

  It "validates inputs and processes files"
    test_boolean_input "fix-only"
    # Use test_input_validation helper for path and security validations
    test_input_validation "$ACTION_DIR" "working-directory" "." "success"
    test_input_validation "$ACTION_DIR" "custom-command" "echo test" "success"

    # Mock file processing
    echo "files_changed=3" >> "$GITHUB_OUTPUT"
    echo "status=changes_made" >> "$GITHUB_OUTPUT"

    When call validate_action_output "status" "changes_made"
    The status should be success
  End
End

Build Actions (docker-build, go-build, etc.)

Focus on build processes and artifact generation:

Context "build process"
  BeforeEach "setup_test_env 'build-test'"
  AfterEach "cleanup_test_env 'build-test'"

  It "validates build inputs"
    # Use test_input_validation helper for Docker inputs
    test_input_validation "$ACTION_DIR" "image-name" "myapp:latest" "success"
    test_input_validation "$ACTION_DIR" "tag" "1.0.0" "success"
    test_input_validation "$ACTION_DIR" "platforms" "linux/amd64,linux/arm64" "success"
    test_input_validation "$ACTION_DIR" "parallel-builds" "8" "success"

    # Mock successful build
    echo "build-status=success" >> "$GITHUB_OUTPUT"
    echo "build-time=45" >> "$GITHUB_OUTPUT"

    When call validate_action_output "build-status" "success"
    The status should be success
  End
End

Publishing Actions (npm-publish, docker-publish, etc.)

Focus on registry interactions using mocks:

Context "publishing"
  BeforeEach "setup_mock_environment"
  AfterEach "cleanup_mock_environment"

  It "validates publishing inputs"
    # Use test_input_validation helper for version, security, and enum validations
    test_input_validation "$ACTION_DIR" "package-version" "1.0.0" "success"
    test_input_validation "$ACTION_DIR" "registry-token" "ghp_test123" "success"
    test_input_validation "$ACTION_DIR" "registry" "npm" "success"
    test_input_validation "$ACTION_DIR" "registry" "github" "success"

    # Mock successful publish
    echo "publish-status=success" >> "$GITHUB_OUTPUT"
    echo "registry-url=https://registry.npmjs.org/" >> "$GITHUB_OUTPUT"

    When call validate_action_output "publish-status" "success"
    The status should be success
  End
End

🔧 Running Tests

Command Line Interface

# Basic usage
./_tests/run-tests.sh [OPTIONS] [ACTION_NAME...]

# Examples
./_tests/run-tests.sh                           # All tests, all actions
./_tests/run-tests.sh -t unit                   # Unit tests only
./_tests/run-tests.sh -a node-setup             # Specific action
./_tests/run-tests.sh -t integration docker-build  # Integration tests for docker-build
./_tests/run-tests.sh --format json --coverage  # JSON output with coverage

Options

Option	Description
-t, --type TYPE	Test type: unit, integration, e2e, all
-a, --action ACTION	Filter by action name pattern
-j, --jobs JOBS	Number of parallel jobs (default: 4)
-c, --coverage	Enable coverage reporting
-f, --format FORMAT	Output format: console, json, junit
-v, --verbose	Enable verbose output
-h, --help	Show help message

Make Targets

make test                    # Run all tests
make test-unit              # Unit tests only
make test-integration       # Integration tests only
make test-coverage          # Tests with coverage
make test-action ACTION=name # Test specific action

🤝 Contributing Tests

Adding Tests for New Actions

1. Create Unit Test Directory

   mkdir -p _tests/unit/new-action
   

2. Write Comprehensive Unit Tests

   # Copy template and customize
   cp _tests/unit/version-file-parser/validation.spec.sh \
      _tests/unit/new-action/validation.spec.sh
   

3. Use Validation Helpers

   # Focus on using helpers for comprehensive coverage
   test_boolean_input "verbose"
   # Use test_input_validation helper for numeric, security, and other validations
   test_input_validation "$ACTION_DIR" "timeout" "3600" "success"
   test_input_validation "$ACTION_DIR" "command" "echo test" "success"
   test_standard_action_structure "$ACTION_FILE" "New Action"
   

4. Create Integration Test

   cp _tests/integration/workflows/version-file-parser-test.yml \
      _tests/integration/workflows/new-action-test.yml
   

5. Test Your Tests

   make test-action ACTION=new-action
   

Pull Request Checklist

• Tests use validation helpers for common patterns
• All test types pass locally (make test)
• Integration test workflow created
• Security testing included for user inputs
• Tests are independent and isolated
• Proper cleanup in test teardown
• Documentation updated if needed

💡 Best Practices

1. Use Validation Helpers

✅ Good:

test_boolean_input "verbose"
# Use test_input_validation helper for other validations
test_input_validation "$ACTION_DIR" "timeout" "3600" "success"
test_input_validation "$ACTION_DIR" "format" "json" "success"

❌ Avoid:

# Don't write manual tests for boolean inputs when test_boolean_input exists
When call test_input_validation "$ACTION_DIR" "verbose" "true" "success"
When call test_input_validation "$ACTION_DIR" "verbose" "false" "success"
# Use test_boolean_input "verbose" instead

2. Group Related Validations

✅ Good:

Context "complete input validation"
  It "validates all input types"
    test_boolean_input "verbose"
    # Use test_input_validation helper for other validations
    test_input_validation "$ACTION_DIR" "timeout" "3600" "success"
    test_input_validation "$ACTION_DIR" "format" "json" "success"
    test_input_validation "$ACTION_DIR" "command" "echo test" "success"
  End
End

3. Include Security Testing

✅ Always include:

# Use test_input_validation helper for security and path validations
test_input_validation "$ACTION_DIR" "command" "echo test" "success"
test_input_validation "$ACTION_DIR" "user-script" "#!/bin/bash" "success"
test_input_validation "$ACTION_DIR" "working-directory" "." "success"

4. Write Descriptive Test Names

✅ Good:

It "accepts valid semantic version format"
It "rejects version with invalid characters"
It "falls back to default when no version file exists"

❌ Avoid:

It "validates input"
It "works correctly"

5. Keep Tests Independent

• Each test should work in isolation
• Don't rely on test execution order
• Clean up after each test
• Use proper setup/teardown

🔍 Framework Features

Test Environment Setup

# Setup test environment
setup_test_env "test-name"

# Create mock repositories
create_mock_repo "node"     # Node.js project
create_mock_repo "php"      # PHP project
create_mock_repo "python"   # Python project
create_mock_repo "go"       # Go project
create_mock_repo "dotnet"   # .NET project

# Cleanup
cleanup_test_env "test-name"

Mock Services

Built-in mocks for external services:

• GitHub API - Repository, releases, packages, workflows
• NPM Registry - Package publishing and retrieval
• Docker Registry - Image push/pull operations
• Container Registries - GitHub Container Registry, Docker Hub

Available Environment Variables

# Test environment paths
$TEST_WORKSPACE       # Current test workspace
$GITHUB_OUTPUT         # Mock GitHub outputs file
$GITHUB_ENV           # Mock GitHub environment file
$GITHUB_STEP_SUMMARY  # Mock step summary file

# Test framework paths
$TEST_ROOT            # _tests/ directory
$FRAMEWORK_DIR        # _tests/framework/ directory
$FIXTURES_DIR         # _tests/framework/fixtures/
$MOCKS_DIR           # _tests/framework/mocks/

🚨 Troubleshooting

Common Issues

"ShellSpec command not found"

# Install ShellSpec globally
curl -fsSL https://github.com/shellspec/shellspec/releases/latest/download/shellspec-dist.tar.gz | tar -xz
sudo make -C shellspec-* install

"act command not found"

# Install nektos/act (macOS)
brew install act

# Install nektos/act (Linux)
curl https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash

Tests timeout

# Increase timeout for slow operations
export TEST_TIMEOUT=300

Permission denied on test scripts

# Make test scripts executable
find _tests/ -name "*.sh" -exec chmod +x {} \;

Debugging Tests

1. Enable Verbose Mode

   ./_tests/run-tests.sh -v
   

2. Run Single Test

   shellspec _tests/unit/my-action/validation.spec.sh
   

3. Check Test Output

   # Test results stored in _tests/reports/
   cat _tests/reports/unit/my-action.txt
   

4. Debug Mock Environment

   # Enable mock debugging
   export MOCK_DEBUG=true
   

📚 Resources

• ShellSpec Documentation
• nektos/act Documentation
• GitHub Actions Documentation
• Testing GitHub Actions Best Practices

---

Framework Development

Adding New Framework Features

1. New Test Utilities

   # Add to _tests/framework/utils.sh
   your_new_function() {
     local param="$1"
     # Implementation
   }
   
   # Export for availability
   export -f your_new_function
   

2. New Mock Services

   # Create _tests/framework/mocks/new-service.sh
   # Follow existing patterns in github-api.sh
   

3. New Validation Helpers

   # Add to _tests/framework/validation_helpers.sh
   # Update this documentation
   

Last Updated: August 17, 2025