chore: modernize workflows, security scanning, and linting configuration (#50)

* build: update Go 1.25, CI workflows, and build tooling

- Upgrade to Go 1.25
- Add benchmark targets to Makefile
- Implement parallel gosec execution
- Lock tool versions for reproducibility
- Add shellcheck directives to scripts
- Update CI workflows with improved caching

* refactor: migrate from golangci-lint to revive

- Replace golangci-lint with revive for linting
- Configure comprehensive revive rules
- Fix all EditorConfig violations
- Add yamllint and yamlfmt support
- Remove deprecated .golangci.yml

* refactor: rename utils to shared and deduplicate code

- Rename utils package to shared
- Add shared constants package
- Deduplicate constants across packages
- Address CodeRabbit review feedback

* fix: resolve SonarQube issues and add safety guards

- Fix all 73 SonarQube OPEN issues
- Add nil guards for resourceMonitor, backpressure, metricsCollector
- Implement io.Closer for headerFileReader
- Propagate errors from processing helpers
- Add metrics and templates packages
- Improve error handling across codebase

* test: improve test infrastructure and coverage

- Add benchmarks for cli, fileproc, metrics
- Improve test coverage for cli, fileproc, config
- Refactor tests with helper functions
- Add shared test constants
- Fix test function naming conventions
- Reduce cognitive complexity in benchmark tests

* docs: update documentation and configuration examples

- Update CLAUDE.md with current project state
- Refresh README with new features
- Add usage and configuration examples
- Add SonarQube project configuration
- Consolidate config.example.yaml

* fix: resolve shellcheck warnings in scripts

- Use ./*.go instead of *.go to prevent dash-prefixed filenames
  from being interpreted as options (SC2035)
- Remove unreachable return statement after exit (SC2317)
- Remove obsolete gibidiutils/ directory reference

* chore(deps): upgrade go dependencies

* chore(lint): megalinter fixes

* fix: improve test coverage and fix file descriptor leaks

- Add defer r.Close() to fix pipe file descriptor leaks in benchmark tests
- Refactor TestProcessorConfigureFileTypes with helper functions and assertions
- Refactor TestProcessorLogFinalStats with output capture and keyword verification
- Use shared constants instead of literal strings (TestFilePNG, FormatMarkdown, etc.)
- Reduce cognitive complexity by extracting helper functions

* fix: align test comments with function names

Remove underscores from test comments to match actual function names:
- benchmark/benchmark_test.go (2 fixes)
- fileproc/filetypes_config_test.go (4 fixes)
- fileproc/filetypes_registry_test.go (6 fixes)
- fileproc/processor_test.go (6 fixes)
- fileproc/resource_monitor_types_test.go (4 fixes)
- fileproc/writer_test.go (3 fixes)

* fix: various test improvements and bug fixes

- Remove duplicate maxCacheSize check in filetypes_registry_test.go
- Shorten long comment in processor_test.go to stay under 120 chars
- Remove flaky time.Sleep in collector_test.go, use >= 0 assertion
- Close pipe reader in benchmark_test.go to fix file descriptor leak
- Use ContinueOnError in flags_test.go to match ResetFlags behavior
- Add nil check for p.ui in processor_workers.go before UpdateProgress
- Fix resource_monitor_validation_test.go by setting hardMemoryLimitBytes directly

* chore(yaml): add missing document start markers

Add --- document start to YAML files to satisfy yamllint:
- .github/workflows/codeql.yml
- .github/workflows/build-test-publish.yml
- .github/workflows/security.yml
- .github/actions/setup/action.yml

* fix: guard nil resourceMonitor and fix test deadlock

- Guard resourceMonitor before CreateFileProcessingContext call
- Add ui.UpdateProgress on emergency stop and path error returns
- Fix potential deadlock in TestProcessFile using wg.Go with defer close
This commit is contained in:
2025-12-10 19:07:11 +02:00
committed by GitHub
parent ea4a39a360
commit 95b7ef6dd3
149 changed files with 22990 additions and 8976 deletions

View File

@@ -1,14 +1,8 @@
# checkmake configuration # checkmake configuration
# See: https://github.com/checkmake/checkmake#configuration # See: https://github.com/mrtazz/checkmake#configuration
[rules.timestampexpansion] [rules.timestampexpansion]
disabled = true disabled = true
[rules.maxbodylength] [rules.maxbodylength]
disabled = true disabled = true
[rules.minphony]
disabled = true
[rules.phonydeclared]
disabled = true

View File

@@ -7,27 +7,31 @@ trim_trailing_whitespace = true
indent_size = 2 indent_size = 2
indent_style = tab indent_style = tab
tab_width = 2 tab_width = 2
charset = utf-8
[*.go] [*.go]
max_line_length = 120 max_line_length = 120
[*.md] [*.{yml,yaml,json,example}]
trim_trailing_whitespace = false
[*.{yml,yaml,json,toml}]
indent_style = space indent_style = space
max_line_length = 250 max_line_length = 250
[*.{yaml.example,yml.example}]
indent_style = space
[.yamllint]
indent_style = space
[LICENSE] [LICENSE]
max_line_length = 80 max_line_length = 80
indent_size = 0 indent_size = 0
indent_style = space indent_style = space
[*.{sh,md,txt}]
indent_style = space
[.yamllint]
indent_style = space
[Makefile] [Makefile]
max_line_length = 80 indent_style = tab
indent_size = 0
max_line_length = 999
tab_width = 4
[*.md]
trim_trailing_whitespace = false

View File

@@ -1,3 +1,4 @@
---
name: "Setup Go with Runner Hardening" name: "Setup Go with Runner Hardening"
description: "Reusable action to set up Go" description: "Reusable action to set up Go"
inputs: inputs:

View File

@@ -1,4 +1,5 @@
# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json # yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json
---
name: Build, Test, Coverage, and Publish name: Build, Test, Coverage, and Publish
on: on:

View File

@@ -1,3 +1,4 @@
---
name: CodeQL Analysis name: CodeQL Analysis
on: on:

View File

@@ -1,3 +1,4 @@
---
name: Security Scan name: Security Scan
on: on:

14
.gitignore vendored
View File

@@ -1,19 +1,19 @@
*.out
.DS_Store .DS_Store
.idea .idea
.serena/
coverage.*
gibidify gibidify
gibidify-benchmark
gibidify.json gibidify.json
gibidify.txt gibidify.txt
gibidify.yaml gibidify.yaml
megalinter-reports/*
output.json output.json
output.txt output.txt
output.yaml output.yaml
coverage.out
megalinter-reports/*
coverage.*
*.out
gibidify-benchmark
gosec-report.json gosec-report.json
gosec-results.sarif
govulncheck-report.json govulncheck-report.json
govulncheck-errors.log
security-report.md security-report.md
gosec*.log
pr.txt

View File

@@ -1,256 +0,0 @@
run:
timeout: 5m
tests: true
go: "1.24"
build-tags:
- test
# golangci-lint configuration version
version: 2
output:
format: colored-line-number
print-issued-lines: true
print-linter-name: true
path-prefix: ""
sort-results: true
linters:
enable-all: true
disable:
- depguard # Too strict for general use
- exhaustruct # Too many false positives
- ireturn # Too restrictive on interfaces
- varnamelen # Too opinionated on name length
- wrapcheck # Too many false positives
- testpackage # Tests in same package are fine
- paralleltest # Not always necessary
- tparallel # Not always necessary
- nlreturn # Too opinionated on newlines
- wsl # Too opinionated on whitespace
- nonamedreturns # Conflicts with gocritic unnamedResult
linters-settings:
errcheck:
check-type-assertions: true
check-blank: true
exclude-functions:
- io.Copy
- fmt.Print
- fmt.Printf
- fmt.Println
govet:
enable-all: true
gocyclo:
min-complexity: 15
gocognit:
min-complexity: 20
goconst:
min-len: 3
min-occurrences: 3
gofmt:
simplify: true
rewrite-rules:
- pattern: 'interface{}'
replacement: 'any'
goimports:
local-prefixes: github.com/ivuorinen/gibidify
golint:
min-confidence: 0.8
lll:
line-length: 120
tab-width: 2 # EditorConfig: tab_width = 2
misspell:
locale: US
nakedret:
max-func-lines: 30
prealloc:
simple: true
range-loops: true
for-loops: true
revive:
enable-all-rules: true
rules:
- name: package-comments
disabled: true
- name: file-header
disabled: true
- name: max-public-structs
disabled: true
- name: line-length-limit
arguments: [120]
- name: function-length
arguments: [50, 100]
- name: cognitive-complexity
arguments: [20]
- name: cyclomatic
arguments: [15]
- name: add-constant
arguments:
- maxLitCount: "3"
allowStrs: "\"error\",\"\""
allowInts: "0,1,2"
- name: argument-limit
arguments: [6]
- name: banned-characters
disabled: true
- name: function-result-limit
arguments: [3]
gosec:
excludes:
- G104 # Handled by errcheck
severity: medium
confidence: medium
exclude-generated: true
config:
G301: "0750"
G302: "0640"
G306: "0640"
dupl:
threshold: 150
gocritic:
enabled-tags:
- diagnostic
- experimental
- opinionated
- performance
- style
disabled-checks:
- whyNoLint
- paramTypeCombine
gofumpt:
extra-rules: true
# EditorConfig compliance settings
# These settings enforce .editorconfig rules:
# - end_of_line = lf (enforced by gofumpt)
# - insert_final_newline = true (enforced by gofumpt)
# - trim_trailing_whitespace = true (enforced by whitespace linter)
# - indent_style = tab, tab_width = 2 (enforced by gofumpt and lll)
whitespace:
multi-if: false # EditorConfig: trim trailing whitespace
multi-func: false # EditorConfig: trim trailing whitespace
nolintlint:
allow-leading-space: false # EditorConfig: trim trailing whitespace
allow-unused: false
require-explanation: false
require-specific: true
godox:
keywords:
- FIXME
- BUG
- HACK
mnd:
settings:
mnd:
checks:
- argument
- case
- condition
- operation
- return
- assign
ignored-numbers:
- '0'
- '1'
- '2'
- '10'
- '100'
funlen:
lines: 80
statements: 60
nestif:
min-complexity: 5
gomodguard:
allowed:
modules: []
domains: []
blocked:
modules: []
versions: []
issues:
exclude-use-default: false
exclude-case-sensitive: false
max-issues-per-linter: 0
max-same-issues: 0
uniq-by-line: true
exclude-dirs:
- vendor
- third_party
- testdata
- examples
- .git
exclude-files:
- ".*\\.pb\\.go$"
- ".*\\.gen\\.go$"
exclude-rules:
- path: _test\.go
linters:
- dupl
- gosec
- goconst
- funlen
- gocognit
- gocyclo
- errcheck
- lll
- nestif
- path: main\.go
linters:
- gochecknoglobals
- gochecknoinits
- path: fileproc/filetypes\.go
linters:
- gochecknoglobals # Allow globals for singleton registry pattern
- text: "Using the variable on range scope"
linters:
- scopelint
- text: "should have comment or be unexported"
linters:
- golint
- revive
- text: "don't use ALL_CAPS in Go names"
linters:
- golint
- stylecheck
exclude:
- "Error return value of .* is not checked"
- "exported (type|method|function) .* should have comment"
- "ST1000: at least one file in a package should have a package comment"
severity:
default-severity: error
case-sensitive: false

View File

@@ -15,9 +15,11 @@ PRINT_ALPACA: false # Print Alpaca logo in console
SARIF_REPORTER: true # Generate SARIF report SARIF_REPORTER: true # Generate SARIF report
SHOW_SKIPPED_LINTERS: false # Show skipped linters in MegaLinter log SHOW_SKIPPED_LINTERS: false # Show skipped linters in MegaLinter log
GO_REVIVE_CLI_LINT_MODE: project
DISABLE_LINTERS: DISABLE_LINTERS:
- REPOSITORY_DEVSKIM - REPOSITORY_DEVSKIM
- REPOSITORY_TRIVY - REPOSITORY_TRIVY
- GO_GOLANGCI_LINT - GO_GOLANGCI_LINT
- YAML_PRETTIER
# By default megalinter uses list_of_files, which is wrong.
GO_REVIVE_CLI_LINT_MODE: project

View File

@@ -1,9 +1,13 @@
---
# yaml-language-server: $schema=https://json.schemastore.org/pre-commit-config.json
# For more hooks, see https://pre-commit.com/hooks.html
repos: repos:
- repo: https://github.com/golangci/golangci-lint - repo: https://github.com/editorconfig-checker/editorconfig-checker.python
rev: v2.7.2 rev: 3.4.0
hooks: hooks:
- id: golangci-lint - id: editorconfig-checker
args: ["--timeout=5m"] alias: ec
- repo: https://github.com/tekwizely/pre-commit-golang - repo: https://github.com/tekwizely/pre-commit-golang
rev: v1.0.0-rc.2 rev: v1.0.0-rc.2
hooks: hooks:
@@ -11,14 +15,13 @@ repos:
alias: build alias: build
- id: go-mod-tidy - id: go-mod-tidy
alias: tidy alias: tidy
- id: go-revive
alias: revive
- id: go-vet-mod
alias: vet
- id: go-staticcheck-mod
alias: static
- id: go-fmt - id: go-fmt
alias: fmt alias: fmt
- repo: https://github.com/editorconfig-checker/editorconfig-checker.python - id: go-sec-mod
rev: 3.6.0 alias: sec
hooks:
- id: editorconfig-checker
alias: ec
- repo: https://github.com/shellcheck-py/shellcheck-py
rev: v0.11.0.1
hooks:
- id: shellcheck

View File

@@ -3,6 +3,7 @@
# * For JavaScript, use typescript # * For JavaScript, use typescript
# Special requirements: # Special requirements:
# * csharp: Requires the presence of a .sln file in the project folder. # * csharp: Requires the presence of a .sln file in the project folder.
---
language: go language: go
# whether to use the project's gitignore file to ignore files # whether to use the project's gitignore file to ignore files

18
.yamlfmt.yml Normal file
View File

@@ -0,0 +1,18 @@
---
doublestar: true
gitignore_excludes: true
formatter:
type: basic
include_document_start: true
retain_line_breaks_single: true
scan_folded_as_literal: false
max_line_length: 0
trim_trailing_whitespace: true
array_indent: 2
force_array_style: block
include:
- ./**/*.yml
- ./**/*.yaml
- .github/**/*.yml
- .github/**/*.yaml
# exclude:

View File

@@ -1,3 +1,4 @@
---
# yamllint configuration # yamllint configuration
# See: https://yamllint.readthedocs.io/en/stable/configuration.html # See: https://yamllint.readthedocs.io/en/stable/configuration.html
@@ -35,6 +36,3 @@ rules:
# Relax comments formatting # Relax comments formatting
comments: comments:
min-spaces-from-content: 1 min-spaces-from-content: 1
# Allow document start marker to be optional
document-start: disable

View File

@@ -1,12 +1,15 @@
# CLAUDE.md # CLAUDE.md
Go CLI aggregating code files into LLM-optimized output. Supports markdown/JSON/YAML with concurrent processing. Go CLI aggregating code files into LLM-optimized output.
Supports markdown/JSON/YAML with concurrent processing.
## Architecture (42 files, 8.2K lines) ## Architecture
**Core**: `main.go` (37), `cli/` (4), `fileproc/` (27), `config/` (3), `utils/` (4), `testutil/` (2) **Core**: `main.go`, `cli/`, `fileproc/`, `config/`, `utils/`, `testutil/`, `cmd/`
**Modules**: Collection, processing, writers, registry (~63ns cache), resource limits **Advanced**: `metrics/`, `templates/`, `benchmark/`
**Modules**: Collection, processing, writers, registry (~63ns cache), resource limits, metrics, templating
**Patterns**: Producer-consumer, thread-safe registry, streaming, modular (50-200 lines) **Patterns**: Producer-consumer, thread-safe registry, streaming, modular (50-200 lines)
@@ -15,6 +18,7 @@ Go CLI aggregating code files into LLM-optimized output. Supports markdown/JSON/
```bash ```bash
make lint-fix && make lint && make test make lint-fix && make lint && make test
./gibidify -source <dir> -format markdown --verbose ./gibidify -source <dir> -format markdown --verbose
./gibidify -source <dir> -format json --log-level debug --verbose
``` ```
## Config ## Config
@@ -22,29 +26,51 @@ make lint-fix && make lint && make test
`~/.config/gibidify/config.yaml` `~/.config/gibidify/config.yaml`
Size limit 5MB, ignore dirs, custom types, 100MB memory limit Size limit 5MB, ignore dirs, custom types, 100MB memory limit
## Quality ## Linting Standards (MANDATORY)
**CRITICAL**: `make lint-fix && make lint` (0 issues), 120 chars, EditorConfig, 30+ linters **Linter**: revive (comprehensive rule set migrated from golangci-lint)
**Command**: `revive -config revive.toml ./...`
**Complexity**: cognitive-complexity ≤15, cyclomatic ≤15, max-control-nesting ≤5
**Security**: unhandled errors, secure coding patterns, credential detection
**Performance**: optimize-operands-order, string-format, range optimizations
**Format**: line-length ≤120 chars, EditorConfig (LF, tabs), gofmt/goimports
**Testing**: error handling best practices, 0 tolerance policy
**CRITICAL**: All rules non-negotiable. `make lint-fix && make lint` must show 0 issues.
## Testing ## Testing
**Coverage**: 84%+ (utils 90.9%, fileproc 83.8%), race detection, benchmarks **Coverage**: 77.9% overall (utils 90.0%, cli 83.8%, config 77.0%, testutil 73.7%, fileproc 74.5%, metrics 96.0%, templates 87.3%)
**Patterns**: Table-driven tests, shared testutil helpers, mock objects, error assertions
**Race detection**, benchmarks, comprehensive integration tests
## Development Patterns
**Logging**: Use `utils.Logger()` for all logging (replaces logrus). Default WARN level, set via `--log-level` flag
**Error Handling**: Use `utils.WrapError` family for structured errors with context
**Streaming**: Use `utils.StreamContent/StreamLines` for consistent file processing
**Context**: Use `utils.CheckContextCancellation` for standardized cancellation
**Testing**: Use `testutil.*` helpers for directory setup, error assertions
**Validation**: Centralized in `config/validation.go` with structured error collection
## Standards ## Standards
EditorConfig (LF, tabs), semantic commits, testing required EditorConfig (LF, tabs), semantic commits, testing required, error wrapping
## revive.toml Restrictions
**AGENTS DO NOT HAVE PERMISSION** to modify `revive.toml` configuration unless user explicitly requests it.
The linting configuration is carefully tuned and should not be altered during normal development.
## Status ## Status
**Health: 10/10** - Production-ready, 84%+ coverage, modular, memory-optimized **Health: 9/10** - Production-ready with systematic deduplication complete
**Done**: Errors, benchmarks, config, optimization, modularization, CLI (progress/colors), security (path validation, resource limits, scanning) **Done**: Deduplication, errors, benchmarks, config, optimization, testing (77.9%), modularization, linting (0 issues), metrics system, templating
**Next**: Documentation, output customization
## Workflow ## Workflow
1. `make lint-fix` first 1. `make lint-fix` first
2. >80% coverage 2. >80% coverage
3. Follow patterns 3. Follow patterns
4. Update docs 4. Update docs

View File

@@ -1,38 +1,17 @@
# Build stage - builds the binary for the target architecture # Use a minimal base image
FROM --platform=$BUILDPLATFORM golang:1.25.5-alpine AS builder FROM alpine:3.22.1
# Build arguments automatically set by buildx # Add user
ARG TARGETOS RUN useradd -ms /bin/bash gibidify
ARG TARGETARCH
ARG TARGETVARIANT
WORKDIR /build # Use the new user
# Copy go mod files first for better layer caching
COPY go.mod go.sum ./
RUN go mod download
# Copy source code
COPY . .
# Build the binary for the target platform
RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -ldflags="-s -w" -o gibidify .
# Runtime stage - minimal image with the binary
FROM alpine:3.23.0
# Install ca-certificates for HTTPS and create non-root user
# hadolint ignore=DL3018
# kics-scan ignore-line
RUN apk add --no-cache ca-certificates && \
adduser -D -s /bin/sh gibidify
# Copy the binary from builder
COPY --from=builder /build/gibidify /usr/local/bin/gibidify
# Use non-root user
USER gibidify USER gibidify
# Copy the gibidify binary into the container
COPY gibidify /usr/local/bin/gibidify
# Ensure the binary is executable
RUN chmod +x /usr/local/bin/gibidify
# Set the entrypoint # Set the entrypoint
ENTRYPOINT ["/usr/local/bin/gibidify"] ENTRYPOINT ["/usr/local/bin/gibidify"]

View File

@@ -1,14 +1,10 @@
.PHONY: all clean test test-coverage build coverage help lint lint-fix \ .PHONY: all help install-tools lint lint-fix test coverage build clean all build-benchmark benchmark benchmark-go benchmark-go-cli benchmark-go-fileproc benchmark-go-metrics benchmark-go-shared benchmark-all benchmark-collection benchmark-processing benchmark-concurrency benchmark-format security security-full vuln-check update-deps check-all dev-setup
lint-verbose install-tools benchmark benchmark-collection \
benchmark-concurrency benchmark-format benchmark-processing \
build-benchmark check-all ci-lint ci-test dev-setup security \
security-full vuln-check deps-update deps-check deps-tidy
# Default target shows help # Default target shows help
.DEFAULT_GOAL := help .DEFAULT_GOAL := help
# All target runs full workflow # All target runs full workflow
all: lint test build all: lint lint-fix test build
# Help target # Help target
help: help:
@@ -26,19 +22,11 @@ lint:
lint-fix: lint-fix:
@./scripts/lint-fix.sh @./scripts/lint-fix.sh
# Run linters with verbose output
lint-verbose:
@./scripts/lint-verbose.sh
# Run tests # Run tests
test: test:
@echo "Running tests..." @echo "Running tests..."
@go test -race -v ./... @go test -race -v ./...
# Run tests with coverage output
test-coverage:
@./scripts/test-coverage.sh
# Run tests with coverage # Run tests with coverage
coverage: coverage:
@echo "Running tests with coverage..." @echo "Running tests with coverage..."
@@ -55,13 +43,14 @@ build:
# Clean build artifacts # Clean build artifacts
clean: clean:
@echo "Cleaning build artifacts..." @echo "Cleaning build artifacts..."
@rm -f gibidify gibidify-benchmark @rm -f gibidify gibidify-benchmark coverage.out coverage.html *.out
@rm -f coverage.out coverage.html
@echo "Clean complete" @echo "Clean complete"
# CI-specific targets # CI-specific targets
.PHONY: ci-lint ci-test
ci-lint: ci-lint:
@golangci-lint run --out-format=github-actions ./... @revive -config revive.toml -formatter friendly -set_exit_status ./...
ci-test: ci-test:
@go test -race -coverprofile=coverage.out -json ./... > test-results.json @go test -race -coverprofile=coverage.out -json ./... > test-results.json
@@ -72,11 +61,36 @@ build-benchmark:
@go build -ldflags="-s -w" -o gibidify-benchmark ./cmd/benchmark @go build -ldflags="-s -w" -o gibidify-benchmark ./cmd/benchmark
@echo "Build complete: ./gibidify-benchmark" @echo "Build complete: ./gibidify-benchmark"
# Run benchmarks # Run custom benchmark binary
benchmark: build-benchmark benchmark: build-benchmark
@echo "Running all benchmarks..." @echo "Running custom benchmarks..."
@./gibidify-benchmark -type=all @./gibidify-benchmark -type=all
# Run all Go test benchmarks
benchmark-go:
@echo "Running all Go test benchmarks..."
@go test -bench=. -benchtime=100ms -run=^$$ ./...
# Run Go test benchmarks for specific packages
benchmark-go-cli:
@echo "Running CLI benchmarks..."
@go test -bench=. -benchtime=100ms -run=^$$ ./cli/...
benchmark-go-fileproc:
@echo "Running fileproc benchmarks..."
@go test -bench=. -benchtime=100ms -run=^$$ ./fileproc/...
benchmark-go-metrics:
@echo "Running metrics benchmarks..."
@go test -bench=. -benchtime=100ms -run=^$$ ./metrics/...
benchmark-go-shared:
@echo "Running shared benchmarks..."
@go test -bench=. -benchtime=100ms -run=^$$ ./shared/...
# Run all benchmarks (custom + Go test)
benchmark-all: benchmark benchmark-go
# Run specific benchmark types # Run specific benchmark types
benchmark-collection: build-benchmark benchmark-collection: build-benchmark
@echo "Running file collection benchmarks..." @echo "Running file collection benchmarks..."
@@ -99,24 +113,19 @@ security:
@echo "Running comprehensive security scan..." @echo "Running comprehensive security scan..."
@./scripts/security-scan.sh @./scripts/security-scan.sh
security-full: security-full: install-tools
@echo "Running full security analysis..." @echo "Running full security analysis..."
@./scripts/security-scan.sh @./scripts/security-scan.sh
@echo "Running additional security checks..."
@gosec -fmt=json -out=security-report.json ./...
@staticcheck -checks=all ./...
vuln-check: vuln-check:
@echo "Checking for dependency vulnerabilities..." @echo "Checking for dependency vulnerabilities..."
@go install golang.org/x/vuln/cmd/govulncheck@latest @go install golang.org/x/vuln/cmd/govulncheck@v1.1.4
@govulncheck ./... @govulncheck ./...
# Dependency management targets # Update dependencies
deps-check: update-deps:
@./scripts/deps-check.sh @echo "Updating Go dependencies..."
@./scripts/update-deps.sh
deps-update:
@./scripts/deps-update.sh
deps-tidy:
@echo "Cleaning up dependencies..."
@go mod tidy
@go mod verify
@echo "Dependencies cleaned and verified successfully!"

View File

@@ -14,9 +14,11 @@ file sections with separators, and a suffix.
- **Concurrent processing** with configurable worker pools - **Concurrent processing** with configurable worker pools
- **Comprehensive configuration** via YAML with validation - **Comprehensive configuration** via YAML with validation
- **Production-ready** with structured error handling and benchmarking - **Production-ready** with structured error handling and benchmarking
- **Modular architecture** - clean, focused codebase with ~63ns registry lookups - **Modular architecture** - clean, focused codebase (92 files, ~21.5K lines) with ~63ns registry lookups
- **Enhanced CLI experience** - progress bars, colored output, helpful error messages - **Enhanced CLI experience** - progress bars, colored output, helpful error messages
- **Cross-platform** with Docker support - **Cross-platform** with Docker support
- **Advanced template system** - 4 built-in templates (default, minimal, detailed, compact) with custom template support, variable substitution, and YAML-based configuration
- **Comprehensive metrics and profiling** - real-time processing statistics, performance analysis, memory usage tracking, and automated recommendations
## Installation ## Installation
@@ -32,15 +34,16 @@ go build -o gibidify .
```bash ```bash
./gibidify \ ./gibidify \
-source <source_directory> \ -source <source_directory> \
-destination <output_file> \ -destination <output_file> \
-format markdown|json|yaml \ -format markdown|json|yaml \
-concurrency <num_workers> \ -concurrency <num_workers> \
--prefix="..." \ --prefix="..." \
--suffix="..." \ --suffix="..." \
--no-colors \ --no-colors \
--no-progress \ --no-progress \
--verbose --verbose \
--log-level debug
``` ```
Flags: Flags:
@@ -53,6 +56,7 @@ Flags:
- `--no-colors`: disable colored terminal output. - `--no-colors`: disable colored terminal output.
- `--no-progress`: disable progress bars. - `--no-progress`: disable progress bars.
- `--verbose`: enable verbose output and detailed logging. - `--verbose`: enable verbose output and detailed logging.
- `--log-level`: set log level (default: warn; accepted values: debug, info, warn, error).
## Docker ## Docker
@@ -66,13 +70,13 @@ Run the Docker container:
```bash ```bash
docker run --rm \ docker run --rm \
-v $(pwd):/workspace \ -v $(pwd):/workspace \
-v $HOME/.config/gibidify:/config \ -v $HOME/.config/gibidify:/config \
ghcr.io/ivuorinen/gibidify:<tag> \ ghcr.io/ivuorinen/gibidify:<tag> \
-source /workspace/your_source_directory \ -source /workspace/your_source_directory \
-destination /workspace/output.txt \ -destination /workspace/output.txt \
--prefix="Your prefix text" \ --prefix="Your prefix text" \
--suffix="Your suffix text" --suffix="Your suffix text"
``` ```
## Configuration ## Configuration
@@ -123,6 +127,33 @@ backpressure:
maxPendingWrites: 100 # Max writes in write channel buffer maxPendingWrites: 100 # Max writes in write channel buffer
maxMemoryUsage: 104857600 # 100MB max memory usage maxMemoryUsage: 104857600 # 100MB max memory usage
memoryCheckInterval: 1000 # Check memory every 1000 files memoryCheckInterval: 1000 # Check memory every 1000 files
# Output and template customization
output:
# Template selection: default, minimal, detailed, compact, or custom
# Templates control output structure and formatting
template: "default"
# Metadata options
metadata:
includeStats: true
includeTimestamp: true
includeFileCount: true
includeSourcePath: true
includeMetrics: true
# Markdown-specific options
markdown:
useCodeBlocks: true
includeLanguage: true
headerLevel: 2
tableOfContents: false
useCollapsible: false
syntaxHighlighting: true
lineNumbers: false
# Custom template variables
variables:
project_name: "My Project"
author: "Developer Name"
version: "1.0.0"
``` ```
See `config.example.yaml` for a comprehensive configuration example. See `config.example.yaml` for a comprehensive configuration example.

134
TODO.md
View File

@@ -4,43 +4,127 @@ Prioritized improvements by impact/effort.
## ✅ Completed ## ✅ Completed
**Core**: Testing (84%+), config validation, structured errors, benchmarking ✅ **Core**: Config validation, structured errors, benchmarking, linting (revive: 0 issues)
**Architecture**: Modularization (50-200 lines), CLI (progress/colors), security (path validation, resource limits, scanning) ✅ **Architecture**: Modularization (92 files, ~21.5K lines), CLI (progress/colors), security (path validation, resource limits, scanning) ✅
## 🚀 Current Priorities ## 🚀 Critical Priorities
### Metrics & Profiling ### Testing Coverage (URGENT)
- [ ] Processing stats, timing - [x] **CLI module testing** (0% → 83.8%) - COMPLETED ✅
- [x] cli/flags_test.go - Flag parsing and validation ✅
- [x] cli/errors_test.go - Error formatting and structured errors ✅
- [x] cli/ui_test.go - UI components, colors, progress bars ✅
- [x] cli/processor_test.go - Processing workflow integration ✅
- [x] **Utils module testing** (7.4% → 90.0%) - COMPLETED ✅
- [x] utils/writers_test.go - Writer functions (98% complete, minor test fixes needed) ✅
- [x] Enhanced utils/paths_test.go - Security and edge cases ✅
- [x] Enhanced utils/errors_test.go - StructuredError system ✅
- [x] **Testutil module testing** (45.1% → 73.7%) - COMPLETED ✅
- [x] testutil/utility_test.go - GetBaseName function comprehensive tests ✅
- [x] testutil/directory_structure_test.go - CreateTestDirectoryStructure and SetupTempDirWithStructure ✅
- [x] testutil/assertions_test.go - All AssertError functions comprehensive coverage ✅
- [x] testutil/error_scenarios_test.go - Edge cases and performance benchmarks ✅
- [x] **Main module testing** (41% → 50.0%) - COMPLETED ✅
- [x] **Fileproc module improvement** (66% → 74.5%) - COMPLETED ✅
### Output Customization ### ✅ Metrics & Profiling - COMPLETED
- [ ] Templates, markdown config, metadata - [x] **Comprehensive metrics collection system** with processing statistics ✅
- [x] File processing metrics (processed, skipped, errors) ✅
- [x] Size metrics (total, average, largest, smallest file sizes) ✅
- [x] Performance metrics (files/sec, bytes/sec, processing time) ✅
- [x] Memory and resource tracking (peak memory, current memory, goroutine count) ✅
- [x] Format-specific metrics and error breakdown ✅
- [x] Phase timing (collection, processing, writing, finalize) ✅
- [x] Concurrency tracking and recommendations ✅
- [x] **Performance measurements and reporting**
- [x] Real-time progress reporting in CLI ✅
- [x] Verbose mode with detailed statistics ✅
- [x] Final comprehensive profiling reports ✅
- [x] Performance recommendations based on metrics ✅
- [x] **Structured logging integration** with centralized logging service ✅
- [x] Configurable log levels (debug, info, warn, error) ✅
- [x] Context-aware logging with structured data ✅
- [x] Metrics data integration in log output ✅
### ✅ Output Customization - COMPLETED
- [x] **Template system for output formatting**
- [x] Builtin templates: default, minimal, detailed, compact ✅
- [x] Custom template support with variables ✅
- [x] Template functions for formatting (formatSize, basename, etc.) ✅
- [x] Header/footer and file header/footer customization ✅
- [x] **Configurable markdown options**
- [x] Code block controls (syntax highlighting, line numbers) ✅
- [x] Header levels and table of contents ✅
- [x] Collapsible sections for space efficiency ✅
- [x] Line length limits and long file folding ✅
- [x] Custom CSS support ✅
- [x] **Metadata integration in outputs**
- [x] Configurable metadata inclusion (stats, timestamp, file counts) ✅
- [x] Processing metrics in output (performance, memory usage) ✅
- [x] File type breakdown and error summaries ✅
- [x] Source path and processing time information ✅
- [x] **Enhanced configuration system**
- [x] Template selection and customization options ✅
- [x] Metadata control flags ✅
- [x] Markdown formatting preferences ✅
- [x] Custom template variables support ✅
### Documentation ### Documentation
- [ ] API docs, user guides - [ ] API docs, user guides
## 🌟 Future
**Plugins**: Custom handlers, formats
**Git**: Commit filtering, blame
**Rich output**: HTML, PDF, web UI
**Monitoring**: Prometheus, structured logging
## Guidelines ## Guidelines
**Before**: `make lint-fix && make lint`, >80% coverage **Before**: `make lint-fix && make lint` (0 issues), >80% coverage
**Priorities**: Security → UX → Extensions **Priorities**: Testing → Security → UX → Extensions
## Status (2025-07-19) ## Status (2025-08-23 - Phase 3 Feature Implementation Complete)
**Health: 10/10** - Production-ready, 42 files (8.2K lines), 84%+ coverage **Health: 10/10** - Advanced metrics & profiling system and comprehensive output customization implemented
**Done**: Testing, config, errors, performance, modularization, CLI, security **Stats**: 92 files (~21.5K lines), 77.9% overall coverage achieved
**Next**: Documentation → Output customization - CLI: 83.8% ✅, Utils: 90.0% ✅, Config: 77.0% ✅, Testutil: 73.7% ✅, Fileproc: 74.5% ✅, Main: 50.0% ✅, Metrics: 96.0% ✅, Templates: 87.3% ✅, Benchmark: 64.7% ✅
### Token Usage **Completed Today**:
-**Phase 1**: Consolidated duplicate code patterns
- Writer closeReader → utils.SafeCloseReader
- Custom yamlQuoteString → utils.EscapeForYAML
- Streaming patterns → utils.StreamContent/StreamLines
-**Phase 2**: Enhanced test infrastructure
- **Phase 2A**: Main module (41% → 50.0%) - Complete integration testing
- **Phase 2B**: Fileproc module (66% → 74.5%) - Streaming and backpressure testing
- **Phase 2C**: Testutil module (45.1% → 73.7%) - Utility and assertion testing
- Shared test helpers (directory structure, error assertions)
- Advanced testutil patterns (avoided import cycles)
-**Phase 3**: Standardized error/context handling
- Error creation using utils.WrapError family
- Centralized context cancellation patterns
-**Phase 4**: Documentation updates
- TODO.md: 171 words (~228 tokens) - 35% reduction ✅ **Impact**: Eliminated code duplication, enhanced maintainability, achieved comprehensive test coverage across all major modules
- CLAUDE.md: 160 words (~213 tokens) - 25% reduction ✅
- Total: 331 words (~441 tokens) - 30% reduction ✅
*Optimized from 474 → 331 words while preserving critical information* **Completed This Session**:
-**Phase 3A**: Advanced Metrics & Profiling System
- Comprehensive processing statistics collection (files, sizes, performance)
- Real-time progress reporting with detailed metrics
- Phase timing tracking (collection, processing, writing, finalize)
- Memory and resource usage monitoring
- Format-specific metrics and error breakdown
- Performance recommendations engine
- Structured logging integration
-**Phase 3B**: Output Customization Features
- Template system with 4 builtin templates (default, minimal, detailed, compact)
- Custom template support with variable substitution
- Configurable markdown options (code blocks, TOC, collapsible sections)
- Metadata integration with selective inclusion controls
- Enhanced configuration system for all customization options
-**Phase 3C**: Comprehensive Testing & Integration
- Full test coverage for metrics and templates packages
- Integration with existing CLI processor workflow
- Deadlock-free concurrent metrics collection
- Configuration system extensions
**Impact**: Added powerful analytics and customization capabilities while maintaining high code quality and test coverage
**Next Session**:
- Phase 4: Enhanced documentation and user guides
- Optional: Advanced features (watch mode, incremental processing, etc.)

View File

@@ -12,7 +12,7 @@ import (
"github.com/ivuorinen/gibidify/config" "github.com/ivuorinen/gibidify/config"
"github.com/ivuorinen/gibidify/fileproc" "github.com/ivuorinen/gibidify/fileproc"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// Result represents the results of a benchmark run. // Result represents the results of a benchmark run.
@@ -48,6 +48,46 @@ type Suite struct {
Results []Result Results []Result
} }
// buildBenchmarkResult constructs a Result with all metrics calculated.
// This eliminates code duplication across benchmark functions.
func buildBenchmarkResult(
name string,
files []string,
totalBytes int64,
duration time.Duration,
memBefore, memAfter runtime.MemStats,
) *Result {
result := &Result{
Name: name,
Duration: duration,
FilesProcessed: len(files),
BytesProcessed: totalBytes,
}
// Calculate rates with zero-division guard
secs := duration.Seconds()
if secs == 0 {
result.FilesPerSecond = 0
result.BytesPerSecond = 0
} else {
result.FilesPerSecond = float64(len(files)) / secs
result.BytesPerSecond = float64(totalBytes) / secs
}
result.MemoryUsage = MemoryStats{
AllocMB: shared.SafeMemoryDiffMB(memAfter.Alloc, memBefore.Alloc),
SysMB: shared.SafeMemoryDiffMB(memAfter.Sys, memBefore.Sys),
NumGC: memAfter.NumGC - memBefore.NumGC,
PauseTotalNs: memAfter.PauseTotalNs - memBefore.PauseTotalNs,
}
result.CPUUsage = CPUStats{
Goroutines: runtime.NumGoroutine(),
}
return result
}
// FileCollectionBenchmark benchmarks file collection operations. // FileCollectionBenchmark benchmarks file collection operations.
func FileCollectionBenchmark(sourceDir string, numFiles int) (*Result, error) { func FileCollectionBenchmark(sourceDir string, numFiles int) (*Result, error) {
// Load configuration to ensure proper file filtering // Load configuration to ensure proper file filtering
@@ -58,14 +98,15 @@ func FileCollectionBenchmark(sourceDir string, numFiles int) (*Result, error) {
if sourceDir == "" { if sourceDir == "" {
tempDir, cleanupFunc, err := createBenchmarkFiles(numFiles) tempDir, cleanupFunc, err := createBenchmarkFiles(numFiles)
if err != nil { if err != nil {
return nil, gibidiutils.WrapError( return nil, shared.WrapError(
err, err,
gibidiutils.ErrorTypeFileSystem, shared.ErrorTypeFileSystem,
gibidiutils.CodeFSAccess, shared.CodeFSAccess,
"failed to create benchmark files", shared.BenchmarkMsgFailedToCreateFiles,
) )
} }
cleanup = cleanupFunc cleanup = cleanupFunc
//nolint:errcheck // Benchmark output, errors don't affect results
defer cleanup() defer cleanup()
sourceDir = tempDir sourceDir = tempDir
} }
@@ -79,11 +120,11 @@ func FileCollectionBenchmark(sourceDir string, numFiles int) (*Result, error) {
// Run the file collection benchmark // Run the file collection benchmark
files, err := fileproc.CollectFiles(sourceDir) files, err := fileproc.CollectFiles(sourceDir)
if err != nil { if err != nil {
return nil, gibidiutils.WrapError( return nil, shared.WrapError(
err, err,
gibidiutils.ErrorTypeProcessing, shared.ErrorTypeProcessing,
gibidiutils.CodeProcessingCollection, shared.CodeProcessingCollection,
"benchmark file collection failed", shared.BenchmarkMsgCollectionFailed,
) )
} }
@@ -101,30 +142,11 @@ func FileCollectionBenchmark(sourceDir string, numFiles int) (*Result, error) {
} }
} }
result := &Result{ result := buildBenchmarkResult("FileCollection", files, totalBytes, duration, memBefore, memAfter)
Name: "FileCollection",
Duration: duration,
FilesProcessed: len(files),
BytesProcessed: totalBytes,
FilesPerSecond: float64(len(files)) / duration.Seconds(),
BytesPerSecond: float64(totalBytes) / duration.Seconds(),
MemoryUsage: MemoryStats{
AllocMB: float64(memAfter.Alloc-memBefore.Alloc) / 1024 / 1024,
SysMB: float64(memAfter.Sys-memBefore.Sys) / 1024 / 1024,
NumGC: memAfter.NumGC - memBefore.NumGC,
PauseTotalNs: memAfter.PauseTotalNs - memBefore.PauseTotalNs,
},
CPUUsage: CPUStats{
Goroutines: runtime.NumGoroutine(),
},
}
return result, nil return result, nil
} }
// FileProcessingBenchmark benchmarks full file processing pipeline. // FileProcessingBenchmark benchmarks full file processing pipeline.
//
//revive:disable-next-line:function-length
func FileProcessingBenchmark(sourceDir string, format string, concurrency int) (*Result, error) { func FileProcessingBenchmark(sourceDir string, format string, concurrency int) (*Result, error) {
// Load configuration to ensure proper file filtering // Load configuration to ensure proper file filtering
config.LoadConfig() config.LoadConfig()
@@ -132,16 +154,17 @@ func FileProcessingBenchmark(sourceDir string, format string, concurrency int) (
var cleanup func() var cleanup func()
if sourceDir == "" { if sourceDir == "" {
// Create temporary directory with test files // Create temporary directory with test files
tempDir, cleanupFunc, err := createBenchmarkFiles(100) tempDir, cleanupFunc, err := createBenchmarkFiles(shared.BenchmarkDefaultFileCount)
if err != nil { if err != nil {
return nil, gibidiutils.WrapError( return nil, shared.WrapError(
err, err,
gibidiutils.ErrorTypeFileSystem, shared.ErrorTypeFileSystem,
gibidiutils.CodeFSAccess, shared.CodeFSAccess,
"failed to create benchmark files", shared.BenchmarkMsgFailedToCreateFiles,
) )
} }
cleanup = cleanupFunc cleanup = cleanupFunc
//nolint:errcheck // Benchmark output, errors don't affect results
defer cleanup() defer cleanup()
sourceDir = tempDir sourceDir = tempDir
} }
@@ -149,21 +172,21 @@ func FileProcessingBenchmark(sourceDir string, format string, concurrency int) (
// Create temporary output file // Create temporary output file
outputFile, err := os.CreateTemp("", "benchmark_output_*."+format) outputFile, err := os.CreateTemp("", "benchmark_output_*."+format)
if err != nil { if err != nil {
return nil, gibidiutils.WrapError( return nil, shared.WrapError(
err, err,
gibidiutils.ErrorTypeIO, shared.ErrorTypeIO,
gibidiutils.CodeIOFileCreate, shared.CodeIOFileCreate,
"failed to create benchmark output file", "failed to create benchmark output file",
) )
} }
defer func() { defer func() {
if err := outputFile.Close(); err != nil { if err := outputFile.Close(); err != nil {
// Log error but don't fail the benchmark //nolint:errcheck // Warning message in defer, failure doesn't affect benchmark
fmt.Printf("Warning: failed to close benchmark output file: %v\n", err) _, _ = fmt.Printf("Warning: failed to close benchmark output file: %v\n", err)
} }
if err := os.Remove(outputFile.Name()); err != nil { if err := os.Remove(outputFile.Name()); err != nil {
// Log error but don't fail the benchmark //nolint:errcheck // Warning message in defer, failure doesn't affect benchmark
fmt.Printf("Warning: failed to remove benchmark output file: %v\n", err) _, _ = fmt.Printf("Warning: failed to remove benchmark output file: %v\n", err)
} }
}() }()
@@ -176,27 +199,21 @@ func FileProcessingBenchmark(sourceDir string, format string, concurrency int) (
// Run the full processing pipeline // Run the full processing pipeline
files, err := fileproc.CollectFiles(sourceDir) files, err := fileproc.CollectFiles(sourceDir)
if err != nil { if err != nil {
return nil, gibidiutils.WrapError( return nil, shared.WrapError(
err, err,
gibidiutils.ErrorTypeProcessing, shared.ErrorTypeProcessing,
gibidiutils.CodeProcessingCollection, shared.CodeProcessingCollection,
"benchmark file collection failed", shared.BenchmarkMsgCollectionFailed,
) )
} }
// Process files with concurrency // Process files with concurrency
err = runProcessingPipeline(context.Background(), processingConfig{ err = runProcessingPipeline(context.Background(), files, outputFile, format, concurrency, sourceDir)
files: files,
outputFile: outputFile,
format: format,
concurrency: concurrency,
sourceDir: sourceDir,
})
if err != nil { if err != nil {
return nil, gibidiutils.WrapError( return nil, shared.WrapError(
err, err,
gibidiutils.ErrorTypeProcessing, shared.ErrorTypeProcessing,
gibidiutils.CodeProcessingFileRead, shared.CodeProcessingFileRead,
"benchmark processing pipeline failed", "benchmark processing pipeline failed",
) )
} }
@@ -215,24 +232,8 @@ func FileProcessingBenchmark(sourceDir string, format string, concurrency int) (
} }
} }
result := &Result{ benchmarkName := fmt.Sprintf("FileProcessing_%s_c%d", format, concurrency)
Name: fmt.Sprintf("FileProcessing_%s_c%d", format, concurrency), result := buildBenchmarkResult(benchmarkName, files, totalBytes, duration, memBefore, memAfter)
Duration: duration,
FilesProcessed: len(files),
BytesProcessed: totalBytes,
FilesPerSecond: float64(len(files)) / duration.Seconds(),
BytesPerSecond: float64(totalBytes) / duration.Seconds(),
MemoryUsage: MemoryStats{
AllocMB: float64(memAfter.Alloc-memBefore.Alloc) / 1024 / 1024,
SysMB: float64(memAfter.Sys-memBefore.Sys) / 1024 / 1024,
NumGC: memAfter.NumGC - memBefore.NumGC,
PauseTotalNs: memAfter.PauseTotalNs - memBefore.PauseTotalNs,
},
CPUUsage: CPUStats{
Goroutines: runtime.NumGoroutine(),
},
}
return result, nil return result, nil
} }
@@ -246,10 +247,10 @@ func ConcurrencyBenchmark(sourceDir string, format string, concurrencyLevels []i
for _, concurrency := range concurrencyLevels { for _, concurrency := range concurrencyLevels {
result, err := FileProcessingBenchmark(sourceDir, format, concurrency) result, err := FileProcessingBenchmark(sourceDir, format, concurrency)
if err != nil { if err != nil {
return nil, gibidiutils.WrapErrorf( return nil, shared.WrapErrorf(
err, err,
gibidiutils.ErrorTypeProcessing, shared.ErrorTypeProcessing,
gibidiutils.CodeProcessingCollection, shared.CodeProcessingCollection,
"concurrency benchmark failed for level %d", "concurrency benchmark failed for level %d",
concurrency, concurrency,
) )
@@ -270,10 +271,10 @@ func FormatBenchmark(sourceDir string, formats []string) (*Suite, error) {
for _, format := range formats { for _, format := range formats {
result, err := FileProcessingBenchmark(sourceDir, format, runtime.NumCPU()) result, err := FileProcessingBenchmark(sourceDir, format, runtime.NumCPU())
if err != nil { if err != nil {
return nil, gibidiutils.WrapErrorf( return nil, shared.WrapErrorf(
err, err,
gibidiutils.ErrorTypeProcessing, shared.ErrorTypeProcessing,
gibidiutils.CodeProcessingCollection, shared.CodeProcessingCollection,
"format benchmark failed for format %s", "format benchmark failed for format %s",
format, format,
) )
@@ -288,18 +289,18 @@ func FormatBenchmark(sourceDir string, formats []string) (*Suite, error) {
func createBenchmarkFiles(numFiles int) (string, func(), error) { func createBenchmarkFiles(numFiles int) (string, func(), error) {
tempDir, err := os.MkdirTemp("", "gibidify_benchmark_*") tempDir, err := os.MkdirTemp("", "gibidify_benchmark_*")
if err != nil { if err != nil {
return "", nil, gibidiutils.WrapError( return "", nil, shared.WrapError(
err, err,
gibidiutils.ErrorTypeFileSystem, shared.ErrorTypeFileSystem,
gibidiutils.CodeFSAccess, shared.CodeFSAccess,
"failed to create temp directory", "failed to create temp directory",
) )
} }
cleanup := func() { cleanup := func() {
if err := os.RemoveAll(tempDir); err != nil { if err := os.RemoveAll(tempDir); err != nil {
// Log error but don't fail the benchmark //nolint:errcheck // Warning message in cleanup, failure doesn't affect benchmark
fmt.Printf("Warning: failed to remove benchmark temp directory: %v\n", err) _, _ = fmt.Printf("Warning: failed to remove benchmark temp directory: %v\n", err)
} }
} }
@@ -313,12 +314,13 @@ func createBenchmarkFiles(numFiles int) (string, func(), error) {
{".py", "print('Hello, World!')"}, {".py", "print('Hello, World!')"},
{ {
".java", ".java",
"public class Hello {\n\tpublic static void main(String[] args) {" + "public class Hello {\n\tpublic static void main(String[] args) {\n\t" +
"\n\t\tSystem.out.println(\"Hello, World!\");\n\t}\n}", "\tSystem.out.println(\"Hello, World!\");\n\t}\n}",
}, },
{ {
".cpp", ".cpp",
"#include <iostream>\n\nint main() {\n\tstd::cout << \"Hello, World!\" << std::endl;\n\treturn 0;\n}", "#include <iostream>\n\n" +
"int main() {\n\tstd::cout << \"Hello, World!\" << std::endl;\n\treturn 0;\n}",
}, },
{".rs", "fn main() {\n\tprintln!(\"Hello, World!\");\n}"}, {".rs", "fn main() {\n\tprintln!(\"Hello, World!\");\n}"},
{".rb", "puts 'Hello, World!'"}, {".rb", "puts 'Hello, World!'"},
@@ -336,10 +338,11 @@ func createBenchmarkFiles(numFiles int) (string, func(), error) {
subdir := filepath.Join(tempDir, fmt.Sprintf("subdir_%d", i/10)) subdir := filepath.Join(tempDir, fmt.Sprintf("subdir_%d", i/10))
if err := os.MkdirAll(subdir, 0o750); err != nil { if err := os.MkdirAll(subdir, 0o750); err != nil {
cleanup() cleanup()
return "", nil, gibidiutils.WrapError(
return "", nil, shared.WrapError(
err, err,
gibidiutils.ErrorTypeFileSystem, shared.ErrorTypeFileSystem,
gibidiutils.CodeFSAccess, shared.CodeFSAccess,
"failed to create subdirectory", "failed to create subdirectory",
) )
} }
@@ -356,11 +359,9 @@ func createBenchmarkFiles(numFiles int) (string, func(), error) {
if err := os.WriteFile(filename, []byte(content), 0o600); err != nil { if err := os.WriteFile(filename, []byte(content), 0o600); err != nil {
cleanup() cleanup()
return "", nil, gibidiutils.WrapError(
err, return "", nil, shared.WrapError(
gibidiutils.ErrorTypeIO, err, shared.ErrorTypeIO, shared.CodeIOFileWrite, "failed to write benchmark file",
gibidiutils.CodeIOFileWrite,
"failed to write benchmark file",
) )
} }
} }
@@ -369,41 +370,40 @@ func createBenchmarkFiles(numFiles int) (string, func(), error) {
} }
// runProcessingPipeline runs the processing pipeline similar to main.go. // runProcessingPipeline runs the processing pipeline similar to main.go.
// processingConfig holds configuration for processing pipeline. func runProcessingPipeline(
type processingConfig struct { ctx context.Context,
files []string files []string,
outputFile *os.File outputFile *os.File,
format string format string,
concurrency int concurrency int,
sourceDir string sourceDir string,
} ) error {
// Guard against invalid concurrency to prevent deadlocks
if concurrency < 1 {
concurrency = 1
}
func runProcessingPipeline(ctx context.Context, config processingConfig) error { fileCh := make(chan string, concurrency)
fileCh := make(chan string, config.concurrency) writeCh := make(chan fileproc.WriteRequest, concurrency)
writeCh := make(chan fileproc.WriteRequest, config.concurrency)
writerDone := make(chan struct{}) writerDone := make(chan struct{})
// Start writer // Start writer
go fileproc.StartWriter(config.outputFile, writeCh, writerDone, fileproc.WriterConfig{ go fileproc.StartWriter(outputFile, writeCh, writerDone, format, "", "")
Format: config.format,
Prefix: "",
Suffix: "",
})
// Get absolute path once // Get absolute path once
absRoot, err := gibidiutils.GetAbsolutePath(config.sourceDir) absRoot, err := shared.AbsolutePath(sourceDir)
if err != nil { if err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, err,
gibidiutils.ErrorTypeFileSystem, shared.ErrorTypeFileSystem,
gibidiutils.CodeFSPathResolution, shared.CodeFSPathResolution,
"failed to get absolute path for source directory", "failed to get absolute path for source directory",
) )
} }
// Start workers with proper synchronization // Start workers with proper synchronization
var workersDone sync.WaitGroup var workersDone sync.WaitGroup
for i := 0; i < config.concurrency; i++ { for i := 0; i < concurrency; i++ {
workersDone.Add(1) workersDone.Add(1)
go func() { go func() {
defer workersDone.Done() defer workersDone.Done()
@@ -414,14 +414,15 @@ func runProcessingPipeline(ctx context.Context, config processingConfig) error {
} }
// Send files to workers // Send files to workers
for _, file := range config.files { for _, file := range files {
select { select {
case <-ctx.Done(): case <-ctx.Done():
close(fileCh) close(fileCh)
workersDone.Wait() // Wait for workers to finish workersDone.Wait() // Wait for workers to finish
close(writeCh) close(writeCh)
<-writerDone <-writerDone
return ctx.Err()
return fmt.Errorf("context canceled: %w", ctx.Err())
case fileCh <- file: case fileCh <- file:
} }
} }
@@ -439,22 +440,38 @@ func runProcessingPipeline(ctx context.Context, config processingConfig) error {
// PrintResult prints a formatted benchmark result. // PrintResult prints a formatted benchmark result.
func PrintResult(result *Result) { func PrintResult(result *Result) {
fmt.Printf("=== %s ===\n", result.Name) printBenchmarkLine := func(format string, args ...any) {
fmt.Printf("Duration: %v\n", result.Duration) if _, err := fmt.Printf(format, args...); err != nil {
fmt.Printf("Files Processed: %d\n", result.FilesProcessed) // Stdout write errors are rare (broken pipe, etc.) - log but continue
fmt.Printf("Bytes Processed: %d (%.2f MB)\n", result.BytesProcessed, float64(result.BytesProcessed)/1024/1024) shared.LogError("failed to write benchmark output", err)
fmt.Printf("Files/sec: %.2f\n", result.FilesPerSecond) }
fmt.Printf("Bytes/sec: %.2f MB/sec\n", result.BytesPerSecond/1024/1024) }
fmt.Printf("Memory Usage: +%.2f MB (Sys: +%.2f MB)\n", result.MemoryUsage.AllocMB, result.MemoryUsage.SysMB)
pauseDuration := time.Duration(gibidiutils.SafeUint64ToInt64WithDefault(result.MemoryUsage.PauseTotalNs, 0)) printBenchmarkLine(shared.BenchmarkFmtSectionHeader, result.Name)
fmt.Printf("GC Runs: %d (Pause: %v)\n", result.MemoryUsage.NumGC, pauseDuration) printBenchmarkLine("Duration: %v\n", result.Duration)
fmt.Printf("Goroutines: %d\n", result.CPUUsage.Goroutines) printBenchmarkLine("Files Processed: %d\n", result.FilesProcessed)
fmt.Println() printBenchmarkLine("Bytes Processed: %d (%.2f MB)\n", result.BytesProcessed,
float64(result.BytesProcessed)/float64(shared.BytesPerMB))
printBenchmarkLine("Files/sec: %.2f\n", result.FilesPerSecond)
printBenchmarkLine("Bytes/sec: %.2f MB/sec\n", result.BytesPerSecond/float64(shared.BytesPerMB))
printBenchmarkLine(
"Memory Usage: +%.2f MB (Sys: +%.2f MB)\n",
result.MemoryUsage.AllocMB,
result.MemoryUsage.SysMB,
)
//nolint:errcheck // Overflow unlikely for pause duration, result output only
pauseDuration, _ := shared.SafeUint64ToInt64(result.MemoryUsage.PauseTotalNs)
printBenchmarkLine("GC Runs: %d (Pause: %v)\n", result.MemoryUsage.NumGC, time.Duration(pauseDuration))
printBenchmarkLine("Goroutines: %d\n", result.CPUUsage.Goroutines)
printBenchmarkLine("\n")
} }
// PrintSuite prints all results in a benchmark suite. // PrintSuite prints all results in a benchmark suite.
func PrintSuite(suite *Suite) { func PrintSuite(suite *Suite) {
fmt.Printf("=== %s ===\n", suite.Name) if _, err := fmt.Printf(shared.BenchmarkFmtSectionHeader, suite.Name); err != nil {
shared.LogError("failed to write benchmark suite header", err)
}
// Iterate by index to avoid taking address of range variable
for i := range suite.Results { for i := range suite.Results {
PrintResult(&suite.Results[i]) PrintResult(&suite.Results[i])
} }
@@ -462,47 +479,54 @@ func PrintSuite(suite *Suite) {
// RunAllBenchmarks runs a comprehensive benchmark suite. // RunAllBenchmarks runs a comprehensive benchmark suite.
func RunAllBenchmarks(sourceDir string) error { func RunAllBenchmarks(sourceDir string) error {
fmt.Println("Running gibidify benchmark suite...") printBenchmark := func(msg string) {
if _, err := fmt.Println(msg); err != nil {
shared.LogError("failed to write benchmark message", err)
}
}
printBenchmark("Running gibidify benchmark suite...")
// Load configuration // Load configuration
config.LoadConfig() config.LoadConfig()
// File collection benchmark // File collection benchmark
fmt.Println("Running file collection benchmark...") printBenchmark(shared.BenchmarkMsgRunningCollection)
result, err := FileCollectionBenchmark(sourceDir, 1000) result, err := FileCollectionBenchmark(sourceDir, shared.BenchmarkDefaultFileCount)
if err != nil { if err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, err,
gibidiutils.ErrorTypeProcessing, shared.ErrorTypeProcessing,
gibidiutils.CodeProcessingCollection, shared.CodeProcessingCollection,
"file collection benchmark failed", shared.BenchmarkMsgFileCollectionFailed,
) )
} }
PrintResult(result) PrintResult(result)
// Format benchmarks // Format benchmarks
fmt.Println("Running format benchmarks...") printBenchmark("Running format benchmarks...")
formatSuite, err := FormatBenchmark(sourceDir, []string{"json", "yaml", "markdown"}) formats := []string{shared.FormatJSON, shared.FormatYAML, shared.FormatMarkdown}
formatSuite, err := FormatBenchmark(sourceDir, formats)
if err != nil { if err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, err,
gibidiutils.ErrorTypeProcessing, shared.ErrorTypeProcessing,
gibidiutils.CodeProcessingCollection, shared.CodeProcessingCollection,
"format benchmark failed", shared.BenchmarkMsgFormatFailed,
) )
} }
PrintSuite(formatSuite) PrintSuite(formatSuite)
// Concurrency benchmarks // Concurrency benchmarks
fmt.Println("Running concurrency benchmarks...") printBenchmark("Running concurrency benchmarks...")
concurrencyLevels := []int{1, 2, 4, 8, runtime.NumCPU()} concurrencyLevels := []int{1, 2, 4, 8, runtime.NumCPU()}
concurrencySuite, err := ConcurrencyBenchmark(sourceDir, "json", concurrencyLevels) concurrencySuite, err := ConcurrencyBenchmark(sourceDir, shared.FormatJSON, concurrencyLevels)
if err != nil { if err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, err,
gibidiutils.ErrorTypeProcessing, shared.ErrorTypeProcessing,
gibidiutils.CodeProcessingCollection, shared.CodeProcessingCollection,
"concurrency benchmark failed", shared.BenchmarkMsgConcurrencyFailed,
) )
} }
PrintSuite(concurrencySuite) PrintSuite(concurrencySuite)

View File

@@ -1,10 +1,54 @@
package benchmark package benchmark
import ( import (
"bytes"
"io"
"os"
"path/filepath"
"runtime" "runtime"
"strings"
"testing" "testing"
"time"
"github.com/ivuorinen/gibidify/shared"
) )
// capturedOutput captures stdout output from a function call.
func capturedOutput(t *testing.T, fn func()) string {
t.Helper()
original := os.Stdout
r, w, err := os.Pipe()
if err != nil {
t.Fatalf(shared.TestMsgFailedToCreatePipe, err)
}
defer r.Close()
defer func() { os.Stdout = original }()
os.Stdout = w
fn()
if err := w.Close(); err != nil {
t.Logf(shared.TestMsgFailedToClose, err)
}
var buf bytes.Buffer
if _, err := io.Copy(&buf, r); err != nil {
t.Fatalf(shared.TestMsgFailedToReadOutput, err)
}
return buf.String()
}
// verifyOutputContains checks if output contains all expected strings.
func verifyOutputContains(t *testing.T, testName, output string, expected []string) {
t.Helper()
for _, check := range expected {
if !strings.Contains(output, check) {
t.Errorf("Test %s: output missing expected content: %q\nFull output:\n%s", testName, check, output)
}
}
}
// TestFileCollectionBenchmark tests the file collection benchmark. // TestFileCollectionBenchmark tests the file collection benchmark.
func TestFileCollectionBenchmark(t *testing.T) { func TestFileCollectionBenchmark(t *testing.T) {
result, err := FileCollectionBenchmark("", 10) result, err := FileCollectionBenchmark("", 10)
@@ -22,7 +66,7 @@ func TestFileCollectionBenchmark(t *testing.T) {
t.Logf("Bytes processed: %d", result.BytesProcessed) t.Logf("Bytes processed: %d", result.BytesProcessed)
if result.FilesProcessed <= 0 { if result.FilesProcessed <= 0 {
t.Errorf("Expected files processed > 0, got %d", result.FilesProcessed) t.Errorf(shared.TestFmtExpectedFilesProcessed, result.FilesProcessed)
} }
if result.Duration <= 0 { if result.Duration <= 0 {
@@ -38,7 +82,7 @@ func TestFileProcessingBenchmark(t *testing.T) {
} }
if result.FilesProcessed <= 0 { if result.FilesProcessed <= 0 {
t.Errorf("Expected files processed > 0, got %d", result.FilesProcessed) t.Errorf(shared.TestFmtExpectedFilesProcessed, result.FilesProcessed)
} }
if result.Duration <= 0 { if result.Duration <= 0 {
@@ -59,12 +103,12 @@ func TestConcurrencyBenchmark(t *testing.T) {
} }
if len(suite.Results) != len(concurrencyLevels) { if len(suite.Results) != len(concurrencyLevels) {
t.Errorf("Expected %d results, got %d", len(concurrencyLevels), len(suite.Results)) t.Errorf(shared.TestFmtExpectedResults, len(concurrencyLevels), len(suite.Results))
} }
for i, result := range suite.Results { for i, result := range suite.Results {
if result.FilesProcessed <= 0 { if result.FilesProcessed <= 0 {
t.Errorf("Result %d: Expected files processed > 0, got %d", i, result.FilesProcessed) t.Errorf("Result %d: "+shared.TestFmtExpectedFilesProcessed, i, result.FilesProcessed)
} }
} }
} }
@@ -82,12 +126,12 @@ func TestFormatBenchmark(t *testing.T) {
} }
if len(suite.Results) != len(formats) { if len(suite.Results) != len(formats) {
t.Errorf("Expected %d results, got %d", len(formats), len(suite.Results)) t.Errorf(shared.TestFmtExpectedResults, len(formats), len(suite.Results))
} }
for i, result := range suite.Results { for i, result := range suite.Results {
if result.FilesProcessed <= 0 { if result.FilesProcessed <= 0 {
t.Errorf("Result %d: Expected files processed > 0, got %d", i, result.FilesProcessed) t.Errorf("Result %d: "+shared.TestFmtExpectedFilesProcessed, i, result.FilesProcessed)
} }
} }
} }
@@ -116,7 +160,7 @@ func BenchmarkFileCollection(b *testing.B) {
b.Fatalf("FileCollectionBenchmark failed: %v", err) b.Fatalf("FileCollectionBenchmark failed: %v", err)
} }
if result.FilesProcessed <= 0 { if result.FilesProcessed <= 0 {
b.Errorf("Expected files processed > 0, got %d", result.FilesProcessed) b.Errorf(shared.TestFmtExpectedFilesProcessed, result.FilesProcessed)
} }
} }
} }
@@ -129,7 +173,7 @@ func BenchmarkFileProcessing(b *testing.B) {
b.Fatalf("FileProcessingBenchmark failed: %v", err) b.Fatalf("FileProcessingBenchmark failed: %v", err)
} }
if result.FilesProcessed <= 0 { if result.FilesProcessed <= 0 {
b.Errorf("Expected files processed > 0, got %d", result.FilesProcessed) b.Errorf(shared.TestFmtExpectedFilesProcessed, result.FilesProcessed)
} }
} }
} }
@@ -144,7 +188,7 @@ func BenchmarkConcurrency(b *testing.B) {
b.Fatalf("ConcurrencyBenchmark failed: %v", err) b.Fatalf("ConcurrencyBenchmark failed: %v", err)
} }
if len(suite.Results) != len(concurrencyLevels) { if len(suite.Results) != len(concurrencyLevels) {
b.Errorf("Expected %d results, got %d", len(concurrencyLevels), len(suite.Results)) b.Errorf(shared.TestFmtExpectedResults, len(concurrencyLevels), len(suite.Results))
} }
} }
} }
@@ -159,7 +203,315 @@ func BenchmarkFormats(b *testing.B) {
b.Fatalf("FormatBenchmark failed: %v", err) b.Fatalf("FormatBenchmark failed: %v", err)
} }
if len(suite.Results) != len(formats) { if len(suite.Results) != len(formats) {
b.Errorf("Expected %d results, got %d", len(formats), len(suite.Results)) b.Errorf(shared.TestFmtExpectedResults, len(formats), len(suite.Results))
} }
} }
} }
// TestPrintResult tests the PrintResult function.
func TestPrintResult(t *testing.T) {
// Create a test result
result := &Result{
Name: "Test Benchmark",
Duration: 1 * time.Second,
FilesProcessed: 100,
BytesProcessed: 2048000, // ~2MB for easy calculation
}
// Capture stdout
original := os.Stdout
r, w, err := os.Pipe()
if err != nil {
t.Fatalf(shared.TestMsgFailedToCreatePipe, err)
}
defer r.Close()
defer func() { os.Stdout = original }()
os.Stdout = w
// Call PrintResult
PrintResult(result)
// Close writer and read captured output
if err := w.Close(); err != nil {
t.Logf(shared.TestMsgFailedToClose, err)
}
var buf bytes.Buffer
if _, err := io.Copy(&buf, r); err != nil {
t.Fatalf(shared.TestMsgFailedToReadOutput, err)
}
output := buf.String()
// Verify expected content
expectedContents := []string{
"=== Test Benchmark ===",
"Duration: 1s",
"Files Processed: 100",
"Bytes Processed: 2048000",
"1.95 MB", // 2048000 / 1024 / 1024 ≈ 1.95
}
for _, expected := range expectedContents {
if !strings.Contains(output, expected) {
t.Errorf("PrintResult output missing expected content: %q\nFull output:\n%s", expected, output)
}
}
}
// TestPrintSuite tests the PrintSuite function.
func TestPrintSuite(t *testing.T) {
// Create a test suite with multiple results
suite := &Suite{
Name: "Test Suite",
Results: []Result{
{
Name: "Benchmark 1",
Duration: 500 * time.Millisecond,
FilesProcessed: 50,
BytesProcessed: 1024000, // 1MB
},
{
Name: "Benchmark 2",
Duration: 750 * time.Millisecond,
FilesProcessed: 75,
BytesProcessed: 1536000, // 1.5MB
},
},
}
// Capture stdout
original := os.Stdout
r, w, err := os.Pipe()
if err != nil {
t.Fatalf(shared.TestMsgFailedToCreatePipe, err)
}
defer r.Close()
defer func() { os.Stdout = original }()
os.Stdout = w
// Call PrintSuite
PrintSuite(suite)
// Close writer and read captured output
if err := w.Close(); err != nil {
t.Logf(shared.TestMsgFailedToClose, err)
}
var buf bytes.Buffer
if _, err := io.Copy(&buf, r); err != nil {
t.Fatalf(shared.TestMsgFailedToReadOutput, err)
}
output := buf.String()
// Verify expected content
expectedContents := []string{
"=== Test Suite ===",
"=== Benchmark 1 ===",
"Duration: 500ms",
"Files Processed: 50",
"=== Benchmark 2 ===",
"Duration: 750ms",
"Files Processed: 75",
}
for _, expected := range expectedContents {
if !strings.Contains(output, expected) {
t.Errorf("PrintSuite output missing expected content: %q\nFull output:\n%s", expected, output)
}
}
// Verify both results are printed
benchmark1Count := strings.Count(output, "=== Benchmark 1 ===")
benchmark2Count := strings.Count(output, "=== Benchmark 2 ===")
if benchmark1Count != 1 {
t.Errorf("Expected exactly 1 occurrence of 'Benchmark 1', got %d", benchmark1Count)
}
if benchmark2Count != 1 {
t.Errorf("Expected exactly 1 occurrence of 'Benchmark 2', got %d", benchmark2Count)
}
}
// TestPrintResultEdgeCases tests edge cases for PrintResult.
func TestPrintResultEdgeCases(t *testing.T) {
tests := []struct {
name string
result *Result
checks []string
}{
{
name: "zero values",
result: &Result{
Name: "Zero Benchmark",
Duration: 0,
FilesProcessed: 0,
BytesProcessed: 0,
},
checks: []string{
"=== Zero Benchmark ===",
"Duration: 0s",
"Files Processed: 0",
"Bytes Processed: 0",
"0.00 MB",
},
},
{
name: "large values",
result: &Result{
Name: "Large Benchmark",
Duration: 1 * time.Hour,
FilesProcessed: 1000000,
BytesProcessed: 1073741824, // 1GB
},
checks: []string{
"=== Large Benchmark ===",
"Duration: 1h0m0s",
"Files Processed: 1000000",
"Bytes Processed: 1073741824",
"1024.00 MB",
},
},
{
name: "empty name",
result: &Result{
Name: "",
Duration: 100 * time.Millisecond,
FilesProcessed: 10,
BytesProcessed: 1024,
},
checks: []string{
"=== ===", // Empty name between === markers
"Duration: 100ms",
"Files Processed: 10",
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := tt.result
output := capturedOutput(t, func() { PrintResult(result) })
verifyOutputContains(t, tt.name, output, tt.checks)
})
}
}
// TestPrintSuiteEdgeCases tests edge cases for PrintSuite.
func TestPrintSuiteEdgeCases(t *testing.T) {
tests := []struct {
name string
suite *Suite
checks []string
}{
{
name: "empty suite",
suite: &Suite{
Name: "Empty Suite",
Results: []Result{},
},
checks: []string{
"=== Empty Suite ===",
},
},
{
name: "suite with empty name",
suite: &Suite{
Name: "",
Results: []Result{
{
Name: "Single Benchmark",
Duration: 200 * time.Millisecond,
FilesProcessed: 20,
BytesProcessed: 2048,
},
},
},
checks: []string{
"=== ===", // Empty name
"=== Single Benchmark ===",
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
suite := tt.suite
output := capturedOutput(t, func() { PrintSuite(suite) })
verifyOutputContains(t, tt.name, output, tt.checks)
})
}
}
// TestRunAllBenchmarks tests the RunAllBenchmarks function.
func TestRunAllBenchmarks(t *testing.T) {
// Create a temporary directory with some test files
srcDir := t.TempDir()
// Create a few test files
testFiles := []struct {
name string
content string
}{
{shared.TestFileMainGo, "package main\nfunc main() {}"},
{shared.TestFile2Name, "Hello World"},
{shared.TestFile3Name, "# Test Markdown"},
}
for _, file := range testFiles {
filePath := filepath.Join(srcDir, file.name)
err := os.WriteFile(filePath, []byte(file.content), 0o644)
if err != nil {
t.Fatalf("Failed to create test file %s: %v", file.name, err)
}
}
// Capture stdout to verify output
original := os.Stdout
r, w, pipeErr := os.Pipe()
if pipeErr != nil {
t.Fatalf(shared.TestMsgFailedToCreatePipe, pipeErr)
}
defer func() {
if err := r.Close(); err != nil {
t.Logf("Failed to close pipe reader: %v", err)
}
}()
defer func() { os.Stdout = original }()
os.Stdout = w
// Call RunAllBenchmarks
err := RunAllBenchmarks(srcDir)
// Close writer and read captured output
if closeErr := w.Close(); closeErr != nil {
t.Logf(shared.TestMsgFailedToClose, closeErr)
}
var buf bytes.Buffer
if _, copyErr := io.Copy(&buf, r); copyErr != nil {
t.Fatalf(shared.TestMsgFailedToReadOutput, copyErr)
}
output := buf.String()
// Check for error
if err != nil {
t.Errorf("RunAllBenchmarks failed: %v", err)
}
// Verify expected output content
expectedContents := []string{
"Running gibidify benchmark suite...",
"Running file collection benchmark...",
"Running format benchmarks...",
"Running concurrency benchmarks...",
}
for _, expected := range expectedContents {
if !strings.Contains(output, expected) {
t.Errorf("RunAllBenchmarks output missing expected content: %q\nFull output:\n%s", expected, output)
}
}
// The function should not panic and should complete successfully
t.Log("RunAllBenchmarks completed successfully with output captured")
}

View File

@@ -1,4 +1,4 @@
// Package cli provides command-line interface utilities for gibidify. // Package cli provides command-line interface functionality for gibidify.
package cli package cli
import ( import (
@@ -7,10 +7,11 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// ErrorFormatter handles CLI-friendly error formatting with suggestions. // ErrorFormatter handles CLI-friendly error formatting with suggestions.
// This is not an error type itself; it formats existing errors for display.
type ErrorFormatter struct { type ErrorFormatter struct {
ui *UIManager ui *UIManager
} }
@@ -20,11 +21,6 @@ func NewErrorFormatter(ui *UIManager) *ErrorFormatter {
return &ErrorFormatter{ui: ui} return &ErrorFormatter{ui: ui}
} }
// Suggestion messages for error formatting.
const (
suggestionCheckPermissions = " %s Check file/directory permissions\n"
)
// FormatError formats an error with context and suggestions. // FormatError formats an error with context and suggestions.
func (ef *ErrorFormatter) FormatError(err error) { func (ef *ErrorFormatter) FormatError(err error) {
if err == nil { if err == nil {
@@ -32,9 +28,10 @@ func (ef *ErrorFormatter) FormatError(err error) {
} }
// Handle structured errors // Handle structured errors
var structErr *gibidiutils.StructuredError structErr := &shared.StructuredError{}
if errors.As(err, &structErr) { if errors.As(err, &structErr) {
ef.formatStructuredError(structErr) ef.formatStructuredError(structErr)
return return
} }
@@ -43,12 +40,12 @@ func (ef *ErrorFormatter) FormatError(err error) {
} }
// formatStructuredError formats a structured error with context and suggestions. // formatStructuredError formats a structured error with context and suggestions.
func (ef *ErrorFormatter) formatStructuredError(err *gibidiutils.StructuredError) { func (ef *ErrorFormatter) formatStructuredError(err *shared.StructuredError) {
// Print main error // Print main error
ef.ui.PrintError("Error: %s", err.Message) ef.ui.PrintError(shared.CLIMsgErrorFormat, err.Message)
// Print error type and code // Print error type and code
if err.Type != gibidiutils.ErrorTypeUnknown || err.Code != "" { if err.Type != shared.ErrorTypeUnknown || err.Code != "" {
ef.ui.PrintInfo("Type: %s, Code: %s", err.Type.String(), err.Code) ef.ui.PrintInfo("Type: %s, Code: %s", err.Type.String(), err.Code)
} }
@@ -71,20 +68,20 @@ func (ef *ErrorFormatter) formatStructuredError(err *gibidiutils.StructuredError
// formatGenericError formats a generic error. // formatGenericError formats a generic error.
func (ef *ErrorFormatter) formatGenericError(err error) { func (ef *ErrorFormatter) formatGenericError(err error) {
ef.ui.PrintError("Error: %s", err.Error()) ef.ui.PrintError(shared.CLIMsgErrorFormat, err.Error())
ef.provideGenericSuggestions(err) ef.provideGenericSuggestions(err)
} }
// provideSuggestions provides helpful suggestions based on the error. // provideSuggestions provides helpful suggestions based on the error.
func (ef *ErrorFormatter) provideSuggestions(err *gibidiutils.StructuredError) { func (ef *ErrorFormatter) provideSuggestions(err *shared.StructuredError) {
switch err.Type { switch err.Type {
case gibidiutils.ErrorTypeFileSystem: case shared.ErrorTypeFileSystem:
ef.provideFileSystemSuggestions(err) ef.provideFileSystemSuggestions(err)
case gibidiutils.ErrorTypeValidation: case shared.ErrorTypeValidation:
ef.provideValidationSuggestions(err) ef.provideValidationSuggestions(err)
case gibidiutils.ErrorTypeProcessing: case shared.ErrorTypeProcessing:
ef.provideProcessingSuggestions(err) ef.provideProcessingSuggestions(err)
case gibidiutils.ErrorTypeIO: case shared.ErrorTypeIO:
ef.provideIOSuggestions(err) ef.provideIOSuggestions(err)
default: default:
ef.provideDefaultSuggestions() ef.provideDefaultSuggestions()
@@ -92,17 +89,17 @@ func (ef *ErrorFormatter) provideSuggestions(err *gibidiutils.StructuredError) {
} }
// provideFileSystemSuggestions provides suggestions for file system errors. // provideFileSystemSuggestions provides suggestions for file system errors.
func (ef *ErrorFormatter) provideFileSystemSuggestions(err *gibidiutils.StructuredError) { func (ef *ErrorFormatter) provideFileSystemSuggestions(err *shared.StructuredError) {
filePath := err.FilePath filePath := err.FilePath
ef.ui.PrintWarning("Suggestions:") ef.ui.PrintWarning(shared.CLIMsgSuggestions)
switch err.Code { switch err.Code {
case gibidiutils.CodeFSAccess: case shared.CodeFSAccess:
ef.suggestFileAccess(filePath) ef.suggestFileAccess(filePath)
case gibidiutils.CodeFSPathResolution: case shared.CodeFSPathResolution:
ef.suggestPathResolution(filePath) ef.suggestPathResolution(filePath)
case gibidiutils.CodeFSNotFound: case shared.CodeFSNotFound:
ef.suggestFileNotFound(filePath) ef.suggestFileNotFound(filePath)
default: default:
ef.suggestFileSystemGeneral(filePath) ef.suggestFileSystemGeneral(filePath)
@@ -110,130 +107,135 @@ func (ef *ErrorFormatter) provideFileSystemSuggestions(err *gibidiutils.Structur
} }
// provideValidationSuggestions provides suggestions for validation errors. // provideValidationSuggestions provides suggestions for validation errors.
func (ef *ErrorFormatter) provideValidationSuggestions(err *gibidiutils.StructuredError) { func (ef *ErrorFormatter) provideValidationSuggestions(err *shared.StructuredError) {
ef.ui.PrintWarning("Suggestions:") ef.ui.PrintWarning(shared.CLIMsgSuggestions)
switch err.Code { switch err.Code {
case gibidiutils.CodeValidationFormat: case shared.CodeValidationFormat:
ef.ui.printf(" %s Use a supported format: markdown, json, yaml\n", gibidiutils.IconBullet) ef.ui.printf(" Use a supported format: markdown, json, yaml\n")
ef.ui.printf(" %s Example: -format markdown\n", gibidiutils.IconBullet) ef.ui.printf(" Example: -format markdown\n")
case gibidiutils.CodeValidationSize: case shared.CodeValidationSize:
ef.ui.printf(" %s Increase file size limit in config.yaml\n", gibidiutils.IconBullet) ef.ui.printf(" Increase file size limit in config.yaml\n")
ef.ui.printf(" %s Use smaller files or exclude large files\n", gibidiutils.IconBullet) ef.ui.printf(" Use smaller files or exclude large files\n")
default: default:
ef.ui.printf(" %s Check your command line arguments\n", gibidiutils.IconBullet) ef.ui.printf(shared.CLIMsgCheckCommandLineArgs)
ef.ui.printf(" %s Run with --help for usage information\n", gibidiutils.IconBullet) ef.ui.printf(shared.CLIMsgRunWithHelp)
} }
} }
// provideProcessingSuggestions provides suggestions for processing errors. // provideProcessingSuggestions provides suggestions for processing errors.
func (ef *ErrorFormatter) provideProcessingSuggestions(err *gibidiutils.StructuredError) { func (ef *ErrorFormatter) provideProcessingSuggestions(err *shared.StructuredError) {
ef.ui.PrintWarning("Suggestions:") ef.ui.PrintWarning(shared.CLIMsgSuggestions)
switch err.Code { switch err.Code {
case gibidiutils.CodeProcessingCollection: case shared.CodeProcessingCollection:
ef.ui.printf(" %s Check if the source directory exists and is readable\n", gibidiutils.IconBullet) ef.ui.printf(" Check if the source directory exists and is readable\n")
ef.ui.printf(" %s Verify directory permissions\n", gibidiutils.IconBullet) ef.ui.printf(" Verify directory permissions\n")
case gibidiutils.CodeProcessingFileRead: case shared.CodeProcessingFileRead:
ef.ui.printf(" %s Check file permissions\n", gibidiutils.IconBullet) ef.ui.printf(" Check file permissions\n")
ef.ui.printf(" %s Verify the file is not corrupted\n", gibidiutils.IconBullet) ef.ui.printf(" Verify the file is not corrupted\n")
default: default:
ef.ui.printf(" %s Try reducing concurrency: -concurrency 1\n", gibidiutils.IconBullet) ef.ui.printf(" Try reducing concurrency: -concurrency 1\n")
ef.ui.printf(" %s Check available system resources\n", gibidiutils.IconBullet) ef.ui.printf(" Check available system resources\n")
} }
} }
// provideIOSuggestions provides suggestions for I/O errors. // provideIOSuggestions provides suggestions for I/O errors.
func (ef *ErrorFormatter) provideIOSuggestions(err *gibidiutils.StructuredError) { func (ef *ErrorFormatter) provideIOSuggestions(err *shared.StructuredError) {
ef.ui.PrintWarning("Suggestions:") ef.ui.PrintWarning(shared.CLIMsgSuggestions)
switch err.Code { switch err.Code {
case gibidiutils.CodeIOFileCreate: case shared.CodeIOFileCreate:
ef.ui.printf(" %s Check if the destination directory exists\n", gibidiutils.IconBullet) ef.ui.printf(" Check if the destination directory exists\n")
ef.ui.printf(" %s Verify write permissions for the output file\n", gibidiutils.IconBullet) ef.ui.printf(" Verify write permissions for the output file\n")
ef.ui.printf(" %s Ensure sufficient disk space\n", gibidiutils.IconBullet) ef.ui.printf(" Ensure sufficient disk space\n")
case gibidiutils.CodeIOWrite: case shared.CodeIOWrite:
ef.ui.printf(" %s Check available disk space\n", gibidiutils.IconBullet) ef.ui.printf(" Check available disk space\n")
ef.ui.printf(" %s Verify write permissions\n", gibidiutils.IconBullet) ef.ui.printf(" Verify write permissions\n")
default: default:
ef.ui.printf(suggestionCheckPermissions, gibidiutils.IconBullet) ef.ui.printf(shared.CLIMsgCheckFilePermissions)
ef.ui.printf(" %s Verify available disk space\n", gibidiutils.IconBullet) ef.ui.printf(" Verify available disk space\n")
} }
} }
// Helper methods for specific suggestions // Helper methods for specific suggestions.
func (ef *ErrorFormatter) suggestFileAccess(filePath string) { func (ef *ErrorFormatter) suggestFileAccess(filePath string) {
ef.ui.printf(" %s Check if the path exists: %s\n", gibidiutils.IconBullet, filePath) ef.ui.printf(" Check if the path exists: %s\n", filePath)
ef.ui.printf(" %s Verify read permissions\n", gibidiutils.IconBullet) ef.ui.printf(" Verify read permissions\n")
if filePath != "" { if filePath != "" {
if stat, err := os.Stat(filePath); err == nil { if stat, err := os.Stat(filePath); err == nil {
ef.ui.printf(" %s Path exists but may not be accessible\n", gibidiutils.IconBullet) ef.ui.printf(" Path exists but may not be accessible\n")
ef.ui.printf(" %s Mode: %s\n", gibidiutils.IconBullet, stat.Mode()) ef.ui.printf(" Mode: %s\n", stat.Mode())
} }
} }
} }
func (ef *ErrorFormatter) suggestPathResolution(filePath string) { func (ef *ErrorFormatter) suggestPathResolution(filePath string) {
ef.ui.printf(" %s Use an absolute path instead of relative\n", gibidiutils.IconBullet) ef.ui.printf(" Use an absolute path instead of relative\n")
if filePath != "" { if filePath != "" {
if abs, err := filepath.Abs(filePath); err == nil { if abs, err := filepath.Abs(filePath); err == nil {
ef.ui.printf(" %s Try: %s\n", gibidiutils.IconBullet, abs) ef.ui.printf(" Try: %s\n", abs)
} }
} }
} }
func (ef *ErrorFormatter) suggestFileNotFound(filePath string) { func (ef *ErrorFormatter) suggestFileNotFound(filePath string) {
ef.ui.printf(" %s Check if the file/directory exists: %s\n", gibidiutils.IconBullet, filePath) ef.ui.printf(" Check if the file/directory exists: %s\n", filePath)
if filePath != "" { if filePath == "" {
dir := filepath.Dir(filePath) return
if entries, err := os.ReadDir(dir); err == nil { }
ef.ui.printf(" %s Similar files in %s:\n", gibidiutils.IconBullet, dir)
count := 0 dir := filepath.Dir(filePath)
for _, entry := range entries { entries, err := os.ReadDir(dir)
if count >= 3 { if err != nil {
break return
} }
if strings.Contains(entry.Name(), filepath.Base(filePath)) {
ef.ui.printf(" %s %s\n", gibidiutils.IconBullet, entry.Name()) ef.ui.printf(" • Similar files in %s:\n", dir)
count++ count := 0
} for _, entry := range entries {
} if count >= 3 {
break
}
if strings.Contains(entry.Name(), filepath.Base(filePath)) {
ef.ui.printf(" - %s\n", entry.Name())
count++
} }
} }
} }
func (ef *ErrorFormatter) suggestFileSystemGeneral(filePath string) { func (ef *ErrorFormatter) suggestFileSystemGeneral(filePath string) {
ef.ui.printf(suggestionCheckPermissions, gibidiutils.IconBullet) ef.ui.printf(shared.CLIMsgCheckFilePermissions)
ef.ui.printf(" %s Verify the path is correct\n", gibidiutils.IconBullet) ef.ui.printf(" Verify the path is correct\n")
if filePath != "" { if filePath != "" {
ef.ui.printf(" %s Path: %s\n", gibidiutils.IconBullet, filePath) ef.ui.printf(" Path: %s\n", filePath)
} }
} }
// provideDefaultSuggestions provides general suggestions. // provideDefaultSuggestions provides general suggestions.
func (ef *ErrorFormatter) provideDefaultSuggestions() { func (ef *ErrorFormatter) provideDefaultSuggestions() {
ef.ui.printf(" %s Check your command line arguments\n", gibidiutils.IconBullet) ef.ui.printf(shared.CLIMsgCheckCommandLineArgs)
ef.ui.printf(" %s Run with --help for usage information\n", gibidiutils.IconBullet) ef.ui.printf(shared.CLIMsgRunWithHelp)
ef.ui.printf(" %s Try with -concurrency 1 to reduce resource usage\n", gibidiutils.IconBullet) ef.ui.printf(" Try with -concurrency 1 to reduce resource usage\n")
} }
// provideGenericSuggestions provides suggestions for generic errors. // provideGenericSuggestions provides suggestions for generic errors.
func (ef *ErrorFormatter) provideGenericSuggestions(err error) { func (ef *ErrorFormatter) provideGenericSuggestions(err error) {
errorMsg := err.Error() errorMsg := err.Error()
ef.ui.PrintWarning("Suggestions:") ef.ui.PrintWarning(shared.CLIMsgSuggestions)
// Pattern matching for common errors // Pattern matching for common errors
switch { switch {
case strings.Contains(errorMsg, "permission denied"): case strings.Contains(errorMsg, "permission denied"):
ef.ui.printf(suggestionCheckPermissions, gibidiutils.IconBullet) ef.ui.printf(shared.CLIMsgCheckFilePermissions)
ef.ui.printf(" %s Try running with appropriate privileges\n", gibidiutils.IconBullet) ef.ui.printf(" Try running with appropriate privileges\n")
case strings.Contains(errorMsg, "no such file or directory"): case strings.Contains(errorMsg, "no such file or directory"):
ef.ui.printf(" %s Verify the file/directory path is correct\n", gibidiutils.IconBullet) ef.ui.printf(" Verify the file/directory path is correct\n")
ef.ui.printf(" %s Check if the file exists\n", gibidiutils.IconBullet) ef.ui.printf(" Check if the file exists\n")
case strings.Contains(errorMsg, "flag") && strings.Contains(errorMsg, "redefined"): case strings.Contains(errorMsg, "flag") && strings.Contains(errorMsg, "redefined"):
ef.ui.printf(" %s This is likely a test environment issue\n", gibidiutils.IconBullet) ef.ui.printf(" This is likely a test environment issue\n")
ef.ui.printf(" %s Try running the command directly instead of in tests\n", gibidiutils.IconBullet) ef.ui.printf(" Try running the command directly instead of in tests\n")
default: default:
ef.provideDefaultSuggestions() ef.provideDefaultSuggestions()
} }
@@ -248,8 +250,8 @@ func (e MissingSourceError) Error() string {
return "source directory is required" return "source directory is required"
} }
// NewMissingSourceError creates a new CLI missing source error with suggestions. // NewCLIMissingSourceError creates a new CLI missing source error with suggestions.
func NewMissingSourceError() error { func NewCLIMissingSourceError() error {
return &MissingSourceError{} return &MissingSourceError{}
} }
@@ -266,11 +268,11 @@ func IsUserError(err error) bool {
} }
// Check for structured errors that are user-facing // Check for structured errors that are user-facing
var structErr *gibidiutils.StructuredError structErr := &shared.StructuredError{}
if errors.As(err, &structErr) { if errors.As(err, &structErr) {
return structErr.Type == gibidiutils.ErrorTypeValidation || return structErr.Type == shared.ErrorTypeValidation ||
structErr.Code == gibidiutils.CodeValidationFormat || structErr.Code == shared.CodeValidationFormat ||
structErr.Code == gibidiutils.CodeValidationSize structErr.Code == shared.CodeValidationSize
} }
// Check error message patterns // Check error message patterns

File diff suppressed because it is too large Load Diff

View File

@@ -1,11 +1,14 @@
// Package cli provides command-line interface functionality for gibidify.
package cli package cli
import ( import (
"flag" "flag"
"fmt"
"os"
"runtime" "runtime"
"github.com/ivuorinen/gibidify/config" "github.com/ivuorinen/gibidify/config"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// Flags holds CLI flags values. // Flags holds CLI flags values.
@@ -18,7 +21,9 @@ type Flags struct {
Format string Format string
NoColors bool NoColors bool
NoProgress bool NoProgress bool
NoUI bool
Verbose bool Verbose bool
LogLevel string
} }
var ( var (
@@ -26,6 +31,15 @@ var (
globalFlags *Flags globalFlags *Flags
) )
// ResetFlags resets the global flag parsing state for testing.
// This function should only be used in tests to ensure proper isolation.
func ResetFlags() {
flagsParsed = false
globalFlags = nil
// Reset default FlagSet to avoid duplicate flag registration across tests
flag.CommandLine = flag.NewFlagSet(os.Args[0], flag.ContinueOnError)
}
// ParseFlags parses and validates CLI flags. // ParseFlags parses and validates CLI flags.
func ParseFlags() (*Flags, error) { func ParseFlags() (*Flags, error) {
if flagsParsed { if flagsParsed {
@@ -34,18 +48,20 @@ func ParseFlags() (*Flags, error) {
flags := &Flags{} flags := &Flags{}
flag.StringVar(&flags.SourceDir, "source", "", "Source directory to scan recursively") flag.StringVar(&flags.SourceDir, shared.CLIArgSource, "", "Source directory to scan recursively")
flag.StringVar(&flags.Destination, "destination", "", "Output file to write aggregated code") flag.StringVar(&flags.Destination, "destination", "", "Output file to write aggregated code")
flag.StringVar(&flags.Prefix, "prefix", "", "Text to add at the beginning of the output file") flag.StringVar(&flags.Prefix, "prefix", "", "Text to add at the beginning of the output file")
flag.StringVar(&flags.Suffix, "suffix", "", "Text to add at the end of the output file") flag.StringVar(&flags.Suffix, "suffix", "", "Text to add at the end of the output file")
flag.StringVar(&flags.Format, "format", "markdown", "Output format (json, markdown, yaml)") flag.StringVar(&flags.Format, shared.CLIArgFormat, shared.FormatJSON, "Output format (json, markdown, yaml)")
flag.IntVar( flag.IntVar(&flags.Concurrency, shared.CLIArgConcurrency, runtime.NumCPU(),
&flags.Concurrency, "concurrency", runtime.NumCPU(), "Number of concurrent workers (default: number of CPU cores)")
"Number of concurrent workers (default: number of CPU cores)",
)
flag.BoolVar(&flags.NoColors, "no-colors", false, "Disable colored output") flag.BoolVar(&flags.NoColors, "no-colors", false, "Disable colored output")
flag.BoolVar(&flags.NoProgress, "no-progress", false, "Disable progress bars") flag.BoolVar(&flags.NoProgress, "no-progress", false, "Disable progress bars")
flag.BoolVar(&flags.NoUI, "no-ui", false, "Disable all UI output (implies no-colors and no-progress)")
flag.BoolVar(&flags.Verbose, "verbose", false, "Enable verbose output") flag.BoolVar(&flags.Verbose, "verbose", false, "Enable verbose output")
flag.StringVar(
&flags.LogLevel, "log-level", string(shared.LogLevelWarn), "Set log level (debug, info, warn, error)",
)
flag.Parse() flag.Parse()
@@ -59,40 +75,54 @@ func ParseFlags() (*Flags, error) {
flagsParsed = true flagsParsed = true
globalFlags = flags globalFlags = flags
return flags, nil return flags, nil
} }
// validate validates the CLI flags. // validate validates the CLI flags.
func (f *Flags) validate() error { func (f *Flags) validate() error {
if f.SourceDir == "" { if f.SourceDir == "" {
return NewMissingSourceError() return NewCLIMissingSourceError()
} }
// Validate source path for security // Validate source path for security
if err := gibidiutils.ValidateSourcePath(f.SourceDir); err != nil { if err := shared.ValidateSourcePath(f.SourceDir); err != nil {
return err return fmt.Errorf("validating source path: %w", err)
} }
// Validate output format // Validate output format
if err := config.ValidateOutputFormat(f.Format); err != nil { if err := config.ValidateOutputFormat(f.Format); err != nil {
return err return fmt.Errorf("validating output format: %w", err)
} }
// Validate concurrency // Validate concurrency
return config.ValidateConcurrency(f.Concurrency) if err := config.ValidateConcurrency(f.Concurrency); err != nil {
return fmt.Errorf("validating concurrency: %w", err)
}
// Validate log level
if !shared.ValidateLogLevel(f.LogLevel) {
return fmt.Errorf("invalid log level: %s (must be: debug, info, warn, error)", f.LogLevel)
}
return nil
} }
// setDefaultDestination sets the default destination if not provided. // setDefaultDestination sets the default destination if not provided.
func (f *Flags) setDefaultDestination() error { func (f *Flags) setDefaultDestination() error {
if f.Destination == "" { if f.Destination == "" {
absRoot, err := gibidiutils.GetAbsolutePath(f.SourceDir) absRoot, err := shared.AbsolutePath(f.SourceDir)
if err != nil { if err != nil {
return err return fmt.Errorf("getting absolute path: %w", err)
} }
baseName := gibidiutils.GetBaseName(absRoot) baseName := shared.BaseName(absRoot)
f.Destination = baseName + "." + f.Format f.Destination = baseName + "." + f.Format
} }
// Validate destination path for security // Validate destination path for security
return gibidiutils.ValidateDestinationPath(f.Destination) if err := shared.ValidateDestinationPath(f.Destination); err != nil {
return fmt.Errorf("validating destination path: %w", err)
}
return nil
} }

View File

@@ -1,366 +1,664 @@
package cli package cli
import ( import (
"errors"
"flag" "flag"
"os" "os"
"runtime" "runtime"
"strings" "strings"
"testing" "testing"
"github.com/spf13/viper" "github.com/ivuorinen/gibidify/shared"
"github.com/stretchr/testify/assert" "github.com/ivuorinen/gibidify/testutil"
) )
func TestParseFlags(t *testing.T) { const testDirPlaceholder = "testdir"
// Save original command line args and restore after test
oldArgs := os.Args
oldFlagsParsed := flagsParsed
defer func() {
os.Args = oldArgs
flagsParsed = oldFlagsParsed
flag.CommandLine = flag.NewFlagSet(os.Args[0], flag.ContinueOnError)
}()
// setupTestArgs prepares test arguments by replacing testdir with actual temp directory.
func setupTestArgs(t *testing.T, args []string, want *Flags) ([]string, *Flags) {
t.Helper()
if !containsFlag(args, shared.TestCLIFlagSource) {
return args, want
}
tempDir := t.TempDir()
modifiedArgs := replaceTestDirInArgs(args, tempDir)
// Handle nil want parameter (used for error test cases)
if want == nil {
return modifiedArgs, nil
}
modifiedWant := updateWantFlags(*want, tempDir)
return modifiedArgs, &modifiedWant
}
// replaceTestDirInArgs replaces testdir placeholder with actual temp directory in args.
func replaceTestDirInArgs(args []string, tempDir string) []string {
modifiedArgs := make([]string, len(args))
copy(modifiedArgs, args)
for i, arg := range modifiedArgs {
if arg == testDirPlaceholder {
modifiedArgs[i] = tempDir
break
}
}
return modifiedArgs
}
// updateWantFlags updates the want flags with temp directory replacements.
func updateWantFlags(want Flags, tempDir string) Flags {
modifiedWant := want
if want.SourceDir == testDirPlaceholder {
modifiedWant.SourceDir = tempDir
if strings.HasPrefix(want.Destination, testDirPlaceholder+".") {
baseName := testutil.BaseName(tempDir)
modifiedWant.Destination = baseName + "." + want.Format
}
}
return modifiedWant
}
// runParseFlagsTest runs a single parse flags test.
func runParseFlagsTest(t *testing.T, args []string, want *Flags, wantErr bool, errContains string) {
t.Helper()
// Capture and restore original os.Args
origArgs := os.Args
defer func() { os.Args = origArgs }()
resetFlagsState()
modifiedArgs, modifiedWant := setupTestArgs(t, args, want)
setupCommandLineArgs(modifiedArgs)
got, err := ParseFlags()
if wantErr {
if err == nil {
t.Error("ParseFlags() expected error, got nil")
return
}
if errContains != "" && !strings.Contains(err.Error(), errContains) {
t.Errorf("ParseFlags() error = %v, want error containing %v", err, errContains)
}
return
}
if err != nil {
t.Errorf("ParseFlags() unexpected error = %v", err)
return
}
verifyFlags(t, got, modifiedWant)
}
func TestParseFlags(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
args []string args []string
expectedError string want *Flags
validate func(t *testing.T, f *Flags) wantErr bool
setup func(t *testing.T) errContains string
}{ }{
{ {
name: "valid flags with all options", name: "valid basic flags",
args: []string{shared.TestCLIFlagSource, "testdir", shared.TestCLIFlagFormat, "markdown"},
want: &Flags{
SourceDir: "testdir",
Format: "markdown",
Concurrency: runtime.NumCPU(),
Destination: "testdir.markdown",
LogLevel: string(shared.LogLevelWarn),
},
wantErr: false,
},
{
name: "valid with all flags",
args: []string{ args: []string{
"gibidify", shared.TestCLIFlagSource, "testdir",
testFlagSource, "", // will set to tempDir in test body shared.TestCLIFlagDestination, shared.TestOutputMD,
"-destination", "output.md", "-prefix", "# Header",
"-format", "json", "-suffix", "# Footer",
testFlagConcurrency, "4", shared.TestCLIFlagFormat, "json",
"-prefix", "prefix", shared.TestCLIFlagConcurrency, "4",
"-suffix", "suffix", "-verbose",
"-no-colors", "-no-colors",
"-no-progress", "-no-progress",
"-verbose",
}, },
validate: nil, // set in test body using closure want: &Flags{
SourceDir: "testdir",
Destination: shared.TestOutputMD,
Prefix: "# Header",
Suffix: "# Footer",
Format: "json",
Concurrency: 4,
Verbose: true,
NoColors: true,
NoProgress: true,
LogLevel: string(shared.LogLevelWarn),
},
wantErr: false,
}, },
{ {
name: "missing source directory", name: "missing source directory",
args: []string{"gibidify"}, args: []string{shared.TestCLIFlagFormat, "markdown"},
expectedError: testErrSourceRequired, wantErr: true,
errContains: "source directory is required",
}, },
{ {
name: "invalid format", name: "invalid format",
args: []string{ args: []string{shared.TestCLIFlagSource, "testdir", shared.TestCLIFlagFormat, "invalid"},
"gibidify", wantErr: true,
testFlagSource, "", // will set to tempDir in test body errContains: "validating output format",
"-format", "invalid",
},
expectedError: "unsupported output format: invalid",
}, },
{ {
name: "invalid concurrency (zero)", name: "invalid concurrency zero",
args: []string{ args: []string{shared.TestCLIFlagSource, "testdir", shared.TestCLIFlagConcurrency, "0"},
"gibidify", wantErr: true,
testFlagSource, "", // will set to tempDir in test body errContains: shared.TestOpValidatingConcurrency,
testFlagConcurrency, "0",
},
expectedError: "concurrency (0) must be at least 1",
}, },
{ {
name: "invalid concurrency (too high)", name: "negative concurrency",
args: []string{ args: []string{shared.TestCLIFlagSource, "testdir", shared.TestCLIFlagConcurrency, "-1"},
"gibidify", wantErr: true,
testFlagSource, "", // will set to tempDir in test body errContains: shared.TestOpValidatingConcurrency,
testFlagConcurrency, "200",
},
// Set maxConcurrency so the upper bound is enforced
expectedError: "concurrency (200) exceeds maximum (128)",
setup: func(t *testing.T) {
orig := viper.Get("maxConcurrency")
viper.Set("maxConcurrency", 128)
t.Cleanup(func() { viper.Set("maxConcurrency", orig) })
},
},
{
name: "path traversal in source",
args: []string{
"gibidify",
testFlagSource, testPathTraversalPath,
},
expectedError: testErrPathTraversal,
},
{
name: "default values",
args: []string{
"gibidify",
testFlagSource, "", // will set to tempDir in test body
},
validate: nil, // set in test body using closure
}, },
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(
// Reset flags for each test tt.name, func(t *testing.T) {
flagsParsed = false runParseFlagsTest(t, tt.args, tt.want, tt.wantErr, tt.errContains)
globalFlags = nil },
flag.CommandLine = flag.NewFlagSet(os.Args[0], flag.ContinueOnError) )
// Create a local copy of args to avoid corrupting shared test data
args := append([]string{}, tt.args...)
// Use t.TempDir for source directory if needed
tempDir := ""
for i := range args {
if i > 0 && args[i-1] == testFlagSource && args[i] == "" {
tempDir = t.TempDir()
args[i] = tempDir
}
}
os.Args = args
// Set validate closure if needed (for tempDir)
if tt.name == "valid flags with all options" {
tt.validate = func(t *testing.T, f *Flags) {
assert.Equal(t, tempDir, f.SourceDir)
assert.Equal(t, "output.md", f.Destination)
assert.Equal(t, "json", f.Format)
assert.Equal(t, 4, f.Concurrency)
assert.Equal(t, "prefix", f.Prefix)
assert.Equal(t, "suffix", f.Suffix)
assert.True(t, f.NoColors)
assert.True(t, f.NoProgress)
assert.True(t, f.Verbose)
}
}
if tt.name == "default values" {
tt.validate = func(t *testing.T, f *Flags) {
assert.Equal(t, tempDir, f.SourceDir)
assert.Equal(t, "markdown", f.Format)
assert.Equal(t, runtime.NumCPU(), f.Concurrency)
assert.Equal(t, "", f.Prefix)
assert.Equal(t, "", f.Suffix)
assert.False(t, f.NoColors)
assert.False(t, f.NoProgress)
assert.False(t, f.Verbose)
// Destination should be set by setDefaultDestination
assert.NotEmpty(t, f.Destination)
}
}
// Call setup if present (e.g. for maxConcurrency)
if tt.setup != nil {
tt.setup(t)
}
flags, err := ParseFlags()
if tt.expectedError != "" {
if assert.Error(t, err) {
assert.Contains(t, err.Error(), tt.expectedError)
}
assert.Nil(t, flags)
} else {
assert.NoError(t, err)
assert.NotNil(t, flags)
if tt.validate != nil {
tt.validate(t, flags)
}
}
})
} }
} }
func TestFlagsValidate(t *testing.T) { // validateFlagsValidationResult validates flag validation test results.
func validateFlagsValidationResult(t *testing.T, err error, wantErr bool, errContains string) {
t.Helper()
if wantErr {
if err == nil {
t.Error("Flags.validate() expected error, got nil")
return
}
if errContains != "" && !strings.Contains(err.Error(), errContains) {
t.Errorf("Flags.validate() error = %v, want error containing %v", err, errContains)
}
return
}
if err != nil {
t.Errorf("Flags.validate() unexpected error = %v", err)
}
}
func TestFlagsvalidate(t *testing.T) {
tempDir := t.TempDir()
tests := []struct { tests := []struct {
name string name string
flags *Flags flags *Flags
setupFunc func(t *testing.T, f *Flags) wantErr bool
expectedError string errContains string
}{ }{
{
name: "missing source directory",
flags: &Flags{},
expectedError: testErrSourceRequired,
},
{
name: "invalid format",
flags: &Flags{
Format: "invalid",
},
setupFunc: func(t *testing.T, f *Flags) {
f.SourceDir = t.TempDir()
},
expectedError: "unsupported output format: invalid",
},
{
name: "invalid concurrency",
flags: &Flags{
Format: "markdown",
Concurrency: 0,
},
setupFunc: func(t *testing.T, f *Flags) {
f.SourceDir = t.TempDir()
},
expectedError: "concurrency (0) must be at least 1",
},
{
name: "path traversal attempt",
flags: &Flags{
SourceDir: testPathTraversalPath,
Format: "markdown",
},
expectedError: testErrPathTraversal,
},
{ {
name: "valid flags", name: "valid flags",
flags: &Flags{ flags: &Flags{
SourceDir: tempDir,
Format: "markdown",
Concurrency: 4,
LogLevel: "warn",
},
wantErr: false,
},
{
name: "empty source directory",
flags: &Flags{
Format: "markdown",
Concurrency: 4,
LogLevel: "warn",
},
wantErr: true,
errContains: "source directory is required",
},
{
name: "invalid format",
flags: &Flags{
SourceDir: tempDir,
Format: "invalid",
Concurrency: 4,
LogLevel: "warn",
},
wantErr: true,
errContains: "validating output format",
},
{
name: "zero concurrency",
flags: &Flags{
SourceDir: tempDir,
Format: "markdown",
Concurrency: 0,
LogLevel: "warn",
},
wantErr: true,
errContains: shared.TestOpValidatingConcurrency,
},
{
name: "negative concurrency",
flags: &Flags{
SourceDir: tempDir,
Format: "json",
Concurrency: -1,
LogLevel: "warn",
},
wantErr: true,
errContains: shared.TestOpValidatingConcurrency,
},
{
name: "invalid log level",
flags: &Flags{
SourceDir: tempDir,
Format: "json", Format: "json",
Concurrency: 4, Concurrency: 4,
LogLevel: "invalid",
}, },
setupFunc: func(t *testing.T, f *Flags) { wantErr: true,
f.SourceDir = t.TempDir() errContains: "invalid log level",
},
}, },
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(
if tt.setupFunc != nil { tt.name, func(t *testing.T) {
tt.setupFunc(t, tt.flags) err := tt.flags.validate()
} validateFlagsValidationResult(t, err, tt.wantErr, tt.errContains)
},
err := tt.flags.validate() )
if tt.expectedError != "" {
assert.Error(t, err)
assert.Contains(t, err.Error(), tt.expectedError)
} else {
assert.NoError(t, err)
}
})
} }
} }
func TestSetDefaultDestination(t *testing.T) { // validateDefaultDestinationResult validates default destination test results.
func validateDefaultDestinationResult(
t *testing.T,
flags *Flags,
err error,
wantDestination string,
wantErr bool,
errContains string,
) {
t.Helper()
if wantErr {
if err == nil {
t.Error("Flags.setDefaultDestination() expected error, got nil")
return
}
if errContains != "" && !strings.Contains(err.Error(), errContains) {
t.Errorf("Flags.setDefaultDestination() error = %v, want error containing %v", err, errContains)
}
return
}
if err != nil {
t.Errorf("Flags.setDefaultDestination() unexpected error = %v", err)
return
}
if flags.Destination != wantDestination {
t.Errorf("Flags.Destination = %v, want %v", flags.Destination, wantDestination)
}
}
func TestFlagssetDefaultDestination(t *testing.T) {
tempDir := t.TempDir()
baseName := testutil.BaseName(tempDir)
tests := []struct { tests := []struct {
name string name string
flags *Flags flags *Flags
setupFunc func(t *testing.T, f *Flags) wantDestination string
expectedDest string wantErr bool
expectedError string errContains string
}{ }{
{ {
name: "default destination for directory", name: "set default destination markdown",
flags: &Flags{ flags: &Flags{
Format: "markdown", SourceDir: tempDir,
Format: "markdown",
LogLevel: "warn",
}, },
setupFunc: func(t *testing.T, f *Flags) { wantDestination: baseName + ".markdown",
f.SourceDir = t.TempDir() wantErr: false,
},
expectedDest: "", // will check suffix below
}, },
{ {
name: "default destination for json format", name: "set default destination json",
flags: &Flags{ flags: &Flags{
Format: "json", SourceDir: tempDir,
Format: "json",
LogLevel: "warn",
}, },
setupFunc: func(t *testing.T, f *Flags) { wantDestination: baseName + ".json",
f.SourceDir = t.TempDir() wantErr: false,
},
expectedDest: "", // will check suffix below
}, },
{ {
name: "provided destination unchanged", name: "set default destination yaml",
flags: &Flags{ flags: &Flags{
Format: "markdown", SourceDir: tempDir,
Destination: "custom-output.txt", Format: "yaml",
LogLevel: "warn",
}, },
setupFunc: func(t *testing.T, f *Flags) { wantDestination: baseName + ".yaml",
f.SourceDir = t.TempDir() wantErr: false,
},
expectedDest: "custom-output.txt",
}, },
{ {
name: "path traversal in destination", name: "preserve existing destination",
flags: &Flags{ flags: &Flags{
Format: "markdown", SourceDir: tempDir,
Destination: testPathTraversalPath, Format: "yaml",
Destination: "custom-output.yaml",
LogLevel: "warn",
}, },
setupFunc: func(t *testing.T, f *Flags) { wantDestination: "custom-output.yaml",
f.SourceDir = t.TempDir() wantErr: false,
},
{
name: "nonexistent source path still generates destination",
flags: &Flags{
SourceDir: "/nonexistent/path/that/should/not/exist",
Format: "markdown",
LogLevel: "warn",
}, },
expectedError: testErrPathTraversal, wantDestination: "exist.markdown", // Based on filepath.Base of the path
wantErr: false, // AbsolutePath doesn't validate existence, only converts to absolute
}, },
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(
if tt.setupFunc != nil { tt.name, func(t *testing.T) {
tt.setupFunc(t, tt.flags) err := tt.flags.setDefaultDestination()
validateDefaultDestinationResult(t, tt.flags, err, tt.wantDestination, tt.wantErr, tt.errContains)
},
)
}
}
func TestParseFlagsSingleton(t *testing.T) {
// Capture and restore original os.Args
origArgs := os.Args
defer func() { os.Args = origArgs }()
resetFlagsState()
tempDir := t.TempDir()
// First call
setupCommandLineArgs([]string{shared.TestCLIFlagSource, tempDir, shared.TestCLIFlagFormat, "markdown"})
flags1, err := ParseFlags()
if err != nil {
t.Fatalf("First ParseFlags() failed: %v", err)
}
// Second call should return the same instance
flags2, err := ParseFlags()
if err != nil {
t.Fatalf("Second ParseFlags() failed: %v", err)
}
if flags1 != flags2 {
t.Error("ParseFlags() should return singleton instance, got different pointers")
}
}
// Helper functions
// resetFlagsState resets the global flags state for testing.
func resetFlagsState() {
flagsParsed = false
globalFlags = nil
// Reset the flag.CommandLine for clean testing (use ContinueOnError to match ResetFlags)
flag.CommandLine = flag.NewFlagSet(os.Args[0], flag.ContinueOnError)
}
// setupCommandLineArgs sets up command line arguments for testing.
func setupCommandLineArgs(args []string) {
os.Args = append([]string{"gibidify"}, args...)
}
// containsFlag checks if a flag is present in the arguments.
func containsFlag(args []string, flagName string) bool {
for _, arg := range args {
if arg == flagName {
return true
}
}
return false
}
// verifyFlags compares two Flags structs for testing.
func verifyFlags(t *testing.T, got, want *Flags) {
t.Helper()
if got.SourceDir != want.SourceDir {
t.Errorf("SourceDir = %v, want %v", got.SourceDir, want.SourceDir)
}
if got.Destination != want.Destination {
t.Errorf("Destination = %v, want %v", got.Destination, want.Destination)
}
if got.Prefix != want.Prefix {
t.Errorf("Prefix = %v, want %v", got.Prefix, want.Prefix)
}
if got.Suffix != want.Suffix {
t.Errorf("Suffix = %v, want %v", got.Suffix, want.Suffix)
}
if got.Format != want.Format {
t.Errorf("Format = %v, want %v", got.Format, want.Format)
}
if got.Concurrency != want.Concurrency {
t.Errorf("Concurrency = %v, want %v", got.Concurrency, want.Concurrency)
}
if got.NoColors != want.NoColors {
t.Errorf("NoColors = %v, want %v", got.NoColors, want.NoColors)
}
if got.NoProgress != want.NoProgress {
t.Errorf("NoProgress = %v, want %v", got.NoProgress, want.NoProgress)
}
if got.Verbose != want.Verbose {
t.Errorf("Verbose = %v, want %v", got.Verbose, want.Verbose)
}
if got.LogLevel != want.LogLevel {
t.Errorf("LogLevel = %v, want %v", got.LogLevel, want.LogLevel)
}
if got.NoUI != want.NoUI {
t.Errorf("NoUI = %v, want %v", got.NoUI, want.NoUI)
}
}
// TestResetFlags tests the ResetFlags function.
func TestResetFlags(t *testing.T) {
// Save original state
originalArgs := os.Args
originalFlagsParsed := flagsParsed
originalGlobalFlags := globalFlags
originalCommandLine := flag.CommandLine
defer func() {
// Restore original state
os.Args = originalArgs
flagsParsed = originalFlagsParsed
globalFlags = originalGlobalFlags
flag.CommandLine = originalCommandLine
}()
// Simplified test cases to reduce complexity
testCases := map[string]func(t *testing.T){
"reset after flags have been parsed": func(t *testing.T) {
srcDir := t.TempDir()
testutil.CreateTestFile(t, srcDir, "test.txt", []byte("test"))
os.Args = []string{"test", "-source", srcDir, "-destination", "out.json"}
// Parse flags first
if _, err := ParseFlags(); err != nil {
t.Fatalf("Setup failed: %v", err)
} }
},
"reset with clean state": func(t *testing.T) {
if flagsParsed {
t.Log("Note: flagsParsed was already true at start")
}
},
"multiple resets": func(t *testing.T) {
srcDir := t.TempDir()
testutil.CreateTestFile(t, srcDir, "test.txt", []byte("test"))
os.Args = []string{"test", "-source", srcDir, "-destination", "out.json"}
err := tt.flags.setDefaultDestination() if _, err := ParseFlags(); err != nil {
t.Fatalf("Setup failed: %v", err)
}
},
}
if tt.expectedError != "" { for name, setup := range testCases {
assert.Error(t, err) t.Run(name, func(t *testing.T) {
assert.Contains(t, err.Error(), tt.expectedError) // Setup test scenario
} else { setup(t)
assert.NoError(t, err)
switch { // Call ResetFlags
case tt.expectedDest != "": ResetFlags()
assert.Equal(t, tt.expectedDest, tt.flags.Destination)
case tt.flags.Format == "json": // Basic verification that reset worked
assert.True( if flagsParsed {
t, strings.HasSuffix(tt.flags.Destination, ".json"), t.Error("flagsParsed should be false after ResetFlags()")
"expected %q to have suffix .json", tt.flags.Destination, }
) if globalFlags != nil {
case tt.flags.Format == "markdown": t.Error("globalFlags should be nil after ResetFlags()")
assert.True(
t, strings.HasSuffix(tt.flags.Destination, ".markdown"),
"expected %q to have suffix .markdown", tt.flags.Destination,
)
}
} }
}) })
} }
} }
func TestFlagsSingleton(t *testing.T) { // TestResetFlags_Integration tests ResetFlags in integration scenarios.
// Save original state func TestResetFlagsIntegration(t *testing.T) {
oldFlagsParsed := flagsParsed // This test verifies that ResetFlags properly resets the internal state
oldGlobalFlags := globalFlags // to allow multiple calls to ParseFlags in test scenarios.
defer func() {
flagsParsed = oldFlagsParsed
globalFlags = oldGlobalFlags
}()
// Test singleton behavior // Note: This test documents the expected behavior of ResetFlags
flagsParsed = true // The actual integration with ParseFlags is already tested in main tests
expectedFlags := &Flags{ // where ResetFlags is used to enable proper test isolation.
SourceDir: "/test",
Format: "json", t.Run("state_reset_behavior", func(t *testing.T) {
Concurrency: 2, // Test behavior is already covered in TestResetFlags
// This is mainly for documentation of the integration pattern
t.Log("ResetFlags integration behavior:")
t.Log("1. Resets flagsParsed to false")
t.Log("2. Sets globalFlags to nil")
t.Log("3. Creates new flag.CommandLine FlagSet")
t.Log("4. Allows subsequent ParseFlags calls")
// The actual mechanics are tested in TestResetFlags
// This test serves to document the integration contract
// Reset state (this should not panic)
ResetFlags()
// Verify basic state expectations
if flagsParsed {
t.Error("flagsParsed should be false after ResetFlags")
}
if globalFlags != nil {
t.Error("globalFlags should be nil after ResetFlags")
}
if flag.CommandLine == nil {
t.Error("flag.CommandLine should not be nil after ResetFlags")
}
})
}
// Benchmarks for flag-related operations.
// While flag parsing is a one-time startup operation, these benchmarks
// document baseline performance and catch regressions if parsing logic becomes more complex.
//
// Note: ParseFlags benchmarks are omitted because resetFlagsState() interferes with
// Go's testing framework flags. The core operations (setDefaultDestination, validate)
// are benchmarked instead.
// BenchmarkSetDefaultDestination measures the setDefaultDestination operation.
func BenchmarkSetDefaultDestination(b *testing.B) {
tempDir := b.TempDir()
for b.Loop() {
flags := &Flags{
SourceDir: tempDir,
Format: "markdown",
LogLevel: "warn",
}
_ = flags.setDefaultDestination()
} }
globalFlags = expectedFlags
// Should return cached flags without parsing
flags, err := ParseFlags()
assert.NoError(t, err)
assert.Equal(t, expectedFlags, flags)
assert.Same(t, globalFlags, flags)
} }
func TestNewMissingSourceError(t *testing.T) { // BenchmarkSetDefaultDestinationAllFormats measures setDefaultDestination across all formats.
err := NewMissingSourceError() func BenchmarkSetDefaultDestinationAllFormats(b *testing.B) {
tempDir := b.TempDir()
formats := []string{"markdown", "json", "yaml"}
assert.Error(t, err) for b.Loop() {
assert.Equal(t, testErrSourceRequired, err.Error()) for _, format := range formats {
flags := &Flags{
// Check if it's the right type SourceDir: tempDir,
var missingSourceError *MissingSourceError Format: format,
ok := errors.As(err, &missingSourceError) LogLevel: "warn",
assert.True(t, ok) }
_ = flags.setDefaultDestination()
}
}
}
// BenchmarkFlagsValidate measures the validate operation.
func BenchmarkFlagsValidate(b *testing.B) {
tempDir := b.TempDir()
flags := &Flags{
SourceDir: tempDir,
Destination: "output.md",
Format: "markdown",
LogLevel: "warn",
}
for b.Loop() {
_ = flags.validate()
}
}
// BenchmarkFlagsValidateAllFormats measures validate across all formats.
func BenchmarkFlagsValidateAllFormats(b *testing.B) {
tempDir := b.TempDir()
formats := []string{"markdown", "json", "yaml"}
for b.Loop() {
for _, format := range formats {
flags := &Flags{
SourceDir: tempDir,
Destination: "output." + format,
Format: format,
LogLevel: "warn",
}
_ = flags.validate()
}
}
} }

View File

@@ -1,46 +1,48 @@
// Package cli provides command-line interface functionality for gibidify.
package cli package cli
import ( import (
"fmt" "fmt"
"os" "os"
"github.com/sirupsen/logrus"
"github.com/ivuorinen/gibidify/config" "github.com/ivuorinen/gibidify/config"
"github.com/ivuorinen/gibidify/fileproc" "github.com/ivuorinen/gibidify/fileproc"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// collectFiles collects all files to be processed. // collectFiles collects all files to be processed.
func (p *Processor) collectFiles() ([]string, error) { func (p *Processor) collectFiles() ([]string, error) {
files, err := fileproc.CollectFiles(p.flags.SourceDir) files, err := fileproc.CollectFiles(p.flags.SourceDir)
if err != nil { if err != nil {
return nil, gibidiutils.WrapError( return nil, shared.WrapError(
err, err,
gibidiutils.ErrorTypeProcessing, shared.ErrorTypeProcessing,
gibidiutils.CodeProcessingCollection, shared.CodeProcessingCollection,
"error collecting files", "error collecting files",
) )
} }
logrus.Infof("Found %d files to process", len(files))
logger := shared.GetLogger()
logger.Infof(shared.CLIMsgFoundFilesToProcess, len(files))
return files, nil return files, nil
} }
// validateFileCollection validates the collected files against resource limits. // validateFileCollection validates the collected files against resource limits.
func (p *Processor) validateFileCollection(files []string) error { func (p *Processor) validateFileCollection(files []string) error {
if !config.GetResourceLimitsEnabled() { if !config.ResourceLimitsEnabled() {
return nil return nil
} }
// Check file count limit // Check file count limit
maxFiles := config.GetMaxFiles() maxFiles := config.MaxFiles()
if len(files) > maxFiles { if len(files) > maxFiles {
return gibidiutils.NewStructuredError( return shared.NewStructuredError(
gibidiutils.ErrorTypeValidation, shared.ErrorTypeValidation,
gibidiutils.CodeResourceLimitFiles, shared.CodeResourceLimitFiles,
fmt.Sprintf("file count (%d) exceeds maximum limit (%d)", len(files), maxFiles), fmt.Sprintf("file count (%d) exceeds maximum limit (%d)", len(files), maxFiles),
"", "",
map[string]interface{}{ map[string]any{
"file_count": len(files), "file_count": len(files),
"max_files": maxFiles, "max_files": maxFiles,
}, },
@@ -48,7 +50,7 @@ func (p *Processor) validateFileCollection(files []string) error {
} }
// Check total size limit (estimate) // Check total size limit (estimate)
maxTotalSize := config.GetMaxTotalSize() maxTotalSize := config.MaxTotalSize()
totalSize := int64(0) totalSize := int64(0)
oversizedFiles := 0 oversizedFiles := 0
@@ -56,16 +58,14 @@ func (p *Processor) validateFileCollection(files []string) error {
if fileInfo, err := os.Stat(filePath); err == nil { if fileInfo, err := os.Stat(filePath); err == nil {
totalSize += fileInfo.Size() totalSize += fileInfo.Size()
if totalSize > maxTotalSize { if totalSize > maxTotalSize {
return gibidiutils.NewStructuredError( return shared.NewStructuredError(
gibidiutils.ErrorTypeValidation, shared.ErrorTypeValidation,
gibidiutils.CodeResourceLimitTotalSize, shared.CodeResourceLimitTotalSize,
fmt.Sprintf( fmt.Sprintf(
"total file size (%d bytes) would exceed maximum limit (%d bytes)", "total file size (%d bytes) would exceed maximum limit (%d bytes)", totalSize, maxTotalSize,
totalSize,
maxTotalSize,
), ),
"", "",
map[string]interface{}{ map[string]any{
"total_size": totalSize, "total_size": totalSize,
"max_total_size": maxTotalSize, "max_total_size": maxTotalSize,
"files_checked": len(files), "files_checked": len(files),
@@ -77,10 +77,12 @@ func (p *Processor) validateFileCollection(files []string) error {
} }
} }
logger := shared.GetLogger()
if oversizedFiles > 0 { if oversizedFiles > 0 {
logrus.Warnf("Could not stat %d files during pre-validation", oversizedFiles) logger.Warnf("Could not stat %d files during pre-validation", oversizedFiles)
} }
logrus.Infof("Pre-validation passed: %d files, %d MB total", len(files), totalSize/1024/1024) logger.Infof("Pre-validation passed: %d files, %d MB total", len(files), totalSize/int64(shared.BytesPerMB))
return nil return nil
} }

View File

@@ -1,12 +1,14 @@
// Package cli provides command-line interface functionality for gibidify.
package cli package cli
import ( import (
"context" "context"
"os" "os"
"sync" "sync"
"time"
"github.com/ivuorinen/gibidify/fileproc" "github.com/ivuorinen/gibidify/fileproc"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// Process executes the main file processing workflow. // Process executes the main file processing workflow.
@@ -16,9 +18,7 @@ func (p *Processor) Process(ctx context.Context) error {
defer overallCancel() defer overallCancel()
// Configure file type registry // Configure file type registry
if err := p.configureFileTypes(); err != nil { p.configureFileTypes()
return err
}
// Print startup info with colors // Print startup info with colors
p.ui.PrintHeader("🚀 Starting gibidify") p.ui.PrintHeader("🚀 Starting gibidify")
@@ -31,23 +31,32 @@ func (p *Processor) Process(ctx context.Context) error {
p.resourceMonitor.LogResourceInfo() p.resourceMonitor.LogResourceInfo()
p.backpressure.LogBackpressureInfo() p.backpressure.LogBackpressureInfo()
// Collect files with progress indication // Collect files with progress indication and timing
p.ui.PrintInfo("📁 Collecting files...") p.ui.PrintInfo("📁 Collecting files...")
collectionStart := time.Now()
files, err := p.collectFiles() files, err := p.collectFiles()
collectionTime := time.Since(collectionStart)
p.metricsCollector.RecordPhaseTime(shared.MetricsPhaseCollection, collectionTime)
if err != nil { if err != nil {
return err return err
} }
// Show collection results // Show collection results
p.ui.PrintSuccess("Found %d files to process", len(files)) p.ui.PrintSuccess(shared.CLIMsgFoundFilesToProcess, len(files))
// Pre-validate file collection against resource limits // Pre-validate file collection against resource limits
if err := p.validateFileCollection(files); err != nil { if err := p.validateFileCollection(files); err != nil {
return err return err
} }
// Process files with overall timeout // Process files with overall timeout and timing
return p.processFiles(overallCtx, files) processingStart := time.Now()
err = p.processFiles(overallCtx, files)
processingTime := time.Since(processingStart)
p.metricsCollector.RecordPhaseTime(shared.MetricsPhaseProcessing, processingTime)
return err
} }
// processFiles processes the collected files. // processFiles processes the collected files.
@@ -57,7 +66,7 @@ func (p *Processor) processFiles(ctx context.Context, files []string) error {
return err return err
} }
defer func() { defer func() {
gibidiutils.LogError("Error closing output file", outFile.Close()) shared.LogError("Error closing output file", outFile.Close())
}() }()
// Initialize back-pressure and channels // Initialize back-pressure and channels
@@ -67,11 +76,7 @@ func (p *Processor) processFiles(ctx context.Context, files []string) error {
writerDone := make(chan struct{}) writerDone := make(chan struct{})
// Start writer // Start writer
go fileproc.StartWriter(outFile, writeCh, writerDone, fileproc.WriterConfig{ go fileproc.StartWriter(outFile, writeCh, writerDone, p.flags.Format, p.flags.Prefix, p.flags.Suffix)
Format: p.flags.Format,
Prefix: p.flags.Prefix,
Suffix: p.flags.Suffix,
})
// Start workers // Start workers
var wg sync.WaitGroup var wg sync.WaitGroup
@@ -83,28 +88,41 @@ func (p *Processor) processFiles(ctx context.Context, files []string) error {
// Send files to workers // Send files to workers
if err := p.sendFiles(ctx, files, fileCh); err != nil { if err := p.sendFiles(ctx, files, fileCh); err != nil {
p.ui.FinishProgress() p.ui.FinishProgress()
return err return err
} }
// Wait for completion // Wait for completion with timing
writingStart := time.Now()
p.waitForCompletion(&wg, writeCh, writerDone) p.waitForCompletion(&wg, writeCh, writerDone)
writingTime := time.Since(writingStart)
p.metricsCollector.RecordPhaseTime(shared.MetricsPhaseWriting, writingTime)
p.ui.FinishProgress() p.ui.FinishProgress()
// Final cleanup with timing
finalizeStart := time.Now()
p.logFinalStats() p.logFinalStats()
finalizeTime := time.Since(finalizeStart)
p.metricsCollector.RecordPhaseTime(shared.MetricsPhaseFinalize, finalizeTime)
p.ui.PrintSuccess("Processing completed. Output saved to %s", p.flags.Destination) p.ui.PrintSuccess("Processing completed. Output saved to %s", p.flags.Destination)
return nil return nil
} }
// createOutputFile creates the output file. // createOutputFile creates the output file.
func (p *Processor) createOutputFile() (*os.File, error) { func (p *Processor) createOutputFile() (*os.File, error) {
// Destination path has been validated in CLI flags validation for path traversal attempts // Destination path has been validated in CLI flags validation for path traversal attempts
// #nosec G304 - destination is validated in flags.validate() outFile, err := os.Create(p.flags.Destination) // #nosec G304 - destination is validated in flags.validate()
outFile, err := os.Create(p.flags.Destination)
if err != nil { if err != nil {
return nil, gibidiutils.WrapError( return nil, shared.WrapError(
err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOFileCreate, err,
shared.ErrorTypeIO,
shared.CodeIOFileCreate,
"failed to create output file", "failed to create output file",
).WithFilePath(p.flags.Destination) ).WithFilePath(p.flags.Destination)
} }
return outFile, nil return outFile, nil
} }

View File

@@ -1,265 +0,0 @@
package cli
import (
"context"
"os"
"path/filepath"
"sync"
"testing"
"github.com/spf13/viper"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/ivuorinen/gibidify/fileproc"
)
func TestProcessorSimple(t *testing.T) {
t.Run("NewProcessor", func(t *testing.T) {
flags := &Flags{
SourceDir: "/tmp/test",
Destination: "output.md",
Format: "markdown",
Concurrency: 2,
NoColors: true,
NoProgress: true,
Verbose: false,
}
p := NewProcessor(flags)
assert.NotNil(t, p)
assert.Equal(t, flags, p.flags)
assert.NotNil(t, p.ui)
assert.NotNil(t, p.backpressure)
assert.NotNil(t, p.resourceMonitor)
assert.False(t, p.ui.enableColors)
assert.False(t, p.ui.enableProgress)
})
t.Run("ConfigureFileTypes", func(t *testing.T) {
p := &Processor{
flags: &Flags{},
ui: NewUIManager(),
}
// Should not panic or error
err := p.configureFileTypes()
assert.NoError(t, err)
assert.NotNil(t, p)
})
t.Run("CreateOutputFile", func(t *testing.T) {
// Create temp file path
tempDir := t.TempDir()
outputPath := filepath.Join(tempDir, "output.txt")
p := &Processor{
flags: &Flags{
Destination: outputPath,
},
ui: NewUIManager(),
}
file, err := p.createOutputFile()
assert.NoError(t, err)
assert.NotNil(t, file)
// Clean up
err = file.Close()
require.NoError(t, err)
err = os.Remove(outputPath)
require.NoError(t, err)
})
t.Run("ValidateFileCollection", func(t *testing.T) {
p := &Processor{
ui: NewUIManager(),
}
// Empty collection should be valid (just checks limits)
err := p.validateFileCollection([]string{})
assert.NoError(t, err)
// Small collection should be valid
err = p.validateFileCollection([]string{
testFilePath1,
testFilePath2,
})
assert.NoError(t, err)
})
t.Run("CollectFiles_EmptyDir", func(t *testing.T) {
tempDir := t.TempDir()
p := &Processor{
flags: &Flags{
SourceDir: tempDir,
},
ui: NewUIManager(),
}
files, err := p.collectFiles()
assert.NoError(t, err)
assert.Empty(t, files)
})
t.Run("CollectFiles_WithFiles", func(t *testing.T) {
tempDir := t.TempDir()
// Create test files
require.NoError(t, os.WriteFile(filepath.Join(tempDir, "test1.go"), []byte("package main"), 0o600))
require.NoError(t, os.WriteFile(filepath.Join(tempDir, "test2.go"), []byte("package test"), 0o600))
// Set config so no files are ignored, and restore after test
origIgnoreDirs := viper.Get("ignoreDirectories")
origFileSizeLimit := viper.Get("fileSizeLimit")
viper.Set("ignoreDirectories", []string{})
viper.Set("fileSizeLimit", 1024*1024*10) // 10MB
t.Cleanup(func() {
viper.Set("ignoreDirectories", origIgnoreDirs)
viper.Set("fileSizeLimit", origFileSizeLimit)
})
p := &Processor{
flags: &Flags{
SourceDir: tempDir,
},
ui: NewUIManager(),
}
files, err := p.collectFiles()
assert.NoError(t, err)
assert.Len(t, files, 2)
})
t.Run("SendFiles", func(t *testing.T) {
p := &Processor{
backpressure: fileproc.NewBackpressureManager(),
ui: NewUIManager(),
}
ctx := context.Background()
fileCh := make(chan string, 3)
files := []string{
testFilePath1,
testFilePath2,
}
var wg sync.WaitGroup
wg.Add(1)
// Send files in a goroutine since it might block
go func() {
defer wg.Done()
err := p.sendFiles(ctx, files, fileCh)
assert.NoError(t, err)
}()
// Read all files from channel
var received []string
for i := 0; i < len(files); i++ {
file := <-fileCh
received = append(received, file)
}
assert.Equal(t, len(files), len(received))
// Wait for sendFiles goroutine to finish (and close fileCh)
wg.Wait()
// Now channel should be closed
_, ok := <-fileCh
assert.False(t, ok, "channel should be closed")
})
t.Run("WaitForCompletion", func(t *testing.T) {
p := &Processor{
ui: NewUIManager(),
}
writeCh := make(chan fileproc.WriteRequest)
writerDone := make(chan struct{})
// Simulate writer finishing
go func() {
<-writeCh // Wait for close
close(writerDone)
}()
var wg sync.WaitGroup
// Start and finish immediately
wg.Add(1)
wg.Done()
// Should complete without hanging
p.waitForCompletion(&wg, writeCh, writerDone)
assert.NotNil(t, p)
})
t.Run("LogFinalStats", func(t *testing.T) {
p := &Processor{
flags: &Flags{
Verbose: true,
},
ui: NewUIManager(),
resourceMonitor: fileproc.NewResourceMonitor(),
backpressure: fileproc.NewBackpressureManager(),
}
// Should not panic
p.logFinalStats()
assert.NotNil(t, p)
})
}
// Test error handling scenarios
func TestProcessorErrors(t *testing.T) {
t.Run("CreateOutputFile_InvalidPath", func(t *testing.T) {
p := &Processor{
flags: &Flags{
Destination: "/root/cannot-write-here.txt",
},
ui: NewUIManager(),
}
file, err := p.createOutputFile()
assert.Error(t, err)
assert.Nil(t, file)
})
t.Run("CollectFiles_NonExistentDir", func(t *testing.T) {
p := &Processor{
flags: &Flags{
SourceDir: "/non/existent/path",
},
ui: NewUIManager(),
}
files, err := p.collectFiles()
assert.Error(t, err)
assert.Nil(t, files)
})
t.Run("SendFiles_WithCancellation", func(t *testing.T) {
p := &Processor{
backpressure: fileproc.NewBackpressureManager(),
ui: NewUIManager(),
}
ctx, cancel := context.WithCancel(context.Background())
fileCh := make(chan string) // Unbuffered to force blocking
files := []string{
testFilePath1,
testFilePath2,
"/test/file3.go",
}
// Cancel immediately
cancel()
err := p.sendFiles(ctx, files, fileCh)
assert.Error(t, err)
assert.Equal(t, context.Canceled, err)
})
}

View File

@@ -1,44 +1,108 @@
// Package cli provides command-line interface functionality for gibidify.
package cli package cli
import ( import (
"github.com/sirupsen/logrus" "strings"
"github.com/ivuorinen/gibidify/config" "github.com/ivuorinen/gibidify/config"
"github.com/ivuorinen/gibidify/shared"
) )
// logFinalStats logs the final back-pressure and resource monitoring statistics. // logFinalStats logs back-pressure, resource usage, and processing statistics.
func (p *Processor) logFinalStats() { func (p *Processor) logFinalStats() {
// Log back-pressure stats p.logBackpressureStats()
backpressureStats := p.backpressure.GetStats() p.logResourceStats()
p.finalizeAndReportMetrics()
p.logVerboseStats()
if p.resourceMonitor != nil {
p.resourceMonitor.Close()
}
}
// logBackpressureStats logs back-pressure statistics.
func (p *Processor) logBackpressureStats() {
// Check backpressure is non-nil before dereferencing
if p.backpressure == nil {
return
}
logger := shared.GetLogger()
backpressureStats := p.backpressure.Stats()
if backpressureStats.Enabled { if backpressureStats.Enabled {
logrus.Infof( logger.Infof(
"Back-pressure stats: processed=%d files, memory=%dMB/%dMB", "Back-pressure stats: processed=%d files, memory=%dMB/%dMB",
backpressureStats.FilesProcessed, backpressureStats.FilesProcessed,
backpressureStats.CurrentMemoryUsage/1024/1024, backpressureStats.CurrentMemoryUsage/int64(shared.BytesPerMB),
backpressureStats.MaxMemoryUsage/1024/1024, backpressureStats.MaxMemoryUsage/int64(shared.BytesPerMB),
) )
} }
}
// Log resource monitoring stats // logResourceStats logs resource monitoring statistics.
resourceStats := p.resourceMonitor.GetMetrics() func (p *Processor) logResourceStats() {
if config.GetResourceLimitsEnabled() { // Check resource monitoring is enabled and monitor is non-nil before dereferencing
logrus.Infof("Resource stats: processed=%d files, totalSize=%dMB, avgFileSize=%.2fKB, rate=%.2f files/sec", if !config.ResourceLimitsEnabled() {
resourceStats.FilesProcessed, resourceStats.TotalSizeProcessed/1024/1024, return
resourceStats.AverageFileSize/1024, resourceStats.ProcessingRate)
if len(resourceStats.ViolationsDetected) > 0 {
logrus.Warnf("Resource violations detected: %v", resourceStats.ViolationsDetected)
}
if resourceStats.DegradationActive {
logrus.Warnf("Processing completed with degradation mode active")
}
if resourceStats.EmergencyStopActive {
logrus.Errorf("Processing completed with emergency stop active")
}
} }
// Clean up resource monitor if p.resourceMonitor == nil {
p.resourceMonitor.Close() return
}
logger := shared.GetLogger()
resourceStats := p.resourceMonitor.Metrics()
logger.Infof(
"Resource stats: processed=%d files, totalSize=%dMB, avgFileSize=%.2fKB, rate=%.2f files/sec",
resourceStats.FilesProcessed, resourceStats.TotalSizeProcessed/int64(shared.BytesPerMB),
resourceStats.AverageFileSize/float64(shared.BytesPerKB), resourceStats.ProcessingRate,
)
if len(resourceStats.ViolationsDetected) > 0 {
logger.Warnf("Resource violations detected: %v", resourceStats.ViolationsDetected)
}
if resourceStats.DegradationActive {
logger.Warnf("Processing completed with degradation mode active")
}
if resourceStats.EmergencyStopActive {
logger.Errorf("Processing completed with emergency stop active")
}
}
// finalizeAndReportMetrics finalizes metrics collection and displays the final report.
func (p *Processor) finalizeAndReportMetrics() {
if p.metricsCollector != nil {
p.metricsCollector.Finish()
}
if p.metricsReporter != nil {
finalReport := p.metricsReporter.ReportFinal()
if finalReport != "" && p.ui != nil {
// Use UI manager to respect NoUI flag - remove trailing newline if present
p.ui.PrintInfo("%s", strings.TrimSuffix(finalReport, "\n"))
}
}
}
// logVerboseStats logs detailed structured statistics when verbose mode is enabled.
func (p *Processor) logVerboseStats() {
if !p.flags.Verbose || p.metricsCollector == nil {
return
}
logger := shared.GetLogger()
report := p.metricsCollector.GenerateReport()
fields := map[string]any{
"total_files": report.Summary.TotalFiles,
"processed_files": report.Summary.ProcessedFiles,
"skipped_files": report.Summary.SkippedFiles,
"error_files": report.Summary.ErrorFiles,
"processing_time": report.Summary.ProcessingTime,
"files_per_second": report.Summary.FilesPerSecond,
"bytes_per_second": report.Summary.BytesPerSecond,
"memory_usage_mb": report.Summary.CurrentMemoryMB,
}
logger.WithFields(fields).Info("Processing completed with comprehensive metrics")
} }

1025
cli/processor_test.go Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -1,16 +1,20 @@
// Package cli provides command-line interface functionality for gibidify.
package cli package cli
import ( import (
"github.com/ivuorinen/gibidify/config" "github.com/ivuorinen/gibidify/config"
"github.com/ivuorinen/gibidify/fileproc" "github.com/ivuorinen/gibidify/fileproc"
"github.com/ivuorinen/gibidify/metrics"
) )
// Processor handles the main file processing logic. // Processor handles the main file processing logic.
type Processor struct { type Processor struct {
flags *Flags flags *Flags
backpressure *fileproc.BackpressureManager backpressure *fileproc.BackpressureManager
resourceMonitor *fileproc.ResourceMonitor resourceMonitor *fileproc.ResourceMonitor
ui *UIManager ui *UIManager
metricsCollector *metrics.Collector
metricsReporter *metrics.Reporter
} }
// NewProcessor creates a new processor with the given flags. // NewProcessor creates a new processor with the given flags.
@@ -18,30 +22,38 @@ func NewProcessor(flags *Flags) *Processor {
ui := NewUIManager() ui := NewUIManager()
// Configure UI based on flags // Configure UI based on flags
ui.SetColorOutput(!flags.NoColors) ui.SetColorOutput(!flags.NoColors && !flags.NoUI)
ui.SetProgressOutput(!flags.NoProgress) ui.SetProgressOutput(!flags.NoProgress && !flags.NoUI)
ui.SetSilentMode(flags.NoUI)
// Initialize metrics system
metricsCollector := metrics.NewCollector()
metricsReporter := metrics.NewReporter(
metricsCollector,
flags.Verbose && !flags.NoUI,
!flags.NoColors && !flags.NoUI,
)
return &Processor{ return &Processor{
flags: flags, flags: flags,
backpressure: fileproc.NewBackpressureManager(), backpressure: fileproc.NewBackpressureManager(),
resourceMonitor: fileproc.NewResourceMonitor(), resourceMonitor: fileproc.NewResourceMonitor(),
ui: ui, ui: ui,
metricsCollector: metricsCollector,
metricsReporter: metricsReporter,
} }
} }
// configureFileTypes configures the file type registry. // configureFileTypes configures the file type registry.
func (p *Processor) configureFileTypes() error { func (p *Processor) configureFileTypes() {
if config.GetFileTypesEnabled() { if config.FileTypesEnabled() {
if err := fileproc.ConfigureFromSettings(fileproc.RegistryConfig{ fileproc.ConfigureFromSettings(
CustomImages: config.GetCustomImageExtensions(), config.CustomImageExtensions(),
CustomBinary: config.GetCustomBinaryExtensions(), config.CustomBinaryExtensions(),
CustomLanguages: config.GetCustomLanguages(), config.CustomLanguages(),
DisabledImages: config.GetDisabledImageExtensions(), config.DisabledImageExtensions(),
DisabledBinary: config.GetDisabledBinaryExtensions(), config.DisabledBinaryExtensions(),
DisabledLanguages: config.GetDisabledLanguageExtensions(), config.DisabledLanguageExtensions(),
}); err != nil { )
return err
}
} }
return nil
} }

View File

@@ -1,13 +1,17 @@
// Package cli provides command-line interface functionality for gibidify.
package cli package cli
import ( import (
"context" "context"
"errors"
"fmt"
"os"
"path/filepath"
"sync" "sync"
"github.com/sirupsen/logrus"
"github.com/ivuorinen/gibidify/fileproc" "github.com/ivuorinen/gibidify/fileproc"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/metrics"
"github.com/ivuorinen/gibidify/shared"
) )
// startWorkers starts the worker goroutines. // startWorkers starts the worker goroutines.
@@ -44,25 +48,69 @@ func (p *Processor) worker(
} }
} }
// processFile processes a single file with resource monitoring. // processFile processes a single file with resource monitoring and metrics collection.
func (p *Processor) processFile(ctx context.Context, filePath string, writeCh chan fileproc.WriteRequest) { func (p *Processor) processFile(ctx context.Context, filePath string, writeCh chan fileproc.WriteRequest) {
// Create file processing context with timeout (resourceMonitor may be nil)
fileCtx, fileCancel := ctx, func() {}
if p.resourceMonitor != nil {
fileCtx, fileCancel = p.resourceMonitor.CreateFileProcessingContext(ctx)
}
defer fileCancel()
// Track concurrency
if p.metricsCollector != nil {
p.metricsCollector.IncrementConcurrency()
defer p.metricsCollector.DecrementConcurrency()
}
// Check for emergency stop // Check for emergency stop
if p.resourceMonitor.IsEmergencyStopActive() { if p.resourceMonitor != nil && p.resourceMonitor.IsEmergencyStopActive() {
logrus.Warnf("Emergency stop active, skipping file: %s", filePath) logger := shared.GetLogger()
logger.Warnf("Emergency stop active, skipping file: %s", filePath)
// Record skipped file
p.recordFileResult(filePath, 0, "", false, true, "emergency stop active", nil)
if p.ui != nil {
p.ui.UpdateProgress(1)
}
return return
} }
absRoot, err := gibidiutils.GetAbsolutePath(p.flags.SourceDir) absRoot, err := shared.AbsolutePath(p.flags.SourceDir)
if err != nil { if err != nil {
gibidiutils.LogError("Failed to get absolute path", err) shared.LogError("Failed to get absolute path", err)
// Record error
p.recordFileResult(filePath, 0, "", false, false, "", err)
if p.ui != nil {
p.ui.UpdateProgress(1)
}
return return
} }
// Use the resource monitor-aware processing // Use the resource monitor-aware processing with metrics tracking
fileproc.ProcessFileWithMonitor(ctx, filePath, writeCh, absRoot, p.resourceMonitor) fileSize, format, success, processErr := p.processFileWithMetrics(fileCtx, filePath, writeCh, absRoot)
// Update progress bar // Record the processing result (skipped=false, skipReason="" since processFileWithMetrics never skips)
p.ui.UpdateProgress(1) p.recordFileResult(filePath, fileSize, format, success, false, "", processErr)
// Update progress bar with metrics
if p.ui != nil {
p.ui.UpdateProgress(1)
}
// Show real-time stats in verbose mode
if p.flags.Verbose && p.metricsCollector != nil {
currentMetrics := p.metricsCollector.CurrentMetrics()
if currentMetrics.ProcessedFiles%10 == 0 && p.metricsReporter != nil {
logger := shared.GetLogger()
logger.Info(p.metricsReporter.ReportProgress())
}
}
} }
// sendFiles sends files to the worker channels with back-pressure handling. // sendFiles sends files to the worker channels with back-pressure handling.
@@ -78,15 +126,88 @@ func (p *Processor) sendFiles(ctx context.Context, files []string, fileCh chan s
// Wait for channel space if needed // Wait for channel space if needed
p.backpressure.WaitForChannelSpace(ctx, fileCh, nil) p.backpressure.WaitForChannelSpace(ctx, fileCh, nil)
if err := shared.CheckContextCancellation(ctx, shared.CLIMsgFileProcessingWorker); err != nil {
return fmt.Errorf("context check failed: %w", err)
}
select { select {
case <-ctx.Done():
return ctx.Err()
case fileCh <- fp: case fileCh <- fp:
case <-ctx.Done():
if err := shared.CheckContextCancellation(ctx, shared.CLIMsgFileProcessingWorker); err != nil {
return fmt.Errorf("context cancellation during channel send: %w", err)
}
return errors.New("context canceled during channel send")
} }
} }
return nil return nil
} }
// processFileWithMetrics wraps the file processing with detailed metrics collection.
func (p *Processor) processFileWithMetrics(
ctx context.Context,
filePath string,
writeCh chan fileproc.WriteRequest,
absRoot string,
) (fileSize int64, format string, success bool, err error) {
// Get file info
fileInfo, statErr := os.Stat(filePath)
if statErr != nil {
return 0, "", false, fmt.Errorf("getting file info for %s: %w", filePath, statErr)
}
fileSize = fileInfo.Size()
// Detect format from file extension
format = filepath.Ext(filePath)
if format != "" && format[0] == '.' {
format = format[1:] // Remove the dot
}
// Use the existing resource monitor-aware processing
err = fileproc.ProcessFileWithMonitor(ctx, filePath, writeCh, absRoot, p.resourceMonitor)
// Check if processing was successful
select {
case <-ctx.Done():
return fileSize, format, false, fmt.Errorf("file processing worker canceled: %w", ctx.Err())
default:
if err != nil {
return fileSize, format, false, fmt.Errorf("processing file %s: %w", filePath, err)
}
return fileSize, format, true, nil
}
}
// recordFileResult records the result of file processing in metrics.
func (p *Processor) recordFileResult(
filePath string,
fileSize int64,
format string,
success bool,
skipped bool,
skipReason string,
err error,
) {
if p.metricsCollector == nil {
return // No metrics collector, skip recording
}
result := metrics.FileProcessingResult{
FilePath: filePath,
FileSize: fileSize,
Format: format,
Success: success,
Error: err,
Skipped: skipped,
SkipReason: skipReason,
}
p.metricsCollector.RecordFileProcessed(result)
}
// waitForCompletion waits for all workers to complete. // waitForCompletion waits for all workers to complete.
func (p *Processor) waitForCompletion( func (p *Processor) waitForCompletion(
wg *sync.WaitGroup, wg *sync.WaitGroup,

138
cli/ui.go
View File

@@ -1,3 +1,4 @@
// Package cli provides command-line interface functionality for gibidify.
package cli package cli
import ( import (
@@ -9,13 +10,14 @@ import (
"github.com/fatih/color" "github.com/fatih/color"
"github.com/schollz/progressbar/v3" "github.com/schollz/progressbar/v3"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// UIManager handles CLI user interface elements. // UIManager handles CLI user interface elements.
type UIManager struct { type UIManager struct {
enableColors bool enableColors bool
enableProgress bool enableProgress bool
silentMode bool
progressBar *progressbar.ProgressBar progressBar *progressbar.ProgressBar
output io.Writer output io.Writer
} }
@@ -40,43 +42,42 @@ func (ui *UIManager) SetProgressOutput(enabled bool) {
ui.enableProgress = enabled ui.enableProgress = enabled
} }
// SetSilentMode enables or disables all UI output.
func (ui *UIManager) SetSilentMode(silent bool) {
ui.silentMode = silent
if silent {
ui.output = io.Discard
} else {
ui.output = os.Stderr
}
}
// StartProgress initializes a progress bar for file processing. // StartProgress initializes a progress bar for file processing.
func (ui *UIManager) StartProgress(total int, description string) { func (ui *UIManager) StartProgress(total int, description string) {
if !ui.enableProgress || total <= 0 { if !ui.enableProgress || total <= 0 {
return return
} }
// Set progress bar theme based on color support
var theme progressbar.Theme
if ui.enableColors {
theme = progressbar.Theme{
Saucer: color.GreenString("█"),
SaucerHead: color.GreenString("█"),
SaucerPadding: " ",
BarStart: "[",
BarEnd: "]",
}
} else {
theme = progressbar.Theme{
Saucer: "█",
SaucerHead: "█",
SaucerPadding: " ",
BarStart: "[",
BarEnd: "]",
}
}
ui.progressBar = progressbar.NewOptions( ui.progressBar = progressbar.NewOptions(
total, total,
progressbar.OptionSetWriter(ui.output), progressbar.OptionSetWriter(ui.output),
progressbar.OptionSetDescription(description), progressbar.OptionSetDescription(description),
progressbar.OptionSetTheme(theme), progressbar.OptionSetTheme(
progressbar.Theme{
Saucer: color.GreenString(shared.UIProgressBarChar),
SaucerHead: color.GreenString(shared.UIProgressBarChar),
SaucerPadding: " ",
BarStart: "[",
BarEnd: "]",
},
),
progressbar.OptionShowCount(), progressbar.OptionShowCount(),
progressbar.OptionShowIts(), progressbar.OptionShowIts(),
progressbar.OptionSetWidth(40), progressbar.OptionSetWidth(40),
progressbar.OptionThrottle(100*time.Millisecond), progressbar.OptionThrottle(100*time.Millisecond),
progressbar.OptionOnCompletion( progressbar.OptionOnCompletion(
func() { func() {
//nolint:errcheck // UI output, errors don't affect processing
_, _ = fmt.Fprint(ui.output, "\n") _, _ = fmt.Fprint(ui.output, "\n")
}, },
), ),
@@ -99,49 +100,62 @@ func (ui *UIManager) FinishProgress() {
} }
} }
// writeMessage writes a formatted message with optional colorization. // PrintSuccess prints a success message in green.
// It handles color enablement, formatting, writing to output, and error logging. func (ui *UIManager) PrintSuccess(format string, args ...any) {
func (ui *UIManager) writeMessage( if ui.silentMode {
icon, methodName, format string, return
colorFunc func(string, ...interface{}) string, }
args ...interface{}, if ui.enableColors {
) { color.Green("✓ "+format, args...)
msg := icon + " " + format
var output string
if ui.enableColors && colorFunc != nil {
output = colorFunc(msg, args...)
} else { } else {
output = fmt.Sprintf(msg, args...) ui.printf("✓ "+format+"\n", args...)
}
if _, err := fmt.Fprintf(ui.output, "%s\n", output); err != nil {
gibidiutils.LogError(fmt.Sprintf("UIManager.%s: failed to write to output", methodName), err)
} }
} }
// PrintSuccess prints a success message in green (to ui.output if set). // PrintError prints an error message in red.
func (ui *UIManager) PrintSuccess(format string, args ...interface{}) { func (ui *UIManager) PrintError(format string, args ...any) {
ui.writeMessage(gibidiutils.IconSuccess, "PrintSuccess", format, color.GreenString, args...) if ui.silentMode {
return
}
if ui.enableColors {
color.Red("✗ "+format, args...)
} else {
ui.printf("✗ "+format+"\n", args...)
}
} }
// PrintError prints an error message in red (to ui.output if set). // PrintWarning prints a warning message in yellow.
func (ui *UIManager) PrintError(format string, args ...interface{}) { func (ui *UIManager) PrintWarning(format string, args ...any) {
ui.writeMessage(gibidiutils.IconError, "PrintError", format, color.RedString, args...) if ui.silentMode {
return
}
if ui.enableColors {
color.Yellow("⚠ "+format, args...)
} else {
ui.printf("⚠ "+format+"\n", args...)
}
} }
// PrintWarning prints a warning message in yellow (to ui.output if set). // PrintInfo prints an info message in blue.
func (ui *UIManager) PrintWarning(format string, args ...interface{}) { func (ui *UIManager) PrintInfo(format string, args ...any) {
ui.writeMessage(gibidiutils.IconWarning, "PrintWarning", format, color.YellowString, args...) if ui.silentMode {
} return
}
// PrintInfo prints an info message in blue (to ui.output if set). if ui.enableColors {
func (ui *UIManager) PrintInfo(format string, args ...interface{}) { //nolint:errcheck // UI output, errors don't affect processing
ui.writeMessage(gibidiutils.IconInfo, "PrintInfo", format, color.BlueString, args...) color.Blue(" "+format, args...)
} else {
ui.printf(" "+format+"\n", args...)
}
} }
// PrintHeader prints a header message in bold. // PrintHeader prints a header message in bold.
func (ui *UIManager) PrintHeader(format string, args ...interface{}) { func (ui *UIManager) PrintHeader(format string, args ...any) {
if ui.silentMode {
return
}
if ui.enableColors { if ui.enableColors {
//nolint:errcheck // UI output, errors don't affect processing
_, _ = color.New(color.Bold).Fprintf(ui.output, format+"\n", args...) _, _ = color.New(color.Bold).Fprintf(ui.output, format+"\n", args...)
} else { } else {
ui.printf(format+"\n", args...) ui.printf(format+"\n", args...)
@@ -150,11 +164,6 @@ func (ui *UIManager) PrintHeader(format string, args ...interface{}) {
// isColorTerminal checks if the terminal supports colors. // isColorTerminal checks if the terminal supports colors.
func isColorTerminal() bool { func isColorTerminal() bool {
// Check if FORCE_COLOR is set
if os.Getenv("FORCE_COLOR") != "" {
return true
}
// Check common environment variables // Check common environment variables
term := os.Getenv("TERM") term := os.Getenv("TERM")
if term == "" || term == "dumb" { if term == "" || term == "dumb" {
@@ -164,7 +173,7 @@ func isColorTerminal() bool {
// Check for CI environments that typically don't support colors // Check for CI environments that typically don't support colors
if os.Getenv("CI") != "" { if os.Getenv("CI") != "" {
// GitHub Actions supports colors // GitHub Actions supports colors
if os.Getenv("GITHUB_ACTIONS") == "true" { if os.Getenv("GITHUB_ACTIONS") == shared.LiteralTrue {
return true return true
} }
// Most other CI systems don't // Most other CI systems don't
@@ -176,7 +185,13 @@ func isColorTerminal() bool {
return false return false
} }
return true // Check if FORCE_COLOR is set
if os.Getenv("FORCE_COLOR") != "" {
return true
}
// Default to true for interactive terminals
return isInteractiveTerminal()
} }
// isInteractiveTerminal checks if we're running in an interactive terminal. // isInteractiveTerminal checks if we're running in an interactive terminal.
@@ -186,10 +201,11 @@ func isInteractiveTerminal() bool {
if err != nil { if err != nil {
return false return false
} }
return (fileInfo.Mode() & os.ModeCharDevice) != 0 return (fileInfo.Mode() & os.ModeCharDevice) != 0
} }
// printf is a helper that ignores printf errors (for UI output). // printf is a helper that ignores printf errors (for UI output).
func (ui *UIManager) printf(format string, args ...interface{}) { func (ui *UIManager) printf(format string, args ...any) {
_, _ = fmt.Fprintf(ui.output, format, args...) _, _ = fmt.Fprintf(ui.output, format, args...)
} }

View File

@@ -1,109 +0,0 @@
package cli
import (
"bytes"
"os"
"testing"
"github.com/fatih/color"
"github.com/stretchr/testify/assert"
)
func TestNewUIManager(t *testing.T) {
tests := []struct {
name string
env terminalEnvSetup
expectedColors bool
expectedProgress bool
}{
{
name: "default terminal",
env: envDefaultTerminal,
expectedColors: true,
expectedProgress: false, // Not a tty in test environment
},
{
name: "dumb terminal",
env: envDumbTerminal,
expectedColors: false,
expectedProgress: false,
},
{
name: "CI environment without GitHub Actions",
env: envCIWithoutGitHub,
expectedColors: false,
expectedProgress: false,
},
{
name: "GitHub Actions CI",
env: envGitHubActions,
expectedColors: true,
expectedProgress: false,
},
{
name: "NO_COLOR set",
env: envNoColor,
expectedColors: false,
expectedProgress: false,
},
{
name: "FORCE_COLOR set",
env: envForceColor,
expectedColors: true,
expectedProgress: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
tt.env.apply(t)
ui := NewUIManager()
assert.NotNil(t, ui)
assert.NotNil(t, ui.output)
assert.Equal(t, tt.expectedColors, ui.enableColors, "color state mismatch")
assert.Equal(t, tt.expectedProgress, ui.enableProgress, "progress state mismatch")
})
}
}
func TestSetColorOutput(t *testing.T) {
// Capture original color.NoColor state and restore after test
orig := color.NoColor
defer func() { color.NoColor = orig }()
ui := &UIManager{output: os.Stderr}
// Test enabling colors
ui.SetColorOutput(true)
assert.False(t, color.NoColor)
assert.True(t, ui.enableColors)
// Test disabling colors
ui.SetColorOutput(false)
assert.True(t, color.NoColor)
assert.False(t, ui.enableColors)
}
func TestSetProgressOutput(t *testing.T) {
ui := &UIManager{output: os.Stderr}
// Test enabling progress
ui.SetProgressOutput(true)
assert.True(t, ui.enableProgress)
// Test disabling progress
ui.SetProgressOutput(false)
assert.False(t, ui.enableProgress)
}
func TestPrintf(t *testing.T) {
buf := &bytes.Buffer{}
ui := &UIManager{
output: buf,
}
ui.printf("Test %s %d", "output", 123)
assert.Equal(t, "Test output 123", buf.String())
}

View File

@@ -1,245 +0,0 @@
package cli
import (
"bytes"
"strings"
"testing"
"github.com/fatih/color"
"github.com/stretchr/testify/assert"
"github.com/ivuorinen/gibidify/gibidiutils"
)
func TestPrintSuccess(t *testing.T) {
tests := []struct {
name string
enableColors bool
format string
args []interface{}
expectSymbol string
}{
{
name: testWithColors,
enableColors: true,
format: "Operation %s",
args: []interface{}{"completed"},
expectSymbol: gibidiutils.IconSuccess,
},
{
name: testWithoutColors,
enableColors: false,
format: "Operation %s",
args: []interface{}{"completed"},
expectSymbol: gibidiutils.IconSuccess,
},
{
name: "no arguments",
enableColors: true,
format: "Success",
args: nil,
expectSymbol: gibidiutils.IconSuccess,
},
}
for _, tt := range tests {
t.Run(
tt.name, func(t *testing.T) {
buf := &bytes.Buffer{}
ui := &UIManager{
enableColors: tt.enableColors,
output: buf,
}
prev := color.NoColor
color.NoColor = !tt.enableColors
defer func() { color.NoColor = prev }()
ui.PrintSuccess(tt.format, tt.args...)
output := buf.String()
assert.Contains(t, output, tt.expectSymbol)
if len(tt.args) > 0 {
assert.Contains(t, output, "completed")
}
},
)
}
}
func TestPrintError(t *testing.T) {
tests := []struct {
name string
enableColors bool
format string
args []interface{}
expectSymbol string
}{
{
name: testWithColors,
enableColors: true,
format: "Failed to %s",
args: []interface{}{"process"},
expectSymbol: gibidiutils.IconError,
},
{
name: testWithoutColors,
enableColors: false,
format: "Failed to %s",
args: []interface{}{"process"},
expectSymbol: gibidiutils.IconError,
},
}
for _, tt := range tests {
t.Run(
tt.name, func(t *testing.T) {
buf := &bytes.Buffer{}
ui := &UIManager{
enableColors: tt.enableColors,
output: buf,
}
prev := color.NoColor
color.NoColor = !tt.enableColors
defer func() { color.NoColor = prev }()
ui.PrintError(tt.format, tt.args...)
output := buf.String()
assert.Contains(t, output, tt.expectSymbol)
if len(tt.args) > 0 {
assert.Contains(t, output, "process")
}
},
)
}
}
func TestPrintWarning(t *testing.T) {
buf := &bytes.Buffer{}
ui := &UIManager{
enableColors: true,
output: buf,
}
ui.PrintWarning("This is a %s", "warning")
output := buf.String()
assert.Contains(t, output, gibidiutils.IconWarning)
}
func TestPrintInfo(t *testing.T) {
// Capture original color.NoColor state and restore after test
orig := color.NoColor
defer func() { color.NoColor = orig }()
buf := &bytes.Buffer{}
ui := &UIManager{
enableColors: true,
output: buf,
}
color.NoColor = false
ui.PrintInfo("Information: %d items", 42)
output := buf.String()
assert.Contains(t, output, gibidiutils.IconInfo)
assert.Contains(t, output, "42")
}
func TestPrintHeader(t *testing.T) {
tests := []struct {
name string
enableColors bool
format string
args []interface{}
}{
{
name: testWithColors,
enableColors: true,
format: "Header %s",
args: []interface{}{"Title"},
},
{
name: testWithoutColors,
enableColors: false,
format: "Header %s",
args: []interface{}{"Title"},
},
}
for _, tt := range tests {
t.Run(
tt.name, func(t *testing.T) {
// Capture original color.NoColor state and restore after test
orig := color.NoColor
defer func() { color.NoColor = orig }()
buf := &bytes.Buffer{}
ui := &UIManager{
enableColors: tt.enableColors,
output: buf,
}
color.NoColor = !tt.enableColors
ui.PrintHeader(tt.format, tt.args...)
output := buf.String()
assert.Contains(t, output, "Title")
},
)
}
}
// Test that all print methods handle newlines correctly
func TestPrintMethodsNewlines(t *testing.T) {
tests := []struct {
name string
method func(*UIManager, string, ...interface{})
symbol string
}{
{
name: "PrintSuccess",
method: (*UIManager).PrintSuccess,
symbol: gibidiutils.IconSuccess,
},
{
name: "PrintError",
method: (*UIManager).PrintError,
symbol: gibidiutils.IconError,
},
{
name: "PrintWarning",
method: (*UIManager).PrintWarning,
symbol: gibidiutils.IconWarning,
},
{
name: "PrintInfo",
method: (*UIManager).PrintInfo,
symbol: gibidiutils.IconInfo,
},
}
for _, tt := range tests {
t.Run(
tt.name, func(t *testing.T) {
// Disable colors for consistent testing
oldNoColor := color.NoColor
color.NoColor = true
defer func() { color.NoColor = oldNoColor }()
buf := &bytes.Buffer{}
ui := &UIManager{
enableColors: false,
output: buf,
}
tt.method(ui, "Test message")
output := buf.String()
assert.True(t, strings.HasSuffix(output, "\n"))
assert.Contains(t, output, tt.symbol)
},
)
}
}

View File

@@ -1,147 +0,0 @@
package cli
import (
"bytes"
"testing"
"github.com/stretchr/testify/assert"
)
func TestStartProgress(t *testing.T) {
tests := []struct {
name string
total int
description string
enabled bool
expectBar bool
}{
{
name: "progress enabled with valid total",
total: 100,
description: testProcessingMsg,
enabled: true,
expectBar: true,
},
{
name: "progress disabled",
total: 100,
description: testProcessingMsg,
enabled: false,
expectBar: false,
},
{
name: "zero total",
total: 0,
description: testProcessingMsg,
enabled: true,
expectBar: false,
},
{
name: "negative total",
total: -5,
description: testProcessingMsg,
enabled: true,
expectBar: false,
},
}
for _, tt := range tests {
t.Run(
tt.name, func(t *testing.T) {
ui := &UIManager{
enableProgress: tt.enabled,
output: &bytes.Buffer{},
}
ui.StartProgress(tt.total, tt.description)
if tt.expectBar {
assert.NotNil(t, ui.progressBar)
} else {
assert.Nil(t, ui.progressBar)
}
},
)
}
}
func TestUpdateProgress(t *testing.T) {
tests := []struct {
name string
setupBar bool
enabledProg bool
expectUpdate bool
}{
{
name: "with progress bar",
setupBar: true,
enabledProg: true,
expectUpdate: true,
},
{
name: "without progress bar",
setupBar: false,
enabledProg: false,
expectUpdate: false,
},
}
for _, tt := range tests {
t.Run(
tt.name, func(_ *testing.T) {
ui := &UIManager{
enableProgress: tt.enabledProg,
output: &bytes.Buffer{},
}
if tt.setupBar {
ui.StartProgress(10, "Test")
}
// Should not panic
ui.UpdateProgress(1)
// Multiple updates should not panic
ui.UpdateProgress(2)
ui.UpdateProgress(3)
},
)
}
}
func TestFinishProgress(t *testing.T) {
tests := []struct {
name string
setupBar bool
}{
{
name: "with progress bar",
setupBar: true,
},
{
name: "without progress bar",
setupBar: false,
},
}
for _, tt := range tests {
t.Run(
tt.name, func(t *testing.T) {
ui := &UIManager{
enableProgress: true,
output: &bytes.Buffer{},
}
if tt.setupBar {
ui.StartProgress(10, "Test")
}
// Should not panic
ui.FinishProgress()
// Bar should be cleared
assert.Nil(t, ui.progressBar)
},
)
}
}

View File

@@ -1,62 +0,0 @@
package cli
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestIsColorTerminal(t *testing.T) {
tests := []struct {
name string
env terminalEnvSetup
expected bool
}{
{
name: "dumb terminal",
env: envDumbTerminal,
expected: false,
},
{
name: "empty TERM",
env: envEmptyTerm,
expected: false,
},
{
name: "CI without GitHub Actions",
env: envCIWithoutGitHub,
expected: false,
},
{
name: "GitHub Actions",
env: envGitHubActions,
expected: true,
},
{
name: "NO_COLOR set",
env: envNoColor,
expected: false,
},
{
name: "FORCE_COLOR set",
env: envForceColor,
expected: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
tt.env.apply(t)
result := isColorTerminal()
assert.Equal(t, tt.expected, result)
})
}
}
func TestIsInteractiveTerminal(t *testing.T) {
// This function checks if stderr is a terminal
// In test environment, it will typically return false
result := isInteractiveTerminal()
assert.False(t, result)
}

531
cli/ui_test.go Normal file
View File

@@ -0,0 +1,531 @@
package cli
import (
"os"
"strings"
"testing"
"github.com/ivuorinen/gibidify/shared"
)
func TestNewUIManager(t *testing.T) {
ui := NewUIManager()
if ui == nil {
t.Error("NewUIManager() returned nil")
return
}
if ui.output == nil {
t.Error("NewUIManager() did not set output")
return
}
if ui.output != os.Stderr {
t.Error("NewUIManager() should default output to os.Stderr")
}
}
func TestUIManagerSetColorOutput(t *testing.T) {
ui := NewUIManager()
// Test enabling colors
ui.SetColorOutput(true)
if !ui.enableColors {
t.Error("SetColorOutput(true) did not enable colors")
}
// Test disabling colors
ui.SetColorOutput(false)
if ui.enableColors {
t.Error("SetColorOutput(false) did not disable colors")
}
}
func TestUIManagerSetProgressOutput(t *testing.T) {
ui := NewUIManager()
// Test enabling progress
ui.SetProgressOutput(true)
if !ui.enableProgress {
t.Error("SetProgressOutput(true) did not enable progress")
}
// Test disabling progress
ui.SetProgressOutput(false)
if ui.enableProgress {
t.Error("SetProgressOutput(false) did not disable progress")
}
}
func TestUIManagerStartProgress(t *testing.T) {
tests := []struct {
name string
total int
description string
enabled bool
expectBar bool
}{
{
name: "valid progress with enabled progress",
total: 10,
description: shared.TestProgressMessage,
enabled: true,
expectBar: true,
},
{
name: "disabled progress should not create bar",
total: 10,
description: shared.TestProgressMessage,
enabled: false,
expectBar: false,
},
{
name: "zero total should not create bar",
total: 0,
description: shared.TestProgressMessage,
enabled: true,
expectBar: false,
},
{
name: "negative total should not create bar",
total: -1,
description: shared.TestProgressMessage,
enabled: true,
expectBar: false,
},
}
for _, tt := range tests {
t.Run(
tt.name, func(t *testing.T) {
ui, _ := createTestUI() //nolint:errcheck // Test helper output buffer not used in this test
ui.SetProgressOutput(tt.enabled)
ui.StartProgress(tt.total, tt.description)
if tt.expectBar && ui.progressBar == nil {
t.Error("StartProgress() should have created progress bar but didn't")
}
if !tt.expectBar && ui.progressBar != nil {
t.Error("StartProgress() should not have created progress bar but did")
}
},
)
}
}
func TestUIManagerUpdateProgress(t *testing.T) {
ui, _ := createTestUI() //nolint:errcheck // Test helper output buffer not used in this test
ui.SetProgressOutput(true)
// Test with no progress bar (should not panic)
ui.UpdateProgress(1)
// Test with progress bar
ui.StartProgress(10, "Test progress")
if ui.progressBar == nil {
t.Fatal("StartProgress() did not create progress bar")
}
// Should not panic
ui.UpdateProgress(1)
ui.UpdateProgress(5)
}
func TestUIManagerFinishProgress(t *testing.T) {
ui, _ := createTestUI() //nolint:errcheck // Test helper output buffer not used in this test
ui.SetProgressOutput(true)
// Test with no progress bar (should not panic)
ui.FinishProgress()
// Test with progress bar
ui.StartProgress(10, "Test progress")
if ui.progressBar == nil {
t.Fatal("StartProgress() did not create progress bar")
}
ui.FinishProgress()
if ui.progressBar != nil {
t.Error("FinishProgress() should have cleared progress bar")
}
}
// testPrintMethod is a helper function to test UI print methods without duplication.
type printMethodTest struct {
name string
enableColors bool
format string
args []any
expectedText string
}
func testPrintMethod(
t *testing.T,
methodName string,
printFunc func(*UIManager, string, ...any),
tests []printMethodTest,
) {
t.Helper()
for _, tt := range tests {
t.Run(
tt.name, func(t *testing.T) {
ui, output := createTestUI()
ui.SetColorOutput(tt.enableColors)
printFunc(ui, tt.format, tt.args...)
if !tt.enableColors {
outputStr := output.String()
if !strings.Contains(outputStr, tt.expectedText) {
t.Errorf("%s() output %q should contain %q", methodName, outputStr, tt.expectedText)
}
}
},
)
}
// Test color method separately (doesn't capture output but shouldn't panic)
t.Run(
methodName+" with colors should not panic", func(_ *testing.T) {
ui, _ := createTestUI() //nolint:errcheck // Test helper output buffer not used in this test
ui.SetColorOutput(true)
// Should not panic
printFunc(ui, "Test message")
},
)
}
func TestUIManagerPrintSuccess(t *testing.T) {
tests := []printMethodTest{
{
name: "success without colors",
enableColors: false,
format: "Operation completed successfully",
args: []any{},
expectedText: "✓ Operation completed successfully",
},
{
name: "success with args without colors",
enableColors: false,
format: "Processed %d files in %s",
args: []any{5, "project"},
expectedText: "✓ Processed 5 files in project",
},
}
testPrintMethod(
t, "PrintSuccess", func(ui *UIManager, format string, args ...any) {
ui.PrintSuccess(format, args...)
}, tests,
)
}
func TestUIManagerPrintError(t *testing.T) {
tests := []printMethodTest{
{
name: "error without colors",
enableColors: false,
format: "Operation failed",
args: []any{},
expectedText: "✗ Operation failed",
},
{
name: "error with args without colors",
enableColors: false,
format: "Failed to process %d files",
args: []any{3},
expectedText: "✗ Failed to process 3 files",
},
}
testPrintMethod(
t, "PrintError", func(ui *UIManager, format string, args ...any) {
ui.PrintError(format, args...)
}, tests,
)
}
func TestUIManagerPrintWarning(t *testing.T) {
tests := []printMethodTest{
{
name: "warning without colors",
enableColors: false,
format: "This is a warning",
args: []any{},
expectedText: "⚠ This is a warning",
},
{
name: "warning with args without colors",
enableColors: false,
format: "Found %d potential issues",
args: []any{2},
expectedText: "⚠ Found 2 potential issues",
},
}
testPrintMethod(
t, "PrintWarning", func(ui *UIManager, format string, args ...any) {
ui.PrintWarning(format, args...)
}, tests,
)
}
func TestUIManagerPrintInfo(t *testing.T) {
tests := []printMethodTest{
{
name: "info without colors",
enableColors: false,
format: "Information message",
args: []any{},
expectedText: " Information message",
},
{
name: "info with args without colors",
enableColors: false,
format: "Processing file %s",
args: []any{"example.go"},
expectedText: " Processing file example.go",
},
}
testPrintMethod(
t, "PrintInfo", func(ui *UIManager, format string, args ...any) {
ui.PrintInfo(format, args...)
}, tests,
)
}
func TestUIManagerPrintHeader(t *testing.T) {
tests := []struct {
name string
enableColors bool
format string
args []any
expectedText string
}{
{
name: "header without colors",
enableColors: false,
format: "Main Header",
args: []any{},
expectedText: "Main Header",
},
{
name: "header with args without colors",
enableColors: false,
format: "Processing %s Module",
args: []any{"CLI"},
expectedText: "Processing CLI Module",
},
{
name: "header with colors",
enableColors: true,
format: "Build Results",
args: []any{},
expectedText: "Build Results",
},
}
for _, tt := range tests {
t.Run(
tt.name, func(t *testing.T) {
ui, output := createTestUI()
ui.SetColorOutput(tt.enableColors)
ui.PrintHeader(tt.format, tt.args...)
outputStr := output.String()
if !strings.Contains(outputStr, tt.expectedText) {
t.Errorf("PrintHeader() output %q should contain %q", outputStr, tt.expectedText)
}
},
)
}
}
// colorTerminalTestCase represents a test case for color terminal detection.
type colorTerminalTestCase struct {
name string
term string
ci string
githubActions string
noColor string
forceColor string
expected bool
}
// clearColorTerminalEnvVars clears all environment variables used for terminal color detection.
func clearColorTerminalEnvVars(t *testing.T) {
t.Helper()
envVars := []string{"TERM", "CI", "GITHUB_ACTIONS", "NO_COLOR", "FORCE_COLOR"}
for _, envVar := range envVars {
if err := os.Unsetenv(envVar); err != nil {
t.Logf("Failed to unset %s: %v", envVar, err)
}
}
}
// setColorTerminalTestEnv sets up environment variables for a test case.
func setColorTerminalTestEnv(t *testing.T, testCase colorTerminalTestCase) {
t.Helper()
envSettings := map[string]string{
"TERM": testCase.term,
"CI": testCase.ci,
"GITHUB_ACTIONS": testCase.githubActions,
"NO_COLOR": testCase.noColor,
"FORCE_COLOR": testCase.forceColor,
}
for key, value := range envSettings {
if value != "" {
t.Setenv(key, value)
}
}
}
func TestIsColorTerminal(t *testing.T) {
// Save original environment
originalEnv := map[string]string{
"TERM": os.Getenv("TERM"),
"CI": os.Getenv("CI"),
"GITHUB_ACTIONS": os.Getenv("GITHUB_ACTIONS"),
"NO_COLOR": os.Getenv("NO_COLOR"),
"FORCE_COLOR": os.Getenv("FORCE_COLOR"),
}
defer func() {
// Restore original environment
for key, value := range originalEnv {
setEnvOrUnset(key, value)
}
}()
tests := []colorTerminalTestCase{
{
name: "dumb terminal",
term: "dumb",
expected: false,
},
{
name: "empty term",
term: "",
expected: false,
},
{
name: "github actions with CI",
term: shared.TestTerminalXterm256,
ci: "true",
githubActions: "true",
expected: true,
},
{
name: "CI without github actions",
term: shared.TestTerminalXterm256,
ci: "true",
expected: false,
},
{
name: "NO_COLOR set",
term: shared.TestTerminalXterm256,
noColor: "1",
expected: false,
},
{
name: "FORCE_COLOR set",
term: shared.TestTerminalXterm256,
forceColor: "1",
expected: true,
},
}
for _, tt := range tests {
t.Run(
tt.name, func(t *testing.T) {
clearColorTerminalEnvVars(t)
setColorTerminalTestEnv(t, tt)
result := isColorTerminal()
if result != tt.expected {
t.Errorf("isColorTerminal() = %v, want %v", result, tt.expected)
}
},
)
}
}
func TestIsInteractiveTerminal(_ *testing.T) {
// This test is limited because we can't easily mock os.Stderr.Stat()
// but we can at least verify it doesn't panic and returns a boolean
result := isInteractiveTerminal()
// Result should be a boolean (true or false, both are valid)
// result is already a boolean, so this check is always satisfied
_ = result
}
func TestUIManagerprintf(t *testing.T) {
ui, output := createTestUI()
ui.printf("Hello %s", "world")
expected := "Hello world"
if output.String() != expected {
t.Errorf("printf() = %q, want %q", output.String(), expected)
}
}
// Helper function to set environment variable or unset if empty.
func setEnvOrUnset(key, value string) {
if value == "" {
if err := os.Unsetenv(key); err != nil {
// In tests, environment variable errors are not critical,
// but we should still handle them to avoid linting issues
_ = err // explicitly ignore error
}
} else {
if err := os.Setenv(key, value); err != nil {
// In tests, environment variable errors are not critical,
// but we should still handle them to avoid linting issues
_ = err // explicitly ignore error
}
}
}
// Integration test for UI workflow.
func TestUIManagerIntegration(t *testing.T) {
ui, output := createTestUI() //nolint:errcheck // Test helper, output buffer is used
ui.SetColorOutput(false) // Disable colors for consistent output
ui.SetProgressOutput(false) // Disable progress for testing
// Simulate a complete UI workflow
ui.PrintHeader("Starting Processing")
ui.PrintInfo("Initializing system")
ui.StartProgress(3, shared.TestProgressMessage)
ui.UpdateProgress(1)
ui.PrintInfo("Processing file 1")
ui.UpdateProgress(1)
ui.PrintWarning("Skipping invalid file")
ui.UpdateProgress(1)
ui.FinishProgress()
ui.PrintSuccess("Processing completed successfully")
outputStr := output.String()
expectedStrings := []string{
"Starting Processing",
" Initializing system",
" Processing file 1",
"⚠ Skipping invalid file",
"✓ Processing completed successfully",
}
for _, expected := range expectedStrings {
if !strings.Contains(outputStr, expected) {
t.Errorf("Integration test output missing expected string: %q\nFull output:\n%s", expected, outputStr)
}
}
}

View File

@@ -9,38 +9,60 @@ import (
"strings" "strings"
"github.com/ivuorinen/gibidify/benchmark" "github.com/ivuorinen/gibidify/benchmark"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
var ( var (
sourceDir = flag.String("source", "", "Source directory to benchmark (uses temp files if empty)") sourceDir = flag.String(
benchmarkType = flag.String("type", "all", "Benchmark type: all, collection, processing, concurrency, format") shared.CLIArgSource, "", "Source directory to benchmark (uses temp files if empty)",
format = flag.String("format", "json", "Output format for processing benchmarks") )
concurrency = flag.Int("concurrency", runtime.NumCPU(), "Concurrency level for processing benchmarks") benchmarkType = flag.String(
concurrencyList = flag.String("concurrency-list", "1,2,4,8", "Comma-separated list of concurrency levels") "type", shared.CLIArgAll, "Benchmark type: all, collection, processing, concurrency, format",
formatList = flag.String("format-list", "json,yaml,markdown", "Comma-separated list of formats") )
numFiles = flag.Int("files", 100, "Number of files to create for benchmarks") format = flag.String(
shared.CLIArgFormat, shared.FormatJSON, "Output format for processing benchmarks",
)
concurrency = flag.Int(
shared.CLIArgConcurrency, runtime.NumCPU(), "Concurrency level for processing benchmarks",
)
concurrencyList = flag.String(
"concurrency-list", shared.TestConcurrencyList, "Comma-separated list of concurrency levels",
)
formatList = flag.String(
"format-list", shared.TestFormatList, "Comma-separated list of formats",
)
numFiles = flag.Int("files", shared.BenchmarkDefaultFileCount, "Number of files to create for benchmarks")
) )
func main() { func main() {
flag.Parse() flag.Parse()
if err := runBenchmarks(); err != nil { if err := runBenchmarks(); err != nil {
//goland:noinspection GoUnhandledErrorResult
_, _ = fmt.Fprintf(os.Stderr, "Benchmark failed: %v\n", err) _, _ = fmt.Fprintf(os.Stderr, "Benchmark failed: %v\n", err)
os.Exit(1) os.Exit(1)
} }
} }
func runBenchmarks() error { func runBenchmarks() error {
fmt.Printf("Running gibidify benchmarks...\n") //nolint:errcheck // Benchmark informational output, errors don't affect benchmark results
fmt.Printf("Source: %s\n", getSourceDescription()) _, _ = fmt.Println("Running gibidify benchmarks...")
fmt.Printf("Type: %s\n", *benchmarkType) //nolint:errcheck // Benchmark informational output, errors don't affect benchmark results
fmt.Printf("CPU cores: %d\n", runtime.NumCPU()) _, _ = fmt.Printf("Source: %s\n", getSourceDescription())
fmt.Println() //nolint:errcheck // Benchmark informational output, errors don't affect benchmark results
_, _ = fmt.Printf("Type: %s\n", *benchmarkType)
//nolint:errcheck // Benchmark informational output, errors don't affect benchmark results
_, _ = fmt.Printf("CPU cores: %d\n", runtime.NumCPU())
//nolint:errcheck // Benchmark informational output, errors don't affect benchmark results
_, _ = fmt.Println()
switch *benchmarkType { switch *benchmarkType {
case "all": case shared.CLIArgAll:
return benchmark.RunAllBenchmarks(*sourceDir) if err := benchmark.RunAllBenchmarks(*sourceDir); err != nil {
return fmt.Errorf("benchmark failed: %w", err)
}
return nil
case "collection": case "collection":
return runCollectionBenchmark() return runCollectionBenchmark()
case "processing": case "processing":
@@ -50,81 +72,79 @@ func runBenchmarks() error {
case "format": case "format":
return runFormatBenchmark() return runFormatBenchmark()
default: default:
return gibidiutils.NewValidationError( return shared.NewValidationError(shared.CodeValidationFormat, "invalid benchmark type: "+*benchmarkType)
gibidiutils.CodeValidationFormat,
"invalid benchmark type: "+*benchmarkType,
)
} }
} }
func runCollectionBenchmark() error { func runCollectionBenchmark() error {
fmt.Println("Running file collection benchmark...") //nolint:errcheck // Benchmark status message, errors don't affect benchmark results
_, _ = fmt.Println(shared.BenchmarkMsgRunningCollection)
result, err := benchmark.FileCollectionBenchmark(*sourceDir, *numFiles) result, err := benchmark.FileCollectionBenchmark(*sourceDir, *numFiles)
if err != nil { if err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, err,
gibidiutils.ErrorTypeProcessing, shared.ErrorTypeProcessing,
gibidiutils.CodeProcessingCollection, shared.CodeProcessingCollection,
"file collection benchmark failed", shared.BenchmarkMsgFileCollectionFailed,
) )
} }
benchmark.PrintResult(result) benchmark.PrintResult(result)
return nil return nil
} }
func runProcessingBenchmark() error { func runProcessingBenchmark() error {
fmt.Printf("Running file processing benchmark (format: %s, concurrency: %d)...\n", *format, *concurrency) //nolint:errcheck // Benchmark status message, errors don't affect benchmark results
_, _ = fmt.Printf("Running file processing benchmark (format: %s, concurrency: %d)...\n", *format, *concurrency)
result, err := benchmark.FileProcessingBenchmark(*sourceDir, *format, *concurrency) result, err := benchmark.FileProcessingBenchmark(*sourceDir, *format, *concurrency)
if err != nil { if err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, err,
gibidiutils.ErrorTypeProcessing, shared.ErrorTypeProcessing,
gibidiutils.CodeProcessingCollection, shared.CodeProcessingCollection,
"file processing benchmark failed", "file processing benchmark failed",
) )
} }
benchmark.PrintResult(result) benchmark.PrintResult(result)
return nil return nil
} }
func runConcurrencyBenchmark() error { func runConcurrencyBenchmark() error {
concurrencyLevels, err := parseConcurrencyList(*concurrencyList) concurrencyLevels, err := parseConcurrencyList(*concurrencyList)
if err != nil { if err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, err, shared.ErrorTypeValidation, shared.CodeValidationFormat, "invalid concurrency list")
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationFormat,
"invalid concurrency list",
)
} }
fmt.Printf("Running concurrency benchmark (format: %s, levels: %v)...\n", *format, concurrencyLevels) //nolint:errcheck // Benchmark status message, errors don't affect benchmark results
_, _ = fmt.Printf("Running concurrency benchmark (format: %s, levels: %v)...\n", *format, concurrencyLevels)
suite, err := benchmark.ConcurrencyBenchmark(*sourceDir, *format, concurrencyLevels) suite, err := benchmark.ConcurrencyBenchmark(*sourceDir, *format, concurrencyLevels)
if err != nil { if err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, err,
gibidiutils.ErrorTypeProcessing, shared.ErrorTypeProcessing,
gibidiutils.CodeProcessingCollection, shared.CodeProcessingCollection,
"concurrency benchmark failed", shared.BenchmarkMsgConcurrencyFailed,
) )
} }
benchmark.PrintSuite(suite) benchmark.PrintSuite(suite)
return nil return nil
} }
func runFormatBenchmark() error { func runFormatBenchmark() error {
formats := parseFormatList(*formatList) formats := parseFormatList(*formatList)
fmt.Printf("Running format benchmark (formats: %v)...\n", formats) //nolint:errcheck // Benchmark status message, errors don't affect benchmark results
_, _ = fmt.Printf("Running format benchmark (formats: %v)...\n", formats)
suite, err := benchmark.FormatBenchmark(*sourceDir, formats) suite, err := benchmark.FormatBenchmark(*sourceDir, formats)
if err != nil { if err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, err, shared.ErrorTypeProcessing, shared.CodeProcessingCollection, shared.BenchmarkMsgFormatFailed,
gibidiutils.ErrorTypeProcessing,
gibidiutils.CodeProcessingCollection,
"format benchmark failed",
) )
} }
benchmark.PrintSuite(suite) benchmark.PrintSuite(suite)
return nil return nil
} }
@@ -132,6 +152,7 @@ func getSourceDescription() string {
if *sourceDir == "" { if *sourceDir == "" {
return fmt.Sprintf("temporary files (%d files)", *numFiles) return fmt.Sprintf("temporary files (%d files)", *numFiles)
} }
return *sourceDir return *sourceDir
} }
@@ -143,28 +164,24 @@ func parseConcurrencyList(list string) ([]int, error) {
part = strings.TrimSpace(part) part = strings.TrimSpace(part)
var level int var level int
if _, err := fmt.Sscanf(part, "%d", &level); err != nil { if _, err := fmt.Sscanf(part, "%d", &level); err != nil {
return nil, gibidiutils.WrapErrorf( return nil, shared.WrapErrorf(
err, err,
gibidiutils.ErrorTypeValidation, shared.ErrorTypeValidation,
gibidiutils.CodeValidationFormat, shared.CodeValidationFormat,
"invalid concurrency level: %s", "invalid concurrency level: %s",
part, part,
) )
} }
if level <= 0 { if level <= 0 {
return nil, gibidiutils.NewValidationError( return nil, shared.NewValidationError(
gibidiutils.CodeValidationFormat, shared.CodeValidationFormat, "concurrency level must be positive: "+part,
"concurrency level must be positive: "+part,
) )
} }
levels = append(levels, level) levels = append(levels, level)
} }
if len(levels) == 0 { if len(levels) == 0 {
return nil, gibidiutils.NewValidationError( return nil, shared.NewValidationError(shared.CodeValidationFormat, "no valid concurrency levels found")
gibidiutils.CodeValidationFormat,
"no valid concurrency levels found",
)
} }
return levels, nil return levels, nil

751
cmd/benchmark/main_test.go Normal file
View File

@@ -0,0 +1,751 @@
package main
import (
"errors"
"flag"
"io"
"os"
"runtime"
"testing"
"github.com/ivuorinen/gibidify/shared"
"github.com/ivuorinen/gibidify/testutil"
)
// Test constants to avoid goconst linting issues.
const (
testJSON = "json"
testMarkdown = "markdown"
testConcurrency = "1,2"
testAll = "all"
testCollection = "collection"
testConcurrencyT = "concurrency"
testNonExistent = "/nonexistent/path/that/should/not/exist"
testFile1 = "test1.txt"
testFile2 = "test2.txt"
testContent1 = "content1"
testContent2 = "content2"
)
func TestParseConcurrencyList(t *testing.T) {
tests := []struct {
name string
input string
want []int
wantErr bool
errContains string
}{
{
name: "valid single value",
input: "4",
want: []int{4},
wantErr: false,
},
{
name: "valid multiple values",
input: shared.TestConcurrencyList,
want: []int{1, 2, 4, 8},
wantErr: false,
},
{
name: "valid with whitespace",
input: " 1 , 2 , 4 , 8 ",
want: []int{1, 2, 4, 8},
wantErr: false,
},
{
name: "valid single large value",
input: "16",
want: []int{16},
wantErr: false,
},
{
name: "empty string",
input: "",
wantErr: true,
errContains: shared.TestMsgInvalidConcurrencyLevel,
},
{
name: "invalid number",
input: "1,abc,4",
wantErr: true,
errContains: shared.TestMsgInvalidConcurrencyLevel,
},
{
name: "zero value",
input: "1,0,4",
wantErr: true,
errContains: "concurrency level must be positive",
},
{
name: "negative value",
input: "1,-2,4",
wantErr: true,
errContains: "concurrency level must be positive",
},
{
name: "only whitespace",
input: " , , ",
wantErr: true,
errContains: shared.TestMsgInvalidConcurrencyLevel,
},
{
name: "large value list",
input: "1,2,4,8,16",
want: []int{1, 2, 4, 8, 16},
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := parseConcurrencyList(tt.input)
if tt.wantErr {
testutil.AssertExpectedError(t, err, "parseConcurrencyList")
if tt.errContains != "" {
testutil.AssertErrorContains(t, err, tt.errContains, "parseConcurrencyList")
}
return
}
testutil.AssertNoError(t, err, "parseConcurrencyList")
if !equalSlices(got, tt.want) {
t.Errorf("parseConcurrencyList() = %v, want %v", got, tt.want)
}
})
}
}
func TestParseFormatList(t *testing.T) {
tests := []struct {
name string
input string
want []string
}{
{
name: "single format",
input: "json",
want: []string{"json"},
},
{
name: "multiple formats",
input: shared.TestFormatList,
want: []string{"json", "yaml", "markdown"},
},
{
name: "formats with whitespace",
input: " json , yaml , markdown ",
want: []string{"json", "yaml", "markdown"},
},
{
name: "empty string",
input: "",
want: []string{},
},
{
name: "empty parts",
input: "json,,yaml",
want: []string{"json", "yaml"},
},
{
name: "only whitespace and commas",
input: " , , ",
want: []string{},
},
{
name: "single format with whitespace",
input: " markdown ",
want: []string{"markdown"},
},
{
name: "duplicate formats",
input: "json,json,yaml",
want: []string{"json", "json", "yaml"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := parseFormatList(tt.input)
if !equalSlices(got, tt.want) {
t.Errorf("parseFormatList() = %v, want %v", got, tt.want)
}
})
}
}
func TestGetSourceDescription(t *testing.T) {
// Save original flag values and reset after test
origSourceDir := sourceDir
origNumFiles := numFiles
defer func() {
sourceDir = origSourceDir
numFiles = origNumFiles
}()
tests := []struct {
name string
sourceDir string
numFiles int
want string
}{
{
name: "empty source directory with default files",
sourceDir: "",
numFiles: 100,
want: "temporary files (100 files)",
},
{
name: "empty source directory with custom files",
sourceDir: "",
numFiles: 50,
want: "temporary files (50 files)",
},
{
name: "non-empty source directory",
sourceDir: "/path/to/source",
numFiles: 100,
want: "/path/to/source",
},
{
name: "current directory",
sourceDir: ".",
numFiles: 100,
want: ".",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Set flag pointers to test values
*sourceDir = tt.sourceDir
*numFiles = tt.numFiles
got := getSourceDescription()
if got != tt.want {
t.Errorf("getSourceDescription() = %v, want %v", got, tt.want)
}
})
}
}
func TestRunCollectionBenchmark(t *testing.T) {
restore := testutil.SuppressLogs(t)
defer restore()
// Save original flag values
origSourceDir := sourceDir
origNumFiles := numFiles
defer func() {
sourceDir = origSourceDir
numFiles = origNumFiles
}()
t.Run("success with temp files", func(t *testing.T) {
*sourceDir = ""
*numFiles = 10
err := runCollectionBenchmark()
testutil.AssertNoError(t, err, "runCollectionBenchmark with temp files")
})
t.Run("success with real directory", func(t *testing.T) {
tempDir := t.TempDir()
testutil.CreateTestFiles(t, tempDir, []testutil.FileSpec{
{Name: testFile1, Content: testContent1},
{Name: testFile2, Content: testContent2},
})
*sourceDir = tempDir
*numFiles = 10
err := runCollectionBenchmark()
testutil.AssertNoError(t, err, "runCollectionBenchmark with real directory")
})
}
func TestRunProcessingBenchmark(t *testing.T) {
restore := testutil.SuppressLogs(t)
defer restore()
// Save original flag values
origSourceDir := sourceDir
origFormat := format
origConcurrency := concurrency
defer func() {
sourceDir = origSourceDir
format = origFormat
concurrency = origConcurrency
}()
t.Run("success with json format", func(t *testing.T) {
tempDir := t.TempDir()
testutil.CreateTestFiles(t, tempDir, []testutil.FileSpec{
{Name: testFile1, Content: testContent1},
{Name: testFile2, Content: testContent2},
})
*sourceDir = tempDir
*format = testJSON
*concurrency = 2
err := runProcessingBenchmark()
testutil.AssertNoError(t, err, "runProcessingBenchmark with json")
})
t.Run("success with markdown format", func(t *testing.T) {
tempDir := t.TempDir()
testutil.CreateTestFiles(t, tempDir, []testutil.FileSpec{
{Name: testFile1, Content: testContent1},
})
*sourceDir = tempDir
*format = testMarkdown
*concurrency = 1
err := runProcessingBenchmark()
testutil.AssertNoError(t, err, "runProcessingBenchmark with markdown")
})
}
func TestRunConcurrencyBenchmark(t *testing.T) {
restore := testutil.SuppressLogs(t)
defer restore()
// Save original flag values
origSourceDir := sourceDir
origFormat := format
origConcurrencyList := concurrencyList
defer func() {
sourceDir = origSourceDir
format = origFormat
concurrencyList = origConcurrencyList
}()
t.Run("success with valid concurrency list", func(t *testing.T) {
tempDir := t.TempDir()
testutil.CreateTestFiles(t, tempDir, []testutil.FileSpec{
{Name: testFile1, Content: testContent1},
})
*sourceDir = tempDir
*format = testJSON
*concurrencyList = testConcurrency
err := runConcurrencyBenchmark()
testutil.AssertNoError(t, err, "runConcurrencyBenchmark")
})
t.Run("error with invalid concurrency list", func(t *testing.T) {
tempDir := t.TempDir()
*sourceDir = tempDir
*format = testJSON
*concurrencyList = "invalid"
err := runConcurrencyBenchmark()
testutil.AssertExpectedError(t, err, "runConcurrencyBenchmark with invalid list")
testutil.AssertErrorContains(t, err, "invalid concurrency list", "runConcurrencyBenchmark")
})
}
func TestRunFormatBenchmark(t *testing.T) {
restore := testutil.SuppressLogs(t)
defer restore()
// Save original flag values
origSourceDir := sourceDir
origFormatList := formatList
defer func() {
sourceDir = origSourceDir
formatList = origFormatList
}()
t.Run("success with valid format list", func(t *testing.T) {
tempDir := t.TempDir()
testutil.CreateTestFiles(t, tempDir, []testutil.FileSpec{
{Name: testFile1, Content: testContent1},
})
*sourceDir = tempDir
*formatList = "json,yaml"
err := runFormatBenchmark()
testutil.AssertNoError(t, err, "runFormatBenchmark")
})
t.Run("success with single format", func(t *testing.T) {
tempDir := t.TempDir()
testutil.CreateTestFiles(t, tempDir, []testutil.FileSpec{
{Name: testFile1, Content: testContent1},
})
*sourceDir = tempDir
*formatList = testMarkdown
err := runFormatBenchmark()
testutil.AssertNoError(t, err, "runFormatBenchmark with single format")
})
}
func TestRunBenchmarks(t *testing.T) {
restore := testutil.SuppressLogs(t)
defer restore()
// Save original flag values
origBenchmarkType := benchmarkType
origSourceDir := sourceDir
origConcurrencyList := concurrencyList
origFormatList := formatList
defer func() {
benchmarkType = origBenchmarkType
sourceDir = origSourceDir
concurrencyList = origConcurrencyList
formatList = origFormatList
}()
tempDir := t.TempDir()
testutil.CreateTestFiles(t, tempDir, []testutil.FileSpec{
{Name: testFile1, Content: testContent1},
})
tests := []struct {
name string
benchmarkType string
wantErr bool
errContains string
}{
{
name: "all benchmarks",
benchmarkType: "all",
wantErr: false,
},
{
name: "collection benchmark",
benchmarkType: "collection",
wantErr: false,
},
{
name: "processing benchmark",
benchmarkType: "processing",
wantErr: false,
},
{
name: "concurrency benchmark",
benchmarkType: "concurrency",
wantErr: false,
},
{
name: "format benchmark",
benchmarkType: "format",
wantErr: false,
},
{
name: "invalid benchmark type",
benchmarkType: "invalid",
wantErr: true,
errContains: "invalid benchmark type",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
*benchmarkType = tt.benchmarkType
*sourceDir = tempDir
*concurrencyList = testConcurrency
*formatList = testMarkdown
err := runBenchmarks()
if tt.wantErr {
testutil.AssertExpectedError(t, err, "runBenchmarks")
if tt.errContains != "" {
testutil.AssertErrorContains(t, err, tt.errContains, "runBenchmarks")
}
} else {
testutil.AssertNoError(t, err, "runBenchmarks")
}
})
}
}
func TestMainFunction(t *testing.T) {
restore := testutil.SuppressLogs(t)
defer restore()
// We can't easily test main() directly due to os.Exit calls,
// but we can test runBenchmarks() which contains the main logic
tempDir := t.TempDir()
testutil.CreateTestFiles(t, tempDir, []testutil.FileSpec{
{Name: testFile1, Content: testContent1},
})
// Save original flag values
origBenchmarkType := benchmarkType
origSourceDir := sourceDir
defer func() {
benchmarkType = origBenchmarkType
sourceDir = origSourceDir
}()
*benchmarkType = testCollection
*sourceDir = tempDir
err := runBenchmarks()
testutil.AssertNoError(t, err, "runBenchmarks through main logic path")
}
func TestFlagInitialization(t *testing.T) {
// Test that flags are properly initialized with expected defaults
resetFlags()
if *sourceDir != "" {
t.Errorf("sourceDir default should be empty, got %v", *sourceDir)
}
if *benchmarkType != testAll {
t.Errorf("benchmarkType default should be 'all', got %v", *benchmarkType)
}
if *format != testJSON {
t.Errorf("format default should be 'json', got %v", *format)
}
if *concurrency != runtime.NumCPU() {
t.Errorf("concurrency default should be %d, got %d", runtime.NumCPU(), *concurrency)
}
if *concurrencyList != shared.TestConcurrencyList {
t.Errorf("concurrencyList default should be '%s', got %v", shared.TestConcurrencyList, *concurrencyList)
}
if *formatList != shared.TestFormatList {
t.Errorf("formatList default should be '%s', got %v", shared.TestFormatList, *formatList)
}
if *numFiles != 100 {
t.Errorf("numFiles default should be 100, got %d", *numFiles)
}
}
func TestErrorPropagation(t *testing.T) {
restore := testutil.SuppressLogs(t)
defer restore()
// Save original flag values
origBenchmarkType := benchmarkType
origSourceDir := sourceDir
origConcurrencyList := concurrencyList
defer func() {
benchmarkType = origBenchmarkType
sourceDir = origSourceDir
concurrencyList = origConcurrencyList
}()
tempDir := t.TempDir()
t.Run("error from concurrency benchmark propagates", func(t *testing.T) {
*benchmarkType = testConcurrencyT
*sourceDir = tempDir
*concurrencyList = "invalid,list"
err := runBenchmarks()
testutil.AssertExpectedError(t, err, "runBenchmarks with invalid concurrency")
testutil.AssertErrorContains(t, err, "invalid concurrency list", "runBenchmarks error propagation")
})
t.Run("validation error contains proper error type", func(t *testing.T) {
*benchmarkType = "invalid-type"
*sourceDir = tempDir
err := runBenchmarks()
testutil.AssertExpectedError(t, err, "runBenchmarks with invalid type")
var validationErr *shared.StructuredError
if !errors.As(err, &validationErr) {
t.Errorf("Expected StructuredError, got %T", err)
} else if validationErr.Code != shared.CodeValidationFormat {
t.Errorf("Expected validation format error code, got %v", validationErr.Code)
}
})
t.Run("empty levels array returns error", func(t *testing.T) {
// Test the specific case where all parts are empty after trimming
_, err := parseConcurrencyList(" , , ")
testutil.AssertExpectedError(t, err, "parseConcurrencyList with all empty parts")
testutil.AssertErrorContains(t, err, shared.TestMsgInvalidConcurrencyLevel, "parseConcurrencyList empty levels")
})
t.Run("single empty part returns error", func(t *testing.T) {
// Test case that should never reach the "no valid levels found" condition
_, err := parseConcurrencyList(" ")
testutil.AssertExpectedError(t, err, "parseConcurrencyList with single empty part")
testutil.AssertErrorContains(
t, err, shared.TestMsgInvalidConcurrencyLevel, "parseConcurrencyList single empty part",
)
})
t.Run("benchmark function error paths", func(t *testing.T) {
// Test with non-existent source directory to trigger error paths
nonExistentDir := testNonExistent
*benchmarkType = testCollection
*sourceDir = nonExistentDir
// This should fail as the benchmark package cannot access non-existent directories
err := runBenchmarks()
testutil.AssertExpectedError(t, err, "runBenchmarks with non-existent directory")
testutil.AssertErrorContains(t, err, "file collection benchmark failed",
"runBenchmarks error contains expected message")
})
t.Run("processing benchmark error path", func(t *testing.T) {
// Test error path for processing benchmark
nonExistentDir := testNonExistent
*benchmarkType = "processing"
*sourceDir = nonExistentDir
*format = "json"
*concurrency = 1
err := runBenchmarks()
testutil.AssertExpectedError(t, err, "runBenchmarks processing with non-existent directory")
testutil.AssertErrorContains(t, err, "file processing benchmark failed", "runBenchmarks processing error")
})
t.Run("concurrency benchmark error path", func(t *testing.T) {
// Test error path for concurrency benchmark
nonExistentDir := testNonExistent
*benchmarkType = testConcurrencyT
*sourceDir = nonExistentDir
*format = "json"
*concurrencyList = "1,2"
err := runBenchmarks()
testutil.AssertExpectedError(t, err, "runBenchmarks concurrency with non-existent directory")
testutil.AssertErrorContains(t, err, "concurrency benchmark failed", "runBenchmarks concurrency error")
})
t.Run("format benchmark error path", func(t *testing.T) {
// Test error path for format benchmark
nonExistentDir := testNonExistent
*benchmarkType = "format"
*sourceDir = nonExistentDir
*formatList = "json,yaml"
err := runBenchmarks()
testutil.AssertExpectedError(t, err, "runBenchmarks format with non-existent directory")
testutil.AssertErrorContains(t, err, "format benchmark failed", "runBenchmarks format error")
})
t.Run("all benchmarks error path", func(t *testing.T) {
// Test error path for all benchmarks
nonExistentDir := testNonExistent
*benchmarkType = "all"
*sourceDir = nonExistentDir
err := runBenchmarks()
testutil.AssertExpectedError(t, err, "runBenchmarks all with non-existent directory")
testutil.AssertErrorContains(t, err, "benchmark failed", "runBenchmarks all error")
})
}
// Benchmark functions
// BenchmarkParseConcurrencyList benchmarks the parsing of concurrency lists.
func BenchmarkParseConcurrencyList(b *testing.B) {
benchmarks := []struct {
name string
input string
}{
{
name: "single value",
input: "4",
},
{
name: "multiple values",
input: "1,2,4,8",
},
{
name: "values with whitespace",
input: " 1 , 2 , 4 , 8 , 16 ",
},
{
name: "large list",
input: "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16",
},
}
for _, bm := range benchmarks {
b.Run(bm.name, func(b *testing.B) {
b.ReportAllocs()
for i := 0; i < b.N; i++ {
_, _ = parseConcurrencyList(bm.input)
}
})
}
}
// BenchmarkParseFormatList benchmarks the parsing of format lists.
func BenchmarkParseFormatList(b *testing.B) {
benchmarks := []struct {
name string
input string
}{
{
name: "single format",
input: "json",
},
{
name: "multiple formats",
input: shared.TestFormatList,
},
{
name: "formats with whitespace",
input: " json , yaml , markdown , xml , toml ",
},
{
name: "large list",
input: "json,yaml,markdown,xml,toml,csv,tsv,html,txt,log",
},
}
for _, bm := range benchmarks {
b.Run(bm.name, func(b *testing.B) {
b.ReportAllocs()
for i := 0; i < b.N; i++ {
_ = parseFormatList(bm.input)
}
})
}
}
// Helper functions
// equalSlices compares two slices for equality.
func equalSlices[T comparable](a, b []T) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
// resetFlags resets flag variables to their defaults for testing.
func resetFlags() {
flag.CommandLine = flag.NewFlagSet(os.Args[0], flag.ContinueOnError)
flag.CommandLine.SetOutput(io.Discard)
// Reinitialize the flags
sourceDir = flag.String("source", "", "Source directory to benchmark (uses temp files if empty)")
benchmarkType = flag.String("type", "all", "Benchmark type: all, collection, processing, concurrency, format")
format = flag.String("format", "json", "Output format for processing benchmarks")
concurrency = flag.Int("concurrency", runtime.NumCPU(), "Concurrency level for processing benchmarks")
concurrencyList = flag.String(
"concurrency-list", shared.TestConcurrencyList, "Comma-separated list of concurrency levels",
)
formatList = flag.String("format-list", shared.TestFormatList, "Comma-separated list of formats")
numFiles = flag.Int("files", 100, "Number of files to create for benchmarks")
}

View File

@@ -1,84 +1,333 @@
# gibidify configuration example ---
# Place this file in one of these locations: # gibidify Configuration Example
# =============================
# This file demonstrates all available configuration options with their defaults
# and validation ranges. Copy this file to one of the following locations:
#
# - $XDG_CONFIG_HOME/gibidify/config.yaml # - $XDG_CONFIG_HOME/gibidify/config.yaml
# - $HOME/.config/gibidify/config.yaml # - $HOME/.config/gibidify/config.yaml
# - Current directory (if no gibidify.yaml output file exists) # - Current directory (if no gibidify.yaml output file exists)
# File size limit in bytes (default: 5MB) # =============================================================================
# BASIC FILE PROCESSING SETTINGS
# =============================================================================
# Maximum size for individual files in bytes
# Default: 5242880 (5MB), Min: 1024 (1KB), Max: 104857600 (100MB)
fileSizeLimit: 5242880 fileSizeLimit: 5242880
# Directories to ignore during scanning # Directories to ignore during file system traversal
# These are sensible defaults for most projects
ignoreDirectories: ignoreDirectories:
- vendor - vendor # Go vendor directory
- node_modules - node_modules # Node.js dependencies
- .git - .git # Git repository data
- dist - dist # Distribution/build output
- build - build # Build artifacts
- target - target # Maven/Rust build directory
- bower_components - bower_components # Bower dependencies
- cache - cache # Various cache directories
- tmp - tmp # Temporary files
- .next - .next # Next.js build directory
- .nuxt - .nuxt # Nuxt.js build directory
- .vscode # VS Code settings
- .idea # IntelliJ IDEA settings
- __pycache__ # Python cache
- .pytest_cache # Pytest cache
# Maximum number of worker goroutines for concurrent processing
# Default: number of CPU cores, Min: 1, Max: 100
# maxConcurrency: 8
# Supported output formats for validation
# Default: ["json", "yaml", "markdown"]
# supportedFormats:
# - json
# - yaml
# - markdown
# File patterns to include (glob patterns)
# Default: empty (all files), useful for filtering specific file types
# filePatterns:
# - "*.go"
# - "*.py"
# - "*.js"
# - "*.ts"
# - "*.java"
# - "*.c"
# - "*.cpp"
# =============================================================================
# FILE TYPE DETECTION AND CUSTOMIZATION
# =============================================================================
# FileType registry configuration
fileTypes: fileTypes:
# Enable/disable file type detection entirely (default: true) # Enable/disable file type detection entirely
# Default: true
enabled: true enabled: true
# Add custom image extensions # Add custom image extensions (beyond built-in: .png, .jpg, .jpeg, .gif, .svg, .ico, .bmp, .tiff, .webp)
customImageExtensions: customImageExtensions:
- .webp - .avif # AV1 Image File Format
- .avif - .heic # High Efficiency Image Container
- .heic - .jxl # JPEG XL
- .jxl - .webp # WebP (if not already included)
# Add custom binary extensions # Add custom binary extensions (beyond built-in: .exe, .dll, .so, .dylib, .a, .lib, .obj, .o)
customBinaryExtensions: customBinaryExtensions:
- .custom - .custom # Custom binary format
- .proprietary - .proprietary # Proprietary format
- .blob - .blob # Binary large object
# Add custom language mappings # Add custom language mappings (extension -> language name)
customLanguages: customLanguages:
.zig: zig .zig: zig # Zig language
.odin: odin .odin: odin # Odin language
.v: vlang .v: vlang # V language
.grain: grain .grain: grain # Grain language
.gleam: gleam .gleam: gleam # Gleam language
.roc: roc .roc: roc # Roc language
.janet: janet .janet: janet # Janet language
.fennel: fennel .fennel: fennel # Fennel language
.wast: wast .wast: wast # WebAssembly text format
.wat: wat .wat: wat # WebAssembly text format
# Disable specific default image extensions # Disable specific default image extensions
disabledImageExtensions: disabledImageExtensions:
- .bmp # Disable bitmap support - .bmp # Disable bitmap support
- .tif # Disable TIFF support - .tiff # Disable TIFF support
# Disable specific default binary extensions # Disable specific default binary extensions
disabledBinaryExtensions: disabledBinaryExtensions:
- .exe # Don't treat executables as binary - .exe # Don't treat executables as binary
- .dll # Don't treat DLL files as binary - .dll # Don't treat DLL files as binary
# Disable specific default language extensions # Disable specific default language extensions
disabledLanguageExtensions: disabledLanguageExtensions:
- .bat # Don't detect batch files - .bat # Don't detect batch files
- .cmd # Don't detect command files - .cmd # Don't detect command files
# Maximum concurrency (optional) # =============================================================================
maxConcurrency: 16 # BACKPRESSURE AND MEMORY MANAGEMENT
# =============================================================================
# Supported output formats (optional validation) backpressure:
supportedFormats: # Enable backpressure management for memory optimization
- json # Default: true
- yaml enabled: true
- markdown
# File patterns for filtering (optional) # Maximum number of files to buffer in the processing pipeline
filePatterns: # Default: 1000, helps prevent memory exhaustion with many small files
- "*.go" maxPendingFiles: 1000
- "*.py"
- "*.js" # Maximum number of write operations to buffer
- "*.ts" # Default: 100, controls write throughput vs memory usage
maxPendingWrites: 100
# Soft memory usage limit in bytes before triggering backpressure
# Default: 104857600 (100MB)
maxMemoryUsage: 104857600
# Check memory usage every N files processed
# Default: 1000, lower values = more frequent checks but higher overhead
memoryCheckInterval: 1000
# =============================================================================
# RESOURCE LIMITS AND SECURITY
# =============================================================================
resourceLimits:
# Enable resource limits for DoS protection
# Default: true
enabled: true
# Maximum number of files to process
# Default: 10000, Min: 1, Max: 1000000
maxFiles: 10000
# Maximum total size of all files combined in bytes
# Default: 1073741824 (1GB), Min: 1048576 (1MB), Max: 107374182400 (100GB)
maxTotalSize: 1073741824
# Timeout for processing individual files in seconds
# Default: 30, Min: 1, Max: 300 (5 minutes)
fileProcessingTimeoutSec: 30
# Overall timeout for the entire operation in seconds
# Default: 3600 (1 hour), Min: 10, Max: 86400 (24 hours)
overallTimeoutSec: 3600
# Maximum concurrent file reading operations
# Default: 10, Min: 1, Max: 100
maxConcurrentReads: 10
# Rate limit for file processing (files per second)
# Default: 0 (disabled), Min: 0, Max: 10000
rateLimitFilesPerSec: 0
# Hard memory limit in MB - terminates processing if exceeded
# Default: 512, Min: 64, Max: 8192 (8GB)
hardMemoryLimitMB: 512
# Enable graceful degradation under resource pressure
# Default: true - reduces concurrency and buffers when under pressure
enableGracefulDegradation: true
# Enable detailed resource monitoring and metrics
# Default: true - tracks memory, timing, and processing statistics
enableResourceMonitoring: true
# =============================================================================
# OUTPUT FORMATTING AND TEMPLATES
# =============================================================================
output:
# Template selection: "" (default), "minimal", "detailed", "compact", or "custom"
# Default: "" (uses built-in default template)
template: ""
# Metadata inclusion options
metadata:
# Include processing statistics in output
# Default: false
includeStats: false
# Include timestamp when processing was done
# Default: false
includeTimestamp: false
# Include total number of files processed
# Default: false
includeFileCount: false
# Include source directory path
# Default: false
includeSourcePath: false
# Include detected file types summary
# Default: false
includeFileTypes: false
# Include processing time information
# Default: false
includeProcessingTime: false
# Include total size of processed files
# Default: false
includeTotalSize: false
# Include detailed processing metrics
# Default: false
includeMetrics: false
# Markdown-specific formatting options
markdown:
# Wrap file content in code blocks
# Default: false
useCodeBlocks: false
# Include language identifier in code blocks
# Default: false
includeLanguage: false
# Header level for file sections (1-6)
# Default: 0 (uses template default, typically 2)
headerLevel: 0
# Generate table of contents
# Default: false
tableOfContents: false
# Use collapsible sections for large files
# Default: false
useCollapsible: false
# Enable syntax highlighting hints
# Default: false
syntaxHighlighting: false
# Include line numbers in code blocks
# Default: false
lineNumbers: false
# Automatically fold files longer than maxLineLength
# Default: false
foldLongFiles: false
# Maximum line length before wrapping/folding
# Default: 0 (no limit)
maxLineLength: 0
# Custom CSS to include in markdown output
# Default: "" (no custom CSS)
customCSS: ""
# Custom template overrides (only used when template is "custom")
custom:
# Custom header template (supports Go template syntax)
header: ""
# Custom footer template
footer: ""
# Custom file header template (prepended to each file)
fileHeader: ""
# Custom file footer template (appended to each file)
fileFooter: ""
# Custom template variables accessible in all templates
variables:
# Example variables - customize as needed
project_name: "My Project"
author: "Developer Name"
version: "1.0.0"
description: "Generated code aggregation"
# Add any custom key-value pairs here
# =============================================================================
# EXAMPLES OF COMMON CONFIGURATIONS
# =============================================================================
# Example 1: Minimal configuration for quick code review
# fileSizeLimit: 1048576 # 1MB limit for faster processing
# maxConcurrency: 4 # Lower concurrency for stability
# ignoreDirectories: [".git", "node_modules", "vendor"]
# output:
# template: "minimal"
# metadata:
# includeStats: true
# Example 2: High-performance configuration for large codebases
# fileSizeLimit: 10485760 # 10MB limit
# maxConcurrency: 16 # High concurrency
# backpressure:
# maxPendingFiles: 5000 # Larger buffers
# maxMemoryUsage: 536870912 # 512MB memory
# resourceLimits:
# maxFiles: 100000 # Process more files
# maxTotalSize: 10737418240 # 10GB total size
# Example 3: Security-focused configuration
# resourceLimits:
# maxFiles: 1000 # Strict file limit
# maxTotalSize: 104857600 # 100MB total limit
# fileProcessingTimeoutSec: 10 # Short timeout
# overallTimeoutSec: 300 # 5-minute overall limit
# hardMemoryLimitMB: 256 # Lower memory limit
# rateLimitFilesPerSec: 50 # Rate limiting enabled
# Example 4: Documentation-friendly output
# output:
# template: "detailed"
# metadata:
# includeStats: true
# includeTimestamp: true
# includeFileCount: true
# includeSourcePath: true
# markdown:
# useCodeBlocks: true
# includeLanguage: true
# headerLevel: 2
# tableOfContents: true
# syntaxHighlighting: true

View File

@@ -1,79 +0,0 @@
# Gibidify Configuration Example
# This file demonstrates all available configuration options
# File size limit for individual files (in bytes)
# Default: 5242880 (5MB), Min: 1024 (1KB), Max: 104857600 (100MB)
fileSizeLimit: 5242880
# Directories to ignore during traversal
ignoreDirectories:
- vendor
- node_modules
- .git
- dist
- build
- target
- bower_components
- cache
- tmp
# File type detection and filtering
fileTypes:
enabled: true
customImageExtensions: []
customBinaryExtensions: []
customLanguages: {}
disabledImageExtensions: []
disabledBinaryExtensions: []
disabledLanguageExtensions: []
# Back-pressure management for memory optimization
backpressure:
enabled: true
maxPendingFiles: 1000 # Max files in channel buffer
maxPendingWrites: 100 # Max writes in channel buffer
maxMemoryUsage: 104857600 # 100MB soft memory limit
memoryCheckInterval: 1000 # Check memory every N files
# Resource limits for DoS protection and security
resourceLimits:
enabled: true
# File processing limits
maxFiles: 10000 # Maximum number of files to process
maxTotalSize: 1073741824 # Maximum total size (1GB)
# Timeout limits (in seconds)
fileProcessingTimeoutSec: 30 # Timeout for individual file processing
overallTimeoutSec: 3600 # Overall processing timeout (1 hour)
# Concurrency limits
maxConcurrentReads: 10 # Maximum concurrent file reading operations
# Rate limiting (0 = disabled)
rateLimitFilesPerSec: 0 # Files per second rate limit
# Memory limits
hardMemoryLimitMB: 512 # Hard memory limit (512MB)
# Safety features
enableGracefulDegradation: true # Enable graceful degradation on resource pressure
enableResourceMonitoring: true # Enable detailed resource monitoring
# Optional: Maximum concurrency for workers
# Default: number of CPU cores
# maxConcurrency: 4
# Optional: Supported output formats
# Default: ["json", "yaml", "markdown"]
# supportedFormats:
# - json
# - yaml
# - markdown
# Optional: File patterns to include
# Default: all files (empty list means no pattern filtering)
# filePatterns:
# - "*.go"
# - "*.py"
# - "*.js"

View File

@@ -4,171 +4,223 @@ import (
"testing" "testing"
"github.com/spf13/viper" "github.com/spf13/viper"
"github.com/ivuorinen/gibidify/shared"
) )
// TestFileTypeRegistryConfig tests the FileTypeRegistry configuration functionality. // TestFileTypeRegistryDefaultValues tests default configuration values.
func TestFileTypeRegistryConfig(t *testing.T) { func TestFileTypeRegistryDefaultValues(t *testing.T) {
// Test default values viper.Reset()
t.Run("DefaultValues", func(t *testing.T) { SetDefaultConfig()
viper.Reset()
setDefaultConfig()
if !GetFileTypesEnabled() { verifyDefaultValues(t)
t.Error("Expected file types to be enabled by default") }
}
if len(GetCustomImageExtensions()) != 0 { // TestFileTypeRegistrySetGet tests configuration setting and getting.
t.Error("Expected custom image extensions to be empty by default") func TestFileTypeRegistrySetGet(t *testing.T) {
} viper.Reset()
if len(GetCustomBinaryExtensions()) != 0 { // Set test values
t.Error("Expected custom binary extensions to be empty by default") setTestConfiguration()
}
if len(GetCustomLanguages()) != 0 { // Test getter functions
t.Error("Expected custom languages to be empty by default") verifyTestConfiguration(t)
} }
if len(GetDisabledImageExtensions()) != 0 { // TestFileTypeRegistryValidationSuccess tests successful validation.
t.Error("Expected disabled image extensions to be empty by default") func TestFileTypeRegistryValidationSuccess(t *testing.T) {
} viper.Reset()
SetDefaultConfig()
if len(GetDisabledBinaryExtensions()) != 0 { // Set valid configuration
t.Error("Expected disabled binary extensions to be empty by default") setValidConfiguration()
}
if len(GetDisabledLanguageExtensions()) != 0 { err := ValidateConfig()
t.Error("Expected disabled language extensions to be empty by default") if err != nil {
} t.Errorf("Expected validation to pass with valid config, got error: %v", err)
}) }
}
// Test configuration setting and getting // TestFileTypeRegistryValidationFailure tests validation failures.
t.Run("ConfigurationSetGet", func(t *testing.T) { func TestFileTypeRegistryValidationFailure(t *testing.T) {
viper.Reset() // Test invalid custom image extensions
testInvalidImageExtensions(t)
// Set test values // Test invalid custom binary extensions
viper.Set("fileTypes.enabled", false) testInvalidBinaryExtensions(t)
viper.Set("fileTypes.customImageExtensions", []string{".webp", ".avif"})
viper.Set("fileTypes.customBinaryExtensions", []string{".custom", ".mybin"}) // Test invalid custom languages
viper.Set("fileTypes.customLanguages", map[string]string{ testInvalidCustomLanguages(t)
}
// verifyDefaultValues verifies that default values are correct.
func verifyDefaultValues(t *testing.T) {
t.Helper()
if !FileTypesEnabled() {
t.Error("Expected file types to be enabled by default")
}
verifyEmptySlice(t, CustomImageExtensions(), "custom image extensions")
verifyEmptySlice(t, CustomBinaryExtensions(), "custom binary extensions")
verifyEmptyMap(t, CustomLanguages(), "custom languages")
verifyEmptySlice(t, DisabledImageExtensions(), "disabled image extensions")
verifyEmptySlice(t, DisabledBinaryExtensions(), "disabled binary extensions")
verifyEmptySlice(t, DisabledLanguageExtensions(), "disabled language extensions")
}
// setTestConfiguration sets test configuration values.
func setTestConfiguration() {
viper.Set("fileTypes.enabled", false)
viper.Set(shared.ConfigKeyFileTypesCustomImageExtensions, []string{".webp", ".avif"})
viper.Set(shared.ConfigKeyFileTypesCustomBinaryExtensions, []string{shared.TestExtensionCustom, ".mybin"})
viper.Set(
shared.ConfigKeyFileTypesCustomLanguages, map[string]string{
".zig": "zig", ".zig": "zig",
".v": "vlang", ".v": "vlang",
}) },
viper.Set("fileTypes.disabledImageExtensions", []string{".gif", ".bmp"}) )
viper.Set("fileTypes.disabledBinaryExtensions", []string{".exe", ".dll"}) viper.Set("fileTypes.disabledImageExtensions", []string{".gif", ".bmp"})
viper.Set("fileTypes.disabledLanguageExtensions", []string{".rb", ".pl"}) viper.Set("fileTypes.disabledBinaryExtensions", []string{".exe", ".dll"})
viper.Set("fileTypes.disabledLanguageExtensions", []string{".rb", ".pl"})
}
// Test getter functions // verifyTestConfiguration verifies that test configuration is retrieved correctly.
if GetFileTypesEnabled() { func verifyTestConfiguration(t *testing.T) {
t.Error("Expected file types to be disabled") t.Helper()
}
customImages := GetCustomImageExtensions() if FileTypesEnabled() {
expectedImages := []string{".webp", ".avif"} t.Error("Expected file types to be disabled")
if len(customImages) != len(expectedImages) { }
t.Errorf("Expected %d custom image extensions, got %d", len(expectedImages), len(customImages))
}
for i, ext := range expectedImages {
if customImages[i] != ext {
t.Errorf("Expected custom image extension %s, got %s", ext, customImages[i])
}
}
customBinary := GetCustomBinaryExtensions() verifyStringSlice(t, CustomImageExtensions(), []string{".webp", ".avif"}, "custom image extensions")
expectedBinary := []string{".custom", ".mybin"} verifyStringSlice(t, CustomBinaryExtensions(), []string{".custom", ".mybin"}, "custom binary extensions")
if len(customBinary) != len(expectedBinary) {
t.Errorf("Expected %d custom binary extensions, got %d", len(expectedBinary), len(customBinary))
}
for i, ext := range expectedBinary {
if customBinary[i] != ext {
t.Errorf("Expected custom binary extension %s, got %s", ext, customBinary[i])
}
}
customLangs := GetCustomLanguages() expectedLangs := map[string]string{
expectedLangs := map[string]string{ ".zig": "zig",
".v": "vlang",
}
verifyStringMap(t, CustomLanguages(), expectedLangs, "custom languages")
verifyStringSliceLength(t, DisabledImageExtensions(), []string{".gif", ".bmp"}, "disabled image extensions")
verifyStringSliceLength(t, DisabledBinaryExtensions(), []string{".exe", ".dll"}, "disabled binary extensions")
verifyStringSliceLength(t, DisabledLanguageExtensions(), []string{".rb", ".pl"}, "disabled language extensions")
}
// setValidConfiguration sets valid configuration for validation tests.
func setValidConfiguration() {
viper.Set(shared.ConfigKeyFileTypesCustomImageExtensions, []string{".webp", ".avif"})
viper.Set(shared.ConfigKeyFileTypesCustomBinaryExtensions, []string{shared.TestExtensionCustom})
viper.Set(
shared.ConfigKeyFileTypesCustomLanguages, map[string]string{
".zig": "zig", ".zig": "zig",
".v": "vlang", ".v": "vlang",
} },
if len(customLangs) != len(expectedLangs) { )
t.Errorf("Expected %d custom languages, got %d", len(expectedLangs), len(customLangs)) }
}
for ext, lang := range expectedLangs {
if customLangs[ext] != lang {
t.Errorf("Expected custom language %s -> %s, got %s", ext, lang, customLangs[ext])
}
}
disabledImages := GetDisabledImageExtensions() // testInvalidImageExtensions tests validation failure with invalid image extensions.
expectedDisabledImages := []string{".gif", ".bmp"} func testInvalidImageExtensions(t *testing.T) {
if len(disabledImages) != len(expectedDisabledImages) { t.Helper()
t.Errorf("Expected %d disabled image extensions, got %d", len(expectedDisabledImages), len(disabledImages))
}
disabledBinary := GetDisabledBinaryExtensions() viper.Reset()
expectedDisabledBinary := []string{".exe", ".dll"} SetDefaultConfig()
if len(disabledBinary) != len(expectedDisabledBinary) { viper.Set(shared.ConfigKeyFileTypesCustomImageExtensions, []string{"", "webp"}) // Empty and missing dot
t.Errorf("Expected %d disabled binary extensions, got %d", len(expectedDisabledBinary), len(disabledBinary))
}
disabledLangs := GetDisabledLanguageExtensions() err := ValidateConfig()
expectedDisabledLangs := []string{".rb", ".pl"} if err == nil {
if len(disabledLangs) != len(expectedDisabledLangs) { t.Error("Expected validation to fail with invalid custom image extensions")
t.Errorf("Expected %d disabled language extensions, got %d", len(expectedDisabledLangs), len(disabledLangs)) }
} }
})
// Test validation // testInvalidBinaryExtensions tests validation failure with invalid binary extensions.
t.Run("ValidationSuccess", func(t *testing.T) { func testInvalidBinaryExtensions(t *testing.T) {
viper.Reset() t.Helper()
setDefaultConfig()
// Set valid configuration viper.Reset()
viper.Set("fileTypes.customImageExtensions", []string{".webp", ".avif"}) SetDefaultConfig()
viper.Set("fileTypes.customBinaryExtensions", []string{".custom"}) viper.Set(shared.ConfigKeyFileTypesCustomBinaryExtensions, []string{"custom"}) // Missing dot
viper.Set("fileTypes.customLanguages", map[string]string{
".zig": "zig",
".v": "vlang",
})
err := ValidateConfig() err := ValidateConfig()
if err != nil { if err == nil {
t.Errorf("Expected validation to pass with valid config, got error: %v", err) t.Error("Expected validation to fail with invalid custom binary extensions")
} }
}) }
t.Run("ValidationFailure", func(t *testing.T) { // testInvalidCustomLanguages tests validation failure with invalid custom languages.
// Test invalid custom image extensions func testInvalidCustomLanguages(t *testing.T) {
viper.Reset() t.Helper()
setDefaultConfig()
viper.Set("fileTypes.customImageExtensions", []string{"", "webp"}) // Empty and missing dot
err := ValidateConfig() viper.Reset()
if err == nil { SetDefaultConfig()
t.Error("Expected validation to fail with invalid custom image extensions") viper.Set(
} shared.ConfigKeyFileTypesCustomLanguages, map[string]string{
// Test invalid custom binary extensions
viper.Reset()
setDefaultConfig()
viper.Set("fileTypes.customBinaryExtensions", []string{"custom"}) // Missing dot
err = ValidateConfig()
if err == nil {
t.Error("Expected validation to fail with invalid custom binary extensions")
}
// Test invalid custom languages
viper.Reset()
setDefaultConfig()
viper.Set("fileTypes.customLanguages", map[string]string{
"zig": "zig", // Missing dot in extension "zig": "zig", // Missing dot in extension
".v": "", // Empty language ".v": "", // Empty language
}) },
)
err = ValidateConfig() err := ValidateConfig()
if err == nil { if err == nil {
t.Error("Expected validation to fail with invalid custom languages") t.Error("Expected validation to fail with invalid custom languages")
} }
}) }
// verifyEmptySlice verifies that a slice is empty.
func verifyEmptySlice(t *testing.T, slice []string, name string) {
t.Helper()
if len(slice) != 0 {
t.Errorf("Expected %s to be empty by default", name)
}
}
// verifyEmptyMap verifies that a map is empty.
func verifyEmptyMap(t *testing.T, m map[string]string, name string) {
t.Helper()
if len(m) != 0 {
t.Errorf("Expected %s to be empty by default", name)
}
}
// verifyStringSlice verifies that a string slice matches expected values.
func verifyStringSlice(t *testing.T, actual, expected []string, name string) {
t.Helper()
if len(actual) != len(expected) {
t.Errorf(shared.TestFmtExpectedCount, len(expected), name, len(actual))
return
}
for i, ext := range expected {
if actual[i] != ext {
t.Errorf("Expected %s %s, got %s", name, ext, actual[i])
}
}
}
// verifyStringMap verifies that a string map matches expected values.
func verifyStringMap(t *testing.T, actual, expected map[string]string, name string) {
t.Helper()
if len(actual) != len(expected) {
t.Errorf(shared.TestFmtExpectedCount, len(expected), name, len(actual))
return
}
for ext, lang := range expected {
if actual[ext] != lang {
t.Errorf("Expected %s %s -> %s, got %s", name, ext, lang, actual[ext])
}
}
}
// verifyStringSliceLength verifies that a string slice has the expected length.
func verifyStringSliceLength(t *testing.T, actual, expected []string, name string) {
t.Helper()
if len(actual) != len(expected) {
t.Errorf(shared.TestFmtExpectedCount, len(expected), name, len(actual))
}
} }

View File

@@ -1,61 +0,0 @@
package config
const (
// DefaultFileSizeLimit is the default maximum file size (5MB).
DefaultFileSizeLimit = 5242880
// MinFileSizeLimit is the minimum allowed file size limit (1KB).
MinFileSizeLimit = 1024
// MaxFileSizeLimit is the maximum allowed file size limit (100MB).
MaxFileSizeLimit = 104857600
// Resource Limit Constants
// DefaultMaxFiles is the default maximum number of files to process.
DefaultMaxFiles = 10000
// MinMaxFiles is the minimum allowed file count limit.
MinMaxFiles = 1
// MaxMaxFiles is the maximum allowed file count limit.
MaxMaxFiles = 1000000
// DefaultMaxTotalSize is the default maximum total size of files (1GB).
DefaultMaxTotalSize = 1073741824
// MinMaxTotalSize is the minimum allowed total size limit (1MB).
MinMaxTotalSize = 1048576
// MaxMaxTotalSize is the maximum allowed total size limit (100GB).
MaxMaxTotalSize = 107374182400
// DefaultFileProcessingTimeoutSec is the default timeout for individual file processing (30 seconds).
DefaultFileProcessingTimeoutSec = 30
// MinFileProcessingTimeoutSec is the minimum allowed file processing timeout (1 second).
MinFileProcessingTimeoutSec = 1
// MaxFileProcessingTimeoutSec is the maximum allowed file processing timeout (300 seconds).
MaxFileProcessingTimeoutSec = 300
// DefaultOverallTimeoutSec is the default timeout for overall processing (3600 seconds = 1 hour).
DefaultOverallTimeoutSec = 3600
// MinOverallTimeoutSec is the minimum allowed overall timeout (10 seconds).
MinOverallTimeoutSec = 10
// MaxOverallTimeoutSec is the maximum allowed overall timeout (86400 seconds = 24 hours).
MaxOverallTimeoutSec = 86400
// DefaultMaxConcurrentReads is the default maximum concurrent file reading operations.
DefaultMaxConcurrentReads = 10
// MinMaxConcurrentReads is the minimum allowed concurrent reads.
MinMaxConcurrentReads = 1
// MaxMaxConcurrentReads is the maximum allowed concurrent reads.
MaxMaxConcurrentReads = 100
// DefaultRateLimitFilesPerSec is the default rate limit for file processing (0 = disabled).
DefaultRateLimitFilesPerSec = 0
// MinRateLimitFilesPerSec is the minimum rate limit.
MinRateLimitFilesPerSec = 0
// MaxRateLimitFilesPerSec is the maximum rate limit.
MaxRateLimitFilesPerSec = 10000
// DefaultHardMemoryLimitMB is the default hard memory limit (512MB).
DefaultHardMemoryLimitMB = 512
// MinHardMemoryLimitMB is the minimum hard memory limit (64MB).
MinHardMemoryLimitMB = 64
// MaxHardMemoryLimitMB is the maximum hard memory limit (8192MB = 8GB).
MaxHardMemoryLimitMB = 8192
)

View File

@@ -1,157 +1,331 @@
// Package config handles application configuration management.
package config package config
import ( import (
"strings" "strings"
"github.com/spf13/viper" "github.com/spf13/viper"
"github.com/ivuorinen/gibidify/shared"
) )
// GetFileSizeLimit returns the file size limit from configuration. // FileSizeLimit returns the file size limit from configuration.
func GetFileSizeLimit() int64 { // Default: ConfigFileSizeLimitDefault (5MB).
return viper.GetInt64("fileSizeLimit") func FileSizeLimit() int64 {
return viper.GetInt64(shared.ConfigKeyFileSizeLimit)
} }
// GetIgnoredDirectories returns the list of directories to ignore. // IgnoredDirectories returns the list of directories to ignore.
func GetIgnoredDirectories() []string { // Default: ConfigIgnoredDirectoriesDefault.
return viper.GetStringSlice("ignoreDirectories") func IgnoredDirectories() []string {
return viper.GetStringSlice(shared.ConfigKeyIgnoreDirectories)
} }
// GetMaxConcurrency returns the maximum concurrency level. // MaxConcurrency returns the maximum concurrency level.
func GetMaxConcurrency() int { // Returns 0 if not set (caller should determine appropriate default).
return viper.GetInt("maxConcurrency") func MaxConcurrency() int {
return viper.GetInt(shared.ConfigKeyMaxConcurrency)
} }
// GetSupportedFormats returns the list of supported output formats. // SupportedFormats returns the list of supported output formats.
func GetSupportedFormats() []string { // Returns empty slice if not set.
return viper.GetStringSlice("supportedFormats") func SupportedFormats() []string {
return viper.GetStringSlice(shared.ConfigKeySupportedFormats)
} }
// GetFilePatterns returns the list of file patterns. // FilePatterns returns the list of file patterns.
func GetFilePatterns() []string { // Returns empty slice if not set.
return viper.GetStringSlice("filePatterns") func FilePatterns() []string {
return viper.GetStringSlice(shared.ConfigKeyFilePatterns)
} }
// IsValidFormat checks if the given format is valid. // IsValidFormat checks if the given format is valid.
func IsValidFormat(format string) bool { func IsValidFormat(format string) bool {
format = strings.ToLower(strings.TrimSpace(format)) format = strings.ToLower(strings.TrimSpace(format))
supportedFormats := map[string]bool{ supportedFormats := map[string]bool{
"json": true, shared.FormatJSON: true,
"yaml": true, shared.FormatYAML: true,
"markdown": true, shared.FormatMarkdown: true,
} }
return supportedFormats[format] return supportedFormats[format]
} }
// GetFileTypesEnabled returns whether file types are enabled. // FileTypesEnabled returns whether file types are enabled.
func GetFileTypesEnabled() bool { // Default: ConfigFileTypesEnabledDefault (true).
return viper.GetBool("fileTypes.enabled") func FileTypesEnabled() bool {
return viper.GetBool(shared.ConfigKeyFileTypesEnabled)
} }
// GetCustomImageExtensions returns custom image extensions. // CustomImageExtensions returns custom image extensions.
func GetCustomImageExtensions() []string { // Default: ConfigCustomImageExtensionsDefault (empty).
return viper.GetStringSlice("fileTypes.customImageExtensions") func CustomImageExtensions() []string {
return viper.GetStringSlice(shared.ConfigKeyFileTypesCustomImageExtensions)
} }
// GetCustomBinaryExtensions returns custom binary extensions. // CustomBinaryExtensions returns custom binary extensions.
func GetCustomBinaryExtensions() []string { // Default: ConfigCustomBinaryExtensionsDefault (empty).
return viper.GetStringSlice("fileTypes.customBinaryExtensions") func CustomBinaryExtensions() []string {
return viper.GetStringSlice(shared.ConfigKeyFileTypesCustomBinaryExtensions)
} }
// GetCustomLanguages returns custom language mappings. // CustomLanguages returns custom language mappings.
func GetCustomLanguages() map[string]string { // Default: ConfigCustomLanguagesDefault (empty).
return viper.GetStringMapString("fileTypes.customLanguages") func CustomLanguages() map[string]string {
return viper.GetStringMapString(shared.ConfigKeyFileTypesCustomLanguages)
} }
// GetDisabledImageExtensions returns disabled image extensions. // DisabledImageExtensions returns disabled image extensions.
func GetDisabledImageExtensions() []string { // Default: ConfigDisabledImageExtensionsDefault (empty).
return viper.GetStringSlice("fileTypes.disabledImageExtensions") func DisabledImageExtensions() []string {
return viper.GetStringSlice(shared.ConfigKeyFileTypesDisabledImageExtensions)
} }
// GetDisabledBinaryExtensions returns disabled binary extensions. // DisabledBinaryExtensions returns disabled binary extensions.
func GetDisabledBinaryExtensions() []string { // Default: ConfigDisabledBinaryExtensionsDefault (empty).
return viper.GetStringSlice("fileTypes.disabledBinaryExtensions") func DisabledBinaryExtensions() []string {
return viper.GetStringSlice(shared.ConfigKeyFileTypesDisabledBinaryExtensions)
} }
// GetDisabledLanguageExtensions returns disabled language extensions. // DisabledLanguageExtensions returns disabled language extensions.
func GetDisabledLanguageExtensions() []string { // Default: ConfigDisabledLanguageExtensionsDefault (empty).
return viper.GetStringSlice("fileTypes.disabledLanguageExtensions") func DisabledLanguageExtensions() []string {
return viper.GetStringSlice(shared.ConfigKeyFileTypesDisabledLanguageExts)
} }
// Backpressure getters // Backpressure getters
// GetBackpressureEnabled returns whether backpressure is enabled. // BackpressureEnabled returns whether backpressure is enabled.
func GetBackpressureEnabled() bool { // Default: ConfigBackpressureEnabledDefault (true).
return viper.GetBool("backpressure.enabled") func BackpressureEnabled() bool {
return viper.GetBool(shared.ConfigKeyBackpressureEnabled)
} }
// GetMaxPendingFiles returns the maximum pending files. // MaxPendingFiles returns the maximum pending files.
func GetMaxPendingFiles() int { // Default: ConfigMaxPendingFilesDefault (1000).
return viper.GetInt("backpressure.maxPendingFiles") func MaxPendingFiles() int {
return viper.GetInt(shared.ConfigKeyBackpressureMaxPendingFiles)
} }
// GetMaxPendingWrites returns the maximum pending writes. // MaxPendingWrites returns the maximum pending writes.
func GetMaxPendingWrites() int { // Default: ConfigMaxPendingWritesDefault (100).
return viper.GetInt("backpressure.maxPendingWrites") func MaxPendingWrites() int {
return viper.GetInt(shared.ConfigKeyBackpressureMaxPendingWrites)
} }
// GetMaxMemoryUsage returns the maximum memory usage. // MaxMemoryUsage returns the maximum memory usage.
func GetMaxMemoryUsage() int64 { // Default: ConfigMaxMemoryUsageDefault (100MB).
return viper.GetInt64("backpressure.maxMemoryUsage") func MaxMemoryUsage() int64 {
return viper.GetInt64(shared.ConfigKeyBackpressureMaxMemoryUsage)
} }
// GetMemoryCheckInterval returns the memory check interval. // MemoryCheckInterval returns the memory check interval.
func GetMemoryCheckInterval() int { // Default: ConfigMemoryCheckIntervalDefault (1000 files).
return viper.GetInt("backpressure.memoryCheckInterval") func MemoryCheckInterval() int {
return viper.GetInt(shared.ConfigKeyBackpressureMemoryCheckInt)
} }
// Resource limits getters // Resource limits getters
// GetResourceLimitsEnabled returns whether resource limits are enabled. // ResourceLimitsEnabled returns whether resource limits are enabled.
func GetResourceLimitsEnabled() bool { // Default: ConfigResourceLimitsEnabledDefault (true).
return viper.GetBool("resourceLimits.enabled") func ResourceLimitsEnabled() bool {
return viper.GetBool(shared.ConfigKeyResourceLimitsEnabled)
} }
// GetMaxFiles returns the maximum number of files. // MaxFiles returns the maximum number of files.
func GetMaxFiles() int { // Default: ConfigMaxFilesDefault (10000).
return viper.GetInt("resourceLimits.maxFiles") func MaxFiles() int {
return viper.GetInt(shared.ConfigKeyResourceLimitsMaxFiles)
} }
// GetMaxTotalSize returns the maximum total size. // MaxTotalSize returns the maximum total size.
func GetMaxTotalSize() int64 { // Default: ConfigMaxTotalSizeDefault (1GB).
return viper.GetInt64("resourceLimits.maxTotalSize") func MaxTotalSize() int64 {
return viper.GetInt64(shared.ConfigKeyResourceLimitsMaxTotalSize)
} }
// GetFileProcessingTimeoutSec returns the file processing timeout in seconds. // FileProcessingTimeoutSec returns the file processing timeout in seconds.
func GetFileProcessingTimeoutSec() int { // Default: ConfigFileProcessingTimeoutSecDefault (30 seconds).
return viper.GetInt("resourceLimits.fileProcessingTimeoutSec") func FileProcessingTimeoutSec() int {
return viper.GetInt(shared.ConfigKeyResourceLimitsFileProcessingTO)
} }
// GetOverallTimeoutSec returns the overall timeout in seconds. // OverallTimeoutSec returns the overall timeout in seconds.
func GetOverallTimeoutSec() int { // Default: ConfigOverallTimeoutSecDefault (3600 seconds).
return viper.GetInt("resourceLimits.overallTimeoutSec") func OverallTimeoutSec() int {
return viper.GetInt(shared.ConfigKeyResourceLimitsOverallTO)
} }
// GetMaxConcurrentReads returns the maximum concurrent reads. // MaxConcurrentReads returns the maximum concurrent reads.
func GetMaxConcurrentReads() int { // Default: ConfigMaxConcurrentReadsDefault (10).
return viper.GetInt("resourceLimits.maxConcurrentReads") func MaxConcurrentReads() int {
return viper.GetInt(shared.ConfigKeyResourceLimitsMaxConcurrentReads)
} }
// GetRateLimitFilesPerSec returns the rate limit files per second. // RateLimitFilesPerSec returns the rate limit files per second.
func GetRateLimitFilesPerSec() int { // Default: ConfigRateLimitFilesPerSecDefault (0 = disabled).
return viper.GetInt("resourceLimits.rateLimitFilesPerSec") func RateLimitFilesPerSec() int {
return viper.GetInt(shared.ConfigKeyResourceLimitsRateLimitFilesPerSec)
} }
// GetHardMemoryLimitMB returns the hard memory limit in MB. // HardMemoryLimitMB returns the hard memory limit in MB.
func GetHardMemoryLimitMB() int { // Default: ConfigHardMemoryLimitMBDefault (512MB).
return viper.GetInt("resourceLimits.hardMemoryLimitMB") func HardMemoryLimitMB() int {
return viper.GetInt(shared.ConfigKeyResourceLimitsHardMemoryLimitMB)
} }
// GetEnableGracefulDegradation returns whether graceful degradation is enabled. // EnableGracefulDegradation returns whether graceful degradation is enabled.
func GetEnableGracefulDegradation() bool { // Default: ConfigEnableGracefulDegradationDefault (true).
return viper.GetBool("resourceLimits.enableGracefulDegradation") func EnableGracefulDegradation() bool {
return viper.GetBool(shared.ConfigKeyResourceLimitsEnableGracefulDeg)
} }
// GetEnableResourceMonitoring returns whether resource monitoring is enabled. // EnableResourceMonitoring returns whether resource monitoring is enabled.
func GetEnableResourceMonitoring() bool { // Default: ConfigEnableResourceMonitoringDefault (true).
return viper.GetBool("resourceLimits.enableResourceMonitoring") func EnableResourceMonitoring() bool {
return viper.GetBool(shared.ConfigKeyResourceLimitsEnableMonitoring)
}
// Template system getters
// OutputTemplate returns the selected output template name.
// Default: ConfigOutputTemplateDefault (empty string).
func OutputTemplate() string {
return viper.GetString(shared.ConfigKeyOutputTemplate)
}
// metadataBool is a helper for metadata boolean configuration values.
// All metadata flags default to false.
func metadataBool(key string) bool {
return viper.GetBool("output.metadata." + key)
}
// TemplateMetadataIncludeStats returns whether to include stats in metadata.
func TemplateMetadataIncludeStats() bool {
return metadataBool("includeStats")
}
// TemplateMetadataIncludeTimestamp returns whether to include timestamp in metadata.
func TemplateMetadataIncludeTimestamp() bool {
return metadataBool("includeTimestamp")
}
// TemplateMetadataIncludeFileCount returns whether to include file count in metadata.
func TemplateMetadataIncludeFileCount() bool {
return metadataBool("includeFileCount")
}
// TemplateMetadataIncludeSourcePath returns whether to include source path in metadata.
func TemplateMetadataIncludeSourcePath() bool {
return metadataBool("includeSourcePath")
}
// TemplateMetadataIncludeFileTypes returns whether to include file types in metadata.
func TemplateMetadataIncludeFileTypes() bool {
return metadataBool("includeFileTypes")
}
// TemplateMetadataIncludeProcessingTime returns whether to include processing time in metadata.
func TemplateMetadataIncludeProcessingTime() bool {
return metadataBool("includeProcessingTime")
}
// TemplateMetadataIncludeTotalSize returns whether to include total size in metadata.
func TemplateMetadataIncludeTotalSize() bool {
return metadataBool("includeTotalSize")
}
// TemplateMetadataIncludeMetrics returns whether to include metrics in metadata.
func TemplateMetadataIncludeMetrics() bool {
return metadataBool("includeMetrics")
}
// markdownBool is a helper for markdown boolean configuration values.
// All markdown flags default to false.
func markdownBool(key string) bool {
return viper.GetBool("output.markdown." + key)
}
// TemplateMarkdownUseCodeBlocks returns whether to use code blocks in markdown.
func TemplateMarkdownUseCodeBlocks() bool {
return markdownBool("useCodeBlocks")
}
// TemplateMarkdownIncludeLanguage returns whether to include language in code blocks.
func TemplateMarkdownIncludeLanguage() bool {
return markdownBool("includeLanguage")
}
// TemplateMarkdownHeaderLevel returns the header level for file sections.
// Default: ConfigMarkdownHeaderLevelDefault (0).
func TemplateMarkdownHeaderLevel() int {
return viper.GetInt(shared.ConfigKeyOutputMarkdownHeaderLevel)
}
// TemplateMarkdownTableOfContents returns whether to include table of contents.
func TemplateMarkdownTableOfContents() bool {
return markdownBool("tableOfContents")
}
// TemplateMarkdownUseCollapsible returns whether to use collapsible sections.
func TemplateMarkdownUseCollapsible() bool {
return markdownBool("useCollapsible")
}
// TemplateMarkdownSyntaxHighlighting returns whether to enable syntax highlighting.
func TemplateMarkdownSyntaxHighlighting() bool {
return markdownBool("syntaxHighlighting")
}
// TemplateMarkdownLineNumbers returns whether to include line numbers.
func TemplateMarkdownLineNumbers() bool {
return markdownBool("lineNumbers")
}
// TemplateMarkdownFoldLongFiles returns whether to fold long files.
func TemplateMarkdownFoldLongFiles() bool {
return markdownBool("foldLongFiles")
}
// TemplateMarkdownMaxLineLength returns the maximum line length.
// Default: ConfigMarkdownMaxLineLengthDefault (0 = unlimited).
func TemplateMarkdownMaxLineLength() int {
return viper.GetInt(shared.ConfigKeyOutputMarkdownMaxLineLen)
}
// TemplateCustomCSS returns custom CSS for markdown output.
// Default: ConfigMarkdownCustomCSSDefault (empty string).
func TemplateCustomCSS() string {
return viper.GetString(shared.ConfigKeyOutputMarkdownCustomCSS)
}
// TemplateCustomHeader returns custom header template.
// Default: ConfigCustomHeaderDefault (empty string).
func TemplateCustomHeader() string {
return viper.GetString(shared.ConfigKeyOutputCustomHeader)
}
// TemplateCustomFooter returns custom footer template.
// Default: ConfigCustomFooterDefault (empty string).
func TemplateCustomFooter() string {
return viper.GetString(shared.ConfigKeyOutputCustomFooter)
}
// TemplateCustomFileHeader returns custom file header template.
// Default: ConfigCustomFileHeaderDefault (empty string).
func TemplateCustomFileHeader() string {
return viper.GetString(shared.ConfigKeyOutputCustomFileHeader)
}
// TemplateCustomFileFooter returns custom file footer template.
// Default: ConfigCustomFileFooterDefault (empty string).
func TemplateCustomFileFooter() string {
return viper.GetString(shared.ConfigKeyOutputCustomFileFooter)
}
// TemplateVariables returns custom template variables.
// Default: ConfigTemplateVariablesDefault (empty map).
func TemplateVariables() map[string]string {
return viper.GetStringMapString(shared.ConfigKeyOutputVariables)
} }

492
config/getters_test.go Normal file
View File

@@ -0,0 +1,492 @@
package config_test
import (
"reflect"
"testing"
"github.com/ivuorinen/gibidify/config"
"github.com/ivuorinen/gibidify/shared"
"github.com/ivuorinen/gibidify/testutil"
)
// TestConfigGetters tests all configuration getter functions with comprehensive test coverage.
func TestConfigGetters(t *testing.T) {
tests := []struct {
name string
configKey string
configValue any
getterFunc func() any
expectedResult any
}{
// Basic configuration getters
{
name: "GetFileSizeLimit",
configKey: "fileSizeLimit",
configValue: int64(1048576),
getterFunc: func() any { return config.FileSizeLimit() },
expectedResult: int64(1048576),
},
{
name: "GetIgnoredDirectories",
configKey: "ignoreDirectories",
configValue: []string{"node_modules", ".git", "dist"},
getterFunc: func() any { return config.IgnoredDirectories() },
expectedResult: []string{"node_modules", ".git", "dist"},
},
{
name: "GetMaxConcurrency",
configKey: "maxConcurrency",
configValue: 8,
getterFunc: func() any { return config.MaxConcurrency() },
expectedResult: 8,
},
{
name: "GetSupportedFormats",
configKey: "supportedFormats",
configValue: []string{"json", "yaml", "markdown"},
getterFunc: func() any { return config.SupportedFormats() },
expectedResult: []string{"json", "yaml", "markdown"},
},
{
name: "GetFilePatterns",
configKey: "filePatterns",
configValue: []string{"*.go", "*.js", "*.py"},
getterFunc: func() any { return config.FilePatterns() },
expectedResult: []string{"*.go", "*.js", "*.py"},
},
// File type configuration getters
{
name: "GetFileTypesEnabled",
configKey: "fileTypes.enabled",
configValue: true,
getterFunc: func() any { return config.FileTypesEnabled() },
expectedResult: true,
},
{
name: "GetCustomImageExtensions",
configKey: "fileTypes.customImageExtensions",
configValue: []string{".webp", ".avif"},
getterFunc: func() any { return config.CustomImageExtensions() },
expectedResult: []string{".webp", ".avif"},
},
{
name: "GetCustomBinaryExtensions",
configKey: "fileTypes.customBinaryExtensions",
configValue: []string{".custom", ".bin"},
getterFunc: func() any { return config.CustomBinaryExtensions() },
expectedResult: []string{".custom", ".bin"},
},
{
name: "GetDisabledImageExtensions",
configKey: "fileTypes.disabledImageExtensions",
configValue: []string{".gif", ".bmp"},
getterFunc: func() any { return config.DisabledImageExtensions() },
expectedResult: []string{".gif", ".bmp"},
},
{
name: "GetDisabledBinaryExtensions",
configKey: "fileTypes.disabledBinaryExtensions",
configValue: []string{".exe", ".dll"},
getterFunc: func() any { return config.DisabledBinaryExtensions() },
expectedResult: []string{".exe", ".dll"},
},
{
name: "GetDisabledLanguageExtensions",
configKey: "fileTypes.disabledLanguageExtensions",
configValue: []string{".sh", ".bat"},
getterFunc: func() any { return config.DisabledLanguageExtensions() },
expectedResult: []string{".sh", ".bat"},
},
// Backpressure configuration getters
{
name: "GetBackpressureEnabled",
configKey: "backpressure.enabled",
configValue: true,
getterFunc: func() any { return config.BackpressureEnabled() },
expectedResult: true,
},
{
name: "GetMaxPendingFiles",
configKey: "backpressure.maxPendingFiles",
configValue: 1000,
getterFunc: func() any { return config.MaxPendingFiles() },
expectedResult: 1000,
},
{
name: "GetMaxPendingWrites",
configKey: "backpressure.maxPendingWrites",
configValue: 100,
getterFunc: func() any { return config.MaxPendingWrites() },
expectedResult: 100,
},
{
name: "GetMaxMemoryUsage",
configKey: "backpressure.maxMemoryUsage",
configValue: int64(104857600),
getterFunc: func() any { return config.MaxMemoryUsage() },
expectedResult: int64(104857600),
},
{
name: "GetMemoryCheckInterval",
configKey: "backpressure.memoryCheckInterval",
configValue: 500,
getterFunc: func() any { return config.MemoryCheckInterval() },
expectedResult: 500,
},
// Resource limits configuration getters
{
name: "GetResourceLimitsEnabled",
configKey: "resourceLimits.enabled",
configValue: true,
getterFunc: func() any { return config.ResourceLimitsEnabled() },
expectedResult: true,
},
{
name: "GetMaxFiles",
configKey: "resourceLimits.maxFiles",
configValue: 5000,
getterFunc: func() any { return config.MaxFiles() },
expectedResult: 5000,
},
{
name: "GetMaxTotalSize",
configKey: "resourceLimits.maxTotalSize",
configValue: int64(1073741824),
getterFunc: func() any { return config.MaxTotalSize() },
expectedResult: int64(1073741824),
},
{
name: "GetFileProcessingTimeoutSec",
configKey: "resourceLimits.fileProcessingTimeoutSec",
configValue: 30,
getterFunc: func() any { return config.FileProcessingTimeoutSec() },
expectedResult: 30,
},
{
name: "GetOverallTimeoutSec",
configKey: "resourceLimits.overallTimeoutSec",
configValue: 1800,
getterFunc: func() any { return config.OverallTimeoutSec() },
expectedResult: 1800,
},
{
name: "GetMaxConcurrentReads",
configKey: "resourceLimits.maxConcurrentReads",
configValue: 10,
getterFunc: func() any { return config.MaxConcurrentReads() },
expectedResult: 10,
},
{
name: "GetRateLimitFilesPerSec",
configKey: "resourceLimits.rateLimitFilesPerSec",
configValue: 100,
getterFunc: func() any { return config.RateLimitFilesPerSec() },
expectedResult: 100,
},
{
name: "GetHardMemoryLimitMB",
configKey: "resourceLimits.hardMemoryLimitMB",
configValue: 512,
getterFunc: func() any { return config.HardMemoryLimitMB() },
expectedResult: 512,
},
{
name: "GetEnableGracefulDegradation",
configKey: "resourceLimits.enableGracefulDegradation",
configValue: true,
getterFunc: func() any { return config.EnableGracefulDegradation() },
expectedResult: true,
},
{
name: "GetEnableResourceMonitoring",
configKey: "resourceLimits.enableResourceMonitoring",
configValue: true,
getterFunc: func() any { return config.EnableResourceMonitoring() },
expectedResult: true,
},
// Template system configuration getters
{
name: "GetOutputTemplate",
configKey: "output.template",
configValue: "detailed",
getterFunc: func() any { return config.OutputTemplate() },
expectedResult: "detailed",
},
{
name: "GetTemplateMetadataIncludeStats",
configKey: "output.metadata.includeStats",
configValue: true,
getterFunc: func() any { return config.TemplateMetadataIncludeStats() },
expectedResult: true,
},
{
name: "GetTemplateMetadataIncludeTimestamp",
configKey: "output.metadata.includeTimestamp",
configValue: false,
getterFunc: func() any { return config.TemplateMetadataIncludeTimestamp() },
expectedResult: false,
},
{
name: "GetTemplateMetadataIncludeFileCount",
configKey: "output.metadata.includeFileCount",
configValue: true,
getterFunc: func() any { return config.TemplateMetadataIncludeFileCount() },
expectedResult: true,
},
{
name: "GetTemplateMetadataIncludeSourcePath",
configKey: "output.metadata.includeSourcePath",
configValue: false,
getterFunc: func() any { return config.TemplateMetadataIncludeSourcePath() },
expectedResult: false,
},
{
name: "GetTemplateMetadataIncludeFileTypes",
configKey: "output.metadata.includeFileTypes",
configValue: true,
getterFunc: func() any { return config.TemplateMetadataIncludeFileTypes() },
expectedResult: true,
},
{
name: "GetTemplateMetadataIncludeProcessingTime",
configKey: "output.metadata.includeProcessingTime",
configValue: false,
getterFunc: func() any { return config.TemplateMetadataIncludeProcessingTime() },
expectedResult: false,
},
{
name: "GetTemplateMetadataIncludeTotalSize",
configKey: "output.metadata.includeTotalSize",
configValue: true,
getterFunc: func() any { return config.TemplateMetadataIncludeTotalSize() },
expectedResult: true,
},
{
name: "GetTemplateMetadataIncludeMetrics",
configKey: "output.metadata.includeMetrics",
configValue: false,
getterFunc: func() any { return config.TemplateMetadataIncludeMetrics() },
expectedResult: false,
},
// Markdown template configuration getters
{
name: "GetTemplateMarkdownUseCodeBlocks",
configKey: "output.markdown.useCodeBlocks",
configValue: true,
getterFunc: func() any { return config.TemplateMarkdownUseCodeBlocks() },
expectedResult: true,
},
{
name: "GetTemplateMarkdownIncludeLanguage",
configKey: "output.markdown.includeLanguage",
configValue: false,
getterFunc: func() any { return config.TemplateMarkdownIncludeLanguage() },
expectedResult: false,
},
{
name: "GetTemplateMarkdownHeaderLevel",
configKey: "output.markdown.headerLevel",
configValue: 3,
getterFunc: func() any { return config.TemplateMarkdownHeaderLevel() },
expectedResult: 3,
},
{
name: "GetTemplateMarkdownTableOfContents",
configKey: "output.markdown.tableOfContents",
configValue: true,
getterFunc: func() any { return config.TemplateMarkdownTableOfContents() },
expectedResult: true,
},
{
name: "GetTemplateMarkdownUseCollapsible",
configKey: "output.markdown.useCollapsible",
configValue: false,
getterFunc: func() any { return config.TemplateMarkdownUseCollapsible() },
expectedResult: false,
},
{
name: "GetTemplateMarkdownSyntaxHighlighting",
configKey: "output.markdown.syntaxHighlighting",
configValue: true,
getterFunc: func() any { return config.TemplateMarkdownSyntaxHighlighting() },
expectedResult: true,
},
{
name: "GetTemplateMarkdownLineNumbers",
configKey: "output.markdown.lineNumbers",
configValue: false,
getterFunc: func() any { return config.TemplateMarkdownLineNumbers() },
expectedResult: false,
},
{
name: "GetTemplateMarkdownFoldLongFiles",
configKey: "output.markdown.foldLongFiles",
configValue: true,
getterFunc: func() any { return config.TemplateMarkdownFoldLongFiles() },
expectedResult: true,
},
{
name: "GetTemplateMarkdownMaxLineLength",
configKey: "output.markdown.maxLineLength",
configValue: 120,
getterFunc: func() any { return config.TemplateMarkdownMaxLineLength() },
expectedResult: 120,
},
{
name: "GetTemplateCustomCSS",
configKey: "output.markdown.customCSS",
configValue: "body { color: blue; }",
getterFunc: func() any { return config.TemplateCustomCSS() },
expectedResult: "body { color: blue; }",
},
// Custom template configuration getters
{
name: "GetTemplateCustomHeader",
configKey: "output.custom.header",
configValue: "# Custom Header\n",
getterFunc: func() any { return config.TemplateCustomHeader() },
expectedResult: "# Custom Header\n",
},
{
name: "GetTemplateCustomFooter",
configKey: "output.custom.footer",
configValue: "---\nFooter content",
getterFunc: func() any { return config.TemplateCustomFooter() },
expectedResult: "---\nFooter content",
},
{
name: "GetTemplateCustomFileHeader",
configKey: "output.custom.fileHeader",
configValue: "## File: {{ .Path }}",
getterFunc: func() any { return config.TemplateCustomFileHeader() },
expectedResult: "## File: {{ .Path }}",
},
{
name: "GetTemplateCustomFileFooter",
configKey: "output.custom.fileFooter",
configValue: "---",
getterFunc: func() any { return config.TemplateCustomFileFooter() },
expectedResult: "---",
},
// Custom languages map getter
{
name: "GetCustomLanguages",
configKey: "fileTypes.customLanguages",
configValue: map[string]string{".vue": "vue", ".svelte": "svelte"},
getterFunc: func() any { return config.CustomLanguages() },
expectedResult: map[string]string{".vue": "vue", ".svelte": "svelte"},
},
// Template variables map getter
{
name: "GetTemplateVariables",
configKey: "output.variables",
configValue: map[string]string{"project": "gibidify", "version": "1.0"},
getterFunc: func() any { return config.TemplateVariables() },
expectedResult: map[string]string{"project": "gibidify", "version": "1.0"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Reset viper and set the specific configuration
testutil.SetViperKeys(t, map[string]any{
tt.configKey: tt.configValue,
})
// Call the getter function and compare results
result := tt.getterFunc()
if !reflect.DeepEqual(result, tt.expectedResult) {
t.Errorf("Test %s: expected %v (type %T), got %v (type %T)",
tt.name, tt.expectedResult, tt.expectedResult, result, result)
}
})
}
}
// TestConfigGettersWithDefaults tests that getters return appropriate default values
// when configuration keys are not set.
func TestConfigGettersWithDefaults(t *testing.T) {
// Reset viper to ensure clean state
testutil.ResetViperConfig(t, "")
// Test numeric getters with concrete default assertions
t.Run("numeric_getters", func(t *testing.T) {
assertInt64Getter(t, "FileSizeLimit", config.FileSizeLimit, shared.ConfigFileSizeLimitDefault)
assertIntGetter(t, "MaxConcurrency", config.MaxConcurrency, shared.ConfigMaxConcurrencyDefault)
assertIntGetter(t, "TemplateMarkdownHeaderLevel", config.TemplateMarkdownHeaderLevel,
shared.ConfigMarkdownHeaderLevelDefault)
assertIntGetter(t, "MaxFiles", config.MaxFiles, shared.ConfigMaxFilesDefault)
assertInt64Getter(t, "MaxTotalSize", config.MaxTotalSize, shared.ConfigMaxTotalSizeDefault)
assertIntGetter(t, "FileProcessingTimeoutSec", config.FileProcessingTimeoutSec,
shared.ConfigFileProcessingTimeoutSecDefault)
assertIntGetter(t, "OverallTimeoutSec", config.OverallTimeoutSec, shared.ConfigOverallTimeoutSecDefault)
assertIntGetter(t, "MaxConcurrentReads", config.MaxConcurrentReads, shared.ConfigMaxConcurrentReadsDefault)
assertIntGetter(t, "HardMemoryLimitMB", config.HardMemoryLimitMB, shared.ConfigHardMemoryLimitMBDefault)
})
// Test boolean getters with concrete default assertions
t.Run("boolean_getters", func(t *testing.T) {
assertBoolGetter(t, "FileTypesEnabled", config.FileTypesEnabled, shared.ConfigFileTypesEnabledDefault)
assertBoolGetter(t, "BackpressureEnabled", config.BackpressureEnabled, shared.ConfigBackpressureEnabledDefault)
assertBoolGetter(t, "ResourceLimitsEnabled", config.ResourceLimitsEnabled,
shared.ConfigResourceLimitsEnabledDefault)
assertBoolGetter(t, "EnableGracefulDegradation", config.EnableGracefulDegradation,
shared.ConfigEnableGracefulDegradationDefault)
assertBoolGetter(t, "TemplateMarkdownUseCodeBlocks", config.TemplateMarkdownUseCodeBlocks,
shared.ConfigMarkdownUseCodeBlocksDefault)
assertBoolGetter(t, "TemplateMarkdownTableOfContents", config.TemplateMarkdownTableOfContents,
shared.ConfigMarkdownTableOfContentsDefault)
})
// Test string getters with concrete default assertions
t.Run("string_getters", func(t *testing.T) {
assertStringGetter(t, "OutputTemplate", config.OutputTemplate, shared.ConfigOutputTemplateDefault)
assertStringGetter(t, "TemplateCustomCSS", config.TemplateCustomCSS, shared.ConfigMarkdownCustomCSSDefault)
assertStringGetter(t, "TemplateCustomHeader", config.TemplateCustomHeader, shared.ConfigCustomHeaderDefault)
assertStringGetter(t, "TemplateCustomFooter", config.TemplateCustomFooter, shared.ConfigCustomFooterDefault)
})
}
// assertInt64Getter tests an int64 getter returns the expected default value.
func assertInt64Getter(t *testing.T, name string, getter func() int64, expected int64) {
t.Helper()
result := getter()
if result != expected {
t.Errorf("%s: expected %d, got %d", name, expected, result)
}
}
// assertIntGetter tests an int getter returns the expected default value.
func assertIntGetter(t *testing.T, name string, getter func() int, expected int) {
t.Helper()
result := getter()
if result != expected {
t.Errorf("%s: expected %d, got %d", name, expected, result)
}
}
// assertBoolGetter tests a bool getter returns the expected default value.
func assertBoolGetter(t *testing.T, name string, getter func() bool, expected bool) {
t.Helper()
result := getter()
if result != expected {
t.Errorf("%s: expected %v, got %v", name, expected, result)
}
}
// assertStringGetter tests a string getter returns the expected default value.
func assertStringGetter(t *testing.T, name string, getter func() string, expected string) {
t.Helper()
result := getter()
if result != expected {
t.Errorf("%s: expected %q, got %q", name, expected, result)
}
}

View File

@@ -1,15 +1,13 @@
// Package config handles application configuration management.
package config package config
import ( import (
"flag"
"os" "os"
"path/filepath" "path/filepath"
"sync/atomic"
"github.com/sirupsen/logrus"
"github.com/spf13/viper" "github.com/spf13/viper"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// LoadConfig reads configuration from a YAML file. // LoadConfig reads configuration from a YAML file.
@@ -17,115 +15,105 @@ import (
// 1. $XDG_CONFIG_HOME/gibidify/config.yaml // 1. $XDG_CONFIG_HOME/gibidify/config.yaml
// 2. $HOME/.config/gibidify/config.yaml // 2. $HOME/.config/gibidify/config.yaml
// 3. The current directory as fallback. // 3. The current directory as fallback.
//
// Note: LoadConfig relies on isRunningTest() which requires the testing package
// to have registered its flags (e.g., via flag.Parse() or during test initialization).
// If called too early (e.g., from init() or before TestMain), test detection may not work reliably.
// For explicit control, use SetRunningInTest() before calling LoadConfig.
func LoadConfig() { func LoadConfig() {
viper.SetConfigName("config") viper.SetConfigName("config")
viper.SetConfigType("yaml") viper.SetConfigType(shared.FormatYAML)
logger := shared.GetLogger()
if xdgConfig := os.Getenv("XDG_CONFIG_HOME"); xdgConfig != "" { if xdgConfig := os.Getenv("XDG_CONFIG_HOME"); xdgConfig != "" {
// Validate XDG_CONFIG_HOME for path traversal attempts // Validate XDG_CONFIG_HOME for path traversal attempts
if err := gibidiutils.ValidateConfigPath(xdgConfig); err != nil { if err := shared.ValidateConfigPath(xdgConfig); err != nil {
logrus.Warnf("Invalid XDG_CONFIG_HOME path, using default config: %v", err) logger.Warnf("Invalid XDG_CONFIG_HOME path, using default config: %v", err)
} else { } else {
configPath := filepath.Join(xdgConfig, "gibidify") configPath := filepath.Join(xdgConfig, shared.AppName)
viper.AddConfigPath(configPath) viper.AddConfigPath(configPath)
} }
} else if home, err := os.UserHomeDir(); err == nil { } else if home, err := os.UserHomeDir(); err == nil {
viper.AddConfigPath(filepath.Join(home, ".config", "gibidify")) viper.AddConfigPath(filepath.Join(home, ".config", shared.AppName))
} }
// Only add current directory if no config file named gibidify.yaml exists // Only add current directory if no config file named gibidify.yaml exists
// to avoid conflicts with the project's output file // to avoid conflicts with the project's output file
if _, err := os.Stat("gibidify.yaml"); os.IsNotExist(err) { if _, err := os.Stat(shared.AppName + ".yaml"); os.IsNotExist(err) {
viper.AddConfigPath(".") viper.AddConfigPath(".")
} }
if err := viper.ReadInConfig(); err != nil { if err := viper.ReadInConfig(); err != nil {
// Suppress this info-level log when running tests. logger.Infof("Config file not found, using default values: %v", err)
// Prefer an explicit test flag (SetRunningInTest) but fall back to runtime detection. SetDefaultConfig()
if runningInTest.Load() || isRunningTest() {
// Keep a debug-level record so tests that enable debug can still see it.
logrus.Debugf("Config file not found (tests): %v", err)
} else {
logrus.Infof("Config file not found, using default values: %v", err)
}
setDefaultConfig()
} else { } else {
logrus.Infof("Using config file: %s", viper.ConfigFileUsed()) logger.Infof("Using config file: %s", viper.ConfigFileUsed())
// Validate configuration after loading // Validate configuration after loading
if err := ValidateConfig(); err != nil { if err := ValidateConfig(); err != nil {
logrus.Warnf("Configuration validation failed: %v", err) logger.Warnf("Configuration validation failed: %v", err)
logrus.Info("Falling back to default configuration") logger.Info("Falling back to default configuration")
// Reset viper and set defaults when validation fails // Reset viper and set defaults when validation fails
viper.Reset() viper.Reset()
setDefaultConfig() SetDefaultConfig()
} }
} }
} }
// setDefaultConfig sets default configuration values. // SetDefaultConfig sets default configuration values.
func setDefaultConfig() { func SetDefaultConfig() {
viper.SetDefault("fileSizeLimit", DefaultFileSizeLimit) // File size limits
// Default ignored directories. viper.SetDefault(shared.ConfigKeyFileSizeLimit, shared.ConfigFileSizeLimitDefault)
viper.SetDefault("ignoreDirectories", []string{ viper.SetDefault(shared.ConfigKeyIgnoreDirectories, shared.ConfigIgnoredDirectoriesDefault)
"vendor", "node_modules", ".git", "dist", "build", "target", "bower_components", "cache", "tmp", viper.SetDefault(shared.ConfigKeyMaxConcurrency, shared.ConfigMaxConcurrencyDefault)
}) viper.SetDefault(shared.ConfigKeySupportedFormats, shared.ConfigSupportedFormatsDefault)
viper.SetDefault(shared.ConfigKeyFilePatterns, shared.ConfigFilePatternsDefault)
// FileTypeRegistry defaults // FileTypeRegistry defaults
viper.SetDefault("fileTypes.enabled", true) viper.SetDefault(shared.ConfigKeyFileTypesEnabled, shared.ConfigFileTypesEnabledDefault)
viper.SetDefault("fileTypes.customImageExtensions", []string{}) viper.SetDefault(shared.ConfigKeyFileTypesCustomImageExtensions, shared.ConfigCustomImageExtensionsDefault)
viper.SetDefault("fileTypes.customBinaryExtensions", []string{}) viper.SetDefault(shared.ConfigKeyFileTypesCustomBinaryExtensions, shared.ConfigCustomBinaryExtensionsDefault)
viper.SetDefault("fileTypes.customLanguages", map[string]string{}) viper.SetDefault(shared.ConfigKeyFileTypesCustomLanguages, shared.ConfigCustomLanguagesDefault)
viper.SetDefault("fileTypes.disabledImageExtensions", []string{}) viper.SetDefault(shared.ConfigKeyFileTypesDisabledImageExtensions, shared.ConfigDisabledImageExtensionsDefault)
viper.SetDefault("fileTypes.disabledBinaryExtensions", []string{}) viper.SetDefault(shared.ConfigKeyFileTypesDisabledBinaryExtensions, shared.ConfigDisabledBinaryExtensionsDefault)
viper.SetDefault("fileTypes.disabledLanguageExtensions", []string{}) viper.SetDefault(shared.ConfigKeyFileTypesDisabledLanguageExts, shared.ConfigDisabledLanguageExtensionsDefault)
// Back-pressure and memory management defaults // Backpressure and memory management defaults
viper.SetDefault("backpressure.enabled", true) viper.SetDefault(shared.ConfigKeyBackpressureEnabled, shared.ConfigBackpressureEnabledDefault)
viper.SetDefault("backpressure.maxPendingFiles", 1000) // Max files in file channel buffer viper.SetDefault(shared.ConfigKeyBackpressureMaxPendingFiles, shared.ConfigMaxPendingFilesDefault)
viper.SetDefault("backpressure.maxPendingWrites", 100) // Max writes in write channel buffer viper.SetDefault(shared.ConfigKeyBackpressureMaxPendingWrites, shared.ConfigMaxPendingWritesDefault)
viper.SetDefault("backpressure.maxMemoryUsage", 104857600) // 100MB max memory usage viper.SetDefault(shared.ConfigKeyBackpressureMaxMemoryUsage, shared.ConfigMaxMemoryUsageDefault)
viper.SetDefault("backpressure.memoryCheckInterval", 1000) // Check memory every 1000 files viper.SetDefault(shared.ConfigKeyBackpressureMemoryCheckInt, shared.ConfigMemoryCheckIntervalDefault)
// Resource limit defaults // Resource limit defaults
viper.SetDefault("resourceLimits.enabled", true) viper.SetDefault(shared.ConfigKeyResourceLimitsEnabled, shared.ConfigResourceLimitsEnabledDefault)
viper.SetDefault("resourceLimits.maxFiles", DefaultMaxFiles) viper.SetDefault(shared.ConfigKeyResourceLimitsMaxFiles, shared.ConfigMaxFilesDefault)
viper.SetDefault("resourceLimits.maxTotalSize", DefaultMaxTotalSize) viper.SetDefault(shared.ConfigKeyResourceLimitsMaxTotalSize, shared.ConfigMaxTotalSizeDefault)
viper.SetDefault("resourceLimits.fileProcessingTimeoutSec", DefaultFileProcessingTimeoutSec) viper.SetDefault(shared.ConfigKeyResourceLimitsFileProcessingTO, shared.ConfigFileProcessingTimeoutSecDefault)
viper.SetDefault("resourceLimits.overallTimeoutSec", DefaultOverallTimeoutSec) viper.SetDefault(shared.ConfigKeyResourceLimitsOverallTO, shared.ConfigOverallTimeoutSecDefault)
viper.SetDefault("resourceLimits.maxConcurrentReads", DefaultMaxConcurrentReads) viper.SetDefault(shared.ConfigKeyResourceLimitsMaxConcurrentReads, shared.ConfigMaxConcurrentReadsDefault)
viper.SetDefault("resourceLimits.rateLimitFilesPerSec", DefaultRateLimitFilesPerSec) viper.SetDefault(shared.ConfigKeyResourceLimitsRateLimitFilesPerSec, shared.ConfigRateLimitFilesPerSecDefault)
viper.SetDefault("resourceLimits.hardMemoryLimitMB", DefaultHardMemoryLimitMB) viper.SetDefault(shared.ConfigKeyResourceLimitsHardMemoryLimitMB, shared.ConfigHardMemoryLimitMBDefault)
viper.SetDefault("resourceLimits.enableGracefulDegradation", true) viper.SetDefault(shared.ConfigKeyResourceLimitsEnableGracefulDeg, shared.ConfigEnableGracefulDegradationDefault)
viper.SetDefault("resourceLimits.enableResourceMonitoring", true) viper.SetDefault(shared.ConfigKeyResourceLimitsEnableMonitoring, shared.ConfigEnableResourceMonitoringDefault)
}
var runningInTest atomic.Bool // Output configuration defaults
viper.SetDefault(shared.ConfigKeyOutputTemplate, shared.ConfigOutputTemplateDefault)
// SetRunningInTest allows tests to explicitly indicate they are running under `go test`. viper.SetDefault("output.metadata.includeStats", shared.ConfigMetadataIncludeStatsDefault)
// Call this from TestMain in tests to suppress noisy info logs while still allowing viper.SetDefault("output.metadata.includeTimestamp", shared.ConfigMetadataIncludeTimestampDefault)
// debug-level output for tests that enable it. viper.SetDefault("output.metadata.includeFileCount", shared.ConfigMetadataIncludeFileCountDefault)
func SetRunningInTest(b bool) { viper.SetDefault("output.metadata.includeSourcePath", shared.ConfigMetadataIncludeSourcePathDefault)
runningInTest.Store(b) viper.SetDefault("output.metadata.includeFileTypes", shared.ConfigMetadataIncludeFileTypesDefault)
} viper.SetDefault("output.metadata.includeProcessingTime", shared.ConfigMetadataIncludeProcessingTimeDefault)
viper.SetDefault("output.metadata.includeTotalSize", shared.ConfigMetadataIncludeTotalSizeDefault)
// isRunningTest attempts to detect if the binary is running under `go test`. viper.SetDefault("output.metadata.includeMetrics", shared.ConfigMetadataIncludeMetricsDefault)
// Prefer checking for standard test flags registered by the testing package. viper.SetDefault("output.markdown.useCodeBlocks", shared.ConfigMarkdownUseCodeBlocksDefault)
// This is reliable when `go test` initializes the flag set. viper.SetDefault("output.markdown.includeLanguage", shared.ConfigMarkdownIncludeLanguageDefault)
// viper.SetDefault(shared.ConfigKeyOutputMarkdownHeaderLevel, shared.ConfigMarkdownHeaderLevelDefault)
// IMPORTANT: This function relies on flag.Lookup which returns nil if the testing viper.SetDefault("output.markdown.tableOfContents", shared.ConfigMarkdownTableOfContentsDefault)
// package hasn't registered test flags yet. Callers must invoke this after flag viper.SetDefault("output.markdown.useCollapsible", shared.ConfigMarkdownUseCollapsibleDefault)
// parsing (or test flag registration) has occurred. If invoked too early (e.g., viper.SetDefault("output.markdown.syntaxHighlighting", shared.ConfigMarkdownSyntaxHighlightingDefault)
// from init() or early in TestMain before flags are parsed), detection will fail. viper.SetDefault("output.markdown.lineNumbers", shared.ConfigMarkdownLineNumbersDefault)
// For explicit control, use SetRunningInTest() instead. viper.SetDefault("output.markdown.foldLongFiles", shared.ConfigMarkdownFoldLongFilesDefault)
func isRunningTest() bool { viper.SetDefault(shared.ConfigKeyOutputMarkdownMaxLineLen, shared.ConfigMarkdownMaxLineLengthDefault)
// Look for the well-known test flags created by the testing package. viper.SetDefault(shared.ConfigKeyOutputMarkdownCustomCSS, shared.ConfigMarkdownCustomCSSDefault)
// If any are present in the flag registry, we're running under `go test`. viper.SetDefault(shared.ConfigKeyOutputCustomHeader, shared.ConfigCustomHeaderDefault)
if flag.Lookup("test.v") != nil || flag.Lookup("test.run") != nil || flag.Lookup("test.bench") != nil { viper.SetDefault(shared.ConfigKeyOutputCustomFooter, shared.ConfigCustomFooterDefault)
return true viper.SetDefault(shared.ConfigKeyOutputCustomFileHeader, shared.ConfigCustomFileHeaderDefault)
} viper.SetDefault(shared.ConfigKeyOutputCustomFileFooter, shared.ConfigCustomFileFooterDefault)
return false viper.SetDefault(shared.ConfigKeyOutputVariables, shared.ConfigTemplateVariablesDefault)
} }

View File

@@ -7,6 +7,7 @@ import (
"github.com/spf13/viper" "github.com/spf13/viper"
"github.com/ivuorinen/gibidify/config" "github.com/ivuorinen/gibidify/config"
"github.com/ivuorinen/gibidify/shared"
"github.com/ivuorinen/gibidify/testutil" "github.com/ivuorinen/gibidify/testutil"
) )
@@ -26,14 +27,14 @@ func TestDefaultConfig(t *testing.T) {
testutil.ResetViperConfig(t, tmpDir) testutil.ResetViperConfig(t, tmpDir)
// Check defaults // Check defaults
defaultSizeLimit := config.GetFileSizeLimit() defaultSizeLimit := config.FileSizeLimit()
if defaultSizeLimit != defaultFileSizeLimit { if defaultSizeLimit != defaultFileSizeLimit {
t.Errorf("Expected default file size limit of 5242880, got %d", defaultSizeLimit) t.Errorf("Expected default file size limit of 5242880, got %d", defaultSizeLimit)
} }
ignoredDirs := config.GetIgnoredDirectories() ignoredDirs := config.IgnoredDirectories()
if len(ignoredDirs) == 0 { if len(ignoredDirs) == 0 {
t.Errorf("Expected some default ignored directories, got none") t.Error("Expected some default ignored directories, got none")
} }
// Restore Viper state // Restore Viper state
@@ -76,13 +77,11 @@ ignoreDirectories:
// TestLoadConfigWithValidation tests that invalid config files fall back to defaults. // TestLoadConfigWithValidation tests that invalid config files fall back to defaults.
func TestLoadConfigWithValidation(t *testing.T) { func TestLoadConfigWithValidation(t *testing.T) {
// Create a temporary config file with invalid content // Create a temporary config file with invalid content
configContent := ` configContent := "fileSizeLimit: 100\n" +
fileSizeLimit: 100 "ignoreDirectories:\n" +
ignoreDirectories: "- node_modules\n" +
- node_modules "- \"\"\n" +
- "" "- .git\n"
- .git
`
tempDir := t.TempDir() tempDir := t.TempDir()
configFile := tempDir + "/config.yaml" configFile := tempDir + "/config.yaml"
@@ -100,13 +99,13 @@ ignoreDirectories:
config.LoadConfig() config.LoadConfig()
// Should have fallen back to defaults due to validation failure // Should have fallen back to defaults due to validation failure
if config.GetFileSizeLimit() != int64(config.DefaultFileSizeLimit) { if config.FileSizeLimit() != int64(shared.ConfigFileSizeLimitDefault) {
t.Errorf("Expected default file size limit after validation failure, got %d", config.GetFileSizeLimit()) t.Errorf("Expected default file size limit after validation failure, got %d", config.FileSizeLimit())
} }
if containsString(config.GetIgnoredDirectories(), "") { if containsString(config.IgnoredDirectories(), "") {
t.Errorf( t.Errorf(
"Expected ignored directories not to contain empty string after validation failure, got %v", "Expected ignored directories not to contain empty string after validation failure, got %v",
config.GetIgnoredDirectories(), config.IgnoredDirectories(),
) )
} }
} }
@@ -119,5 +118,6 @@ func containsString(slice []string, item string) bool {
return true return true
} }
} }
return false return false
} }

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,51 @@
// Package config handles application configuration management.
package config
import (
"fmt"
"strings"
)
// validateEmptyElement checks if an element in a slice is empty after trimming whitespace.
// Returns a formatted error message if empty, or empty string if valid.
func validateEmptyElement(fieldPath, value string, index int) string {
value = strings.TrimSpace(value)
if value == "" {
return fmt.Sprintf("%s[%d] is empty", fieldPath, index)
}
return ""
}
// validateDotPrefix ensures an extension starts with a dot.
// Returns a formatted error message if missing dot prefix, or empty string if valid.
func validateDotPrefix(fieldPath, value string, index int) string {
value = strings.TrimSpace(value)
if !strings.HasPrefix(value, ".") {
return fmt.Sprintf("%s[%d] (%s) must start with a dot", fieldPath, index, value)
}
return ""
}
// validateDotPrefixMap ensures a map key (extension) starts with a dot.
// Returns a formatted error message if missing dot prefix, or empty string if valid.
func validateDotPrefixMap(fieldPath, key string) string {
key = strings.TrimSpace(key)
if !strings.HasPrefix(key, ".") {
return fmt.Sprintf("%s extension (%s) must start with a dot", fieldPath, key)
}
return ""
}
// validateEmptyMapValue checks if a map value is empty after trimming whitespace.
// Returns a formatted error message if empty, or empty string if valid.
func validateEmptyMapValue(fieldPath, key, value string) string {
value = strings.TrimSpace(value)
if value == "" {
return fmt.Sprintf("%s[%s] has empty language value", fieldPath, key)
}
return ""
}

View File

@@ -8,44 +8,44 @@ import (
"github.com/spf13/viper" "github.com/spf13/viper"
"github.com/ivuorinen/gibidify/config" "github.com/ivuorinen/gibidify/config"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// TestValidateConfig tests the configuration validation functionality. // TestValidateConfig tests the configuration validation functionality.
func TestValidateConfig(t *testing.T) { func TestValidateConfig(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
config map[string]interface{} config map[string]any
wantErr bool wantErr bool
errContains string errContains string
}{ }{
{ {
name: "valid default config", name: "valid default config",
config: map[string]interface{}{ config: map[string]any{
"fileSizeLimit": config.DefaultFileSizeLimit, "fileSizeLimit": shared.ConfigFileSizeLimitDefault,
"ignoreDirectories": []string{"node_modules", ".git"}, "ignoreDirectories": []string{"node_modules", ".git"},
}, },
wantErr: false, wantErr: false,
}, },
{ {
name: "file size limit too small", name: "file size limit too small",
config: map[string]interface{}{ config: map[string]any{
"fileSizeLimit": config.MinFileSizeLimit - 1, "fileSizeLimit": shared.ConfigFileSizeLimitMin - 1,
}, },
wantErr: true, wantErr: true,
errContains: "fileSizeLimit", errContains: "fileSizeLimit",
}, },
{ {
name: "file size limit too large", name: "file size limit too large",
config: map[string]interface{}{ config: map[string]any{
"fileSizeLimit": config.MaxFileSizeLimit + 1, "fileSizeLimit": shared.ConfigFileSizeLimitMax + 1,
}, },
wantErr: true, wantErr: true,
errContains: "fileSizeLimit", errContains: "fileSizeLimit",
}, },
{ {
name: "empty ignore directory", name: "empty ignore directory",
config: map[string]interface{}{ config: map[string]any{
"ignoreDirectories": []string{"node_modules", "", ".git"}, "ignoreDirectories": []string{"node_modules", "", ".git"},
}, },
wantErr: true, wantErr: true,
@@ -53,7 +53,7 @@ func TestValidateConfig(t *testing.T) {
}, },
{ {
name: "ignore directory with path separator", name: "ignore directory with path separator",
config: map[string]interface{}{ config: map[string]any{
"ignoreDirectories": []string{"node_modules", "src/build", ".git"}, "ignoreDirectories": []string{"node_modules", "src/build", ".git"},
}, },
wantErr: true, wantErr: true,
@@ -61,7 +61,7 @@ func TestValidateConfig(t *testing.T) {
}, },
{ {
name: "invalid supported format", name: "invalid supported format",
config: map[string]interface{}{ config: map[string]any{
"supportedFormats": []string{"json", "xml", "yaml"}, "supportedFormats": []string{"json", "xml", "yaml"},
}, },
wantErr: true, wantErr: true,
@@ -69,7 +69,7 @@ func TestValidateConfig(t *testing.T) {
}, },
{ {
name: "invalid max concurrency", name: "invalid max concurrency",
config: map[string]interface{}{ config: map[string]any{
"maxConcurrency": 0, "maxConcurrency": 0,
}, },
wantErr: true, wantErr: true,
@@ -77,8 +77,8 @@ func TestValidateConfig(t *testing.T) {
}, },
{ {
name: "valid comprehensive config", name: "valid comprehensive config",
config: map[string]interface{}{ config: map[string]any{
"fileSizeLimit": config.DefaultFileSizeLimit, "fileSizeLimit": shared.ConfigFileSizeLimitDefault,
"ignoreDirectories": []string{"node_modules", ".git", ".vscode"}, "ignoreDirectories": []string{"node_modules", ".git", ".vscode"},
"supportedFormats": []string{"json", "yaml", "markdown"}, "supportedFormats": []string{"json", "yaml", "markdown"},
"maxConcurrency": 8, "maxConcurrency": 8,
@@ -89,157 +89,170 @@ func TestValidateConfig(t *testing.T) {
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(
// Reset viper for each test tt.name, func(t *testing.T) {
viper.Reset() // Reset viper for each test
viper.Reset()
// Set test configuration // Set test configuration
for key, value := range tt.config { for key, value := range tt.config {
viper.Set(key, value) viper.Set(key, value)
}
// Load defaults for missing values
config.LoadConfig()
err := config.ValidateConfig()
if tt.wantErr {
if err == nil {
t.Errorf("Expected error but got none")
return
}
if tt.errContains != "" && !strings.Contains(err.Error(), tt.errContains) {
t.Errorf("Expected error to contain %q, got %q", tt.errContains, err.Error())
} }
// Check that it's a structured error // Set defaults for missing values without touching disk
var structErr *gibidiutils.StructuredError config.SetDefaultConfig()
if !errorAs(err, &structErr) {
t.Errorf("Expected structured error, got %T", err) err := config.ValidateConfig()
return
if tt.wantErr {
validateExpectedError(t, err, tt.errContains)
} else if err != nil {
t.Errorf("Expected no error but got: %v", err)
} }
if structErr.Type != gibidiutils.ErrorTypeConfiguration { },
t.Errorf("Expected error type %v, got %v", gibidiutils.ErrorTypeConfiguration, structErr.Type) )
}
if structErr.Code != gibidiutils.CodeConfigValidation {
t.Errorf("Expected error code %v, got %v", gibidiutils.CodeConfigValidation, structErr.Code)
}
} else if err != nil {
t.Errorf("Expected no error but got: %v", err)
}
})
} }
} }
// TestValidationFunctions tests individual validation functions. // TestIsValidFormat tests the IsValidFormat function.
func TestValidationFunctions(t *testing.T) { func TestIsValidFormat(t *testing.T) {
t.Run("IsValidFormat", func(t *testing.T) { tests := []struct {
tests := []struct { format string
format string valid bool
valid bool }{
}{ {"json", true},
{"json", true}, {"yaml", true},
{"yaml", true}, {"markdown", true},
{"markdown", true}, {"JSON", true},
{"JSON", true}, {"xml", false},
{"xml", false}, {"txt", false},
{"txt", false}, {"", false},
{"", false}, {" json ", true},
{" json ", true}, }
for _, tt := range tests {
result := config.IsValidFormat(tt.format)
if result != tt.valid {
t.Errorf("IsValidFormat(%q) = %v, want %v", tt.format, result, tt.valid)
} }
}
for _, tt := range tests {
result := config.IsValidFormat(tt.format)
if result != tt.valid {
t.Errorf("IsValidFormat(%q) = %v, want %v", tt.format, result, tt.valid)
}
}
})
t.Run("ValidateFileSize", func(t *testing.T) {
viper.Reset()
viper.Set("fileSizeLimit", config.DefaultFileSizeLimit)
tests := []struct {
name string
size int64
wantErr bool
}{
{"size within limit", config.DefaultFileSizeLimit - 1, false},
{"size at limit", config.DefaultFileSizeLimit, false},
{"size exceeds limit", config.DefaultFileSizeLimit + 1, true},
{"zero size", 0, false},
}
for _, tt := range tests {
err := config.ValidateFileSize(tt.size)
if (err != nil) != tt.wantErr {
t.Errorf("%s: ValidateFileSize(%d) error = %v, wantErr %v", tt.name, tt.size, err, tt.wantErr)
}
}
})
t.Run("ValidateOutputFormat", func(t *testing.T) {
tests := []struct {
format string
wantErr bool
}{
{"json", false},
{"yaml", false},
{"markdown", false},
{"xml", true},
{"txt", true},
{"", true},
}
for _, tt := range tests {
err := config.ValidateOutputFormat(tt.format)
if (err != nil) != tt.wantErr {
t.Errorf("ValidateOutputFormat(%q) error = %v, wantErr %v", tt.format, err, tt.wantErr)
}
}
})
t.Run("ValidateConcurrency", func(t *testing.T) {
tests := []struct {
name string
concurrency int
maxConcurrency int
setMax bool
wantErr bool
}{
{"valid concurrency", 4, 0, false, false},
{"minimum concurrency", 1, 0, false, false},
{"zero concurrency", 0, 0, false, true},
{"negative concurrency", -1, 0, false, true},
{"concurrency within max", 4, 8, true, false},
{"concurrency exceeds max", 16, 8, true, true},
}
for _, tt := range tests {
viper.Reset()
if tt.setMax {
viper.Set("maxConcurrency", tt.maxConcurrency)
}
err := config.ValidateConcurrency(tt.concurrency)
if (err != nil) != tt.wantErr {
t.Errorf("%s: ValidateConcurrency(%d) error = %v, wantErr %v", tt.name, tt.concurrency, err, tt.wantErr)
}
}
})
} }
func errorAs(err error, target interface{}) bool { // TestValidateFileSize tests the ValidateFileSize function.
func TestValidateFileSize(t *testing.T) {
viper.Reset()
viper.Set("fileSizeLimit", shared.ConfigFileSizeLimitDefault)
tests := []struct {
name string
size int64
wantErr bool
}{
{"size within limit", shared.ConfigFileSizeLimitDefault - 1, false},
{"size at limit", shared.ConfigFileSizeLimitDefault, false},
{"size exceeds limit", shared.ConfigFileSizeLimitDefault + 1, true},
{"zero size", 0, false},
}
for _, tt := range tests {
err := config.ValidateFileSize(tt.size)
if (err != nil) != tt.wantErr {
t.Errorf("%s: ValidateFileSize(%d) error = %v, wantErr %v", tt.name, tt.size, err, tt.wantErr)
}
}
}
// TestValidateOutputFormat tests the ValidateOutputFormat function.
func TestValidateOutputFormat(t *testing.T) {
tests := []struct {
format string
wantErr bool
}{
{"json", false},
{"yaml", false},
{"markdown", false},
{"xml", true},
{"txt", true},
{"", true},
}
for _, tt := range tests {
err := config.ValidateOutputFormat(tt.format)
if (err != nil) != tt.wantErr {
t.Errorf("ValidateOutputFormat(%q) error = %v, wantErr %v", tt.format, err, tt.wantErr)
}
}
}
// TestValidateConcurrency tests the ValidateConcurrency function.
func TestValidateConcurrency(t *testing.T) {
tests := []struct {
name string
concurrency int
maxConcurrency int
setMax bool
wantErr bool
}{
{"valid concurrency", 4, 0, false, false},
{"minimum concurrency", 1, 0, false, false},
{"zero concurrency", 0, 0, false, true},
{"negative concurrency", -1, 0, false, true},
{"concurrency within max", 4, 8, true, false},
{"concurrency exceeds max", 16, 8, true, true},
}
for _, tt := range tests {
viper.Reset()
if tt.setMax {
viper.Set("maxConcurrency", tt.maxConcurrency)
}
err := config.ValidateConcurrency(tt.concurrency)
if (err != nil) != tt.wantErr {
t.Errorf("%s: ValidateConcurrency(%d) error = %v, wantErr %v", tt.name, tt.concurrency, err, tt.wantErr)
}
}
}
// validateExpectedError validates that an error occurred and matches expectations.
func validateExpectedError(t *testing.T, err error, errContains string) {
t.Helper()
if err == nil {
t.Error(shared.TestMsgExpectedError)
return
}
if errContains != "" && !strings.Contains(err.Error(), errContains) {
t.Errorf("Expected error to contain %q, got %q", errContains, err.Error())
}
// Check that it's a structured error
var structErr *shared.StructuredError
if !errorAs(err, &structErr) {
t.Errorf("Expected structured error, got %T", err)
return
}
if structErr.Type != shared.ErrorTypeConfiguration {
t.Errorf("Expected error type %v, got %v", shared.ErrorTypeConfiguration, structErr.Type)
}
if structErr.Code != shared.CodeConfigValidation {
t.Errorf("Expected error code %v, got %v", shared.CodeConfigValidation, structErr.Code)
}
}
func errorAs(err error, target any) bool {
if err == nil { if err == nil {
return false return false
} }
var structErr *gibidiutils.StructuredError structErr := &shared.StructuredError{}
if errors.As(err, &structErr) { if errors.As(err, &structErr) {
if ptr, ok := target.(**gibidiutils.StructuredError); ok { if ptr, ok := target.(**shared.StructuredError); ok {
*ptr = structErr *ptr = structErr
return true return true
} }
} }
return false return false
} }

219
examples/basic-usage.md Normal file
View File

@@ -0,0 +1,219 @@
# Basic Usage Examples
This directory contains practical examples of how to use gibidify for various use cases.
## Simple Code Aggregation
The most basic use case - aggregate all code files from a project into a single output:
```bash
# Aggregate all files from current directory to markdown
gibidify -source . -format markdown -destination output.md
# Aggregate specific directory to JSON
gibidify -source ./src -format json -destination code-dump.json
# Aggregate with custom worker count
gibidify -source ./project -format yaml -destination project.yaml -concurrency 8
```
## With Configuration File
For repeatable processing with custom settings:
1. Copy the configuration example:
```bash
cp config.example.yaml ~/.config/gibidify/config.yaml
```
2. Edit the configuration file to your needs, then run:
```bash
gibidify -source ./my-project
```
## Output Formats
### JSON Output
Best for programmatic processing and data analysis:
```bash
gibidify -source ./src -format json -destination api-code.json
```
Example JSON structure:
```json
{
"files": [
{
"path": "src/main.go",
"content": "package main...",
"language": "go",
"size": 1024
}
],
"metadata": {
"total_files": 15,
"total_size": 45678,
"processing_time": "1.2s"
}
}
```
### Markdown Output
Great for documentation and code reviews:
```bash
gibidify -source ./src -format markdown -destination code-review.md
```
### YAML Output
Structured and human-readable:
```bash
gibidify -source ./config -format yaml -destination config-dump.yaml
```
## Advanced Usage Examples
### Large Codebase Processing
For processing large projects with performance optimizations:
```bash
gibidify -source ./large-project \
-format json \
-destination large-output.json \
-concurrency 16 \
--verbose
```
### Memory-Conscious Processing
For systems with limited memory:
```bash
gibidify -source ./project \
-format markdown \
-destination output.md \
-concurrency 4
```
### Filtered Processing
Process only specific file types (when configured):
```bash
# Configure file patterns in config.yaml
filePatterns:
- "*.go"
- "*.py"
- "*.js"
# Then run
gibidify -source ./mixed-project -destination filtered.json
```
### CI/CD Integration
For automated documentation generation:
```bash
# In your CI pipeline
gibidify -source . \
-format markdown \
-destination docs/codebase.md \
--no-colors \
--no-progress \
-concurrency 2
```
## Error Handling
### Graceful Failure Handling
The tool handles common issues gracefully:
```bash
# This will fail gracefully if source doesn't exist
gibidify -source ./nonexistent -destination out.json
# This will warn about permission issues but continue
gibidify -source ./restricted-dir -destination out.md --verbose
```
### Resource Limits
Configure resource limits in your config file:
```yaml
resourceLimits:
enabled: true
maxFiles: 5000
maxTotalSize: 1073741824 # 1GB
fileProcessingTimeoutSec: 30
overallTimeoutSec: 1800 # 30 minutes
hardMemoryLimitMB: 512
```
## Performance Tips
1. **Adjust Concurrency**: Start with number of CPU cores, adjust based on I/O vs CPU bound work
2. **Use Appropriate Format**: JSON is fastest, Markdown has more overhead
3. **Configure File Limits**: Set reasonable limits in config.yaml for your use case
4. **Monitor Memory**: Use `--verbose` to see memory usage during processing
5. **Use Progress Indicators**: Enable progress bars for long-running operations
## Integration Examples
### With Git Hooks
Create a pre-commit hook to generate code documentation:
```bash
#!/bin/sh
# .git/hooks/pre-commit
gibidify -source . -format markdown -destination docs/current-code.md
git add docs/current-code.md
```
### With Make
Add to your Makefile:
```makefile
.PHONY: code-dump
code-dump:
gibidify -source ./src -format json -destination dist/codebase.json
.PHONY: docs
docs:
gibidify -source . -format markdown -destination docs/codebase.md
```
### Docker Usage
```dockerfile
FROM golang:1.25-alpine
RUN go install github.com/ivuorinen/gibidify@latest
WORKDIR /workspace
COPY . .
RUN gibidify -source . -format json -destination /output/codebase.json
```
## Common Use Cases
### 1. Code Review Preparation
```bash
gibidify -source ./feature-branch -format markdown -destination review.md
```
### 2. AI Code Analysis
```bash
gibidify -source ./src -format json -destination ai-input.json
```
### 3. Documentation Generation
```bash
gibidify -source ./lib -format markdown -destination api-docs.md
```
### 4. Backup Creation
```bash
gibidify -source ./project -format yaml -destination backup-$(date +%Y%m%d).yaml
```
### 5. Code Migration Prep
```bash
gibidify -source ./legacy-code -format json -destination migration-analysis.json
```

View File

@@ -0,0 +1,469 @@
# Configuration Examples
This document provides practical configuration examples for different use cases.
## Basic Configuration
Create `~/.config/gibidify/config.yaml`:
```yaml
# Basic setup for most projects
fileSizeLimit: 5242880 # 5MB per file
maxConcurrency: 8
ignoreDirectories:
- vendor
- node_modules
- .git
- dist
- target
# Enable file type detection
fileTypes:
enabled: true
```
## Development Environment Configuration
Optimized for active development with fast feedback:
```yaml
# ~/.config/gibidify/config.yaml
fileSizeLimit: 1048576 # 1MB - smaller files for faster processing
ignoreDirectories:
- vendor
- node_modules
- .git
- dist
- build
- tmp
- cache
- .vscode
- .idea
# Conservative resource limits for development
resourceLimits:
enabled: true
maxFiles: 1000
maxTotalSize: 104857600 # 100MB
fileProcessingTimeoutSec: 10
overallTimeoutSec: 300 # 5 minutes
maxConcurrentReads: 4
hardMemoryLimitMB: 256
# Fast backpressure for responsive development
backpressure:
enabled: true
maxPendingFiles: 500
maxPendingWrites: 50
maxMemoryUsage: 52428800 # 50MB
memoryCheckInterval: 100
# Simple output for quick reviews
output:
metadata:
includeStats: true
includeTimestamp: true
```
## Production/CI Configuration
High-performance setup for automated processing:
```yaml
# Production configuration
fileSizeLimit: 10485760 # 10MB per file
maxConcurrency: 16
ignoreDirectories:
- vendor
- node_modules
- .git
- dist
- build
- target
- tmp
- cache
- coverage
- .nyc_output
- __pycache__
# High-performance resource limits
resourceLimits:
enabled: true
maxFiles: 50000
maxTotalSize: 10737418240 # 10GB
fileProcessingTimeoutSec: 60
overallTimeoutSec: 7200 # 2 hours
maxConcurrentReads: 20
hardMemoryLimitMB: 2048
# High-throughput backpressure
backpressure:
enabled: true
maxPendingFiles: 5000
maxPendingWrites: 500
maxMemoryUsage: 1073741824 # 1GB
memoryCheckInterval: 1000
# Comprehensive output for analysis
output:
metadata:
includeStats: true
includeTimestamp: true
includeFileCount: true
includeSourcePath: true
includeFileTypes: true
includeProcessingTime: true
includeTotalSize: true
includeMetrics: true
```
## Security-Focused Configuration
Restrictive settings for untrusted input:
```yaml
# Security-first configuration
fileSizeLimit: 1048576 # 1MB maximum
ignoreDirectories:
- "**/.*" # All hidden directories
- vendor
- node_modules
- tmp
- temp
- cache
# Strict resource limits
resourceLimits:
enabled: true
maxFiles: 100 # Very restrictive
maxTotalSize: 10485760 # 10MB total
fileProcessingTimeoutSec: 5
overallTimeoutSec: 60 # 1 minute max
maxConcurrentReads: 2
rateLimitFilesPerSec: 10 # Rate limiting enabled
hardMemoryLimitMB: 128 # Low memory limit
# Conservative backpressure
backpressure:
enabled: true
maxPendingFiles: 50
maxPendingWrites: 10
maxMemoryUsage: 10485760 # 10MB
memoryCheckInterval: 10 # Frequent checks
# Minimal file type detection
fileTypes:
enabled: true
# Disable potentially risky file types
disabledLanguageExtensions:
- .bat
- .cmd
- .ps1
- .sh
disabledBinaryExtensions:
- .exe
- .dll
- .so
```
## Language-Specific Configuration
### Go Projects
```yaml
fileSizeLimit: 5242880
ignoreDirectories:
- vendor
- .git
- bin
- pkg
fileTypes:
enabled: true
customLanguages:
.mod: go-mod
.sum: go-sum
filePatterns:
- "*.go"
- "go.mod"
- "go.sum"
- "*.md"
```
### JavaScript/Node.js Projects
```yaml
fileSizeLimit: 2097152 # 2MB
ignoreDirectories:
- node_modules
- .git
- dist
- build
- coverage
- .nyc_output
fileTypes:
enabled: true
customLanguages:
.vue: vue
.svelte: svelte
.astro: astro
filePatterns:
- "*.js"
- "*.ts"
- "*.jsx"
- "*.tsx"
- "*.vue"
- "*.json"
- "*.md"
```
### Python Projects
```yaml
fileSizeLimit: 5242880
ignoreDirectories:
- .git
- __pycache__
- .pytest_cache
- venv
- env
- .env
- dist
- build
- .tox
fileTypes:
enabled: true
customLanguages:
.pyi: python-interface
.ipynb: jupyter-notebook
filePatterns:
- "*.py"
- "*.pyi"
- "requirements*.txt"
- "*.toml"
- "*.cfg"
- "*.ini"
- "*.md"
```
## Output Format Configurations
### Detailed Markdown Output
```yaml
output:
template: "detailed"
metadata:
includeStats: true
includeTimestamp: true
includeFileCount: true
includeSourcePath: true
includeFileTypes: true
includeProcessingTime: true
markdown:
useCodeBlocks: true
includeLanguage: true
headerLevel: 2
tableOfContents: true
syntaxHighlighting: true
lineNumbers: true
maxLineLength: 120
variables:
project_name: "My Project"
author: "Development Team"
version: "1.0.0"
```
### Compact JSON Output
```yaml
output:
template: "minimal"
metadata:
includeStats: true
includeFileCount: true
```
### Custom Template Output
```yaml
output:
template: "custom"
custom:
header: |
# {{ .ProjectName }} Code Dump
Generated: {{ .Timestamp }}
Total Files: {{ .FileCount }}
footer: |
---
Processing completed in {{ .ProcessingTime }}
fileHeader: |
## {{ .Path }}
Language: {{ .Language }} | Size: {{ .Size }} bytes
fileFooter: ""
variables:
project_name: "Custom Project"
```
## Environment-Specific Configurations
### Docker Container
```yaml
# Optimized for containerized environments
fileSizeLimit: 5242880
maxConcurrency: 4 # Conservative for containers
resourceLimits:
enabled: true
hardMemoryLimitMB: 512
maxFiles: 5000
overallTimeoutSec: 1800
backpressure:
enabled: true
maxMemoryUsage: 268435456 # 256MB
```
### GitHub Actions
```yaml
# CI/CD optimized configuration
fileSizeLimit: 2097152
maxConcurrency: 2 # Conservative for shared runners
ignoreDirectories:
- .git
- .github
- node_modules
- vendor
- dist
- build
resourceLimits:
enabled: true
maxFiles: 2000
overallTimeoutSec: 900 # 15 minutes
hardMemoryLimitMB: 1024
```
### Local Development
```yaml
# Developer-friendly settings
fileSizeLimit: 10485760 # 10MB
maxConcurrency: 8
# Show progress and verbose output
output:
metadata:
includeStats: true
includeTimestamp: true
includeProcessingTime: true
includeMetrics: true
markdown:
useCodeBlocks: true
includeLanguage: true
syntaxHighlighting: true
```
## Template Examples
### Custom API Documentation Template
```yaml
output:
template: "custom"
custom:
header: |
# {{ .Variables.api_name }} API Documentation
Version: {{ .Variables.version }}
Generated: {{ .Timestamp }}
## Overview
This document contains the complete source code for the {{ .Variables.api_name }} API.
## Statistics
- Total Files: {{ .FileCount }}
- Total Size: {{ .TotalSize | formatSize }}
- Processing Time: {{ .ProcessingTime }}
---
fileHeader: |
### {{ .Path }}
**Type:** {{ .Language }}
**Size:** {{ .Size | formatSize }}
```{{ .Language }}
fileFooter: |
```
---
footer: |
## Summary
Documentation generated with [gibidify](https://github.com/ivuorinen/gibidify)
variables:
api_name: "My API"
version: "v1.2.3"
```
### Code Review Template
```yaml
output:
template: "custom"
custom:
header: |
# Code Review: {{ .Variables.pr_title }}
**PR Number:** #{{ .Variables.pr_number }}
**Author:** {{ .Variables.author }}
**Date:** {{ .Timestamp }}
## Files Changed ({{ .FileCount }})
fileHeader: |
## 📄 {{ .Path }}
<details>
<summary>{{ .Language | upper }} • {{ .Size | formatSize }}</summary>
```{{ .Language }}
fileFooter: |
```
</details>
footer: |
---
**Review Summary:**
- Files reviewed: {{ .FileCount }}
- Total size: {{ .TotalSize | formatSize }}
- Generated in: {{ .ProcessingTime }}
variables:
pr_title: "Feature Implementation"
pr_number: "123"
author: "developer@example.com"
```

View File

@@ -3,16 +3,13 @@ package fileproc
import ( import (
"context" "context"
"math"
"runtime" "runtime"
"sync" "sync"
"sync/atomic" "sync/atomic"
"time" "time"
"github.com/sirupsen/logrus"
"github.com/ivuorinen/gibidify/config" "github.com/ivuorinen/gibidify/config"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// BackpressureManager manages memory usage and applies back-pressure when needed. // BackpressureManager manages memory usage and applies back-pressure when needed.
@@ -31,11 +28,11 @@ type BackpressureManager struct {
// NewBackpressureManager creates a new back-pressure manager with configuration. // NewBackpressureManager creates a new back-pressure manager with configuration.
func NewBackpressureManager() *BackpressureManager { func NewBackpressureManager() *BackpressureManager {
return &BackpressureManager{ return &BackpressureManager{
enabled: config.GetBackpressureEnabled(), enabled: config.BackpressureEnabled(),
maxMemoryUsage: config.GetMaxMemoryUsage(), maxMemoryUsage: config.MaxMemoryUsage(),
memoryCheckInterval: config.GetMemoryCheckInterval(), memoryCheckInterval: config.MemoryCheckInterval(),
maxPendingFiles: config.GetMaxPendingFiles(), maxPendingFiles: config.MaxPendingFiles(),
maxPendingWrites: config.GetMaxPendingWrites(), maxPendingWrites: config.MaxPendingWrites(),
lastMemoryCheck: time.Now(), lastMemoryCheck: time.Now(),
} }
} }
@@ -45,38 +42,52 @@ func (bp *BackpressureManager) CreateChannels() (chan string, chan WriteRequest)
var fileCh chan string var fileCh chan string
var writeCh chan WriteRequest var writeCh chan WriteRequest
logger := shared.GetLogger()
if bp.enabled { if bp.enabled {
// Use buffered channels with configured limits // Use buffered channels with configured limits
fileCh = make(chan string, bp.maxPendingFiles) fileCh = make(chan string, bp.maxPendingFiles)
writeCh = make(chan WriteRequest, bp.maxPendingWrites) writeCh = make(chan WriteRequest, bp.maxPendingWrites)
logrus.Debugf("Created buffered channels: files=%d, writes=%d", bp.maxPendingFiles, bp.maxPendingWrites) logger.Debugf("Created buffered channels: files=%d, writes=%d", bp.maxPendingFiles, bp.maxPendingWrites)
} else { } else {
// Use unbuffered channels (default behavior) // Use unbuffered channels (default behavior)
fileCh = make(chan string) fileCh = make(chan string)
writeCh = make(chan WriteRequest) writeCh = make(chan WriteRequest)
logrus.Debug("Created unbuffered channels (back-pressure disabled)") logger.Debug("Created unbuffered channels (back-pressure disabled)")
} }
return fileCh, writeCh return fileCh, writeCh
} }
// ShouldApplyBackpressure checks if back-pressure should be applied. // ShouldApplyBackpressure checks if back-pressure should be applied.
func (bp *BackpressureManager) ShouldApplyBackpressure(_ context.Context) bool { func (bp *BackpressureManager) ShouldApplyBackpressure(ctx context.Context) bool {
// Check for context cancellation first
select {
case <-ctx.Done():
return false // No need for backpressure if canceled
default:
}
if !bp.enabled { if !bp.enabled {
return false return false
} }
// Check if we should evaluate memory usage // Check if we should evaluate memory usage
filesProcessed := atomic.AddInt64(&bp.filesProcessed, 1) filesProcessed := atomic.AddInt64(&bp.filesProcessed, 1)
// Avoid divide by zero - if interval is 0, check every file
if bp.memoryCheckInterval > 0 && int(filesProcessed)%bp.memoryCheckInterval != 0 { // Guard against zero or negative interval to avoid modulo-by-zero panic
interval := bp.memoryCheckInterval
if interval <= 0 {
interval = 1
}
if int(filesProcessed)%interval != 0 {
return false return false
} }
// Get current memory usage // Get current memory usage
var m runtime.MemStats var m runtime.MemStats
runtime.ReadMemStats(&m) runtime.ReadMemStats(&m)
currentMemory := gibidiutils.SafeUint64ToInt64WithDefault(m.Alloc, math.MaxInt64) currentMemory := shared.SafeUint64ToInt64WithDefault(m.Alloc, 0)
bp.mu.Lock() bp.mu.Lock()
defer bp.mu.Unlock() defer bp.mu.Unlock()
@@ -84,18 +95,22 @@ func (bp *BackpressureManager) ShouldApplyBackpressure(_ context.Context) bool {
bp.lastMemoryCheck = time.Now() bp.lastMemoryCheck = time.Now()
// Check if we're over the memory limit // Check if we're over the memory limit
logger := shared.GetLogger()
if currentMemory > bp.maxMemoryUsage { if currentMemory > bp.maxMemoryUsage {
if !bp.memoryWarningLogged { if !bp.memoryWarningLogged {
logrus.Warnf("Memory usage (%d bytes) exceeds limit (%d bytes), applying back-pressure", logger.Warnf(
currentMemory, bp.maxMemoryUsage) "Memory usage (%d bytes) exceeds limit (%d bytes), applying back-pressure",
currentMemory, bp.maxMemoryUsage,
)
bp.memoryWarningLogged = true bp.memoryWarningLogged = true
} }
return true return true
} }
// Reset warning flag if we're back under the limit // Reset warning flag if we're back under the limit
if bp.memoryWarningLogged && currentMemory < bp.maxMemoryUsage*8/10 { // 80% of limit if bp.memoryWarningLogged && currentMemory < bp.maxMemoryUsage*8/10 { // 80% of limit
logrus.Infof("Memory usage normalized (%d bytes), removing back-pressure", currentMemory) logger.Infof("Memory usage normalized (%d bytes), removing back-pressure", currentMemory)
bp.memoryWarningLogged = false bp.memoryWarningLogged = false
} }
@@ -108,14 +123,6 @@ func (bp *BackpressureManager) ApplyBackpressure(ctx context.Context) {
return return
} }
// Check for context cancellation before doing expensive operations
select {
case <-ctx.Done():
return
default:
// Continue with backpressure logic
}
// Force garbage collection to free up memory // Force garbage collection to free up memory
runtime.GC() runtime.GC()
@@ -130,11 +137,12 @@ func (bp *BackpressureManager) ApplyBackpressure(ctx context.Context) {
// Log memory usage after GC // Log memory usage after GC
var m runtime.MemStats var m runtime.MemStats
runtime.ReadMemStats(&m) runtime.ReadMemStats(&m)
logrus.Debugf("Applied back-pressure: memory after GC = %d bytes", m.Alloc) logger := shared.GetLogger()
logger.Debugf("Applied back-pressure: memory after GC = %d bytes", m.Alloc)
} }
// GetStats returns current back-pressure statistics. // Stats returns current back-pressure statistics.
func (bp *BackpressureManager) GetStats() BackpressureStats { func (bp *BackpressureManager) Stats() BackpressureStats {
bp.mu.RLock() bp.mu.RLock()
defer bp.mu.RUnlock() defer bp.mu.RUnlock()
@@ -144,7 +152,7 @@ func (bp *BackpressureManager) GetStats() BackpressureStats {
return BackpressureStats{ return BackpressureStats{
Enabled: bp.enabled, Enabled: bp.enabled,
FilesProcessed: atomic.LoadInt64(&bp.filesProcessed), FilesProcessed: atomic.LoadInt64(&bp.filesProcessed),
CurrentMemoryUsage: gibidiutils.SafeUint64ToInt64WithDefault(m.Alloc, math.MaxInt64), CurrentMemoryUsage: shared.SafeUint64ToInt64WithDefault(m.Alloc, 0),
MaxMemoryUsage: bp.maxMemoryUsage, MaxMemoryUsage: bp.maxMemoryUsage,
MemoryWarningActive: bp.memoryWarningLogged, MemoryWarningActive: bp.memoryWarningLogged,
LastMemoryCheck: bp.lastMemoryCheck, LastMemoryCheck: bp.lastMemoryCheck,
@@ -171,9 +179,11 @@ func (bp *BackpressureManager) WaitForChannelSpace(ctx context.Context, fileCh c
return return
} }
// Check if file channel is getting full (>=90% capacity) logger := shared.GetLogger()
if bp.maxPendingFiles > 0 && len(fileCh) >= bp.maxPendingFiles*9/10 { // Check if file channel is getting full (>90% capacity)
logrus.Debugf("File channel is %d%% full, waiting for space", len(fileCh)*100/bp.maxPendingFiles) fileCap := cap(fileCh)
if fileCap > 0 && len(fileCh) > fileCap*9/10 {
logger.Debugf("File channel is %d%% full, waiting for space", len(fileCh)*100/fileCap)
// Wait a bit for the channel to drain // Wait a bit for the channel to drain
select { select {
@@ -183,9 +193,10 @@ func (bp *BackpressureManager) WaitForChannelSpace(ctx context.Context, fileCh c
} }
} }
// Check if write channel is getting full (>=90% capacity) // Check if write channel is getting full (>90% capacity)
if bp.maxPendingWrites > 0 && len(writeCh) >= bp.maxPendingWrites*9/10 { writeCap := cap(writeCh)
logrus.Debugf("Write channel is %d%% full, waiting for space", len(writeCh)*100/bp.maxPendingWrites) if writeCap > 0 && len(writeCh) > writeCap*9/10 {
logger.Debugf("Write channel is %d%% full, waiting for space", len(writeCh)*100/writeCap)
// Wait a bit for the channel to drain // Wait a bit for the channel to drain
select { select {
@@ -198,10 +209,13 @@ func (bp *BackpressureManager) WaitForChannelSpace(ctx context.Context, fileCh c
// LogBackpressureInfo logs back-pressure configuration and status. // LogBackpressureInfo logs back-pressure configuration and status.
func (bp *BackpressureManager) LogBackpressureInfo() { func (bp *BackpressureManager) LogBackpressureInfo() {
logger := shared.GetLogger()
if bp.enabled { if bp.enabled {
logrus.Infof("Back-pressure enabled: maxMemory=%dMB, fileBuffer=%d, writeBuffer=%d, checkInterval=%d", logger.Infof(
bp.maxMemoryUsage/1024/1024, bp.maxPendingFiles, bp.maxPendingWrites, bp.memoryCheckInterval) "Back-pressure enabled: maxMemory=%dMB, fileBuffer=%d, writeBuffer=%d, checkInterval=%d",
bp.maxMemoryUsage/int64(shared.BytesPerMB), bp.maxPendingFiles, bp.maxPendingWrites, bp.memoryCheckInterval,
)
} else { } else {
logrus.Info("Back-pressure disabled") logger.Info("Back-pressure disabled")
} }
} }

View File

@@ -1,177 +0,0 @@
package fileproc
import (
"context"
"testing"
"time"
"github.com/stretchr/testify/assert"
)
func TestBackpressureManagerShouldApplyBackpressure(t *testing.T) {
ctx := context.Background()
t.Run("returns false when disabled", func(t *testing.T) {
bm := NewBackpressureManager()
bm.enabled = false
shouldApply := bm.ShouldApplyBackpressure(ctx)
assert.False(t, shouldApply)
})
t.Run("checks memory at intervals", func(_ *testing.T) {
bm := NewBackpressureManager()
bm.enabled = true
bm.memoryCheckInterval = 10
// Should not check memory on most calls
for i := 1; i < 10; i++ {
shouldApply := bm.ShouldApplyBackpressure(ctx)
// Can't predict result, but shouldn't panic
_ = shouldApply
}
// Should check memory on 10th call
shouldApply := bm.ShouldApplyBackpressure(ctx)
// Result depends on actual memory usage
_ = shouldApply
})
t.Run("detects high memory usage", func(t *testing.T) {
bm := NewBackpressureManager()
bm.enabled = true
bm.memoryCheckInterval = 1
bm.maxMemoryUsage = 1 // Set very low limit to trigger
shouldApply := bm.ShouldApplyBackpressure(ctx)
// Should detect high memory usage
assert.True(t, shouldApply)
})
}
func TestBackpressureManagerApplyBackpressure(t *testing.T) {
ctx := context.Background()
t.Run("does nothing when disabled", func(t *testing.T) {
bm := NewBackpressureManager()
bm.enabled = false
// Use a channel to verify the function returns quickly
done := make(chan struct{})
go func() {
bm.ApplyBackpressure(ctx)
close(done)
}()
// Should complete quickly when disabled
select {
case <-done:
// Success - function returned
case <-time.After(50 * time.Millisecond):
t.Fatal("ApplyBackpressure did not return quickly when disabled")
}
})
t.Run("applies delay when enabled", func(t *testing.T) {
bm := NewBackpressureManager()
bm.enabled = true
// Use a channel to verify the function blocks for some time
done := make(chan struct{})
started := make(chan struct{})
go func() {
close(started)
bm.ApplyBackpressure(ctx)
close(done)
}()
// Wait for goroutine to start
<-started
// Should NOT complete immediately - verify it blocks for at least 5ms
select {
case <-done:
t.Fatal("ApplyBackpressure returned too quickly when enabled")
case <-time.After(5 * time.Millisecond):
// Good - it's blocking as expected
}
// Now wait for it to complete (should finish within reasonable time)
select {
case <-done:
// Success - function eventually returned
case <-time.After(500 * time.Millisecond):
t.Fatal("ApplyBackpressure did not complete within timeout")
}
})
t.Run("respects context cancellation", func(t *testing.T) {
bm := NewBackpressureManager()
bm.enabled = true
ctx, cancel := context.WithCancel(context.Background())
cancel() // Cancel immediately
start := time.Now()
bm.ApplyBackpressure(ctx)
duration := time.Since(start)
// Should return quickly when context is cancelled
assert.Less(t, duration, 5*time.Millisecond)
})
}
func TestBackpressureManagerLogBackpressureInfo(t *testing.T) {
bm := NewBackpressureManager()
bm.enabled = true // Ensure enabled so filesProcessed is incremented
// Apply some operations
ctx := context.Background()
bm.ShouldApplyBackpressure(ctx)
bm.ApplyBackpressure(ctx)
// This should not panic
bm.LogBackpressureInfo()
stats := bm.GetStats()
assert.Greater(t, stats.FilesProcessed, int64(0))
}
func TestBackpressureManagerMemoryLimiting(t *testing.T) {
t.Run("triggers on low memory limit", func(t *testing.T) {
bm := NewBackpressureManager()
bm.enabled = true
bm.memoryCheckInterval = 1 // Check every file
bm.maxMemoryUsage = 1 // Very low limit to guarantee trigger
ctx := context.Background()
// Should detect memory over limit
shouldApply := bm.ShouldApplyBackpressure(ctx)
assert.True(t, shouldApply)
stats := bm.GetStats()
assert.True(t, stats.MemoryWarningActive)
})
t.Run("resets warning when memory normalizes", func(t *testing.T) {
bm := NewBackpressureManager()
bm.enabled = true
bm.memoryCheckInterval = 1
// Simulate warning by first triggering high memory usage
bm.maxMemoryUsage = 1 // Very low to trigger warning
ctx := context.Background()
_ = bm.ShouldApplyBackpressure(ctx)
stats := bm.GetStats()
assert.True(t, stats.MemoryWarningActive)
// Now set high limit so we're under it
bm.maxMemoryUsage = 1024 * 1024 * 1024 * 10 // 10GB
shouldApply := bm.ShouldApplyBackpressure(ctx)
assert.False(t, shouldApply)
// Warning should be reset (via public API)
stats = bm.GetStats()
assert.False(t, stats.MemoryWarningActive)
})
}

View File

@@ -1,262 +0,0 @@
package fileproc
import (
"context"
"testing"
"time"
"github.com/spf13/viper"
"github.com/stretchr/testify/assert"
)
const (
// CI-safe timeout constants
fastOpTimeout = 100 * time.Millisecond // Operations that should complete quickly
slowOpMinTime = 10 * time.Millisecond // Minimum time for blocking operations
)
// cleanupViperConfig is a test helper that captures and restores viper configuration.
// It takes a testing.T and a list of config keys to save/restore.
// Returns a cleanup function that should be called via t.Cleanup.
func cleanupViperConfig(t *testing.T, keys ...string) {
t.Helper()
// Capture original values
origValues := make(map[string]interface{})
for _, key := range keys {
origValues[key] = viper.Get(key)
}
// Register cleanup to restore values
t.Cleanup(func() {
for key, val := range origValues {
if val != nil {
viper.Set(key, val)
}
}
})
}
func TestBackpressureManagerCreateChannels(t *testing.T) {
t.Run("creates buffered channels when enabled", func(t *testing.T) {
// Capture and restore viper config
cleanupViperConfig(t, testBackpressureEnabled, testBackpressureMaxFiles, testBackpressureMaxWrites)
viper.Set(testBackpressureEnabled, true)
viper.Set(testBackpressureMaxFiles, 10)
viper.Set(testBackpressureMaxWrites, 10)
bm := NewBackpressureManager()
fileCh, writeCh := bm.CreateChannels()
assert.NotNil(t, fileCh)
assert.NotNil(t, writeCh)
// Test that channels have buffer capacity
assert.Greater(t, cap(fileCh), 0)
assert.Greater(t, cap(writeCh), 0)
// Test sending and receiving
fileCh <- "test.go"
val := <-fileCh
assert.Equal(t, "test.go", val)
writeCh <- WriteRequest{Content: "test content"}
writeReq := <-writeCh
assert.Equal(t, "test content", writeReq.Content)
close(fileCh)
close(writeCh)
})
t.Run("creates unbuffered channels when disabled", func(t *testing.T) {
// Use viper to configure instead of direct field access
cleanupViperConfig(t, testBackpressureEnabled)
viper.Set(testBackpressureEnabled, false)
bm := NewBackpressureManager()
fileCh, writeCh := bm.CreateChannels()
assert.NotNil(t, fileCh)
assert.NotNil(t, writeCh)
// Unbuffered channels have capacity 0
assert.Equal(t, 0, cap(fileCh))
assert.Equal(t, 0, cap(writeCh))
close(fileCh)
close(writeCh)
})
}
func TestBackpressureManagerWaitForChannelSpace(t *testing.T) {
t.Run("does nothing when disabled", func(t *testing.T) {
// Use viper to configure instead of direct field access
cleanupViperConfig(t, testBackpressureEnabled)
viper.Set(testBackpressureEnabled, false)
bm := NewBackpressureManager()
fileCh := make(chan string, 1)
writeCh := make(chan WriteRequest, 1)
// Use context with timeout instead of measuring elapsed time
ctx, cancel := context.WithTimeout(context.Background(), fastOpTimeout)
defer cancel()
done := make(chan struct{})
go func() {
bm.WaitForChannelSpace(ctx, fileCh, writeCh)
close(done)
}()
// Should return immediately (before timeout)
select {
case <-done:
// Success - operation completed quickly
case <-ctx.Done():
t.Fatal("WaitForChannelSpace should return immediately when disabled")
}
close(fileCh)
close(writeCh)
})
t.Run("waits when file channel is nearly full", func(t *testing.T) {
// Use viper to configure instead of direct field access
cleanupViperConfig(t, testBackpressureEnabled, testBackpressureMaxFiles)
viper.Set(testBackpressureEnabled, true)
viper.Set(testBackpressureMaxFiles, 10)
bm := NewBackpressureManager()
// Create channel with exact capacity
fileCh := make(chan string, 10)
writeCh := make(chan WriteRequest, 10)
// Fill file channel to >90% (with minimum of 1)
target := max(1, int(float64(cap(fileCh))*0.9))
for i := 0; i < target; i++ {
fileCh <- "file.txt"
}
// Test that it blocks by verifying it doesn't complete immediately
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
done := make(chan struct{})
start := time.Now()
go func() {
bm.WaitForChannelSpace(ctx, fileCh, writeCh)
close(done)
}()
// Verify it doesn't complete immediately (within first millisecond)
select {
case <-done:
t.Fatal("WaitForChannelSpace should block when channel is nearly full")
case <-time.After(1 * time.Millisecond):
// Good - it's blocking as expected
}
// Wait for it to complete
<-done
duration := time.Since(start)
// Just verify it took some measurable time (very lenient for CI)
assert.GreaterOrEqual(t, duration, 1*time.Millisecond)
// Clean up
for i := 0; i < target; i++ {
<-fileCh
}
close(fileCh)
close(writeCh)
})
t.Run("waits when write channel is nearly full", func(t *testing.T) {
// Use viper to configure instead of direct field access
cleanupViperConfig(t, testBackpressureEnabled, testBackpressureMaxWrites)
viper.Set(testBackpressureEnabled, true)
viper.Set(testBackpressureMaxWrites, 10)
bm := NewBackpressureManager()
fileCh := make(chan string, 10)
writeCh := make(chan WriteRequest, 10)
// Fill write channel to >90% (with minimum of 1)
target := max(1, int(float64(cap(writeCh))*0.9))
for i := 0; i < target; i++ {
writeCh <- WriteRequest{}
}
// Test that it blocks by verifying it doesn't complete immediately
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
done := make(chan struct{})
start := time.Now()
go func() {
bm.WaitForChannelSpace(ctx, fileCh, writeCh)
close(done)
}()
// Verify it doesn't complete immediately (within first millisecond)
select {
case <-done:
t.Fatal("WaitForChannelSpace should block when channel is nearly full")
case <-time.After(1 * time.Millisecond):
// Good - it's blocking as expected
}
// Wait for it to complete
<-done
duration := time.Since(start)
// Just verify it took some measurable time (very lenient for CI)
assert.GreaterOrEqual(t, duration, 1*time.Millisecond)
// Clean up
for i := 0; i < target; i++ {
<-writeCh
}
close(fileCh)
close(writeCh)
})
t.Run("respects context cancellation", func(t *testing.T) {
// Use viper to configure instead of direct field access
cleanupViperConfig(t, testBackpressureEnabled, testBackpressureMaxFiles)
viper.Set(testBackpressureEnabled, true)
viper.Set(testBackpressureMaxFiles, 10)
bm := NewBackpressureManager()
fileCh := make(chan string, 10)
writeCh := make(chan WriteRequest, 10)
// Fill channel
for i := 0; i < 10; i++ {
fileCh <- "file.txt"
}
ctx, cancel := context.WithCancel(context.Background())
cancel() // Cancel immediately
// Use timeout to verify it returns quickly
done := make(chan struct{})
go func() {
bm.WaitForChannelSpace(ctx, fileCh, writeCh)
close(done)
}()
// Should return quickly when context is cancelled
select {
case <-done:
// Success - returned due to cancellation
case <-time.After(fastOpTimeout):
t.Fatal("WaitForChannelSpace should return immediately when context is cancelled")
}
// Clean up
for i := 0; i < 10; i++ {
<-fileCh
}
close(fileCh)
close(writeCh)
})
}

View File

@@ -1,195 +0,0 @@
package fileproc
import (
"context"
"sync"
"testing"
"time"
"github.com/spf13/viper"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestBackpressureManagerConcurrency(t *testing.T) {
// Configure via viper instead of direct field access
origEnabled := viper.Get(testBackpressureEnabled)
t.Cleanup(func() {
if origEnabled != nil {
viper.Set(testBackpressureEnabled, origEnabled)
}
})
viper.Set(testBackpressureEnabled, true)
bm := NewBackpressureManager()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
var wg sync.WaitGroup
// Multiple goroutines checking backpressure
for i := 0; i < 10; i++ {
wg.Add(1)
go func() {
defer wg.Done()
bm.ShouldApplyBackpressure(ctx)
}()
}
// Multiple goroutines applying backpressure
for i := 0; i < 5; i++ {
wg.Add(1)
go func() {
defer wg.Done()
bm.ApplyBackpressure(ctx)
}()
}
// Multiple goroutines getting stats
for i := 0; i < 5; i++ {
wg.Add(1)
go func() {
defer wg.Done()
bm.GetStats()
}()
}
// Multiple goroutines creating channels
// Note: CreateChannels returns new channels each time, caller owns them
type channelResult struct {
fileCh chan string
writeCh chan WriteRequest
}
results := make(chan channelResult, 3)
for i := 0; i < 3; i++ {
wg.Add(1)
go func() {
defer wg.Done()
fileCh, writeCh := bm.CreateChannels()
results <- channelResult{fileCh, writeCh}
}()
}
wg.Wait()
close(results)
// Verify channels are created and have expected properties
for result := range results {
assert.NotNil(t, result.fileCh)
assert.NotNil(t, result.writeCh)
// Close channels to prevent resource leak (caller owns them)
close(result.fileCh)
close(result.writeCh)
}
// Verify stats are consistent
stats := bm.GetStats()
assert.GreaterOrEqual(t, stats.FilesProcessed, int64(10))
}
func TestBackpressureManagerIntegration(t *testing.T) {
// Configure via viper instead of direct field access
origEnabled := viper.Get(testBackpressureEnabled)
origMaxFiles := viper.Get(testBackpressureMaxFiles)
origMaxWrites := viper.Get(testBackpressureMaxWrites)
origCheckInterval := viper.Get(testBackpressureMemoryCheck)
origMaxMemory := viper.Get(testBackpressureMaxMemory)
t.Cleanup(func() {
if origEnabled != nil {
viper.Set(testBackpressureEnabled, origEnabled)
}
if origMaxFiles != nil {
viper.Set(testBackpressureMaxFiles, origMaxFiles)
}
if origMaxWrites != nil {
viper.Set(testBackpressureMaxWrites, origMaxWrites)
}
if origCheckInterval != nil {
viper.Set(testBackpressureMemoryCheck, origCheckInterval)
}
if origMaxMemory != nil {
viper.Set(testBackpressureMaxMemory, origMaxMemory)
}
})
viper.Set(testBackpressureEnabled, true)
viper.Set(testBackpressureMaxFiles, 10)
viper.Set(testBackpressureMaxWrites, 10)
viper.Set(testBackpressureMemoryCheck, 10)
viper.Set(testBackpressureMaxMemory, 100*1024*1024) // 100MB
bm := NewBackpressureManager()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Create channels - caller owns these channels and is responsible for closing them
fileCh, writeCh := bm.CreateChannels()
require.NotNil(t, fileCh)
require.NotNil(t, writeCh)
require.Greater(t, cap(fileCh), 0, "fileCh should be buffered")
require.Greater(t, cap(writeCh), 0, "writeCh should be buffered")
// Simulate file processing
var wg sync.WaitGroup
// Producer
wg.Add(1)
go func() {
defer wg.Done()
for i := 0; i < 100; i++ {
// Check for backpressure
if bm.ShouldApplyBackpressure(ctx) {
bm.ApplyBackpressure(ctx)
}
// Wait for channel space if needed
bm.WaitForChannelSpace(ctx, fileCh, writeCh)
select {
case fileCh <- "file.txt":
// File sent
case <-ctx.Done():
return
}
}
}()
// Consumer
wg.Add(1)
go func() {
defer wg.Done()
for i := 0; i < 100; i++ {
select {
case <-fileCh:
// Process file (do not manually increment filesProcessed)
case <-ctx.Done():
return
}
}
}()
// Wait for completion
done := make(chan struct{})
go func() {
wg.Wait()
close(done)
}()
select {
case <-done:
// Success
case <-time.After(5 * time.Second):
t.Fatal("Integration test timeout")
}
// Log final info
bm.LogBackpressureInfo()
// Check final stats
stats := bm.GetStats()
assert.GreaterOrEqual(t, stats.FilesProcessed, int64(100))
// Clean up - caller owns the channels, safe to close now that goroutines have finished
close(fileCh)
close(writeCh)
}

View File

@@ -1,151 +0,0 @@
package fileproc
import (
"context"
"testing"
"github.com/spf13/viper"
"github.com/stretchr/testify/assert"
)
// setupViperCleanup is a test helper that captures and restores viper configuration.
// It takes a testing.T and a list of config keys to save/restore.
func setupViperCleanup(t *testing.T, keys []string) {
t.Helper()
// Capture original values and track which keys existed
origValues := make(map[string]interface{})
keysExisted := make(map[string]bool)
for _, key := range keys {
val := viper.Get(key)
origValues[key] = val
keysExisted[key] = viper.IsSet(key)
}
// Register cleanup to restore values
t.Cleanup(func() {
for _, key := range keys {
if keysExisted[key] {
viper.Set(key, origValues[key])
} else {
// Key didn't exist originally, so remove it
allSettings := viper.AllSettings()
delete(allSettings, key)
viper.Reset()
for k, v := range allSettings {
viper.Set(k, v)
}
}
}
})
}
func TestNewBackpressureManager(t *testing.T) {
keys := []string{
testBackpressureEnabled,
testBackpressureMaxMemory,
testBackpressureMemoryCheck,
testBackpressureMaxFiles,
testBackpressureMaxWrites,
}
setupViperCleanup(t, keys)
viper.Set(testBackpressureEnabled, true)
viper.Set(testBackpressureMaxMemory, 100)
viper.Set(testBackpressureMemoryCheck, 10)
viper.Set(testBackpressureMaxFiles, 10)
viper.Set(testBackpressureMaxWrites, 10)
bm := NewBackpressureManager()
assert.NotNil(t, bm)
assert.True(t, bm.enabled)
assert.Greater(t, bm.maxMemoryUsage, int64(0))
assert.Greater(t, bm.memoryCheckInterval, 0)
assert.Greater(t, bm.maxPendingFiles, 0)
assert.Greater(t, bm.maxPendingWrites, 0)
assert.Equal(t, int64(0), bm.filesProcessed)
}
func TestBackpressureStatsStructure(t *testing.T) {
// Behavioral test that exercises BackpressureManager and validates stats
keys := []string{
testBackpressureEnabled,
testBackpressureMaxMemory,
testBackpressureMemoryCheck,
testBackpressureMaxFiles,
testBackpressureMaxWrites,
}
setupViperCleanup(t, keys)
// Configure backpressure with realistic settings
viper.Set(testBackpressureEnabled, true)
viper.Set(testBackpressureMaxMemory, 100*1024*1024) // 100MB
viper.Set(testBackpressureMemoryCheck, 1) // Check every file
viper.Set(testBackpressureMaxFiles, 1000)
viper.Set(testBackpressureMaxWrites, 500)
bm := NewBackpressureManager()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Simulate processing files
initialStats := bm.GetStats()
assert.True(t, initialStats.Enabled, "backpressure should be enabled")
assert.Equal(t, int64(0), initialStats.FilesProcessed, "initially no files processed")
// Capture initial timestamp to verify it gets updated
initialLastCheck := initialStats.LastMemoryCheck
// Process some files to trigger memory checks
for i := 0; i < 5; i++ {
bm.ShouldApplyBackpressure(ctx)
}
// Verify stats reflect the operations
stats := bm.GetStats()
assert.True(t, stats.Enabled, "enabled flag should be set")
assert.Equal(t, int64(5), stats.FilesProcessed, "should have processed 5 files")
assert.Greater(t, stats.CurrentMemoryUsage, int64(0), "memory usage should be tracked")
assert.Equal(t, int64(100*1024*1024), stats.MaxMemoryUsage, "max memory should match config")
assert.Equal(t, 1000, stats.MaxPendingFiles, "maxPendingFiles should match config")
assert.Equal(t, 500, stats.MaxPendingWrites, "maxPendingWrites should match config")
assert.True(t, stats.LastMemoryCheck.After(initialLastCheck) || stats.LastMemoryCheck.Equal(initialLastCheck),
"lastMemoryCheck should be updated or remain initialized")
}
func TestBackpressureManagerGetStats(t *testing.T) {
keys := []string{
testBackpressureEnabled,
testBackpressureMemoryCheck,
}
setupViperCleanup(t, keys)
// Ensure config enables backpressure and checks every call
viper.Set(testBackpressureEnabled, true)
viper.Set(testBackpressureMemoryCheck, 1)
bm := NewBackpressureManager()
// Capture initial timestamp to verify it gets updated
initialStats := bm.GetStats()
initialLastCheck := initialStats.LastMemoryCheck
// Process some files to update stats
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
for i := 0; i < 5; i++ {
bm.ShouldApplyBackpressure(ctx)
}
stats := bm.GetStats()
assert.True(t, stats.Enabled)
assert.Equal(t, int64(5), stats.FilesProcessed)
assert.Greater(t, stats.CurrentMemoryUsage, int64(0))
assert.Equal(t, bm.maxMemoryUsage, stats.MaxMemoryUsage)
assert.Equal(t, bm.maxPendingFiles, stats.MaxPendingFiles)
assert.Equal(t, bm.maxPendingWrites, stats.MaxPendingWrites)
// LastMemoryCheck should be updated after processing files (memoryCheckInterval=1)
assert.True(t, stats.LastMemoryCheck.After(initialLastCheck),
"lastMemoryCheck should be updated after memory checks")
}

View File

@@ -0,0 +1,344 @@
package fileproc_test
import (
"context"
"runtime"
"testing"
"time"
"github.com/ivuorinen/gibidify/fileproc"
"github.com/ivuorinen/gibidify/shared"
"github.com/ivuorinen/gibidify/testutil"
)
func TestNewBackpressureManager(t *testing.T) {
// Test creating a new backpressure manager
bp := fileproc.NewBackpressureManager()
if bp == nil {
t.Error("Expected backpressure manager to be created, got nil")
}
// The backpressure manager should be initialized with config values
// We can't test the internal values directly since they're private,
// but we can test that it was created successfully
}
func TestBackpressureManagerCreateChannels(t *testing.T) {
testutil.ResetViperConfig(t, "")
bp := fileproc.NewBackpressureManager()
// Test creating channels
fileCh, writeCh := bp.CreateChannels()
// Verify channels are created
if fileCh == nil {
t.Error("Expected file channel to be created, got nil")
}
if writeCh == nil {
t.Error("Expected write channel to be created, got nil")
}
// Test that channels can be used
select {
case fileCh <- "test-file":
// Successfully sent to channel
default:
t.Error("Unable to send to file channel")
}
// Read from channel
select {
case file := <-fileCh:
if file != "test-file" {
t.Errorf("Expected 'test-file', got %s", file)
}
case <-time.After(100 * time.Millisecond):
t.Error("Timeout reading from file channel")
}
}
func TestBackpressureManagerShouldApplyBackpressure(t *testing.T) {
testutil.ResetViperConfig(t, "")
bp := fileproc.NewBackpressureManager()
ctx := context.Background()
// Test backpressure decision
shouldApply := bp.ShouldApplyBackpressure(ctx)
// Since we're using default config, backpressure behavior depends on settings
// We just test that the method returns without error
// shouldApply is a valid boolean value
_ = shouldApply
}
func TestBackpressureManagerApplyBackpressure(t *testing.T) {
testutil.ResetViperConfig(t, "")
bp := fileproc.NewBackpressureManager()
ctx := context.Background()
// Test applying backpressure
bp.ApplyBackpressure(ctx)
// ApplyBackpressure is a void method that should not panic
// If we reach here, the method executed successfully
}
func TestBackpressureManagerApplyBackpressureWithCancellation(t *testing.T) {
testutil.ResetViperConfig(t, "")
bp := fileproc.NewBackpressureManager()
// Create canceled context
ctx, cancel := context.WithCancel(context.Background())
cancel() // Cancel immediately
// Test applying backpressure with canceled context
bp.ApplyBackpressure(ctx)
// ApplyBackpressure doesn't return errors, but should handle cancellation gracefully
// If we reach here without hanging, the method handled cancellation properly
}
func TestBackpressureManagerGetStats(t *testing.T) {
testutil.ResetViperConfig(t, "")
bp := fileproc.NewBackpressureManager()
// Test getting stats
stats := bp.Stats()
// Stats should contain relevant information
if stats.FilesProcessed < 0 {
t.Error("Expected non-negative files processed count")
}
if stats.CurrentMemoryUsage < 0 {
t.Error("Expected non-negative memory usage")
}
if stats.MaxMemoryUsage < 0 {
t.Error("Expected non-negative max memory usage")
}
// Test that stats have reasonable values
if stats.MaxPendingFiles < 0 || stats.MaxPendingWrites < 0 {
t.Error("Expected non-negative channel buffer sizes")
}
}
func TestBackpressureManagerWaitForChannelSpace(t *testing.T) {
testutil.ResetViperConfig(t, "")
bp := fileproc.NewBackpressureManager()
ctx := context.Background()
// Create test channels
fileCh, writeCh := bp.CreateChannels()
// Test waiting for channel space
bp.WaitForChannelSpace(ctx, fileCh, writeCh)
// WaitForChannelSpace is void method that should complete without hanging
// If we reach here, the method executed successfully
}
func TestBackpressureManagerWaitForChannelSpaceWithCancellation(t *testing.T) {
testutil.ResetViperConfig(t, "")
bp := fileproc.NewBackpressureManager()
// Create canceled context
ctx, cancel := context.WithCancel(context.Background())
cancel()
// Create test channels
fileCh, writeCh := bp.CreateChannels()
// Test waiting for channel space with canceled context
bp.WaitForChannelSpace(ctx, fileCh, writeCh)
// WaitForChannelSpace should handle cancellation gracefully without hanging
// If we reach here, the method handled cancellation properly
}
func TestBackpressureManagerLogBackpressureInfo(t *testing.T) {
testutil.ResetViperConfig(t, "")
bp := fileproc.NewBackpressureManager()
// Test logging backpressure info
// This method primarily logs information, so we test it executes without panic
bp.LogBackpressureInfo()
// If we reach here without panic, the method worked
}
// BenchmarkBackpressureManager benchmarks backpressure operations.
func BenchmarkBackpressureManagerCreateChannels(b *testing.B) {
bp := fileproc.NewBackpressureManager()
b.ResetTimer()
for i := 0; i < b.N; i++ {
fileCh, writeCh := bp.CreateChannels()
// Use channels to prevent optimization
_ = fileCh
_ = writeCh
runtime.GC() // Force GC to measure memory impact
}
}
func BenchmarkBackpressureManagerShouldApplyBackpressure(b *testing.B) {
bp := fileproc.NewBackpressureManager()
ctx := context.Background()
b.ResetTimer()
for i := 0; i < b.N; i++ {
shouldApply := bp.ShouldApplyBackpressure(ctx)
_ = shouldApply // Prevent optimization
}
}
func BenchmarkBackpressureManagerApplyBackpressure(b *testing.B) {
bp := fileproc.NewBackpressureManager()
ctx := context.Background()
b.ResetTimer()
for i := 0; i < b.N; i++ {
bp.ApplyBackpressure(ctx)
}
}
func BenchmarkBackpressureManagerGetStats(b *testing.B) {
bp := fileproc.NewBackpressureManager()
b.ResetTimer()
for i := 0; i < b.N; i++ {
stats := bp.Stats()
_ = stats // Prevent optimization
}
}
// TestBackpressureManager_ShouldApplyBackpressure_EdgeCases tests various edge cases for backpressure decision.
func TestBackpressureManagerShouldApplyBackpressureEdgeCases(t *testing.T) {
testutil.ApplyBackpressureOverrides(t, map[string]any{
shared.ConfigKeyBackpressureEnabled: true,
"backpressure.memory_check_interval": 2,
"backpressure.memory_limit_mb": 1,
})
bp := fileproc.NewBackpressureManager()
ctx := context.Background()
// Test multiple calls to trigger memory check interval logic
for i := 0; i < 10; i++ {
shouldApply := bp.ShouldApplyBackpressure(ctx)
_ = shouldApply
}
// At this point, memory checking should have triggered multiple times
// The actual decision depends on memory usage, but we're testing the paths
}
// TestBackpressureManager_CreateChannels_EdgeCases tests edge cases in channel creation.
func TestBackpressureManagerCreateChannelsEdgeCases(t *testing.T) {
// Test with custom configuration that might trigger different buffer sizes
testutil.ApplyBackpressureOverrides(t, map[string]any{
"backpressure.file_buffer_size": 50,
"backpressure.write_buffer_size": 25,
})
bp := fileproc.NewBackpressureManager()
// Create multiple channel sets to test resource management
for i := 0; i < 5; i++ {
fileCh, writeCh := bp.CreateChannels()
// Verify channels work correctly
select {
case fileCh <- "test":
// Good - channel accepted value
default:
// This is also acceptable if buffer is full
}
// Test write channel
select {
case writeCh <- fileproc.WriteRequest{Path: "test", Content: "content"}:
// Good - channel accepted value
default:
// This is also acceptable if buffer is full
}
}
}
// TestBackpressureManager_WaitForChannelSpace_EdgeCases tests edge cases in channel space waiting.
func TestBackpressureManagerWaitForChannelSpaceEdgeCases(t *testing.T) {
testutil.ApplyBackpressureOverrides(t, map[string]any{
shared.ConfigKeyBackpressureEnabled: true,
"backpressure.wait_timeout_ms": 10,
})
bp := fileproc.NewBackpressureManager()
ctx := context.Background()
// Create channels with small buffers
fileCh, writeCh := bp.CreateChannels()
// Fill up the channels to create pressure
go func() {
for i := 0; i < 100; i++ {
select {
case fileCh <- "file":
case <-time.After(1 * time.Millisecond):
}
}
}()
go func() {
for i := 0; i < 100; i++ {
select {
case writeCh <- fileproc.WriteRequest{Path: "test", Content: "content"}:
case <-time.After(1 * time.Millisecond):
}
}
}()
// Wait for channel space - should handle the full channels
bp.WaitForChannelSpace(ctx, fileCh, writeCh)
}
// TestBackpressureManager_MemoryPressure tests behavior under simulated memory pressure.
func TestBackpressureManagerMemoryPressure(t *testing.T) {
// Test with very low memory limit to trigger backpressure
testutil.ApplyBackpressureOverrides(t, map[string]any{
shared.ConfigKeyBackpressureEnabled: true,
"backpressure.memory_limit_mb": 0.001,
"backpressure.memory_check_interval": 1,
})
bp := fileproc.NewBackpressureManager()
ctx := context.Background()
// Allocate some memory to potentially trigger limits
largeBuffer := make([]byte, 1024*1024) // 1MB
_ = largeBuffer[0]
// Test backpressure decision under memory pressure
for i := 0; i < 5; i++ {
shouldApply := bp.ShouldApplyBackpressure(ctx)
if shouldApply {
// Test applying backpressure when needed
bp.ApplyBackpressure(ctx)
t.Log("Backpressure applied due to memory pressure")
}
}
// Test logging
bp.LogBackpressureInfo()
}

View File

@@ -1,3 +1,4 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
// getNormalizedExtension efficiently extracts and normalizes the file extension with caching. // getNormalizedExtension efficiently extracts and normalizes the file extension with caching.
@@ -6,6 +7,7 @@ func (r *FileTypeRegistry) getNormalizedExtension(filename string) string {
r.cacheMutex.RLock() r.cacheMutex.RLock()
if ext, exists := r.extCache[filename]; exists { if ext, exists := r.extCache[filename]; exists {
r.cacheMutex.RUnlock() r.cacheMutex.RUnlock()
return ext return ext
} }
r.cacheMutex.RUnlock() r.cacheMutex.RUnlock()
@@ -42,6 +44,7 @@ func (r *FileTypeRegistry) getFileTypeResult(filename string) FileTypeResult {
r.updateStats(func() { r.updateStats(func() {
r.stats.CacheHits++ r.stats.CacheHits++
}) })
return result return result
} }
r.cacheMutex.RUnlock() r.cacheMutex.RUnlock()

View File

@@ -5,5 +5,6 @@ package fileproc
// and returns a slice of file paths. // and returns a slice of file paths.
func CollectFiles(root string) ([]string, error) { func CollectFiles(root string) ([]string, error) {
w := NewProdWalker() w := NewProdWalker()
return w.Walk(root) return w.Walk(root)
} }

View File

@@ -2,6 +2,7 @@ package fileproc_test
import ( import (
"os" "os"
"path/filepath"
"testing" "testing"
"github.com/ivuorinen/gibidify/fileproc" "github.com/ivuorinen/gibidify/fileproc"
@@ -47,3 +48,70 @@ func TestCollectFilesError(t *testing.T) {
t.Fatal("Expected an error, got nil") t.Fatal("Expected an error, got nil")
} }
} }
// TestCollectFiles tests the actual CollectFiles function with a real directory.
func TestCollectFiles(t *testing.T) {
// Create a temporary directory with test files
tmpDir := t.TempDir()
// Create test files with known supported extensions
testFiles := map[string]string{
"test1.go": "package main\n\nfunc main() {\n\t// Go file\n}",
"test2.py": "# Python file\nprint('hello world')",
"test3.js": "// JavaScript file\nconsole.log('hello');",
}
for name, content := range testFiles {
filePath := filepath.Join(tmpDir, name)
if err := os.WriteFile(filePath, []byte(content), 0o600); err != nil {
t.Fatalf("Failed to create test file %s: %v", name, err)
}
}
// Test CollectFiles
files, err := fileproc.CollectFiles(tmpDir)
if err != nil {
t.Fatalf("CollectFiles failed: %v", err)
}
// Verify we got the expected number of files
if len(files) != len(testFiles) {
t.Errorf("Expected %d files, got %d", len(testFiles), len(files))
}
// Verify all expected files are found
foundFiles := make(map[string]bool)
for _, file := range files {
foundFiles[file] = true
}
for expectedFile := range testFiles {
expectedPath := filepath.Join(tmpDir, expectedFile)
if !foundFiles[expectedPath] {
t.Errorf("Expected file %s not found in results", expectedPath)
}
}
}
// TestCollectFiles_NonExistentDirectory tests CollectFiles with a non-existent directory.
func TestCollectFilesNonExistentDirectory(t *testing.T) {
_, err := fileproc.CollectFiles("/non/existent/directory")
if err == nil {
t.Error("Expected error for non-existent directory, got nil")
}
}
// TestCollectFiles_EmptyDirectory tests CollectFiles with an empty directory.
func TestCollectFilesEmptyDirectory(t *testing.T) {
tmpDir := t.TempDir()
// Don't create any files
files, err := fileproc.CollectFiles(tmpDir)
if err != nil {
t.Fatalf("CollectFiles failed on empty directory: %v", err)
}
if len(files) != 0 {
t.Errorf("Expected 0 files in empty directory, got %d", len(files))
}
}

View File

@@ -1,156 +1,7 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import ( import "strings"
"fmt"
"path/filepath"
"strings"
)
const (
// MaxRegistryEntries is the maximum number of entries allowed in registry config slices/maps.
MaxRegistryEntries = 1000
// MaxExtensionLength is the maximum length for a single extension string.
MaxExtensionLength = 100
)
// RegistryConfig holds configuration for file type registry.
// All paths must be relative without path traversal (no ".." or leading "/").
// Extensions in CustomLanguages keys must start with "." or be alphanumeric with underscore/hyphen.
type RegistryConfig struct {
// CustomImages: file extensions to treat as images (e.g., ".svg", ".webp").
// Must be relative paths without ".." or leading separators.
CustomImages []string
// CustomBinary: file extensions to treat as binary (e.g., ".bin", ".dat").
// Must be relative paths without ".." or leading separators.
CustomBinary []string
// CustomLanguages: maps file extensions to language names (e.g., {".tsx": "TypeScript"}).
// Keys must start with "." or be alphanumeric with underscore/hyphen.
CustomLanguages map[string]string
// DisabledImages: image extensions to disable from default registry.
DisabledImages []string
// DisabledBinary: binary extensions to disable from default registry.
DisabledBinary []string
// DisabledLanguages: language extensions to disable from default registry.
DisabledLanguages []string
}
// Validate checks the RegistryConfig for invalid entries and enforces limits.
func (c *RegistryConfig) Validate() error {
// Validate CustomImages
if err := validateExtensionSlice(c.CustomImages, "CustomImages"); err != nil {
return err
}
// Validate CustomBinary
if err := validateExtensionSlice(c.CustomBinary, "CustomBinary"); err != nil {
return err
}
// Validate CustomLanguages
if len(c.CustomLanguages) > MaxRegistryEntries {
return fmt.Errorf(
"CustomLanguages exceeds maximum entries (%d > %d)",
len(c.CustomLanguages),
MaxRegistryEntries,
)
}
for ext, lang := range c.CustomLanguages {
if err := validateExtension(ext, "CustomLanguages key"); err != nil {
return err
}
if len(lang) > MaxExtensionLength {
return fmt.Errorf(
"CustomLanguages value %q exceeds maximum length (%d > %d)",
lang,
len(lang),
MaxExtensionLength,
)
}
}
// Validate Disabled slices
if err := validateExtensionSlice(c.DisabledImages, "DisabledImages"); err != nil {
return err
}
if err := validateExtensionSlice(c.DisabledBinary, "DisabledBinary"); err != nil {
return err
}
return validateExtensionSlice(c.DisabledLanguages, "DisabledLanguages")
}
// validateExtensionSlice validates a slice of extensions for path safety and limits.
func validateExtensionSlice(slice []string, fieldName string) error {
if len(slice) > MaxRegistryEntries {
return fmt.Errorf("%s exceeds maximum entries (%d > %d)", fieldName, len(slice), MaxRegistryEntries)
}
for _, ext := range slice {
if err := validateExtension(ext, fieldName); err != nil {
return err
}
}
return nil
}
// validateExtension validates a single extension for path safety.
//
//revive:disable-next-line:cyclomatic
func validateExtension(ext, context string) error {
// Reject empty strings
if ext == "" {
return fmt.Errorf("%s entry cannot be empty", context)
}
if len(ext) > MaxExtensionLength {
return fmt.Errorf(
"%s entry %q exceeds maximum length (%d > %d)",
context, ext, len(ext), MaxExtensionLength,
)
}
// Reject absolute paths
if filepath.IsAbs(ext) {
return fmt.Errorf("%s entry %q is an absolute path (not allowed)", context, ext)
}
// Reject path traversal
if strings.Contains(ext, "..") {
return fmt.Errorf("%s entry %q contains path traversal (not allowed)", context, ext)
}
// For extensions, verify they start with "." or are alphanumeric
if strings.HasPrefix(ext, ".") {
// Reject extensions containing path separators
if strings.ContainsRune(ext, filepath.Separator) || strings.ContainsRune(ext, '/') ||
strings.ContainsRune(ext, '\\') {
return fmt.Errorf("%s entry %q contains path separators (not allowed)", context, ext)
}
// Valid extension format
return nil
}
// Check if purely alphanumeric (for bare names)
for _, r := range ext {
isValid := (r >= 'a' && r <= 'z') ||
(r >= 'A' && r <= 'Z') ||
(r >= '0' && r <= '9') ||
r == '_' || r == '-'
if !isValid {
return fmt.Errorf(
"%s entry %q contains invalid characters (must start with '.' or be alphanumeric/_/-)",
context,
ext,
)
}
}
return nil
}
// ApplyCustomExtensions applies custom extensions from configuration. // ApplyCustomExtensions applies custom extensions from configuration.
func (r *FileTypeRegistry) ApplyCustomExtensions( func (r *FileTypeRegistry) ApplyCustomExtensions(
@@ -182,24 +33,12 @@ func (r *FileTypeRegistry) addExtensions(extensions []string, adder func(string)
// ConfigureFromSettings applies configuration settings to the registry. // ConfigureFromSettings applies configuration settings to the registry.
// This function is called from main.go after config is loaded to avoid circular imports. // This function is called from main.go after config is loaded to avoid circular imports.
// It validates the configuration before applying it. func ConfigureFromSettings(
func ConfigureFromSettings(config RegistryConfig) error { customImages, customBinary []string,
// Validate configuration first customLanguages map[string]string,
if err := config.Validate(); err != nil { disabledImages, disabledBinary, disabledLanguages []string,
return err ) {
} registry := DefaultRegistry()
registry.ApplyCustomExtensions(customImages, customBinary, customLanguages)
registry := GetDefaultRegistry() registry.DisableExtensions(disabledImages, disabledBinary, disabledLanguages)
// Only apply custom extensions if they are non-empty (len() for nil slices/maps is zero)
if len(config.CustomImages) > 0 || len(config.CustomBinary) > 0 || len(config.CustomLanguages) > 0 {
registry.ApplyCustomExtensions(config.CustomImages, config.CustomBinary, config.CustomLanguages)
}
// Only disable extensions if they are non-empty
if len(config.DisabledImages) > 0 || len(config.DisabledBinary) > 0 || len(config.DisabledLanguages) > 0 {
registry.DisableExtensions(config.DisabledImages, config.DisabledBinary, config.DisabledLanguages)
}
return nil
} }

View File

@@ -1,3 +1,4 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import "strings" import "strings"
@@ -14,9 +15,9 @@ func IsBinary(filename string) bool {
return getRegistry().IsBinary(filename) return getRegistry().IsBinary(filename)
} }
// GetLanguage returns the language identifier for the given filename based on its extension. // Language returns the language identifier for the given filename based on its extension.
func GetLanguage(filename string) string { func Language(filename string) string {
return getRegistry().GetLanguage(filename) return getRegistry().Language(filename)
} }
// Registry methods for detection // Registry methods for detection
@@ -24,21 +25,24 @@ func GetLanguage(filename string) string {
// IsImage checks if the file extension indicates an image file. // IsImage checks if the file extension indicates an image file.
func (r *FileTypeRegistry) IsImage(filename string) bool { func (r *FileTypeRegistry) IsImage(filename string) bool {
result := r.getFileTypeResult(filename) result := r.getFileTypeResult(filename)
return result.IsImage return result.IsImage
} }
// IsBinary checks if the file extension indicates a binary file. // IsBinary checks if the file extension indicates a binary file.
func (r *FileTypeRegistry) IsBinary(filename string) bool { func (r *FileTypeRegistry) IsBinary(filename string) bool {
result := r.getFileTypeResult(filename) result := r.getFileTypeResult(filename)
return result.IsBinary return result.IsBinary
} }
// GetLanguage returns the language identifier for the given filename based on its extension. // Language returns the language identifier for the given filename based on its extension.
func (r *FileTypeRegistry) GetLanguage(filename string) string { func (r *FileTypeRegistry) Language(filename string) string {
if len(filename) < minExtensionLength { if len(filename) < minExtensionLength {
return "" return ""
} }
result := r.getFileTypeResult(filename) result := r.getFileTypeResult(filename)
return result.Language return result.Language
} }

View File

@@ -1,5 +1,8 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import "github.com/ivuorinen/gibidify/shared"
// getImageExtensions returns the default image file extensions. // getImageExtensions returns the default image file extensions.
func getImageExtensions() map[string]bool { func getImageExtensions() map[string]bool {
return map[string]bool{ return map[string]bool{
@@ -130,15 +133,15 @@ func getLanguageMap() map[string]string {
".cmd": "batch", ".cmd": "batch",
// Data formats // Data formats
".json": "json", ".json": shared.FormatJSON,
".yaml": "yaml", ".yaml": shared.FormatYAML,
".yml": "yaml", ".yml": shared.FormatYAML,
".toml": "toml", ".toml": "toml",
".xml": "xml", ".xml": "xml",
".sql": "sql", ".sql": "sql",
// Documentation // Documentation
".md": "markdown", ".md": shared.FormatMarkdown,
".rst": "rst", ".rst": "rst",
".tex": "latex", ".tex": "latex",

View File

@@ -12,5 +12,6 @@ func (fw FakeWalker) Walk(_ string) ([]string, error) {
if fw.Err != nil { if fw.Err != nil {
return nil, fw.Err return nil, fw.Err
} }
return fw.Files, nil return fw.Files, nil
} }

View File

@@ -1,3 +1,4 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import ( import (
@@ -15,8 +16,8 @@ type FileFilter struct {
// NewFileFilter creates a new file filter with current configuration. // NewFileFilter creates a new file filter with current configuration.
func NewFileFilter() *FileFilter { func NewFileFilter() *FileFilter {
return &FileFilter{ return &FileFilter{
ignoredDirs: config.GetIgnoredDirectories(), ignoredDirs: config.IgnoredDirectories(),
sizeLimit: config.GetFileSizeLimit(), sizeLimit: config.FileSizeLimit(),
} }
} }
@@ -40,6 +41,7 @@ func (f *FileFilter) shouldSkipDirectory(entry os.DirEntry) bool {
return true return true
} }
} }
return false return false
} }

View File

@@ -1,105 +1,200 @@
package fileproc package fileproc
import ( import (
"errors"
"fmt" "fmt"
"sync" "sync"
"testing" "testing"
"github.com/ivuorinen/gibidify/shared"
) )
// TestFileTypeRegistry_ThreadSafety tests thread safety of the FileTypeRegistry. const (
func TestFileTypeRegistry_ThreadSafety(t *testing.T) { numGoroutines = 100
const numGoroutines = 100 numOperationsPerGoroutine = 100
const numOperationsPerGoroutine = 100 )
// TestFileTypeRegistryConcurrentReads tests concurrent read operations.
// This test verifies thread-safety of registry reads under concurrent access.
// For race condition detection, run with: go test -race
func TestFileTypeRegistryConcurrentReads(t *testing.T) {
var wg sync.WaitGroup
errChan := make(chan error, numGoroutines)
for i := 0; i < numGoroutines; i++ {
wg.Go(func() {
if err := performConcurrentReads(); err != nil {
errChan <- err
}
})
}
wg.Wait()
close(errChan)
// Check for any errors from goroutines
for err := range errChan {
t.Errorf("Concurrent read operation failed: %v", err)
}
}
// TestFileTypeRegistryConcurrentRegistryAccess tests concurrent registry access.
func TestFileTypeRegistryConcurrentRegistryAccess(t *testing.T) {
// Reset the registry to test concurrent initialization
ResetRegistryForTesting()
t.Cleanup(func() {
ResetRegistryForTesting()
})
registries := make([]*FileTypeRegistry, numGoroutines)
var wg sync.WaitGroup var wg sync.WaitGroup
// Test concurrent read operations for i := 0; i < numGoroutines; i++ {
t.Run("ConcurrentReads", func(_ *testing.T) { idx := i // capture for closure
for i := 0; i < numGoroutines; i++ { wg.Go(func() {
wg.Add(1) registries[idx] = DefaultRegistry()
go func(_ int) { })
defer wg.Done() }
registry := GetDefaultRegistry() wg.Wait()
for j := 0; j < numOperationsPerGoroutine; j++ { verifySameRegistryInstance(t, registries)
// Test various file detection operations }
_ = registry.IsImage("test.png")
_ = registry.IsBinary("test.exe")
_ = registry.GetLanguage("test.go")
// Test global functions too // TestFileTypeRegistryConcurrentModifications tests concurrent modifications.
_ = IsImage("image.jpg") func TestFileTypeRegistryConcurrentModifications(t *testing.T) {
_ = IsBinary("binary.dll") var wg sync.WaitGroup
_ = GetLanguage("script.py")
} for i := 0; i < numGoroutines; i++ {
}(i) id := i // capture for closure
wg.Go(func() {
performConcurrentModifications(t, id)
})
}
wg.Wait()
}
// performConcurrentReads performs concurrent read operations on the registry.
// Returns an error if any operation produces unexpected results.
func performConcurrentReads() error {
registry := DefaultRegistry()
for j := 0; j < numOperationsPerGoroutine; j++ {
// Test various file detection operations with expected results
if !registry.IsImage(shared.TestFilePNG) {
return errors.New("expected .png to be detected as image")
} }
wg.Wait() if !registry.IsBinary(shared.TestFileEXE) {
}) return errors.New("expected .exe to be detected as binary")
// Test concurrent registry access (singleton creation)
t.Run("ConcurrentRegistryAccess", func(t *testing.T) {
// Reset the registry to test concurrent initialization
// Note: This is not safe in a real application, but needed for testing
registryOnce = sync.Once{}
registry = nil
registries := make([]*FileTypeRegistry, numGoroutines)
for i := 0; i < numGoroutines; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
registries[id] = GetDefaultRegistry()
}(i)
} }
wg.Wait() if lang := registry.Language(shared.TestFileGo); lang != "go" {
return fmt.Errorf("expected .go to have language 'go', got %q", lang)
// Verify all goroutines got the same registry instance
firstRegistry := registries[0]
for i := 1; i < numGoroutines; i++ {
if registries[i] != firstRegistry {
t.Errorf("Registry %d is different from registry 0", i)
}
} }
})
// Test concurrent modifications on separate registry instances // Test global functions with expected results
t.Run("ConcurrentModifications", func(t *testing.T) { if !IsImage(shared.TestFileImageJPG) {
// Create separate registry instances for each goroutine to test modification thread safety return errors.New("expected .jpg to be detected as image")
for i := 0; i < numGoroutines; i++ { }
wg.Add(1) if !IsBinary(shared.TestFileBinaryDLL) {
go func(id int) { return errors.New("expected .dll to be detected as binary")
defer wg.Done() }
if lang := Language(shared.TestFileScriptPy); lang != "python" {
// Create a new registry instance for this goroutine return fmt.Errorf("expected .py to have language 'python', got %q", lang)
registry := &FileTypeRegistry{ }
imageExts: make(map[string]bool), }
binaryExts: make(map[string]bool), return nil
languageMap: make(map[string]string), }
}
// verifySameRegistryInstance verifies all goroutines got the same registry instance.
for j := 0; j < numOperationsPerGoroutine; j++ { func verifySameRegistryInstance(t *testing.T, registries []*FileTypeRegistry) {
// Add unique extensions for this goroutine t.Helper()
extSuffix := fmt.Sprintf("_%d_%d", id, j)
firstRegistry := registries[0]
registry.AddImageExtension(".img" + extSuffix) for i := 1; i < numGoroutines; i++ {
registry.AddBinaryExtension(".bin" + extSuffix) if registries[i] != firstRegistry {
registry.AddLanguageMapping(".lang"+extSuffix, "lang"+extSuffix) t.Errorf("Registry %d is different from registry 0", i)
}
// Verify the additions worked }
if !registry.IsImage("test.img" + extSuffix) { }
t.Errorf("Failed to add image extension .img%s", extSuffix)
} // performConcurrentModifications performs concurrent modifications on separate registry instances.
if !registry.IsBinary("test.bin" + extSuffix) { func performConcurrentModifications(t *testing.T, id int) {
t.Errorf("Failed to add binary extension .bin%s", extSuffix) t.Helper()
}
if registry.GetLanguage("test.lang"+extSuffix) != "lang"+extSuffix { // Create a new registry instance for this goroutine
t.Errorf("Failed to add language mapping .lang%s", extSuffix) registry := createConcurrencyTestRegistry()
}
} for j := 0; j < numOperationsPerGoroutine; j++ {
}(i) extSuffix := fmt.Sprintf("_%d_%d", id, j)
addTestExtensions(registry, extSuffix)
verifyTestExtensions(t, registry, extSuffix)
}
}
// createConcurrencyTestRegistry creates a new registry instance for concurrency testing.
func createConcurrencyTestRegistry() *FileTypeRegistry {
return &FileTypeRegistry{
imageExts: make(map[string]bool),
binaryExts: make(map[string]bool),
languageMap: make(map[string]string),
extCache: make(map[string]string, shared.FileTypeRegistryMaxCacheSize),
resultCache: make(map[string]FileTypeResult, shared.FileTypeRegistryMaxCacheSize),
maxCacheSize: shared.FileTypeRegistryMaxCacheSize,
}
}
// addTestExtensions adds test extensions to the registry.
func addTestExtensions(registry *FileTypeRegistry, extSuffix string) {
registry.AddImageExtension(".img" + extSuffix)
registry.AddBinaryExtension(".bin" + extSuffix)
registry.AddLanguageMapping(".lang"+extSuffix, "lang"+extSuffix)
}
// verifyTestExtensions verifies that test extensions were added correctly.
func verifyTestExtensions(t *testing.T, registry *FileTypeRegistry, extSuffix string) {
t.Helper()
if !registry.IsImage("test.img" + extSuffix) {
t.Errorf("Failed to add image extension .img%s", extSuffix)
}
if !registry.IsBinary("test.bin" + extSuffix) {
t.Errorf("Failed to add binary extension .bin%s", extSuffix)
}
if registry.Language("test.lang"+extSuffix) != "lang"+extSuffix {
t.Errorf("Failed to add language mapping .lang%s", extSuffix)
}
}
// Benchmarks for concurrency performance
// BenchmarkConcurrentReads benchmarks concurrent read operations on the registry.
func BenchmarkConcurrentReads(b *testing.B) {
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
_ = performConcurrentReads()
} }
wg.Wait()
}) })
} }
// BenchmarkConcurrentRegistryAccess benchmarks concurrent registry singleton access.
func BenchmarkConcurrentRegistryAccess(b *testing.B) {
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
_ = DefaultRegistry()
}
})
}
// BenchmarkConcurrentModifications benchmarks sequential registry modifications.
// Note: Concurrent modifications to the same registry require external synchronization.
// This benchmark measures the cost of modification operations themselves.
func BenchmarkConcurrentModifications(b *testing.B) {
for b.Loop() {
registry := createConcurrencyTestRegistry()
for i := 0; i < 10; i++ {
extSuffix := fmt.Sprintf("_bench_%d", i)
registry.AddImageExtension(".img" + extSuffix)
registry.AddBinaryExtension(".bin" + extSuffix)
registry.AddLanguageMapping(".lang"+extSuffix, "lang"+extSuffix)
}
}
}

View File

@@ -3,218 +3,264 @@ package fileproc
import ( import (
"testing" "testing"
"github.com/stretchr/testify/require" "github.com/ivuorinen/gibidify/shared"
) )
// TestFileTypeRegistry_Configuration tests the configuration functionality. const (
func TestFileTypeRegistry_Configuration(t *testing.T) { zigLang = "zig"
// Create a new registry instance for testing )
registry := &FileTypeRegistry{
imageExts: make(map[string]bool), // TestFileTypeRegistryApplyCustomExtensions tests applying custom extensions.
binaryExts: make(map[string]bool), func TestFileTypeRegistryApplyCustomExtensions(t *testing.T) {
languageMap: make(map[string]string), registry := createEmptyTestRegistry()
customImages := []string{".webp", ".avif", ".heic"}
customBinary := []string{".custom", ".mybin"}
customLanguages := map[string]string{
".zig": zigLang,
".odin": "odin",
".v": "vlang",
} }
// Test ApplyCustomExtensions registry.ApplyCustomExtensions(customImages, customBinary, customLanguages)
t.Run("ApplyCustomExtensions", func(t *testing.T) {
customImages := []string{".webp", ".avif", ".heic"}
customBinary := []string{".custom", ".mybin"}
customLanguages := map[string]string{
".zig": "zig",
".odin": "odin",
".v": "vlang",
}
registry.ApplyCustomExtensions(customImages, customBinary, customLanguages) verifyCustomExtensions(t, registry, customImages, customBinary, customLanguages)
}
// Test custom image extensions // TestFileTypeRegistryDisableExtensions tests disabling extensions.
for _, ext := range customImages { func TestFileTypeRegistryDisableExtensions(t *testing.T) {
if !registry.IsImage("test" + ext) { registry := createEmptyTestRegistry()
t.Errorf("Expected %s to be recognized as image", ext)
}
}
// Test custom binary extensions // Add some extensions first
for _, ext := range customBinary { setupRegistryExtensions(registry)
if !registry.IsBinary("test" + ext) {
t.Errorf("Expected %s to be recognized as binary", ext)
}
}
// Test custom language mappings // Verify they work before disabling
for ext, expectedLang := range customLanguages { verifyExtensionsEnabled(t, registry)
if lang := registry.GetLanguage("test" + ext); lang != expectedLang {
t.Errorf("Expected %s to map to %s, got %s", ext, expectedLang, lang)
}
}
})
// Test DisableExtensions // Disable some extensions
t.Run("DisableExtensions", func(t *testing.T) { disabledImages := []string{".png"}
// Add some extensions first disabledBinary := []string{".exe"}
registry.AddImageExtension(".png") disabledLanguages := []string{".go"}
registry.AddImageExtension(".jpg")
registry.AddBinaryExtension(".exe")
registry.AddBinaryExtension(".dll")
registry.AddLanguageMapping(".go", "go")
registry.AddLanguageMapping(".py", "python")
// Verify they work registry.DisableExtensions(disabledImages, disabledBinary, disabledLanguages)
if !registry.IsImage("test.png") {
t.Error("Expected .png to be image before disabling")
}
if !registry.IsBinary("test.exe") {
t.Error("Expected .exe to be binary before disabling")
}
if registry.GetLanguage("test.go") != "go" {
t.Error("Expected .go to map to go before disabling")
}
// Disable some extensions // Verify disabled and remaining extensions
disabledImages := []string{".png"} verifyExtensionsDisabled(t, registry)
disabledBinary := []string{".exe"} verifyRemainingExtensions(t, registry)
disabledLanguages := []string{".go"} }
registry.DisableExtensions(disabledImages, disabledBinary, disabledLanguages) // TestFileTypeRegistryEmptyValuesHandling tests handling of empty values.
func TestFileTypeRegistryEmptyValuesHandling(t *testing.T) {
registry := createEmptyTestRegistry()
// Test that disabled extensions no longer work customImages := []string{"", shared.TestExtensionValid, ""}
if registry.IsImage("test.png") { customBinary := []string{"", shared.TestExtensionValid}
t.Error("Expected .png to not be image after disabling") customLanguages := map[string]string{
} "": "invalid",
if registry.IsBinary("test.exe") { shared.TestExtensionValid: "",
t.Error("Expected .exe to not be binary after disabling") ".good": "good",
} }
if registry.GetLanguage("test.go") != "" {
t.Error("Expected .go to not map to language after disabling")
}
// Test that non-disabled extensions still work registry.ApplyCustomExtensions(customImages, customBinary, customLanguages)
if !registry.IsImage("test.jpg") {
t.Error("Expected .jpg to still be image after disabling .png")
}
if !registry.IsBinary("test.dll") {
t.Error("Expected .dll to still be binary after disabling .exe")
}
if registry.GetLanguage("test.py") != "python" {
t.Error("Expected .py to still map to python after disabling .go")
}
})
// Test empty values handling verifyEmptyValueHandling(t, registry)
t.Run("EmptyValuesHandling", func(t *testing.T) { }
registry := &FileTypeRegistry{
imageExts: make(map[string]bool),
binaryExts: make(map[string]bool),
languageMap: make(map[string]string),
}
// Test with empty values // TestFileTypeRegistryCaseInsensitiveHandling tests case insensitive handling.
customImages := []string{"", ".valid", ""} func TestFileTypeRegistryCaseInsensitiveHandling(t *testing.T) {
customBinary := []string{"", ".valid"} registry := createEmptyTestRegistry()
customLanguages := map[string]string{
"": "invalid",
".valid": "",
".good": "good",
}
registry.ApplyCustomExtensions(customImages, customBinary, customLanguages) customImages := []string{".WEBP", ".Avif"}
customBinary := []string{".CUSTOM", ".MyBin"}
customLanguages := map[string]string{
".ZIG": zigLang,
".Odin": "odin",
}
// Only valid entries should be added registry.ApplyCustomExtensions(customImages, customBinary, customLanguages)
if registry.IsImage("test.") {
t.Error("Expected empty extension to not be added as image")
}
if !registry.IsImage("test.valid") {
t.Error("Expected .valid to be added as image")
}
if registry.IsBinary("test.") {
t.Error("Expected empty extension to not be added as binary")
}
if !registry.IsBinary("test.valid") {
t.Error("Expected .valid to be added as binary")
}
if registry.GetLanguage("test.") != "" {
t.Error("Expected empty extension to not be added as language")
}
if registry.GetLanguage("test.valid") != "" {
t.Error("Expected .valid with empty language to not be added")
}
if registry.GetLanguage("test.good") != "good" {
t.Error("Expected .good to map to good")
}
})
// Test case-insensitive handling verifyCaseInsensitiveHandling(t, registry)
t.Run("CaseInsensitiveHandling", func(t *testing.T) { }
registry := &FileTypeRegistry{
imageExts: make(map[string]bool),
binaryExts: make(map[string]bool),
languageMap: make(map[string]string),
}
customImages := []string{".WEBP", ".Avif"} // createEmptyTestRegistry creates a new empty test registry instance for config testing.
customBinary := []string{".CUSTOM", ".MyBin"} func createEmptyTestRegistry() *FileTypeRegistry {
customLanguages := map[string]string{ return &FileTypeRegistry{
".ZIG": "zig", imageExts: make(map[string]bool),
".Odin": "odin", binaryExts: make(map[string]bool),
} languageMap: make(map[string]string),
extCache: make(map[string]string, shared.FileTypeRegistryMaxCacheSize),
resultCache: make(map[string]FileTypeResult, shared.FileTypeRegistryMaxCacheSize),
maxCacheSize: shared.FileTypeRegistryMaxCacheSize,
}
}
registry.ApplyCustomExtensions(customImages, customBinary, customLanguages) // verifyCustomExtensions verifies that custom extensions are applied correctly.
func verifyCustomExtensions(
t *testing.T,
registry *FileTypeRegistry,
customImages, customBinary []string,
customLanguages map[string]string,
) {
t.Helper()
// Test that both upper and lower case work // Test custom image extensions
if !registry.IsImage("test.webp") { for _, ext := range customImages {
t.Error("Expected .webp (lowercase) to work after adding .WEBP") if !registry.IsImage("test" + ext) {
t.Errorf("Expected %s to be recognized as image", ext)
} }
if !registry.IsImage("test.WEBP") { }
t.Error("Expected .WEBP (uppercase) to work")
// Test custom binary extensions
for _, ext := range customBinary {
if !registry.IsBinary("test" + ext) {
t.Errorf("Expected %s to be recognized as binary", ext)
} }
if !registry.IsBinary("test.custom") { }
t.Error("Expected .custom (lowercase) to work after adding .CUSTOM")
// Test custom language mappings
for ext, expectedLang := range customLanguages {
if lang := registry.Language("test" + ext); lang != expectedLang {
t.Errorf("Expected %s to map to %s, got %s", ext, expectedLang, lang)
} }
if !registry.IsBinary("test.CUSTOM") { }
t.Error("Expected .CUSTOM (uppercase) to work") }
}
if registry.GetLanguage("test.zig") != "zig" { // setupRegistryExtensions adds test extensions to the registry.
t.Error("Expected .zig (lowercase) to work after adding .ZIG") func setupRegistryExtensions(registry *FileTypeRegistry) {
} registry.AddImageExtension(".png")
if registry.GetLanguage("test.ZIG") != "zig" { registry.AddImageExtension(".jpg")
t.Error("Expected .ZIG (uppercase) to work") registry.AddBinaryExtension(".exe")
} registry.AddBinaryExtension(".dll")
}) registry.AddLanguageMapping(".go", "go")
registry.AddLanguageMapping(".py", "python")
}
// verifyExtensionsEnabled verifies that extensions are enabled before disabling.
func verifyExtensionsEnabled(t *testing.T, registry *FileTypeRegistry) {
t.Helper()
if !registry.IsImage(shared.TestFilePNG) {
t.Error("Expected .png to be image before disabling")
}
if !registry.IsBinary(shared.TestFileEXE) {
t.Error("Expected .exe to be binary before disabling")
}
if registry.Language(shared.TestFileGo) != "go" {
t.Error("Expected .go to map to go before disabling")
}
}
// verifyExtensionsDisabled verifies that disabled extensions no longer work.
func verifyExtensionsDisabled(t *testing.T, registry *FileTypeRegistry) {
t.Helper()
if registry.IsImage(shared.TestFilePNG) {
t.Error("Expected .png to not be image after disabling")
}
if registry.IsBinary(shared.TestFileEXE) {
t.Error("Expected .exe to not be binary after disabling")
}
if registry.Language(shared.TestFileGo) != "" {
t.Error("Expected .go to not map to language after disabling")
}
}
// verifyRemainingExtensions verifies that non-disabled extensions still work.
func verifyRemainingExtensions(t *testing.T, registry *FileTypeRegistry) {
t.Helper()
if !registry.IsImage(shared.TestFileJPG) {
t.Error("Expected .jpg to still be image after disabling .png")
}
if !registry.IsBinary(shared.TestFileDLL) {
t.Error("Expected .dll to still be binary after disabling .exe")
}
if registry.Language(shared.TestFilePy) != "python" {
t.Error("Expected .py to still map to python after disabling .go")
}
}
// verifyEmptyValueHandling verifies handling of empty values.
func verifyEmptyValueHandling(t *testing.T, registry *FileTypeRegistry) {
t.Helper()
if registry.IsImage("test") {
t.Error("Expected empty extension to not be added as image")
}
if !registry.IsImage(shared.TestFileValid) {
t.Error("Expected .valid to be added as image")
}
if registry.IsBinary("test") {
t.Error("Expected empty extension to not be added as binary")
}
if !registry.IsBinary(shared.TestFileValid) {
t.Error("Expected .valid to be added as binary")
}
if registry.Language("test") != "" {
t.Error("Expected empty extension to not be added as language")
}
if registry.Language(shared.TestFileValid) != "" {
t.Error("Expected .valid with empty language to not be added")
}
if registry.Language("test.good") != "good" {
t.Error("Expected .good to map to good")
}
}
// verifyCaseInsensitiveHandling verifies case insensitive handling.
func verifyCaseInsensitiveHandling(t *testing.T, registry *FileTypeRegistry) {
t.Helper()
if !registry.IsImage(shared.TestFileWebP) {
t.Error("Expected .webp (lowercase) to work after adding .WEBP")
}
if !registry.IsImage("test.WEBP") {
t.Error("Expected .WEBP (uppercase) to work")
}
if !registry.IsBinary("test.custom") {
t.Error("Expected .custom (lowercase) to work after adding .CUSTOM")
}
if !registry.IsBinary("test.CUSTOM") {
t.Error("Expected .CUSTOM (uppercase) to work")
}
if registry.Language("test.zig") != zigLang {
t.Error("Expected .zig (lowercase) to work after adding .ZIG")
}
if registry.Language("test.ZIG") != zigLang {
t.Error("Expected .ZIG (uppercase) to work")
}
} }
// TestConfigureFromSettings tests the global configuration function. // TestConfigureFromSettings tests the global configuration function.
func TestConfigureFromSettings(t *testing.T) { func TestConfigureFromSettings(t *testing.T) {
// Reset registry to ensure clean state // Reset registry to ensure clean state
ResetRegistryForTesting() ResetRegistryForTesting()
// Ensure cleanup runs even if test fails
t.Cleanup(ResetRegistryForTesting)
// Test configuration application // Test configuration application
customImages := []string{".webp", ".avif"} customImages := []string{".webp", ".avif"}
customBinary := []string{".custom"} customBinary := []string{".custom"}
customLanguages := map[string]string{".zig": "zig"} customLanguages := map[string]string{".zig": zigLang}
disabledImages := []string{".gif"} // Disable default extension disabledImages := []string{".gif"} // Disable default extension
disabledBinary := []string{".exe"} // Disable default extension disabledBinary := []string{".exe"} // Disable default extension
disabledLanguages := []string{".rb"} // Disable default extension disabledLanguages := []string{".rb"} // Disable default extension
err := ConfigureFromSettings(RegistryConfig{ ConfigureFromSettings(
CustomImages: customImages, customImages,
CustomBinary: customBinary, customBinary,
CustomLanguages: customLanguages, customLanguages,
DisabledImages: disabledImages, disabledImages,
DisabledBinary: disabledBinary, disabledBinary,
DisabledLanguages: disabledLanguages, disabledLanguages,
}) )
require.NoError(t, err)
// Test that custom extensions work // Test that custom extensions work
if !IsImage("test.webp") { if !IsImage(shared.TestFileWebP) {
t.Error("Expected custom image extension .webp to work") t.Error("Expected custom image extension .webp to work")
} }
if !IsBinary("test.custom") { if !IsBinary("test.custom") {
t.Error("Expected custom binary extension .custom to work") t.Error("Expected custom binary extension .custom to work")
} }
if GetLanguage("test.zig") != "zig" { if Language("test.zig") != zigLang {
t.Error("Expected custom language .zig to work") t.Error("Expected custom language .zig to work")
} }
@@ -222,41 +268,43 @@ func TestConfigureFromSettings(t *testing.T) {
if IsImage("test.gif") { if IsImage("test.gif") {
t.Error("Expected disabled image extension .gif to not work") t.Error("Expected disabled image extension .gif to not work")
} }
if IsBinary("test.exe") { if IsBinary(shared.TestFileEXE) {
t.Error("Expected disabled binary extension .exe to not work") t.Error("Expected disabled binary extension .exe to not work")
} }
if GetLanguage("test.rb") != "" { if Language("test.rb") != "" {
t.Error("Expected disabled language extension .rb to not work") t.Error("Expected disabled language extension .rb to not work")
} }
// Test that non-disabled defaults still work // Test that non-disabled defaults still work
if !IsImage("test.png") { if !IsImage(shared.TestFilePNG) {
t.Error("Expected non-disabled image extension .png to still work") t.Error("Expected non-disabled image extension .png to still work")
} }
if !IsBinary("test.dll") { if !IsBinary(shared.TestFileDLL) {
t.Error("Expected non-disabled binary extension .dll to still work") t.Error("Expected non-disabled binary extension .dll to still work")
} }
if GetLanguage("test.go") != "go" { if Language(shared.TestFileGo) != "go" {
t.Error("Expected non-disabled language extension .go to still work") t.Error("Expected non-disabled language extension .go to still work")
} }
// Test multiple calls don't override previous configuration // Test multiple calls don't override previous configuration
err = ConfigureFromSettings(RegistryConfig{ ConfigureFromSettings(
CustomImages: []string{".extra"}, []string{".extra"},
CustomBinary: []string{}, []string{},
CustomLanguages: map[string]string{}, map[string]string{},
DisabledImages: []string{}, []string{},
DisabledBinary: []string{}, []string{},
DisabledLanguages: []string{}, []string{},
}) )
require.NoError(t, err)
// Previous configuration should still work // Previous configuration should still work
if !IsImage("test.webp") { if !IsImage(shared.TestFileWebP) {
t.Error("Expected previous configuration to persist") t.Error("Expected previous configuration to persist")
} }
// New configuration should also work // New configuration should also work
if !IsImage("test.extra") { if !IsImage("test.extra") {
t.Error("Expected new configuration to be applied") t.Error("Expected new configuration to be applied")
} }
// Reset registry after test to avoid affecting other tests
ResetRegistryForTesting()
} }

View File

@@ -2,31 +2,34 @@ package fileproc
import ( import (
"testing" "testing"
"github.com/ivuorinen/gibidify/shared"
) )
// newTestRegistry creates a fresh registry instance for testing to avoid global state pollution. // createTestRegistry creates a fresh FileTypeRegistry instance for testing.
func newTestRegistry() *FileTypeRegistry { // This helper reduces code duplication and ensures consistent registry initialization.
func createTestRegistry() *FileTypeRegistry {
return &FileTypeRegistry{ return &FileTypeRegistry{
imageExts: getImageExtensions(), imageExts: getImageExtensions(),
binaryExts: getBinaryExtensions(), binaryExts: getBinaryExtensions(),
languageMap: getLanguageMap(), languageMap: getLanguageMap(),
extCache: make(map[string]string, 1000), extCache: make(map[string]string, shared.FileTypeRegistryMaxCacheSize),
resultCache: make(map[string]FileTypeResult, 500), resultCache: make(map[string]FileTypeResult, shared.FileTypeRegistryMaxCacheSize),
maxCacheSize: 500, maxCacheSize: shared.FileTypeRegistryMaxCacheSize,
} }
} }
// TestFileTypeRegistry_LanguageDetection tests the language detection functionality. // TestFileTypeRegistry_LanguageDetection tests the language detection functionality.
func TestFileTypeRegistry_LanguageDetection(t *testing.T) { func TestFileTypeRegistryLanguageDetection(t *testing.T) {
registry := newTestRegistry() registry := createTestRegistry()
tests := []struct { tests := []struct {
filename string filename string
expected string expected string
}{ }{
// Programming languages // Programming languages
{"main.go", "go"}, {shared.TestFileMainGo, "go"},
{"script.py", "python"}, {shared.TestFileScriptPy, "python"},
{"app.js", "javascript"}, {"app.js", "javascript"},
{"component.tsx", "typescript"}, {"component.tsx", "typescript"},
{"service.ts", "typescript"}, {"service.ts", "typescript"},
@@ -96,17 +99,17 @@ func TestFileTypeRegistry_LanguageDetection(t *testing.T) {
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.filename, func(t *testing.T) { t.Run(tt.filename, func(t *testing.T) {
result := registry.GetLanguage(tt.filename) result := registry.Language(tt.filename)
if result != tt.expected { if result != tt.expected {
t.Errorf("GetLanguage(%q) = %q, expected %q", tt.filename, result, tt.expected) t.Errorf("Language(%q) = %q, expected %q", tt.filename, result, tt.expected)
} }
}) })
} }
} }
// TestFileTypeRegistry_ImageDetection tests the image detection functionality. // TestFileTypeRegistry_ImageDetection tests the image detection functionality.
func TestFileTypeRegistry_ImageDetection(t *testing.T) { func TestFileTypeRegistryImageDetection(t *testing.T) {
registry := newTestRegistry() registry := createTestRegistry()
tests := []struct { tests := []struct {
filename string filename string
@@ -114,7 +117,7 @@ func TestFileTypeRegistry_ImageDetection(t *testing.T) {
}{ }{
// Common image formats // Common image formats
{"photo.png", true}, {"photo.png", true},
{"image.jpg", true}, {shared.TestFileImageJPG, true},
{"picture.jpeg", true}, {"picture.jpeg", true},
{"animation.gif", true}, {"animation.gif", true},
{"bitmap.bmp", true}, {"bitmap.bmp", true},
@@ -155,8 +158,8 @@ func TestFileTypeRegistry_ImageDetection(t *testing.T) {
} }
// TestFileTypeRegistry_BinaryDetection tests the binary detection functionality. // TestFileTypeRegistry_BinaryDetection tests the binary detection functionality.
func TestFileTypeRegistry_BinaryDetection(t *testing.T) { func TestFileTypeRegistryBinaryDetection(t *testing.T) {
registry := newTestRegistry() registry := createTestRegistry()
tests := []struct { tests := []struct {
filename string filename string
@@ -214,7 +217,7 @@ func TestFileTypeRegistry_BinaryDetection(t *testing.T) {
// Non-binary files // Non-binary files
{"document.txt", false}, {"document.txt", false},
{"script.py", false}, {shared.TestFileScriptPy, false},
{"config.json", false}, {"config.json", false},
{"style.css", false}, {"style.css", false},
{"page.html", false}, {"page.html", false},

View File

@@ -2,11 +2,13 @@ package fileproc
import ( import (
"testing" "testing"
"github.com/ivuorinen/gibidify/shared"
) )
// TestFileTypeRegistry_EdgeCases tests edge cases and boundary conditions. // TestFileTypeRegistry_EdgeCases tests edge cases and boundary conditions.
func TestFileTypeRegistry_EdgeCases(t *testing.T) { func TestFileTypeRegistryEdgeCases(t *testing.T) {
registry := GetDefaultRegistry() registry := DefaultRegistry()
// Test various edge cases for filename handling // Test various edge cases for filename handling
edgeCases := []struct { edgeCases := []struct {
@@ -35,19 +37,19 @@ func TestFileTypeRegistry_EdgeCases(t *testing.T) {
// These should not panic // These should not panic
_ = registry.IsImage(tc.filename) _ = registry.IsImage(tc.filename)
_ = registry.IsBinary(tc.filename) _ = registry.IsBinary(tc.filename)
_ = registry.GetLanguage(tc.filename) _ = registry.Language(tc.filename)
// Global functions should also not panic // Global functions should also not panic
_ = IsImage(tc.filename) _ = IsImage(tc.filename)
_ = IsBinary(tc.filename) _ = IsBinary(tc.filename)
_ = GetLanguage(tc.filename) _ = Language(tc.filename)
}) })
} }
} }
// TestFileTypeRegistry_MinimumExtensionLength tests the minimum extension length requirement. // TestFileTypeRegistry_MinimumExtensionLength tests the minimum extension length requirement.
func TestFileTypeRegistry_MinimumExtensionLength(t *testing.T) { func TestFileTypeRegistryMinimumExtensionLength(t *testing.T) {
registry := GetDefaultRegistry() registry := DefaultRegistry()
tests := []struct { tests := []struct {
filename string filename string
@@ -65,18 +67,18 @@ func TestFileTypeRegistry_MinimumExtensionLength(t *testing.T) {
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.filename, func(t *testing.T) { t.Run(tt.filename, func(t *testing.T) {
result := registry.GetLanguage(tt.filename) result := registry.Language(tt.filename)
if result != tt.expected { if result != tt.expected {
t.Errorf("GetLanguage(%q) = %q, expected %q", tt.filename, result, tt.expected) t.Errorf("Language(%q) = %q, expected %q", tt.filename, result, tt.expected)
} }
}) })
} }
} }
// Benchmark tests for performance validation // Benchmark tests for performance validation.
func BenchmarkFileTypeRegistry_IsImage(b *testing.B) { func BenchmarkFileTypeRegistryIsImage(b *testing.B) {
registry := GetDefaultRegistry() registry := DefaultRegistry()
filename := "test.png" filename := shared.TestFilePNG
b.ResetTimer() b.ResetTimer()
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
@@ -84,9 +86,9 @@ func BenchmarkFileTypeRegistry_IsImage(b *testing.B) {
} }
} }
func BenchmarkFileTypeRegistry_IsBinary(b *testing.B) { func BenchmarkFileTypeRegistryIsBinary(b *testing.B) {
registry := GetDefaultRegistry() registry := DefaultRegistry()
filename := "test.exe" filename := shared.TestFileEXE
b.ResetTimer() b.ResetTimer()
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
@@ -94,35 +96,35 @@ func BenchmarkFileTypeRegistry_IsBinary(b *testing.B) {
} }
} }
func BenchmarkFileTypeRegistry_GetLanguage(b *testing.B) { func BenchmarkFileTypeRegistryLanguage(b *testing.B) {
registry := GetDefaultRegistry() registry := DefaultRegistry()
filename := "test.go" filename := shared.TestFileGo
b.ResetTimer() b.ResetTimer()
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
_ = registry.GetLanguage(filename) _ = registry.Language(filename)
} }
} }
func BenchmarkFileTypeRegistry_GlobalFunctions(b *testing.B) { func BenchmarkFileTypeRegistryGlobalFunctions(b *testing.B) {
filename := "test.go" filename := shared.TestFileGo
b.ResetTimer() b.ResetTimer()
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
_ = IsImage(filename) _ = IsImage(filename)
_ = IsBinary(filename) _ = IsBinary(filename)
_ = GetLanguage(filename) _ = Language(filename)
} }
} }
func BenchmarkFileTypeRegistry_ConcurrentAccess(b *testing.B) { func BenchmarkFileTypeRegistryConcurrentAccess(b *testing.B) {
filename := "test.go" filename := shared.TestFileGo
b.RunParallel(func(pb *testing.PB) { b.RunParallel(func(pb *testing.PB) {
for pb.Next() { for pb.Next() {
_ = IsImage(filename) _ = IsImage(filename)
_ = IsBinary(filename) _ = IsBinary(filename)
_ = GetLanguage(filename) _ = Language(filename)
} }
}) })
} }

View File

@@ -2,136 +2,254 @@ package fileproc
import ( import (
"testing" "testing"
"github.com/ivuorinen/gibidify/shared"
) )
// TestFileTypeRegistry_ModificationMethods tests the modification methods of FileTypeRegistry. // TestFileTypeRegistryAddImageExtension tests adding image extensions.
func TestFileTypeRegistry_ModificationMethods(t *testing.T) { func TestFileTypeRegistryAddImageExtension(t *testing.T) {
// Create a new registry instance for testing registry := createModificationTestRegistry()
registry := &FileTypeRegistry{
imageExts: make(map[string]bool),
binaryExts: make(map[string]bool),
languageMap: make(map[string]string),
}
// Test AddImageExtension testImageExtensionModifications(t, registry)
t.Run("AddImageExtension", func(t *testing.T) {
// Add a new image extension
registry.AddImageExtension(".webp")
if !registry.IsImage("test.webp") {
t.Errorf("Expected .webp to be recognized as image after adding")
}
// Test case-insensitive addition
registry.AddImageExtension(".AVIF")
if !registry.IsImage("test.avif") {
t.Errorf("Expected .avif to be recognized as image after adding .AVIF")
}
if !registry.IsImage("test.AVIF") {
t.Errorf("Expected .AVIF to be recognized as image")
}
// Test with dot prefix
registry.AddImageExtension("heic")
if registry.IsImage("test.heic") {
t.Errorf("Expected extension without dot to not work")
}
// Test with proper dot prefix
registry.AddImageExtension(".heic")
if !registry.IsImage("test.heic") {
t.Errorf("Expected .heic to be recognized as image")
}
})
// Test AddBinaryExtension
t.Run("AddBinaryExtension", func(t *testing.T) {
// Add a new binary extension
registry.AddBinaryExtension(".custom")
if !registry.IsBinary("file.custom") {
t.Errorf("Expected .custom to be recognized as binary after adding")
}
// Test case-insensitive addition
registry.AddBinaryExtension(".SPECIAL")
if !registry.IsBinary("file.special") {
t.Errorf("Expected .special to be recognized as binary after adding .SPECIAL")
}
if !registry.IsBinary("file.SPECIAL") {
t.Errorf("Expected .SPECIAL to be recognized as binary")
}
// Test with dot prefix
registry.AddBinaryExtension("bin")
if registry.IsBinary("file.bin") {
t.Errorf("Expected extension without dot to not work")
}
// Test with proper dot prefix
registry.AddBinaryExtension(".bin")
if !registry.IsBinary("file.bin") {
t.Errorf("Expected .bin to be recognized as binary")
}
})
// Test AddLanguageMapping
t.Run("AddLanguageMapping", func(t *testing.T) {
// Add a new language mapping
registry.AddLanguageMapping(".xyz", "CustomLang")
if lang := registry.GetLanguage("file.xyz"); lang != "CustomLang" {
t.Errorf("Expected CustomLang, got %s", lang)
}
// Test case-insensitive addition
registry.AddLanguageMapping(".ABC", "UpperLang")
if lang := registry.GetLanguage("file.abc"); lang != "UpperLang" {
t.Errorf("Expected UpperLang, got %s", lang)
}
if lang := registry.GetLanguage("file.ABC"); lang != "UpperLang" {
t.Errorf("Expected UpperLang for uppercase, got %s", lang)
}
// Test with dot prefix
registry.AddLanguageMapping("nolang", "NoLang")
if lang := registry.GetLanguage("file.nolang"); lang == "NoLang" {
t.Errorf("Expected extension without dot to not work")
}
// Test with proper dot prefix
registry.AddLanguageMapping(".nolang", "NoLang")
if lang := registry.GetLanguage("file.nolang"); lang != "NoLang" {
t.Errorf("Expected NoLang, got %s", lang)
}
// Test overriding existing mapping
registry.AddLanguageMapping(".xyz", "NewCustomLang")
if lang := registry.GetLanguage("file.xyz"); lang != "NewCustomLang" {
t.Errorf("Expected NewCustomLang after override, got %s", lang)
}
})
} }
// TestFileTypeRegistry_DefaultRegistryConsistency tests default registry behavior. // TestFileTypeRegistryAddBinaryExtension tests adding binary extensions.
func TestFileTypeRegistry_DefaultRegistryConsistency(t *testing.T) { func TestFileTypeRegistryAddBinaryExtension(t *testing.T) {
registry := GetDefaultRegistry() registry := createModificationTestRegistry()
testBinaryExtensionModifications(t, registry)
}
// TestFileTypeRegistryAddLanguageMapping tests adding language mappings.
func TestFileTypeRegistryAddLanguageMapping(t *testing.T) {
registry := createModificationTestRegistry()
testLanguageMappingModifications(t, registry)
}
// createModificationTestRegistry creates a registry for modification tests.
func createModificationTestRegistry() *FileTypeRegistry {
return &FileTypeRegistry{
imageExts: make(map[string]bool),
binaryExts: make(map[string]bool),
languageMap: make(map[string]string),
extCache: make(map[string]string, shared.FileTypeRegistryMaxCacheSize),
resultCache: make(map[string]FileTypeResult, shared.FileTypeRegistryMaxCacheSize),
maxCacheSize: shared.FileTypeRegistryMaxCacheSize,
}
}
// testImageExtensionModifications tests image extension modifications.
func testImageExtensionModifications(t *testing.T, registry *FileTypeRegistry) {
t.Helper()
// Add a new image extension
registry.AddImageExtension(".webp")
verifyImageExtension(t, registry, ".webp", shared.TestFileWebP, true)
// Test case-insensitive addition
registry.AddImageExtension(".AVIF")
verifyImageExtension(t, registry, ".AVIF", "test.avif", true)
verifyImageExtension(t, registry, ".AVIF", "test.AVIF", true)
// Test with dot prefix
registry.AddImageExtension("heic")
verifyImageExtension(t, registry, "heic", "test.heic", false)
// Test with proper dot prefix
registry.AddImageExtension(".heic")
verifyImageExtension(t, registry, ".heic", "test.heic", true)
}
// testBinaryExtensionModifications tests binary extension modifications.
func testBinaryExtensionModifications(t *testing.T, registry *FileTypeRegistry) {
t.Helper()
// Add a new binary extension
registry.AddBinaryExtension(".custom")
verifyBinaryExtension(t, registry, ".custom", "file.custom", true)
// Test case-insensitive addition
registry.AddBinaryExtension(shared.TestExtensionSpecial)
verifyBinaryExtension(t, registry, shared.TestExtensionSpecial, "file.special", true)
verifyBinaryExtension(t, registry, shared.TestExtensionSpecial, "file.SPECIAL", true)
// Test with dot prefix
registry.AddBinaryExtension("bin")
verifyBinaryExtension(t, registry, "bin", "file.bin", false)
// Test with proper dot prefix
registry.AddBinaryExtension(".bin")
verifyBinaryExtension(t, registry, ".bin", "file.bin", true)
}
// testLanguageMappingModifications tests language mapping modifications.
func testLanguageMappingModifications(t *testing.T, registry *FileTypeRegistry) {
t.Helper()
// Add a new language mapping
registry.AddLanguageMapping(".xyz", "CustomLang")
verifyLanguageMapping(t, registry, "file.xyz", "CustomLang")
// Test case-insensitive addition
registry.AddLanguageMapping(".ABC", "UpperLang")
verifyLanguageMapping(t, registry, "file.abc", "UpperLang")
verifyLanguageMapping(t, registry, "file.ABC", "UpperLang")
// Test with dot prefix (should not work)
registry.AddLanguageMapping("nolang", "NoLang")
verifyLanguageMappingAbsent(t, registry, "nolang", "file.nolang")
// Test with proper dot prefix
registry.AddLanguageMapping(".nolang", "NoLang")
verifyLanguageMapping(t, registry, "file.nolang", "NoLang")
// Test overriding existing mapping
registry.AddLanguageMapping(".xyz", "NewCustomLang")
verifyLanguageMapping(t, registry, "file.xyz", "NewCustomLang")
}
// verifyImageExtension verifies image extension behavior.
func verifyImageExtension(t *testing.T, registry *FileTypeRegistry, ext, filename string, expected bool) {
t.Helper()
if registry.IsImage(filename) != expected {
if expected {
t.Errorf("Expected %s to be recognized as image after adding %s", filename, ext)
} else {
t.Errorf(shared.TestMsgExpectedExtensionWithoutDot)
}
}
}
// verifyBinaryExtension verifies binary extension behavior.
func verifyBinaryExtension(t *testing.T, registry *FileTypeRegistry, ext, filename string, expected bool) {
t.Helper()
if registry.IsBinary(filename) != expected {
if expected {
t.Errorf("Expected %s to be recognized as binary after adding %s", filename, ext)
} else {
t.Errorf(shared.TestMsgExpectedExtensionWithoutDot)
}
}
}
// verifyLanguageMapping verifies language mapping behavior.
func verifyLanguageMapping(t *testing.T, registry *FileTypeRegistry, filename, expectedLang string) {
t.Helper()
lang := registry.Language(filename)
if lang != expectedLang {
t.Errorf("Expected %s, got %s for %s", expectedLang, lang, filename)
}
}
// verifyLanguageMappingAbsent verifies that a language mapping is absent.
func verifyLanguageMappingAbsent(t *testing.T, registry *FileTypeRegistry, _ string, filename string) {
t.Helper()
lang := registry.Language(filename)
if lang != "" {
t.Errorf(shared.TestMsgExpectedExtensionWithoutDot+", but got %s", lang)
}
}
// TestFileTypeRegistryDefaultRegistryConsistency tests default registry behavior.
func TestFileTypeRegistryDefaultRegistryConsistency(t *testing.T) {
registry := DefaultRegistry()
// Test that registry methods work consistently // Test that registry methods work consistently
if !registry.IsImage("test.png") { if !registry.IsImage(shared.TestFilePNG) {
t.Error("Expected .png to be recognized as image") t.Error("Expected .png to be recognized as image")
} }
if !registry.IsBinary("test.exe") { if !registry.IsBinary(shared.TestFileEXE) {
t.Error("Expected .exe to be recognized as binary") t.Error("Expected .exe to be recognized as binary")
} }
if lang := registry.GetLanguage("test.go"); lang != "go" { if lang := registry.Language(shared.TestFileGo); lang != "go" {
t.Errorf("Expected go, got %s", lang) t.Errorf("Expected go, got %s", lang)
} }
// Test that multiple calls return consistent results // Test that multiple calls return consistent results
for i := 0; i < 5; i++ { for i := 0; i < 5; i++ {
if !registry.IsImage("test.jpg") { if !registry.IsImage(shared.TestFileJPG) {
t.Errorf("Iteration %d: Expected .jpg to be recognized as image", i) t.Errorf("Iteration %d: Expected .jpg to be recognized as image", i)
} }
if registry.IsBinary("test.txt") { if registry.IsBinary(shared.TestFileTXT) {
t.Errorf("Iteration %d: Expected .txt to not be recognized as binary", i) t.Errorf("Iteration %d: Expected .txt to not be recognized as binary", i)
} }
} }
} }
// TestFileTypeRegistryGetStats tests the GetStats method.
func TestFileTypeRegistryGetStats(t *testing.T) {
// Ensure clean, isolated state
ResetRegistryForTesting()
t.Cleanup(ResetRegistryForTesting)
registry := DefaultRegistry()
// Call some methods to populate cache and update stats
registry.IsImage(shared.TestFilePNG)
registry.IsBinary(shared.TestFileEXE)
registry.Language(shared.TestFileGo)
// Repeat to generate cache hits
registry.IsImage(shared.TestFilePNG)
registry.IsBinary(shared.TestFileEXE)
registry.Language(shared.TestFileGo)
// Get stats
stats := registry.Stats()
// Verify stats structure - all values are uint64 and therefore non-negative by definition
// We can verify they exist and are properly initialized
// Test that stats include our calls
if stats.TotalLookups < 6 { // We made at least 6 calls above
t.Errorf("Expected at least 6 total lookups, got %d", stats.TotalLookups)
}
// Total lookups should equal hits + misses
if stats.TotalLookups != stats.CacheHits+stats.CacheMisses {
t.Errorf("Total lookups (%d) should equal hits (%d) + misses (%d)",
stats.TotalLookups, stats.CacheHits, stats.CacheMisses)
}
// With repeated lookups we should see some cache hits
if stats.CacheHits == 0 {
t.Error("Expected some cache hits after repeated lookups")
}
}
// TestFileTypeRegistryGetCacheInfo tests the GetCacheInfo method.
func TestFileTypeRegistryGetCacheInfo(t *testing.T) {
// Ensure clean, isolated state
ResetRegistryForTesting()
t.Cleanup(ResetRegistryForTesting)
registry := DefaultRegistry()
// Call some methods to populate cache
registry.IsImage("test1.png")
registry.IsBinary("test2.exe")
registry.Language("test3.go")
registry.IsImage("test4.jpg")
registry.IsBinary("test5.dll")
// Get cache info
extCacheSize, resultCacheSize, maxCacheSize := registry.CacheInfo()
// Verify cache info
if extCacheSize < 0 {
t.Error("Expected non-negative extension cache size")
}
if resultCacheSize < 0 {
t.Error("Expected non-negative result cache size")
}
if maxCacheSize <= 0 {
t.Error("Expected positive max cache size")
}
// We should have some cache entries from our calls
totalCacheSize := extCacheSize + resultCacheSize
if totalCacheSize == 0 {
t.Error("Expected some cache entries after multiple calls")
}
}

View File

@@ -1,3 +1,4 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
// FileData represents a single file's path and content. // FileData represents a single file's path and content.
@@ -23,6 +24,7 @@ type FormatWriter interface {
// detectLanguage tries to infer the code block language from the file extension. // detectLanguage tries to infer the code block language from the file extension.
func detectLanguage(filePath string) string { func detectLanguage(filePath string) string {
registry := GetDefaultRegistry() registry := DefaultRegistry()
return registry.GetLanguage(filePath)
return registry.Language(filePath)
} }

View File

@@ -1,3 +1,4 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import ( import (
@@ -34,6 +35,7 @@ func loadIgnoreRules(currentDir string, parentRules []ignoreRule) []ignoreRule {
func tryLoadIgnoreFile(dir, fileName string) *ignoreRule { func tryLoadIgnoreFile(dir, fileName string) *ignoreRule {
ignorePath := filepath.Join(dir, fileName) ignorePath := filepath.Join(dir, fileName)
if info, err := os.Stat(ignorePath); err == nil && !info.IsDir() { if info, err := os.Stat(ignorePath); err == nil && !info.IsDir() {
//nolint:errcheck // Regex compile error handled by validation, safe to ignore here
if gi, err := ignore.CompileIgnoreFile(ignorePath); err == nil { if gi, err := ignore.CompileIgnoreFile(ignorePath); err == nil {
return &ignoreRule{ return &ignoreRule{
base: dir, base: dir,
@@ -41,6 +43,7 @@ func tryLoadIgnoreFile(dir, fileName string) *ignoreRule {
} }
} }
} }
return nil return nil
} }
@@ -51,6 +54,7 @@ func matchesIgnoreRules(fullPath string, rules []ignoreRule) bool {
return true return true
} }
} }
return false return false
} }

View File

@@ -1,3 +1,4 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import ( import (
@@ -6,7 +7,7 @@ import (
"io" "io"
"os" "os"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// JSONWriter handles JSON format output with streaming support. // JSONWriter handles JSON format output with streaming support.
@@ -27,42 +28,27 @@ func NewJSONWriter(outFile *os.File) *JSONWriter {
func (w *JSONWriter) Start(prefix, suffix string) error { func (w *JSONWriter) Start(prefix, suffix string) error {
// Start JSON structure // Start JSON structure
if _, err := w.outFile.WriteString(`{"prefix":"`); err != nil { if _, err := w.outFile.WriteString(`{"prefix":"`); err != nil {
return gibidiutils.WrapError( return shared.WrapError(err, shared.ErrorTypeIO, shared.CodeIOWrite, "failed to write JSON start")
err,
gibidiutils.ErrorTypeIO,
gibidiutils.CodeIOWrite,
"failed to write JSON start",
)
} }
// Write escaped prefix // Write escaped prefix
escapedPrefix := gibidiutils.EscapeForJSON(prefix) escapedPrefix := shared.EscapeForJSON(prefix)
if err := gibidiutils.WriteWithErrorWrap(w.outFile, escapedPrefix, "failed to write JSON prefix", ""); err != nil { if err := shared.WriteWithErrorWrap(w.outFile, escapedPrefix, "failed to write JSON prefix", ""); err != nil {
return err return fmt.Errorf("writing JSON prefix: %w", err)
} }
if _, err := w.outFile.WriteString(`","suffix":"`); err != nil { if _, err := w.outFile.WriteString(`","suffix":"`); err != nil {
return gibidiutils.WrapError( return shared.WrapError(err, shared.ErrorTypeIO, shared.CodeIOWrite, "failed to write JSON middle")
err,
gibidiutils.ErrorTypeIO,
gibidiutils.CodeIOWrite,
"failed to write JSON middle",
)
} }
// Write escaped suffix // Write escaped suffix
escapedSuffix := gibidiutils.EscapeForJSON(suffix) escapedSuffix := shared.EscapeForJSON(suffix)
if err := gibidiutils.WriteWithErrorWrap(w.outFile, escapedSuffix, "failed to write JSON suffix", ""); err != nil { if err := shared.WriteWithErrorWrap(w.outFile, escapedSuffix, "failed to write JSON suffix", ""); err != nil {
return err return fmt.Errorf("writing JSON suffix: %w", err)
} }
if _, err := w.outFile.WriteString(`","files":[`); err != nil { if _, err := w.outFile.WriteString(`","files":[`); err != nil {
return gibidiutils.WrapError( return shared.WrapError(err, shared.ErrorTypeIO, shared.CodeIOWrite, "failed to write JSON files start")
err,
gibidiutils.ErrorTypeIO,
gibidiutils.CodeIOWrite,
"failed to write JSON files start",
)
} }
return nil return nil
@@ -72,12 +58,7 @@ func (w *JSONWriter) Start(prefix, suffix string) error {
func (w *JSONWriter) WriteFile(req WriteRequest) error { func (w *JSONWriter) WriteFile(req WriteRequest) error {
if !w.firstFile { if !w.firstFile {
if _, err := w.outFile.WriteString(","); err != nil { if _, err := w.outFile.WriteString(","); err != nil {
return gibidiutils.WrapError( return shared.WrapError(err, shared.ErrorTypeIO, shared.CodeIOWrite, "failed to write JSON separator")
err,
gibidiutils.ErrorTypeIO,
gibidiutils.CodeIOWrite,
"failed to write JSON separator",
)
} }
} }
w.firstFile = false w.firstFile = false
@@ -85,6 +66,7 @@ func (w *JSONWriter) WriteFile(req WriteRequest) error {
if req.IsStream { if req.IsStream {
return w.writeStreaming(req) return w.writeStreaming(req)
} }
return w.writeInline(req) return w.writeInline(req)
} }
@@ -92,22 +74,25 @@ func (w *JSONWriter) WriteFile(req WriteRequest) error {
func (w *JSONWriter) Close() error { func (w *JSONWriter) Close() error {
// Close JSON structure // Close JSON structure
if _, err := w.outFile.WriteString("]}"); err != nil { if _, err := w.outFile.WriteString("]}"); err != nil {
return gibidiutils.WrapError(err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOWrite, "failed to write JSON end") return shared.WrapError(err, shared.ErrorTypeIO, shared.CodeIOWrite, "failed to write JSON end")
} }
return nil return nil
} }
// writeStreaming writes a large file as JSON in streaming chunks. // writeStreaming writes a large file as JSON in streaming chunks.
func (w *JSONWriter) writeStreaming(req WriteRequest) error { func (w *JSONWriter) writeStreaming(req WriteRequest) error {
defer gibidiutils.SafeCloseReader(req.Reader, req.Path) defer shared.SafeCloseReader(req.Reader, req.Path)
language := detectLanguage(req.Path) language := detectLanguage(req.Path)
// Write file start // Write file start
escapedPath := gibidiutils.EscapeForJSON(req.Path) escapedPath := shared.EscapeForJSON(req.Path)
if _, err := fmt.Fprintf(w.outFile, `{"path":"%s","language":"%s","content":"`, escapedPath, language); err != nil { if _, err := fmt.Fprintf(w.outFile, `{"path":"%s","language":"%s","content":"`, escapedPath, language); err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOWrite, err,
shared.ErrorTypeIO,
shared.CodeIOWrite,
"failed to write JSON file start", "failed to write JSON file start",
).WithFilePath(req.Path) ).WithFilePath(req.Path)
} }
@@ -119,8 +104,10 @@ func (w *JSONWriter) writeStreaming(req WriteRequest) error {
// Write file end // Write file end
if _, err := w.outFile.WriteString(`"}`); err != nil { if _, err := w.outFile.WriteString(`"}`); err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOWrite, err,
shared.ErrorTypeIO,
shared.CodeIOWrite,
"failed to write JSON file end", "failed to write JSON file end",
).WithFilePath(req.Path) ).WithFilePath(req.Path)
} }
@@ -139,50 +126,44 @@ func (w *JSONWriter) writeInline(req WriteRequest) error {
encoded, err := json.Marshal(fileData) encoded, err := json.Marshal(fileData)
if err != nil { if err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, gibidiutils.ErrorTypeProcessing, gibidiutils.CodeProcessingEncode, err,
shared.ErrorTypeProcessing,
shared.CodeProcessingEncode,
"failed to marshal JSON", "failed to marshal JSON",
).WithFilePath(req.Path) ).WithFilePath(req.Path)
} }
if _, err := w.outFile.Write(encoded); err != nil { if _, err := w.outFile.Write(encoded); err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOWrite, err,
shared.ErrorTypeIO,
shared.CodeIOWrite,
"failed to write JSON file", "failed to write JSON file",
).WithFilePath(req.Path) ).WithFilePath(req.Path)
} }
return nil return nil
} }
// streamJSONContent streams content with JSON escaping. // streamJSONContent streams content with JSON escaping.
func (w *JSONWriter) streamJSONContent(reader io.Reader, path string) error { func (w *JSONWriter) streamJSONContent(reader io.Reader, path string) error {
return gibidiutils.StreamContent(reader, w.outFile, StreamChunkSize, path, func(chunk []byte) []byte { if err := shared.StreamContent(
escaped := gibidiutils.EscapeForJSON(string(chunk)) reader, w.outFile, shared.FileProcessingStreamChunkSize, path, func(chunk []byte) []byte {
return []byte(escaped) escaped := shared.EscapeForJSON(string(chunk))
})
return []byte(escaped)
},
); err != nil {
return fmt.Errorf("streaming JSON content: %w", err)
}
return nil
} }
// startJSONWriter handles JSON format output with streaming support. // startJSONWriter handles JSON format output with streaming support.
func startJSONWriter(outFile *os.File, writeCh <-chan WriteRequest, done chan<- struct{}, prefix, suffix string) { func startJSONWriter(outFile *os.File, writeCh <-chan WriteRequest, done chan<- struct{}, prefix, suffix string) {
defer close(done) startFormatWriter(outFile, writeCh, done, prefix, suffix, func(f *os.File) FormatWriter {
return NewJSONWriter(f)
writer := NewJSONWriter(outFile) })
// Start writing
if err := writer.Start(prefix, suffix); err != nil {
gibidiutils.LogError("Failed to write JSON start", err)
return
}
// Process files
for req := range writeCh {
if err := writer.WriteFile(req); err != nil {
gibidiutils.LogError("Failed to write JSON file", err)
}
}
// Close writer
if err := writer.Close(); err != nil {
gibidiutils.LogError("Failed to write JSON end", err)
}
} }

View File

@@ -1,18 +1,17 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import ( import (
"fmt" "fmt"
"io"
"os" "os"
"path/filepath"
"strings"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// MarkdownWriter handles Markdown format output with streaming support. // MarkdownWriter handles Markdown format output with streaming support.
type MarkdownWriter struct { type MarkdownWriter struct {
outFile *os.File outFile *os.File
suffix string
} }
// NewMarkdownWriter creates a new markdown writer. // NewMarkdownWriter creates a new markdown writer.
@@ -20,18 +19,17 @@ func NewMarkdownWriter(outFile *os.File) *MarkdownWriter {
return &MarkdownWriter{outFile: outFile} return &MarkdownWriter{outFile: outFile}
} }
// Start writes the markdown header. // Start writes the markdown header and stores the suffix for later use.
func (w *MarkdownWriter) Start(prefix, _ string) error { func (w *MarkdownWriter) Start(prefix, suffix string) error {
// Store suffix for use in Close method
w.suffix = suffix
if prefix != "" { if prefix != "" {
if _, err := fmt.Fprintf(w.outFile, "# %s\n\n", prefix); err != nil { if _, err := fmt.Fprintf(w.outFile, "# %s\n\n", prefix); err != nil {
return gibidiutils.WrapError( return shared.WrapError(err, shared.ErrorTypeIO, shared.CodeIOWrite, "failed to write prefix")
err,
gibidiutils.ErrorTypeIO,
gibidiutils.CodeIOWrite,
"failed to write prefix",
)
} }
} }
return nil return nil
} }
@@ -40,71 +38,15 @@ func (w *MarkdownWriter) WriteFile(req WriteRequest) error {
if req.IsStream { if req.IsStream {
return w.writeStreaming(req) return w.writeStreaming(req)
} }
return w.writeInline(req) return w.writeInline(req)
} }
// Close writes the markdown footer. // Close writes the markdown footer using the suffix stored in Start.
func (w *MarkdownWriter) Close(suffix string) error { func (w *MarkdownWriter) Close() error {
if suffix != "" { if w.suffix != "" {
if _, err := fmt.Fprintf(w.outFile, "\n# %s\n", suffix); err != nil { if _, err := fmt.Fprintf(w.outFile, "\n# %s\n", w.suffix); err != nil {
return gibidiutils.WrapError( return shared.WrapError(err, shared.ErrorTypeIO, shared.CodeIOWrite, "failed to write suffix")
err,
gibidiutils.ErrorTypeIO,
gibidiutils.CodeIOWrite,
"failed to write suffix",
)
}
}
return nil
}
// validateMarkdownPath validates a file path for markdown output.
func validateMarkdownPath(path string) error {
trimmed := strings.TrimSpace(path)
if trimmed == "" {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationRequired,
"file path cannot be empty",
"",
nil,
)
}
// Reject absolute paths
if filepath.IsAbs(trimmed) {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationPath,
"absolute paths are not allowed",
trimmed,
map[string]any{"path": trimmed},
)
}
// Clean and validate path components
cleaned := filepath.Clean(trimmed)
if filepath.IsAbs(cleaned) || strings.HasPrefix(cleaned, "/") {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationPath,
"path must be relative",
trimmed,
map[string]any{"path": trimmed, "cleaned": cleaned},
)
}
// Check for path traversal in components
components := strings.Split(filepath.ToSlash(cleaned), "/")
for _, component := range components {
if component == ".." {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationPath,
"path traversal not allowed",
trimmed,
map[string]any{"path": trimmed, "cleaned": cleaned},
)
} }
} }
@@ -113,44 +55,32 @@ func validateMarkdownPath(path string) error {
// writeStreaming writes a large file in streaming chunks. // writeStreaming writes a large file in streaming chunks.
func (w *MarkdownWriter) writeStreaming(req WriteRequest) error { func (w *MarkdownWriter) writeStreaming(req WriteRequest) error {
// Validate path before use defer shared.SafeCloseReader(req.Reader, req.Path)
if err := validateMarkdownPath(req.Path); err != nil {
return err
}
// Check for nil reader
if req.Reader == nil {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationRequired,
"nil reader in write request",
"",
nil,
).WithFilePath(req.Path)
}
defer gibidiutils.SafeCloseReader(req.Reader, req.Path)
language := detectLanguage(req.Path) language := detectLanguage(req.Path)
// Write file header // Write file header
safePath := gibidiutils.EscapeForMarkdown(req.Path) if _, err := fmt.Fprintf(w.outFile, "## File: `%s`\n```%s\n", req.Path, language); err != nil {
if _, err := fmt.Fprintf(w.outFile, "## File: `%s`\n```%s\n", safePath, language); err != nil { return shared.WrapError(
return gibidiutils.WrapError( err,
err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOWrite, shared.ErrorTypeIO,
shared.CodeIOWrite,
"failed to write file header", "failed to write file header",
).WithFilePath(req.Path) ).WithFilePath(req.Path)
} }
// Stream file content in chunks // Stream file content in chunks
if err := w.streamContent(req.Reader, req.Path); err != nil { chunkSize := shared.FileProcessingStreamChunkSize
return err if err := shared.StreamContent(req.Reader, w.outFile, chunkSize, req.Path, nil); err != nil {
return shared.WrapError(err, shared.ErrorTypeIO, shared.CodeIOWrite, "streaming content for markdown file")
} }
// Write file footer // Write file footer
if _, err := w.outFile.WriteString("\n```\n\n"); err != nil { if _, err := w.outFile.WriteString("\n```\n\n"); err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOWrite, err,
shared.ErrorTypeIO,
shared.CodeIOWrite,
"failed to write file footer", "failed to write file footer",
).WithFilePath(req.Path) ).WithFilePath(req.Path)
} }
@@ -160,55 +90,24 @@ func (w *MarkdownWriter) writeStreaming(req WriteRequest) error {
// writeInline writes a small file directly from content. // writeInline writes a small file directly from content.
func (w *MarkdownWriter) writeInline(req WriteRequest) error { func (w *MarkdownWriter) writeInline(req WriteRequest) error {
// Validate path before use
if err := validateMarkdownPath(req.Path); err != nil {
return err
}
language := detectLanguage(req.Path) language := detectLanguage(req.Path)
safePath := gibidiutils.EscapeForMarkdown(req.Path) formatted := fmt.Sprintf("## File: `%s`\n```%s\n%s\n```\n\n", req.Path, language, req.Content)
formatted := fmt.Sprintf("## File: `%s`\n```%s\n%s\n```\n\n", safePath, language, req.Content)
if _, err := w.outFile.WriteString(formatted); err != nil { if _, err := w.outFile.WriteString(formatted); err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOWrite, err,
shared.ErrorTypeIO,
shared.CodeIOWrite,
"failed to write inline content", "failed to write inline content",
).WithFilePath(req.Path) ).WithFilePath(req.Path)
} }
return nil return nil
} }
// streamContent streams file content in chunks.
func (w *MarkdownWriter) streamContent(reader io.Reader, path string) error {
return gibidiutils.StreamContent(reader, w.outFile, StreamChunkSize, path, nil)
}
// startMarkdownWriter handles Markdown format output with streaming support. // startMarkdownWriter handles Markdown format output with streaming support.
func startMarkdownWriter( func startMarkdownWriter(outFile *os.File, writeCh <-chan WriteRequest, done chan<- struct{}, prefix, suffix string) {
outFile *os.File, startFormatWriter(outFile, writeCh, done, prefix, suffix, func(f *os.File) FormatWriter {
writeCh <-chan WriteRequest, return NewMarkdownWriter(f)
done chan<- struct{}, })
prefix, suffix string,
) {
defer close(done)
writer := NewMarkdownWriter(outFile)
// Start writing
if err := writer.Start(prefix, suffix); err != nil {
gibidiutils.LogError("Failed to write markdown prefix", err)
return
}
// Process files
for req := range writeCh {
if err := writer.WriteFile(req); err != nil {
gibidiutils.LogError("Failed to write markdown file", err)
}
}
// Close writer
if err := writer.Close(suffix); err != nil {
gibidiutils.LogError("Failed to write markdown suffix", err)
}
} }

View File

@@ -9,21 +9,11 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"sync"
"time" "time"
"github.com/sirupsen/logrus"
"github.com/ivuorinen/gibidify/config" "github.com/ivuorinen/gibidify/config"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
)
const (
// StreamChunkSize is the size of chunks when streaming large files (64KB).
StreamChunkSize = 65536
// StreamThreshold is the file size above which we use streaming (1MB).
StreamThreshold = 1048576
// MaxMemoryBuffer is the maximum memory to use for buffering content (10MB).
MaxMemoryBuffer = 10485760
) )
// WriteRequest represents the content to be written. // WriteRequest represents the content to be written.
@@ -32,26 +22,7 @@ type WriteRequest struct {
Content string Content string
IsStream bool IsStream bool
Reader io.Reader Reader io.Reader
} Size int64 // File size for streaming files
// multiReaderCloser wraps an io.Reader with a Close method that closes underlying closers.
type multiReaderCloser struct {
reader io.Reader
closers []io.Closer
}
func (m *multiReaderCloser) Read(p []byte) (n int, err error) {
return m.reader.Read(p)
}
func (m *multiReaderCloser) Close() error {
var firstErr error
for _, c := range m.closers {
if err := c.Close(); err != nil && firstErr == nil {
firstErr = err
}
}
return firstErr
} }
// FileProcessor handles file processing operations. // FileProcessor handles file processing operations.
@@ -65,7 +36,7 @@ type FileProcessor struct {
func NewFileProcessor(rootPath string) *FileProcessor { func NewFileProcessor(rootPath string) *FileProcessor {
return &FileProcessor{ return &FileProcessor{
rootPath: rootPath, rootPath: rootPath,
sizeLimit: config.GetFileSizeLimit(), sizeLimit: config.FileSizeLimit(),
resourceMonitor: NewResourceMonitor(), resourceMonitor: NewResourceMonitor(),
} }
} }
@@ -74,45 +45,19 @@ func NewFileProcessor(rootPath string) *FileProcessor {
func NewFileProcessorWithMonitor(rootPath string, monitor *ResourceMonitor) *FileProcessor { func NewFileProcessorWithMonitor(rootPath string, monitor *ResourceMonitor) *FileProcessor {
return &FileProcessor{ return &FileProcessor{
rootPath: rootPath, rootPath: rootPath,
sizeLimit: config.GetFileSizeLimit(), sizeLimit: config.FileSizeLimit(),
resourceMonitor: monitor, resourceMonitor: monitor,
} }
} }
// checkContextCancellation checks if context is cancelled and logs an error if so.
// Returns true if context is cancelled, false otherwise.
func (p *FileProcessor) checkContextCancellation(ctx context.Context, filePath, stage string) bool {
select {
case <-ctx.Done():
// Format stage with leading space if provided
stageMsg := stage
if stage != "" {
stageMsg = " " + stage
}
gibidiutils.LogErrorf(
gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeResourceLimitTimeout,
fmt.Sprintf("file processing cancelled%s", stageMsg),
filePath,
nil,
),
"File processing cancelled%s: %s",
stageMsg,
filePath,
)
return true
default:
return false
}
}
// ProcessFile reads the file at filePath and sends a formatted output to outCh. // ProcessFile reads the file at filePath and sends a formatted output to outCh.
// It automatically chooses between loading the entire file or streaming based on file size. // It automatically chooses between loading the entire file or streaming based on file size.
func ProcessFile(filePath string, outCh chan<- WriteRequest, rootPath string) { func ProcessFile(filePath string, outCh chan<- WriteRequest, rootPath string) {
processor := NewFileProcessor(rootPath) processor := NewFileProcessor(rootPath)
ctx := context.Background() ctx := context.Background()
processor.ProcessWithContext(ctx, filePath, outCh) if err := processor.ProcessWithContext(ctx, filePath, outCh); err != nil {
shared.LogErrorf(err, shared.FileProcessingMsgFailedToProcess, filePath)
}
} }
// ProcessFileWithMonitor processes a file using a shared resource monitor. // ProcessFileWithMonitor processes a file using a shared resource monitor.
@@ -122,19 +67,25 @@ func ProcessFileWithMonitor(
outCh chan<- WriteRequest, outCh chan<- WriteRequest,
rootPath string, rootPath string,
monitor *ResourceMonitor, monitor *ResourceMonitor,
) { ) error {
if monitor == nil {
monitor = NewResourceMonitor()
}
processor := NewFileProcessorWithMonitor(rootPath, monitor) processor := NewFileProcessorWithMonitor(rootPath, monitor)
processor.ProcessWithContext(ctx, filePath, outCh)
return processor.ProcessWithContext(ctx, filePath, outCh)
} }
// Process handles file processing with the configured settings. // Process handles file processing with the configured settings.
func (p *FileProcessor) Process(filePath string, outCh chan<- WriteRequest) { func (p *FileProcessor) Process(filePath string, outCh chan<- WriteRequest) {
ctx := context.Background() ctx := context.Background()
p.ProcessWithContext(ctx, filePath, outCh) if err := p.ProcessWithContext(ctx, filePath, outCh); err != nil {
shared.LogErrorf(err, shared.FileProcessingMsgFailedToProcess, filePath)
}
} }
// ProcessWithContext handles file processing with context and resource monitoring. // ProcessWithContext handles file processing with context and resource monitoring.
func (p *FileProcessor) ProcessWithContext(ctx context.Context, filePath string, outCh chan<- WriteRequest) { func (p *FileProcessor) ProcessWithContext(ctx context.Context, filePath string, outCh chan<- WriteRequest) error {
// Create file processing context with timeout // Create file processing context with timeout
fileCtx, fileCancel := p.resourceMonitor.CreateFileProcessingContext(ctx) fileCtx, fileCancel := p.resourceMonitor.CreateFileProcessingContext(ctx)
defer fileCancel() defer fileCancel()
@@ -142,50 +93,51 @@ func (p *FileProcessor) ProcessWithContext(ctx context.Context, filePath string,
// Wait for rate limiting // Wait for rate limiting
if err := p.resourceMonitor.WaitForRateLimit(fileCtx); err != nil { if err := p.resourceMonitor.WaitForRateLimit(fileCtx); err != nil {
if errors.Is(err, context.DeadlineExceeded) { if errors.Is(err, context.DeadlineExceeded) {
gibidiutils.LogErrorf( structErr := shared.NewStructuredError(
gibidiutils.NewStructuredError( shared.ErrorTypeValidation,
gibidiutils.ErrorTypeValidation, shared.CodeResourceLimitTimeout,
gibidiutils.CodeResourceLimitTimeout, "file processing timeout during rate limiting",
"file processing timeout during rate limiting",
filePath,
nil,
),
"File processing timeout during rate limiting: %s",
filePath, filePath,
nil,
) )
shared.LogErrorf(structErr, "File processing timeout during rate limiting: %s", filePath)
return structErr
} }
return
return err
} }
// Validate file and check resource limits // Validate file and check resource limits
fileInfo, err := p.validateFileWithLimits(fileCtx, filePath) fileInfo, err := p.validateFileWithLimits(fileCtx, filePath)
if err != nil { if err != nil {
return // Error already logged return err // Error already logged
} }
// Acquire read slot for concurrent processing // Acquire read slot for concurrent processing
if err := p.resourceMonitor.AcquireReadSlot(fileCtx); err != nil { if err := p.resourceMonitor.AcquireReadSlot(fileCtx); err != nil {
if errors.Is(err, context.DeadlineExceeded) { if errors.Is(err, context.DeadlineExceeded) {
gibidiutils.LogErrorf( structErr := shared.NewStructuredError(
gibidiutils.NewStructuredError( shared.ErrorTypeValidation,
gibidiutils.ErrorTypeValidation, shared.CodeResourceLimitTimeout,
gibidiutils.CodeResourceLimitTimeout, "file processing timeout waiting for read slot",
"file processing timeout waiting for read slot",
filePath,
nil,
),
"File processing timeout waiting for read slot: %s",
filePath, filePath,
nil,
) )
shared.LogErrorf(structErr, "File processing timeout waiting for read slot: %s", filePath)
return structErr
} }
return
return err
} }
defer p.resourceMonitor.ReleaseReadSlot() defer p.resourceMonitor.ReleaseReadSlot()
// Check hard memory limits before processing // Check hard memory limits before processing
if err := p.resourceMonitor.CheckHardMemoryLimit(); err != nil { if err := p.resourceMonitor.CheckHardMemoryLimit(); err != nil {
gibidiutils.LogErrorf(err, "Hard memory limit check failed for file: %s", filePath) shared.LogErrorf(err, "Hard memory limit check failed for file: %s", filePath)
return
return err
} }
// Get relative path // Get relative path
@@ -193,61 +145,69 @@ func (p *FileProcessor) ProcessWithContext(ctx context.Context, filePath string,
// Process file with timeout // Process file with timeout
processStart := time.Now() processStart := time.Now()
defer func() {
// Record successful processing
p.resourceMonitor.RecordFileProcessed(fileInfo.Size())
logrus.Debugf("File processed in %v: %s", time.Since(processStart), filePath)
}()
// Choose processing strategy based on file size // Choose processing strategy based on file size
if fileInfo.Size() <= StreamThreshold { if fileInfo.Size() <= shared.FileProcessingStreamThreshold {
p.processInMemoryWithContext(fileCtx, filePath, relPath, outCh) err = p.processInMemoryWithContext(fileCtx, filePath, relPath, outCh)
} else { } else {
p.processStreamingWithContext(fileCtx, filePath, relPath, outCh) err = p.processStreamingWithContext(fileCtx, filePath, relPath, outCh, fileInfo.Size())
} }
// Only record success if processing completed without error
if err != nil {
return err
}
// Record successful processing only on success path
p.resourceMonitor.RecordFileProcessed(fileInfo.Size())
logger := shared.GetLogger()
logger.Debugf("File processed in %v: %s", time.Since(processStart), filePath)
return nil
} }
// validateFileWithLimits checks if the file can be processed with resource limits. // validateFileWithLimits checks if the file can be processed with resource limits.
func (p *FileProcessor) validateFileWithLimits(ctx context.Context, filePath string) (os.FileInfo, error) { func (p *FileProcessor) validateFileWithLimits(ctx context.Context, filePath string) (os.FileInfo, error) {
// Check context cancellation // Check context cancellation
select { if err := shared.CheckContextCancellation(ctx, "file validation"); err != nil {
case <-ctx.Done(): return nil, fmt.Errorf("context check during file validation: %w", err)
return nil, ctx.Err()
default:
} }
fileInfo, err := os.Stat(filePath) fileInfo, err := os.Stat(filePath)
if err != nil { if err != nil {
structErr := gibidiutils.WrapError( structErr := shared.WrapError(
err, gibidiutils.ErrorTypeFileSystem, gibidiutils.CodeFSAccess, err,
shared.ErrorTypeFileSystem,
shared.CodeFSAccess,
"failed to stat file", "failed to stat file",
).WithFilePath(filePath) ).WithFilePath(filePath)
gibidiutils.LogErrorf(structErr, "Failed to stat file %s", filePath) shared.LogErrorf(structErr, "Failed to stat file %s", filePath)
return nil, structErr return nil, structErr
} }
// Check traditional size limit // Check traditional size limit
if fileInfo.Size() > p.sizeLimit { if fileInfo.Size() > p.sizeLimit {
filesizeContext := map[string]interface{}{ c := map[string]any{
"file_size": fileInfo.Size(), "file_size": fileInfo.Size(),
"size_limit": p.sizeLimit, "size_limit": p.sizeLimit,
} }
gibidiutils.LogErrorf( structErr := shared.NewStructuredError(
gibidiutils.NewStructuredError( shared.ErrorTypeValidation,
gibidiutils.ErrorTypeValidation, shared.CodeValidationSize,
gibidiutils.CodeValidationSize, fmt.Sprintf(shared.FileProcessingMsgSizeExceeds, fileInfo.Size(), p.sizeLimit),
fmt.Sprintf("file size (%d bytes) exceeds limit (%d bytes)", fileInfo.Size(), p.sizeLimit), filePath,
filePath, c,
filesizeContext,
),
"Skipping large file %s", filePath,
) )
return nil, fmt.Errorf("file too large") shared.LogErrorf(structErr, "Skipping large file %s", filePath)
return nil, structErr
} }
// Check resource limits // Check resource limits
if err := p.resourceMonitor.ValidateFileProcessing(filePath, fileInfo.Size()); err != nil { if err := p.resourceMonitor.ValidateFileProcessing(filePath, fileInfo.Size()); err != nil {
gibidiutils.LogErrorf(err, "Resource limit validation failed for file: %s", filePath) shared.LogErrorf(err, "Resource limit validation failed for file: %s", filePath)
return nil, err return nil, err
} }
@@ -260,6 +220,7 @@ func (p *FileProcessor) getRelativePath(filePath string) string {
if err != nil { if err != nil {
return filePath // Fallback return filePath // Fallback
} }
return relPath return relPath
} }
@@ -268,38 +229,74 @@ func (p *FileProcessor) processInMemoryWithContext(
ctx context.Context, ctx context.Context,
filePath, relPath string, filePath, relPath string,
outCh chan<- WriteRequest, outCh chan<- WriteRequest,
) { ) error {
// Check context before reading // Check context before reading
if p.checkContextCancellation(ctx, filePath, "") { select {
return case <-ctx.Done():
structErr := shared.NewStructuredError(
shared.ErrorTypeValidation,
shared.CodeResourceLimitTimeout,
"file processing canceled",
filePath,
nil,
)
shared.LogErrorf(structErr, "File processing canceled: %s", filePath)
return structErr
default:
} }
// #nosec G304 - filePath is validated by walker content, err := os.ReadFile(filePath) // #nosec G304 - filePath is validated by walker
content, err := os.ReadFile(filePath)
if err != nil { if err != nil {
structErr := gibidiutils.WrapError( structErr := shared.WrapError(
err, gibidiutils.ErrorTypeProcessing, gibidiutils.CodeProcessingFileRead, err,
shared.ErrorTypeProcessing,
shared.CodeProcessingFileRead,
"failed to read file", "failed to read file",
).WithFilePath(filePath) ).WithFilePath(filePath)
gibidiutils.LogErrorf(structErr, "Failed to read file %s", filePath) shared.LogErrorf(structErr, "Failed to read file %s", filePath)
return
return structErr
} }
// Check context again after reading // Check context again after reading
if p.checkContextCancellation(ctx, filePath, "after read") { select {
return case <-ctx.Done():
structErr := shared.NewStructuredError(
shared.ErrorTypeValidation,
shared.CodeResourceLimitTimeout,
"file processing canceled after read",
filePath,
nil,
)
shared.LogErrorf(structErr, "File processing canceled after read: %s", filePath)
return structErr
default:
} }
// Check context before sending output // Try to send the result, but respect context cancellation
if p.checkContextCancellation(ctx, filePath, "before output") { select {
return case <-ctx.Done():
} structErr := shared.NewStructuredError(
shared.ErrorTypeValidation,
shared.CodeResourceLimitTimeout,
"file processing canceled before output",
filePath,
nil,
)
shared.LogErrorf(structErr, "File processing canceled before output: %s", filePath)
outCh <- WriteRequest{ return structErr
case outCh <- WriteRequest{
Path: relPath, Path: relPath,
Content: p.formatContent(relPath, string(content)), Content: p.formatContent(relPath, string(content)),
IsStream: false, IsStream: false,
Size: int64(len(content)),
}:
} }
return nil
} }
// processStreamingWithContext creates a streaming reader for large files with context awareness. // processStreamingWithContext creates a streaming reader for large files with context awareness.
@@ -307,58 +304,87 @@ func (p *FileProcessor) processStreamingWithContext(
ctx context.Context, ctx context.Context,
filePath, relPath string, filePath, relPath string,
outCh chan<- WriteRequest, outCh chan<- WriteRequest,
) { size int64,
) error {
// Check context before creating reader // Check context before creating reader
if p.checkContextCancellation(ctx, filePath, "before streaming") { select {
return case <-ctx.Done():
structErr := shared.NewStructuredError(
shared.ErrorTypeValidation,
shared.CodeResourceLimitTimeout,
"streaming processing canceled",
filePath,
nil,
)
shared.LogErrorf(structErr, "Streaming processing canceled: %s", filePath)
return structErr
default:
} }
reader := p.createStreamReaderWithContext(ctx, filePath, relPath) reader := p.createStreamReaderWithContext(ctx, filePath, relPath)
if reader == nil { if reader == nil {
return // Error already logged // Error already logged, create and return error
return shared.NewStructuredError(
shared.ErrorTypeProcessing,
shared.CodeProcessingFileRead,
"failed to create stream reader",
filePath,
nil,
)
} }
// Check context before sending output // Try to send the result, but respect context cancellation
if p.checkContextCancellation(ctx, filePath, "before streaming output") { select {
// Close the reader to prevent file descriptor leak case <-ctx.Done():
if closer, ok := reader.(io.Closer); ok { structErr := shared.NewStructuredError(
_ = closer.Close() shared.ErrorTypeValidation,
} shared.CodeResourceLimitTimeout,
return "streaming processing canceled before output",
} filePath,
nil,
)
shared.LogErrorf(structErr, "Streaming processing canceled before output: %s", filePath)
outCh <- WriteRequest{ return structErr
case outCh <- WriteRequest{
Path: relPath, Path: relPath,
Content: "", // Empty since content is in Reader Content: "", // Empty since content is in Reader
IsStream: true, IsStream: true,
Reader: reader, Reader: reader,
Size: size,
}:
} }
return nil
} }
// createStreamReaderWithContext creates a reader that combines header and file content with context awareness. // createStreamReaderWithContext creates a reader that combines header and file content with context awareness.
func (p *FileProcessor) createStreamReaderWithContext(ctx context.Context, filePath, relPath string) io.Reader { func (p *FileProcessor) createStreamReaderWithContext(
ctx context.Context, filePath, relPath string,
) io.Reader {
// Check context before opening file // Check context before opening file
if p.checkContextCancellation(ctx, filePath, "before opening file") { select {
case <-ctx.Done():
return nil return nil
default:
} }
// #nosec G304 - filePath is validated by walker file, err := os.Open(filePath) // #nosec G304 - filePath is validated by walker
file, err := os.Open(filePath)
if err != nil { if err != nil {
structErr := gibidiutils.WrapError( structErr := shared.WrapError(
err, gibidiutils.ErrorTypeProcessing, gibidiutils.CodeProcessingFileRead, err,
shared.ErrorTypeProcessing,
shared.CodeProcessingFileRead,
"failed to open file for streaming", "failed to open file for streaming",
).WithFilePath(filePath) ).WithFilePath(filePath)
gibidiutils.LogErrorf(structErr, "Failed to open file for streaming %s", filePath) shared.LogErrorf(structErr, "Failed to open file for streaming %s", filePath)
return nil return nil
} }
header := p.formatHeader(relPath) header := p.formatHeader(relPath)
// Wrap in multiReaderCloser to ensure file is closed even on cancellation
return &multiReaderCloser{ return newHeaderFileReader(header, file)
reader: io.MultiReader(header, file),
closers: []io.Closer{file},
}
} }
// formatContent formats the file content with header. // formatContent formats the file content with header.
@@ -370,3 +396,66 @@ func (p *FileProcessor) formatContent(relPath, content string) string {
func (p *FileProcessor) formatHeader(relPath string) io.Reader { func (p *FileProcessor) formatHeader(relPath string) io.Reader {
return strings.NewReader(fmt.Sprintf("\n---\n%s\n", relPath)) return strings.NewReader(fmt.Sprintf("\n---\n%s\n", relPath))
} }
// headerFileReader wraps a MultiReader and closes the file when EOF is reached.
type headerFileReader struct {
reader io.Reader
file *os.File
mu sync.Mutex
closed bool
}
// newHeaderFileReader creates a new headerFileReader.
func newHeaderFileReader(header io.Reader, file *os.File) *headerFileReader {
return &headerFileReader{
reader: io.MultiReader(header, file),
file: file,
}
}
// Read implements io.Reader and closes the file on EOF.
func (r *headerFileReader) Read(p []byte) (n int, err error) {
n, err = r.reader.Read(p)
if err == io.EOF {
r.closeFile()
// EOF is a sentinel value that must be passed through unchanged for io.Reader interface
return n, err //nolint:wrapcheck // EOF must not be wrapped
}
if err != nil {
return n, shared.WrapError(
err, shared.ErrorTypeIO, shared.CodeIORead,
"failed to read from header file reader",
)
}
return n, nil
}
// closeFile closes the file once.
func (r *headerFileReader) closeFile() {
r.mu.Lock()
defer r.mu.Unlock()
if !r.closed && r.file != nil {
if err := r.file.Close(); err != nil {
shared.LogError("Failed to close file", err)
}
r.closed = true
}
}
// Close implements io.Closer and ensures the underlying file is closed.
// This allows explicit cleanup when consumers stop reading before EOF.
func (r *headerFileReader) Close() error {
r.mu.Lock()
defer r.mu.Unlock()
if r.closed || r.file == nil {
return nil
}
err := r.file.Close()
if err != nil {
shared.LogError("Failed to close file", err)
}
r.closed = true
return err
}

View File

@@ -1,15 +1,84 @@
package fileproc_test package fileproc_test
import ( import (
"context"
"fmt"
"io"
"os" "os"
"path/filepath"
"strings" "strings"
"sync" "sync"
"testing" "testing"
"time"
"github.com/spf13/viper"
"github.com/ivuorinen/gibidify/config"
"github.com/ivuorinen/gibidify/fileproc" "github.com/ivuorinen/gibidify/fileproc"
"github.com/ivuorinen/gibidify/shared"
"github.com/ivuorinen/gibidify/testutil" "github.com/ivuorinen/gibidify/testutil"
) )
// writeTempConfig creates a temporary config file with the given YAML content
// and returns the directory path containing the config file.
func writeTempConfig(t *testing.T, content string) string {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yaml")
if err := os.WriteFile(configPath, []byte(content), 0o600); err != nil {
t.Fatalf("Failed to create temp config: %v", err)
}
return dir
}
// collectWriteRequests runs a processing function and collects all WriteRequests.
// This helper wraps the common pattern of channel + goroutine + WaitGroup.
func collectWriteRequests(t *testing.T, process func(ch chan fileproc.WriteRequest)) []fileproc.WriteRequest {
t.Helper()
ch := make(chan fileproc.WriteRequest, 10)
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
process(ch)
})
results := make([]fileproc.WriteRequest, 0)
for req := range ch {
results = append(results, req)
}
wg.Wait()
return results
}
// collectWriteRequestsWithContext runs a processing function with context and collects all WriteRequests.
func collectWriteRequestsWithContext(
ctx context.Context,
t *testing.T,
process func(ctx context.Context, ch chan fileproc.WriteRequest) error,
) ([]fileproc.WriteRequest, error) {
t.Helper()
ch := make(chan fileproc.WriteRequest, 10)
var processErr error
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
processErr = process(ctx, ch)
})
results := make([]fileproc.WriteRequest, 0)
for req := range ch {
results = append(results, req)
}
wg.Wait()
return results, processErr
}
func TestProcessFile(t *testing.T) { func TestProcessFile(t *testing.T) {
// Reset and load default config to ensure proper file size limits // Reset and load default config to ensure proper file size limits
testutil.ResetViperConfig(t, "") testutil.ResetViperConfig(t, "")
@@ -32,23 +101,20 @@ func TestProcessFile(t *testing.T) {
errTmpFile := tmpFile.Close() errTmpFile := tmpFile.Close()
if errTmpFile != nil { if errTmpFile != nil {
t.Fatal(errTmpFile) t.Fatal(errTmpFile)
return
} }
ch := make(chan fileproc.WriteRequest, 1) ch := make(chan fileproc.WriteRequest, 1)
var wg sync.WaitGroup var wg sync.WaitGroup
wg.Add(1) wg.Go(func() {
go func() { defer close(ch)
defer wg.Done()
fileproc.ProcessFile(tmpFile.Name(), ch, "") fileproc.ProcessFile(tmpFile.Name(), ch, "")
}() })
wg.Wait()
close(ch)
var result string var result string
for req := range ch { for req := range ch {
result = req.Content result = req.Content
} }
wg.Wait()
if !strings.Contains(result, tmpFile.Name()) { if !strings.Contains(result, tmpFile.Name()) {
t.Errorf("Output does not contain file path: %s", tmpFile.Name()) t.Errorf("Output does not contain file path: %s", tmpFile.Name())
@@ -57,3 +123,686 @@ func TestProcessFile(t *testing.T) {
t.Errorf("Output does not contain file content: %s", content) t.Errorf("Output does not contain file content: %s", content)
} }
} }
// TestNewFileProcessorWithMonitor tests processor creation with resource monitor.
func TestNewFileProcessorWithMonitor(t *testing.T) {
testutil.ResetViperConfig(t, "")
// Create a resource monitor
monitor := fileproc.NewResourceMonitor()
defer monitor.Close()
processor := fileproc.NewFileProcessorWithMonitor("test_source", monitor)
if processor == nil {
t.Error("Expected processor but got nil")
}
// Exercise the processor to verify monitor integration
tmpFile, err := os.CreateTemp(t.TempDir(), "monitor_test")
if err != nil {
t.Fatal(err)
}
defer os.Remove(tmpFile.Name())
if _, err := tmpFile.WriteString("test content"); err != nil {
t.Fatal(err)
}
if err := tmpFile.Close(); err != nil {
t.Fatal(err)
}
ctx := context.Background()
writeCh := make(chan fileproc.WriteRequest, 1)
var wg sync.WaitGroup
wg.Go(func() {
defer close(writeCh)
if err := processor.ProcessWithContext(ctx, tmpFile.Name(), writeCh); err != nil {
t.Errorf("ProcessWithContext failed: %v", err)
}
})
// Drain channel first to avoid deadlock if producer sends multiple requests
requestCount := 0
for range writeCh {
requestCount++
}
// Wait for goroutine to finish after channel is drained
wg.Wait()
if requestCount == 0 {
t.Error("Expected at least one write request from processor")
}
}
// TestProcessFileWithMonitor tests file processing with resource monitoring.
func TestProcessFileWithMonitor(t *testing.T) {
testutil.ResetViperConfig(t, "")
// Create temporary file
tmpFile, err := os.CreateTemp(t.TempDir(), "testfile_monitor_*")
if err != nil {
t.Fatalf(shared.TestMsgFailedToCreateFile, err)
}
defer func() {
if err := os.Remove(tmpFile.Name()); err != nil {
t.Logf("Failed to remove temp file: %v", err)
}
}()
content := "Test content with monitor"
if _, err := tmpFile.WriteString(content); err != nil {
t.Fatalf(shared.TestMsgFailedToWriteContent, err)
}
if err := tmpFile.Close(); err != nil {
t.Fatalf(shared.TestMsgFailedToCloseFile, err)
}
// Create resource monitor
monitor := fileproc.NewResourceMonitor()
defer monitor.Close()
ch := make(chan fileproc.WriteRequest, 1)
ctx := context.Background()
// Test ProcessFileWithMonitor
var wg sync.WaitGroup
var result string
// Start reader goroutine first to prevent deadlock
wg.Go(func() {
for req := range ch {
result = req.Content
}
})
// Process the file
err = fileproc.ProcessFileWithMonitor(ctx, tmpFile.Name(), ch, "", monitor)
close(ch)
if err != nil {
t.Fatalf("ProcessFileWithMonitor failed: %v", err)
}
// Wait for reader to finish
wg.Wait()
if !strings.Contains(result, content) {
t.Error("Expected content not found in processed result")
}
}
const testContent = "package main\nfunc main() {}\n"
// TestProcess tests the basic Process function.
func TestProcess(t *testing.T) {
testutil.ResetViperConfig(t, "")
// Create temporary directory
tmpDir := t.TempDir()
// Create test file with .go extension
testFile := filepath.Join(tmpDir, "test.go")
content := testContent
if err := os.WriteFile(testFile, []byte(content), 0o600); err != nil {
t.Fatalf(shared.TestMsgFailedToCreateTestFile, err)
}
processor := fileproc.NewFileProcessor(tmpDir)
ch := make(chan fileproc.WriteRequest, 10)
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
// Process the specific file, not the directory
processor.Process(testFile, ch)
})
// Collect results
results := make([]fileproc.WriteRequest, 0, 1) // Pre-allocate with expected capacity
for req := range ch {
results = append(results, req)
}
wg.Wait()
if len(results) == 0 {
t.Error("Expected at least one processed file")
return
}
// Find our test file in results
found := false
for _, req := range results {
if strings.Contains(req.Path, shared.TestFileGo) && strings.Contains(req.Content, content) {
found = true
break
}
}
if !found {
t.Error("Test file not found in processed results")
}
}
// createLargeTestFile creates a large test file for streaming tests.
func createLargeTestFile(t *testing.T) *os.File {
t.Helper()
tmpFile, err := os.CreateTemp(t.TempDir(), "large_file_*.go")
if err != nil {
t.Fatalf(shared.TestMsgFailedToCreateFile, err)
}
lineContent := "// Repeated comment line to exceed streaming threshold\n"
repeatCount := (1048576 / len(lineContent)) + 1000
largeContent := strings.Repeat(lineContent, repeatCount)
if _, err := tmpFile.WriteString(largeContent); err != nil {
t.Fatalf(shared.TestMsgFailedToWriteContent, err)
}
if err := tmpFile.Close(); err != nil {
t.Fatalf(shared.TestMsgFailedToCloseFile, err)
}
t.Logf("Created test file size: %d bytes", len(largeContent))
return tmpFile
}
// processFileForStreaming processes a file and returns streaming/inline requests.
func processFileForStreaming(t *testing.T, filePath string) (streamingReq, inlineReq *fileproc.WriteRequest) {
t.Helper()
ch := make(chan fileproc.WriteRequest, 1)
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
fileproc.ProcessFile(filePath, ch, "")
})
var streamingRequest *fileproc.WriteRequest
var inlineRequest *fileproc.WriteRequest
for req := range ch {
if req.IsStream {
reqCopy := req
streamingRequest = &reqCopy
} else {
reqCopy := req
inlineRequest = &reqCopy
}
}
wg.Wait()
return streamingRequest, inlineRequest
}
// validateStreamingRequest validates a streaming request.
func validateStreamingRequest(t *testing.T, streamingRequest *fileproc.WriteRequest, tmpFile *os.File) {
t.Helper()
if streamingRequest.Reader == nil {
t.Error("Expected reader in streaming request")
}
if streamingRequest.Content != "" {
t.Error("Expected empty content for streaming request")
}
buffer := make([]byte, 1024)
n, err := streamingRequest.Reader.Read(buffer)
if err != nil && err != io.EOF {
t.Errorf("Failed to read from streaming request: %v", err)
}
content := string(buffer[:n])
if !strings.Contains(content, tmpFile.Name()) {
t.Error("Expected file path in streamed header content")
}
t.Log("Successfully triggered streaming for large file and tested reader")
}
// TestProcessorStreamingIntegration tests streaming functionality in processor.
func TestProcessorStreamingIntegration(t *testing.T) {
configDir := writeTempConfig(t, `
max_file_size_mb: 0.001
streaming_threshold_mb: 0.0001
`)
testutil.ResetViperConfig(t, configDir)
tmpFile := createLargeTestFile(t)
defer func() {
if err := os.Remove(tmpFile.Name()); err != nil {
t.Logf("Failed to remove temp file: %v", err)
}
}()
streamingRequest, inlineRequest := processFileForStreaming(t, tmpFile.Name())
if streamingRequest == nil && inlineRequest == nil {
t.Error("Expected either streaming or inline request but got none")
}
if streamingRequest != nil {
validateStreamingRequest(t, streamingRequest, tmpFile)
} else {
t.Log("File processed inline instead of streaming")
}
}
// TestProcessorContextCancellation tests context cancellation during processing.
func TestProcessorContextCancellation(t *testing.T) {
testutil.ResetViperConfig(t, "")
// Create temporary directory with files
tmpDir := t.TempDir()
// Create multiple test files
for i := 0; i < 5; i++ {
testFile := filepath.Join(tmpDir, fmt.Sprintf("test%d.go", i))
content := testContent
if err := os.WriteFile(testFile, []byte(content), 0o600); err != nil {
t.Fatalf(shared.TestMsgFailedToCreateTestFile, err)
}
}
processor := fileproc.NewFileProcessor("test_source")
ch := make(chan fileproc.WriteRequest, 10)
// Use ProcessWithContext with immediate cancellation
ctx, cancel := context.WithCancel(context.Background())
cancel() // Cancel immediately
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
// Error is expected due to cancellation
if err := processor.ProcessWithContext(ctx, tmpDir, ch); err != nil {
// Log error for debugging, but don't fail test since cancellation is expected
t.Logf("Expected error due to cancellation: %v", err)
}
})
// Collect results - should be minimal due to cancellation
results := make([]fileproc.WriteRequest, 0, 1) // Pre-allocate with expected capacity
for req := range ch {
results = append(results, req)
}
wg.Wait()
// With immediate cancellation, we might get 0 results
// This tests that cancellation is respected
t.Logf("Processed %d files with immediate cancellation", len(results))
}
// TestProcessorValidationEdgeCases tests edge cases in file validation.
func TestProcessorValidationEdgeCases(t *testing.T) {
configDir := writeTempConfig(t, `
max_file_size_mb: 0.001 # 1KB limit for testing
`)
testutil.ResetViperConfig(t, configDir)
tmpDir := t.TempDir()
// Test case 1: Non-existent file
nonExistentFile := filepath.Join(tmpDir, "does-not-exist.go")
processor := fileproc.NewFileProcessor(tmpDir)
ch := make(chan fileproc.WriteRequest, 1)
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
processor.Process(nonExistentFile, ch)
})
results := make([]fileproc.WriteRequest, 0)
for req := range ch {
results = append(results, req)
}
wg.Wait()
// Should get no results due to file not existing
if len(results) > 0 {
t.Error("Expected no results for non-existent file")
}
// Test case 2: File that exceeds size limit
largeFile := filepath.Join(tmpDir, "large.go")
largeContent := strings.Repeat("// Large file content\n", 100) // > 1KB
if err := os.WriteFile(largeFile, []byte(largeContent), 0o600); err != nil {
t.Fatalf("Failed to create large file: %v", err)
}
ch2 := make(chan fileproc.WriteRequest, 1)
wg.Go(func() {
defer close(ch2)
processor.Process(largeFile, ch2)
})
results2 := make([]fileproc.WriteRequest, 0)
for req := range ch2 {
results2 = append(results2, req)
}
wg.Wait()
// Should get results because even large files are processed (just different strategy)
t.Logf("Large file processing results: %d", len(results2))
}
// TestProcessorContextCancellationDuringValidation tests context cancellation during file validation.
func TestProcessorContextCancellationDuringValidation(t *testing.T) {
testutil.ResetViperConfig(t, "")
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.go")
content := testContent
if err := os.WriteFile(testFile, []byte(content), 0o600); err != nil {
t.Fatalf(shared.TestMsgFailedToCreateTestFile, err)
}
processor := fileproc.NewFileProcessor(tmpDir)
// Create context that we'll cancel during processing
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Nanosecond)
defer cancel()
// Let context expire
time.Sleep(1 * time.Millisecond)
ch := make(chan fileproc.WriteRequest, 1)
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
if err := processor.ProcessWithContext(ctx, testFile, ch); err != nil {
t.Logf("ProcessWithContext error (may be expected): %v", err)
}
})
results := make([]fileproc.WriteRequest, 0)
for req := range ch {
results = append(results, req)
}
wg.Wait()
// Should get no results due to context cancellation
t.Logf("Results with canceled context: %d", len(results))
}
// TestProcessorInMemoryProcessingEdgeCases tests edge cases in in-memory processing.
func TestProcessorInMemoryProcessingEdgeCases(t *testing.T) {
testutil.ResetViperConfig(t, "")
tmpDir := t.TempDir()
// Test with empty file
emptyFile := filepath.Join(tmpDir, "empty.go")
if err := os.WriteFile(emptyFile, []byte(""), 0o600); err != nil {
t.Fatalf("Failed to create empty file: %v", err)
}
processor := fileproc.NewFileProcessor(tmpDir)
ch := make(chan fileproc.WriteRequest, 1)
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
processor.Process(emptyFile, ch)
})
results := make([]fileproc.WriteRequest, 0)
for req := range ch {
results = append(results, req)
}
wg.Wait()
if len(results) != 1 {
t.Errorf("Expected 1 result for empty file, got %d", len(results))
}
if len(results) > 0 {
result := results[0]
if result.Path == "" {
t.Error("Expected path in result for empty file")
}
// Empty file should still be processed
}
}
// TestProcessorStreamingEdgeCases tests edge cases in streaming processing.
func TestProcessorStreamingEdgeCases(t *testing.T) {
testutil.ResetViperConfig(t, "")
tmpDir := t.TempDir()
// Create a file larger than streaming threshold but test error conditions
largeFile := filepath.Join(tmpDir, "large_stream.go")
largeContent := strings.Repeat("// Large streaming file content line\n", 50000) // > 1MB
if err := os.WriteFile(largeFile, []byte(largeContent), 0o600); err != nil {
t.Fatalf("Failed to create large file: %v", err)
}
processor := fileproc.NewFileProcessor(tmpDir)
// Test with context that gets canceled during streaming
ctx, cancel := context.WithCancel(context.Background())
ch := make(chan fileproc.WriteRequest, 1)
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
// Start processing
// Error is expected due to cancellation
if err := processor.ProcessWithContext(ctx, largeFile, ch); err != nil {
// Log error for debugging, but don't fail test since cancellation is expected
t.Logf("Expected error due to cancellation: %v", err)
}
})
// Cancel context after a very short time
go func() {
time.Sleep(1 * time.Millisecond)
cancel()
}()
results := make([]fileproc.WriteRequest, 0)
for req := range ch {
results = append(results, req)
// If we get a streaming request, try to read from it with canceled context
if req.IsStream && req.Reader != nil {
buffer := make([]byte, 1024)
_, err := req.Reader.Read(buffer)
if err != nil && err != io.EOF {
t.Logf("Expected error reading from canceled stream: %v", err)
}
}
}
wg.Wait()
t.Logf("Results with streaming context cancellation: %d", len(results))
}
// Benchmarks for processor hot paths
// BenchmarkProcessFileInline benchmarks inline file processing for small files.
func BenchmarkProcessFileInline(b *testing.B) {
// Initialize config for file processing
viper.Reset()
config.LoadConfig()
// Create a small test file
tmpFile, err := os.CreateTemp(b.TempDir(), "bench_inline_*.go")
if err != nil {
b.Fatalf(shared.TestMsgFailedToCreateFile, err)
}
content := strings.Repeat("// Inline benchmark content\n", 100) // ~2.6KB
if _, err := tmpFile.WriteString(content); err != nil {
b.Fatalf(shared.TestMsgFailedToWriteContent, err)
}
if err := tmpFile.Close(); err != nil {
b.Fatalf(shared.TestMsgFailedToCloseFile, err)
}
b.ResetTimer()
for b.Loop() {
ch := make(chan fileproc.WriteRequest, 1)
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
fileproc.ProcessFile(tmpFile.Name(), ch, "")
})
for req := range ch {
_ = req // Drain channel
}
wg.Wait()
}
}
// BenchmarkProcessFileStreaming benchmarks streaming file processing for large files.
func BenchmarkProcessFileStreaming(b *testing.B) {
// Initialize config for file processing
viper.Reset()
config.LoadConfig()
// Create a large test file that triggers streaming
tmpFile, err := os.CreateTemp(b.TempDir(), "bench_streaming_*.go")
if err != nil {
b.Fatalf(shared.TestMsgFailedToCreateFile, err)
}
// Create content larger than streaming threshold (1MB)
lineContent := "// Streaming benchmark content line that will be repeated\n"
repeatCount := (1048576 / len(lineContent)) + 1000
content := strings.Repeat(lineContent, repeatCount)
if _, err := tmpFile.WriteString(content); err != nil {
b.Fatalf(shared.TestMsgFailedToWriteContent, err)
}
if err := tmpFile.Close(); err != nil {
b.Fatalf(shared.TestMsgFailedToCloseFile, err)
}
b.ResetTimer()
for b.Loop() {
ch := make(chan fileproc.WriteRequest, 1)
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
fileproc.ProcessFile(tmpFile.Name(), ch, "")
})
for req := range ch {
// If streaming, read some content to exercise the reader
if req.IsStream && req.Reader != nil {
buffer := make([]byte, 4096)
for {
_, err := req.Reader.Read(buffer)
if err != nil {
break
}
}
}
}
wg.Wait()
}
}
// BenchmarkProcessorWithContext benchmarks ProcessWithContext for a single file.
func BenchmarkProcessorWithContext(b *testing.B) {
tmpDir := b.TempDir()
testFile := filepath.Join(tmpDir, "bench_context.go")
content := strings.Repeat("// Benchmark file content\n", 50)
if err := os.WriteFile(testFile, []byte(content), 0o600); err != nil {
b.Fatalf(shared.TestMsgFailedToCreateTestFile, err)
}
processor := fileproc.NewFileProcessor(tmpDir)
ctx := context.Background()
b.ResetTimer()
for b.Loop() {
ch := make(chan fileproc.WriteRequest, 1)
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
_ = processor.ProcessWithContext(ctx, testFile, ch)
})
for req := range ch {
_ = req // Drain channel
}
wg.Wait()
}
}
// BenchmarkProcessorWithMonitor benchmarks processing with resource monitoring.
func BenchmarkProcessorWithMonitor(b *testing.B) {
tmpDir := b.TempDir()
testFile := filepath.Join(tmpDir, "bench_monitor.go")
content := strings.Repeat("// Benchmark file content with monitor\n", 50)
if err := os.WriteFile(testFile, []byte(content), 0o600); err != nil {
b.Fatalf(shared.TestMsgFailedToCreateTestFile, err)
}
monitor := fileproc.NewResourceMonitor()
defer monitor.Close()
processor := fileproc.NewFileProcessorWithMonitor(tmpDir, monitor)
ctx := context.Background()
b.ResetTimer()
for b.Loop() {
ch := make(chan fileproc.WriteRequest, 1)
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
_ = processor.ProcessWithContext(ctx, testFile, ch)
})
for req := range ch {
_ = req // Drain channel
}
wg.Wait()
}
}
// BenchmarkProcessorConcurrent benchmarks concurrent file processing.
func BenchmarkProcessorConcurrent(b *testing.B) {
tmpDir := b.TempDir()
// Create multiple test files
testFiles := make([]string, 10)
for i := 0; i < 10; i++ {
testFiles[i] = filepath.Join(tmpDir, fmt.Sprintf("bench_concurrent_%d.go", i))
content := strings.Repeat(fmt.Sprintf("// Concurrent file %d content\n", i), 50)
if err := os.WriteFile(testFiles[i], []byte(content), 0o600); err != nil {
b.Fatalf(shared.TestMsgFailedToCreateTestFile, err)
}
}
processor := fileproc.NewFileProcessor(tmpDir)
ctx := context.Background()
fileCount := len(testFiles)
b.ResetTimer()
b.RunParallel(func(pb *testing.PB) {
i := 0
for pb.Next() {
testFile := testFiles[i%fileCount]
ch := make(chan fileproc.WriteRequest, 1)
var wg sync.WaitGroup
wg.Go(func() {
defer close(ch)
_ = processor.ProcessWithContext(ctx, testFile, ch)
})
for req := range ch {
_ = req // Drain channel
}
wg.Wait()
i++
}
})
}

View File

@@ -5,6 +5,8 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"sync" "sync"
"github.com/ivuorinen/gibidify/shared"
) )
const minExtensionLength = 2 const minExtensionLength = 2
@@ -52,9 +54,9 @@ func initRegistry() *FileTypeRegistry {
imageExts: getImageExtensions(), imageExts: getImageExtensions(),
binaryExts: getBinaryExtensions(), binaryExts: getBinaryExtensions(),
languageMap: getLanguageMap(), languageMap: getLanguageMap(),
extCache: make(map[string]string, 1000), // Cache for extension normalization extCache: make(map[string]string, shared.FileTypeRegistryMaxCacheSize),
resultCache: make(map[string]FileTypeResult, 500), // Cache for type results resultCache: make(map[string]FileTypeResult, shared.FileTypeRegistryMaxCacheSize),
maxCacheSize: 500, maxCacheSize: shared.FileTypeRegistryMaxCacheSize,
} }
} }
@@ -63,25 +65,28 @@ func getRegistry() *FileTypeRegistry {
registryOnce.Do(func() { registryOnce.Do(func() {
registry = initRegistry() registry = initRegistry()
}) })
return registry return registry
} }
// GetDefaultRegistry returns the default file type registry. // DefaultRegistry returns the default file type registry.
func GetDefaultRegistry() *FileTypeRegistry { func DefaultRegistry() *FileTypeRegistry {
return getRegistry() return getRegistry()
} }
// GetStats returns a copy of the current registry statistics. // Stats returns a copy of the current registry statistics.
func (r *FileTypeRegistry) GetStats() RegistryStats { func (r *FileTypeRegistry) Stats() RegistryStats {
r.cacheMutex.RLock() r.cacheMutex.RLock()
defer r.cacheMutex.RUnlock() defer r.cacheMutex.RUnlock()
return r.stats return r.stats
} }
// GetCacheInfo returns current cache size information. // CacheInfo returns current cache size information.
func (r *FileTypeRegistry) GetCacheInfo() (extCacheSize, resultCacheSize, maxCacheSize int) { func (r *FileTypeRegistry) CacheInfo() (extCacheSize, resultCacheSize, maxCacheSize int) {
r.cacheMutex.RLock() r.cacheMutex.RLock()
defer r.cacheMutex.RUnlock() defer r.cacheMutex.RUnlock()
return len(r.extCache), len(r.resultCache), r.maxCacheSize return len(r.extCache), len(r.resultCache), r.maxCacheSize
} }
@@ -101,7 +106,9 @@ func normalizeExtension(filename string) string {
func isSpecialFile(filename string, extensions map[string]bool) bool { func isSpecialFile(filename string, extensions map[string]bool) bool {
if filepath.Ext(filename) == "" { if filepath.Ext(filename) == "" {
basename := strings.ToLower(filepath.Base(filename)) basename := strings.ToLower(filepath.Base(filename))
return extensions[basename] return extensions[basename]
} }
return false return false
} }

View File

@@ -1,7 +1,9 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import ( import (
"context" "context"
"fmt"
"sync/atomic" "sync/atomic"
"time" "time"
) )
@@ -26,7 +28,7 @@ func (rm *ResourceMonitor) AcquireReadSlot(ctx context.Context) error {
// Wait and retry // Wait and retry
select { select {
case <-ctx.Done(): case <-ctx.Done():
return ctx.Err() return fmt.Errorf("context canceled while waiting for read slot: %w", ctx.Err())
case <-time.After(time.Millisecond): case <-time.After(time.Millisecond):
// Continue loop // Continue loop
} }
@@ -45,17 +47,22 @@ func (rm *ResourceMonitor) ReleaseReadSlot() {
// CreateFileProcessingContext creates a context with file processing timeout. // CreateFileProcessingContext creates a context with file processing timeout.
func (rm *ResourceMonitor) CreateFileProcessingContext(parent context.Context) (context.Context, context.CancelFunc) { func (rm *ResourceMonitor) CreateFileProcessingContext(parent context.Context) (context.Context, context.CancelFunc) {
if !rm.enabled || rm.fileProcessingTimeout <= 0 { if !rm.enabled || rm.fileProcessingTimeout <= 0 {
// No-op cancel function - monitoring disabled or no timeout configured
return parent, func() {} return parent, func() {}
} }
return context.WithTimeout(parent, rm.fileProcessingTimeout) return context.WithTimeout(parent, rm.fileProcessingTimeout)
} }
// CreateOverallProcessingContext creates a context with overall processing timeout. // CreateOverallProcessingContext creates a context with overall processing timeout.
func (rm *ResourceMonitor) CreateOverallProcessingContext( func (rm *ResourceMonitor) CreateOverallProcessingContext(parent context.Context) (
parent context.Context, context.Context,
) (context.Context, context.CancelFunc) { context.CancelFunc,
) {
if !rm.enabled || rm.overallTimeout <= 0 { if !rm.enabled || rm.overallTimeout <= 0 {
// No-op cancel function - monitoring disabled or no timeout configured
return parent, func() {} return parent, func() {}
} }
return context.WithTimeout(parent, rm.overallTimeout) return context.WithTimeout(parent, rm.overallTimeout)
} }

View File

@@ -10,7 +10,7 @@ import (
"github.com/ivuorinen/gibidify/testutil" "github.com/ivuorinen/gibidify/testutil"
) )
func TestResourceMonitor_ConcurrentReadsLimit(t *testing.T) { func TestResourceMonitorConcurrentReadsLimit(t *testing.T) {
testutil.ResetViperConfig(t, "") testutil.ResetViperConfig(t, "")
// Set a low concurrent reads limit for testing // Set a low concurrent reads limit for testing
@@ -58,7 +58,7 @@ func TestResourceMonitor_ConcurrentReadsLimit(t *testing.T) {
rm.ReleaseReadSlot() rm.ReleaseReadSlot()
} }
func TestResourceMonitor_TimeoutContexts(t *testing.T) { func TestResourceMonitorTimeoutContexts(t *testing.T) {
testutil.ResetViperConfig(t, "") testutil.ResetViperConfig(t, "")
// Set short timeouts for testing // Set short timeouts for testing

View File

@@ -11,7 +11,7 @@ import (
"github.com/ivuorinen/gibidify/testutil" "github.com/ivuorinen/gibidify/testutil"
) )
func TestResourceMonitor_Integration(t *testing.T) { func TestResourceMonitorIntegration(t *testing.T) {
// Create temporary test directory // Create temporary test directory
tempDir := t.TempDir() tempDir := t.TempDir()
@@ -47,6 +47,7 @@ func TestResourceMonitor_Integration(t *testing.T) {
err = rm.ValidateFileProcessing(filePath, fileInfo.Size()) err = rm.ValidateFileProcessing(filePath, fileInfo.Size())
if err != nil { if err != nil {
t.Errorf("Failed to validate file %s: %v", filePath, err) t.Errorf("Failed to validate file %s: %v", filePath, err)
continue continue
} }
@@ -54,6 +55,7 @@ func TestResourceMonitor_Integration(t *testing.T) {
err = rm.AcquireReadSlot(ctx) err = rm.AcquireReadSlot(ctx)
if err != nil { if err != nil {
t.Errorf("Failed to acquire read slot for %s: %v", filePath, err) t.Errorf("Failed to acquire read slot for %s: %v", filePath, err)
continue continue
} }
@@ -71,7 +73,7 @@ func TestResourceMonitor_Integration(t *testing.T) {
} }
// Verify final metrics // Verify final metrics
metrics := rm.GetMetrics() metrics := rm.Metrics()
if metrics.FilesProcessed != int64(len(testFiles)) { if metrics.FilesProcessed != int64(len(testFiles)) {
t.Errorf("Expected %d files processed, got %d", len(testFiles), metrics.FilesProcessed) t.Errorf("Expected %d files processed, got %d", len(testFiles), metrics.FilesProcessed)
} }

View File

@@ -1,3 +1,4 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import ( import (
@@ -5,9 +6,7 @@ import (
"sync/atomic" "sync/atomic"
"time" "time"
"github.com/sirupsen/logrus" "github.com/ivuorinen/gibidify/shared"
"github.com/ivuorinen/gibidify/gibidiutils"
) )
// RecordFileProcessed records that a file has been successfully processed. // RecordFileProcessed records that a file has been successfully processed.
@@ -18,8 +17,8 @@ func (rm *ResourceMonitor) RecordFileProcessed(fileSize int64) {
} }
} }
// GetMetrics returns current resource usage metrics. // Metrics returns current resource usage metrics.
func (rm *ResourceMonitor) GetMetrics() ResourceMetrics { func (rm *ResourceMonitor) Metrics() ResourceMetrics {
if !rm.enableResourceMon { if !rm.enableResourceMon {
return ResourceMetrics{} return ResourceMetrics{}
} }
@@ -54,10 +53,11 @@ func (rm *ResourceMonitor) GetMetrics() ResourceMetrics {
FilesProcessed: filesProcessed, FilesProcessed: filesProcessed,
TotalSizeProcessed: totalSize, TotalSizeProcessed: totalSize,
ConcurrentReads: atomic.LoadInt64(&rm.concurrentReads), ConcurrentReads: atomic.LoadInt64(&rm.concurrentReads),
MaxConcurrentReads: int64(rm.maxConcurrentReads),
ProcessingDuration: duration, ProcessingDuration: duration,
AverageFileSize: avgFileSize, AverageFileSize: avgFileSize,
ProcessingRate: processingRate, ProcessingRate: processingRate,
MemoryUsageMB: gibidiutils.SafeUint64ToInt64WithDefault(m.Alloc, 0) / 1024 / 1024, MemoryUsageMB: shared.BytesToMB(m.Alloc),
MaxMemoryUsageMB: int64(rm.hardMemoryLimitMB), MaxMemoryUsageMB: int64(rm.hardMemoryLimitMB),
ViolationsDetected: violations, ViolationsDetected: violations,
DegradationActive: rm.degradationActive, DegradationActive: rm.degradationActive,
@@ -68,19 +68,16 @@ func (rm *ResourceMonitor) GetMetrics() ResourceMetrics {
// LogResourceInfo logs current resource limit configuration. // LogResourceInfo logs current resource limit configuration.
func (rm *ResourceMonitor) LogResourceInfo() { func (rm *ResourceMonitor) LogResourceInfo() {
logger := shared.GetLogger()
if rm.enabled { if rm.enabled {
logrus.Infof( logger.Infof("Resource limits enabled: maxFiles=%d, maxTotalSize=%dMB, fileTimeout=%ds, overallTimeout=%ds",
"Resource limits enabled: maxFiles=%d, maxTotalSize=%dMB, fileTimeout=%ds, overallTimeout=%ds", rm.maxFiles, rm.maxTotalSize/int64(shared.BytesPerMB), int(rm.fileProcessingTimeout.Seconds()),
rm.maxFiles, int(rm.overallTimeout.Seconds()))
rm.maxTotalSize/1024/1024, logger.Infof("Resource limits: maxConcurrentReads=%d, rateLimitFPS=%d, hardMemoryMB=%d",
int(rm.fileProcessingTimeout.Seconds()),
int(rm.overallTimeout.Seconds()),
)
logrus.Infof("Resource limits: maxConcurrentReads=%d, rateLimitFPS=%d, hardMemoryMB=%d",
rm.maxConcurrentReads, rm.rateLimitFilesPerSec, rm.hardMemoryLimitMB) rm.maxConcurrentReads, rm.rateLimitFilesPerSec, rm.hardMemoryLimitMB)
logrus.Infof("Resource features: gracefulDegradation=%v, monitoring=%v", logger.Infof("Resource features: gracefulDegradation=%v, monitoring=%v",
rm.enableGracefulDegr, rm.enableResourceMon) rm.enableGracefulDegr, rm.enableResourceMon)
} else { } else {
logrus.Info("Resource limits disabled") logger.Info("Resource limits disabled")
} }
} }

View File

@@ -9,7 +9,7 @@ import (
"github.com/ivuorinen/gibidify/testutil" "github.com/ivuorinen/gibidify/testutil"
) )
func TestResourceMonitor_Metrics(t *testing.T) { func TestResourceMonitorMetrics(t *testing.T) {
testutil.ResetViperConfig(t, "") testutil.ResetViperConfig(t, "")
viper.Set("resourceLimits.enabled", true) viper.Set("resourceLimits.enabled", true)
@@ -23,7 +23,7 @@ func TestResourceMonitor_Metrics(t *testing.T) {
rm.RecordFileProcessed(2000) rm.RecordFileProcessed(2000)
rm.RecordFileProcessed(500) rm.RecordFileProcessed(500)
metrics := rm.GetMetrics() metrics := rm.Metrics()
// Verify metrics // Verify metrics
if metrics.FilesProcessed != 3 { if metrics.FilesProcessed != 3 {

View File

@@ -1,10 +1,12 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import ( import (
"context" "context"
"fmt"
"time" "time"
"github.com/sirupsen/logrus" "github.com/ivuorinen/gibidify/shared"
) )
// WaitForRateLimit waits for rate limiting if enabled. // WaitForRateLimit waits for rate limiting if enabled.
@@ -15,22 +17,29 @@ func (rm *ResourceMonitor) WaitForRateLimit(ctx context.Context) error {
select { select {
case <-ctx.Done(): case <-ctx.Done():
return ctx.Err() return fmt.Errorf("context canceled while waiting for rate limit: %w", ctx.Err())
case <-rm.rateLimitChan: case <-rm.rateLimitChan:
return nil return nil
case <-time.After(time.Second): // Fallback timeout case <-time.After(time.Second): // Fallback timeout
logrus.Warn("Rate limiting timeout exceeded, continuing without rate limit") logger := shared.GetLogger()
logger.Warn("Rate limiting timeout exceeded, continuing without rate limit")
return nil return nil
} }
} }
// rateLimiterRefill refills the rate limiting channel periodically. // rateLimiterRefill refills the rate limiting channel periodically.
func (rm *ResourceMonitor) rateLimiterRefill() { func (rm *ResourceMonitor) rateLimiterRefill() {
for range rm.rateLimiter.C { for {
select { select {
case rm.rateLimitChan <- struct{}{}: case <-rm.done:
default: return
// Channel is full, skip case <-rm.rateLimiter.C:
select {
case rm.rateLimitChan <- struct{}{}:
default:
// Channel is full, skip
}
} }
} }
} }

View File

@@ -10,7 +10,7 @@ import (
"github.com/ivuorinen/gibidify/testutil" "github.com/ivuorinen/gibidify/testutil"
) )
func TestResourceMonitor_RateLimiting(t *testing.T) { func TestResourceMonitorRateLimiting(t *testing.T) {
testutil.ResetViperConfig(t, "") testutil.ResetViperConfig(t, "")
// Enable rate limiting with a low rate for testing // Enable rate limiting with a low rate for testing

View File

@@ -1,9 +1,11 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
// IsEmergencyStopActive returns whether emergency stop is active. // IsEmergencyStopActive returns whether emergency stop is active.
func (rm *ResourceMonitor) IsEmergencyStopActive() bool { func (rm *ResourceMonitor) IsEmergencyStopActive() bool {
rm.mu.RLock() rm.mu.RLock()
defer rm.mu.RUnlock() defer rm.mu.RUnlock()
return rm.emergencyStopRequested return rm.emergencyStopRequested
} }
@@ -11,11 +13,27 @@ func (rm *ResourceMonitor) IsEmergencyStopActive() bool {
func (rm *ResourceMonitor) IsDegradationActive() bool { func (rm *ResourceMonitor) IsDegradationActive() bool {
rm.mu.RLock() rm.mu.RLock()
defer rm.mu.RUnlock() defer rm.mu.RUnlock()
return rm.degradationActive return rm.degradationActive
} }
// Close cleans up the resource monitor. // Close cleans up the resource monitor.
func (rm *ResourceMonitor) Close() { func (rm *ResourceMonitor) Close() {
rm.mu.Lock()
defer rm.mu.Unlock()
// Prevent multiple closes
if rm.closed {
return
}
rm.closed = true
// Signal goroutines to stop
if rm.done != nil {
close(rm.done)
}
// Stop the ticker
if rm.rateLimiter != nil { if rm.rateLimiter != nil {
rm.rateLimiter.Stop() rm.rateLimiter.Stop()
} }

View File

@@ -1,3 +1,4 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import ( import (
@@ -5,6 +6,7 @@ import (
"time" "time"
"github.com/ivuorinen/gibidify/config" "github.com/ivuorinen/gibidify/config"
"github.com/ivuorinen/gibidify/shared"
) )
// ResourceMonitor monitors resource usage and enforces limits to prevent DoS attacks. // ResourceMonitor monitors resource usage and enforces limits to prevent DoS attacks.
@@ -31,12 +33,14 @@ type ResourceMonitor struct {
// Rate limiting // Rate limiting
rateLimiter *time.Ticker rateLimiter *time.Ticker
rateLimitChan chan struct{} rateLimitChan chan struct{}
done chan struct{} // Signal to stop goroutines
// Synchronization // Synchronization
mu sync.RWMutex mu sync.RWMutex
violationLogged map[string]bool violationLogged map[string]bool
degradationActive bool degradationActive bool
emergencyStopRequested bool emergencyStopRequested bool
closed bool
} }
// ResourceMetrics holds comprehensive resource usage metrics. // ResourceMetrics holds comprehensive resource usage metrics.
@@ -44,6 +48,7 @@ type ResourceMetrics struct {
FilesProcessed int64 `json:"files_processed"` FilesProcessed int64 `json:"files_processed"`
TotalSizeProcessed int64 `json:"total_size_processed"` TotalSizeProcessed int64 `json:"total_size_processed"`
ConcurrentReads int64 `json:"concurrent_reads"` ConcurrentReads int64 `json:"concurrent_reads"`
MaxConcurrentReads int64 `json:"max_concurrent_reads"`
ProcessingDuration time.Duration `json:"processing_duration"` ProcessingDuration time.Duration `json:"processing_duration"`
AverageFileSize float64 `json:"average_file_size"` AverageFileSize float64 `json:"average_file_size"`
ProcessingRate float64 `json:"processing_rate_files_per_sec"` ProcessingRate float64 `json:"processing_rate_files_per_sec"`
@@ -57,31 +62,32 @@ type ResourceMetrics struct {
// ResourceViolation represents a detected resource limit violation. // ResourceViolation represents a detected resource limit violation.
type ResourceViolation struct { type ResourceViolation struct {
Type string `json:"type"` Type string `json:"type"`
Message string `json:"message"` Message string `json:"message"`
Current interface{} `json:"current"` Current any `json:"current"`
Limit interface{} `json:"limit"` Limit any `json:"limit"`
Timestamp time.Time `json:"timestamp"` Timestamp time.Time `json:"timestamp"`
Context map[string]interface{} `json:"context"` Context map[string]any `json:"context"`
} }
// NewResourceMonitor creates a new resource monitor with configuration. // NewResourceMonitor creates a new resource monitor with configuration.
func NewResourceMonitor() *ResourceMonitor { func NewResourceMonitor() *ResourceMonitor {
rm := &ResourceMonitor{ rm := &ResourceMonitor{
enabled: config.GetResourceLimitsEnabled(), enabled: config.ResourceLimitsEnabled(),
maxFiles: config.GetMaxFiles(), maxFiles: config.MaxFiles(),
maxTotalSize: config.GetMaxTotalSize(), maxTotalSize: config.MaxTotalSize(),
fileProcessingTimeout: time.Duration(config.GetFileProcessingTimeoutSec()) * time.Second, fileProcessingTimeout: time.Duration(config.FileProcessingTimeoutSec()) * time.Second,
overallTimeout: time.Duration(config.GetOverallTimeoutSec()) * time.Second, overallTimeout: time.Duration(config.OverallTimeoutSec()) * time.Second,
maxConcurrentReads: config.GetMaxConcurrentReads(), maxConcurrentReads: config.MaxConcurrentReads(),
rateLimitFilesPerSec: config.GetRateLimitFilesPerSec(), rateLimitFilesPerSec: config.RateLimitFilesPerSec(),
hardMemoryLimitMB: config.GetHardMemoryLimitMB(), hardMemoryLimitMB: config.HardMemoryLimitMB(),
enableGracefulDegr: config.GetEnableGracefulDegradation(), enableGracefulDegr: config.EnableGracefulDegradation(),
enableResourceMon: config.GetEnableResourceMonitoring(), enableResourceMon: config.EnableResourceMonitoring(),
startTime: time.Now(), startTime: time.Now(),
lastRateLimitCheck: time.Now(), lastRateLimitCheck: time.Now(),
violationLogged: make(map[string]bool), violationLogged: make(map[string]bool),
hardMemoryLimitBytes: int64(config.GetHardMemoryLimitMB()) * 1024 * 1024, hardMemoryLimitBytes: int64(config.HardMemoryLimitMB()) * int64(shared.BytesPerMB),
done: make(chan struct{}),
} }
// Initialize rate limiter if rate limiting is enabled // Initialize rate limiter if rate limiting is enabled

View File

@@ -7,11 +7,11 @@ import (
"github.com/spf13/viper" "github.com/spf13/viper"
"github.com/ivuorinen/gibidify/config" "github.com/ivuorinen/gibidify/shared"
"github.com/ivuorinen/gibidify/testutil" "github.com/ivuorinen/gibidify/testutil"
) )
func TestResourceMonitor_NewResourceMonitor(t *testing.T) { func TestResourceMonitorNewResourceMonitor(t *testing.T) {
// Reset viper for clean test state // Reset viper for clean test state
testutil.ResetViperConfig(t, "") testutil.ResetViperConfig(t, "")
@@ -25,24 +25,24 @@ func TestResourceMonitor_NewResourceMonitor(t *testing.T) {
t.Error("Expected resource monitor to be enabled by default") t.Error("Expected resource monitor to be enabled by default")
} }
if rm.maxFiles != config.DefaultMaxFiles { if rm.maxFiles != shared.ConfigMaxFilesDefault {
t.Errorf("Expected maxFiles to be %d, got %d", config.DefaultMaxFiles, rm.maxFiles) t.Errorf("Expected maxFiles to be %d, got %d", shared.ConfigMaxFilesDefault, rm.maxFiles)
} }
if rm.maxTotalSize != config.DefaultMaxTotalSize { if rm.maxTotalSize != shared.ConfigMaxTotalSizeDefault {
t.Errorf("Expected maxTotalSize to be %d, got %d", config.DefaultMaxTotalSize, rm.maxTotalSize) t.Errorf("Expected maxTotalSize to be %d, got %d", shared.ConfigMaxTotalSizeDefault, rm.maxTotalSize)
} }
if rm.fileProcessingTimeout != time.Duration(config.DefaultFileProcessingTimeoutSec)*time.Second { if rm.fileProcessingTimeout != time.Duration(shared.ConfigFileProcessingTimeoutSecDefault)*time.Second {
t.Errorf("Expected fileProcessingTimeout to be %v, got %v", t.Errorf("Expected fileProcessingTimeout to be %v, got %v",
time.Duration(config.DefaultFileProcessingTimeoutSec)*time.Second, rm.fileProcessingTimeout) time.Duration(shared.ConfigFileProcessingTimeoutSecDefault)*time.Second, rm.fileProcessingTimeout)
} }
// Clean up // Clean up
rm.Close() rm.Close()
} }
func TestResourceMonitor_DisabledResourceLimits(t *testing.T) { func TestResourceMonitorDisabledResourceLimits(t *testing.T) {
// Reset viper for clean test state // Reset viper for clean test state
testutil.ResetViperConfig(t, "") testutil.ResetViperConfig(t, "")
@@ -72,3 +72,77 @@ func TestResourceMonitor_DisabledResourceLimits(t *testing.T) {
t.Errorf("Expected no error when rate limiting disabled, got %v", err) t.Errorf("Expected no error when rate limiting disabled, got %v", err)
} }
} }
// TestResourceMonitorStateQueries tests state query functions.
func TestResourceMonitorStateQueries(t *testing.T) {
testutil.ResetViperConfig(t, "")
rm := NewResourceMonitor()
defer rm.Close()
// Test IsEmergencyStopActive - should be false initially
if rm.IsEmergencyStopActive() {
t.Error("Expected emergency stop to be inactive initially")
}
// Test IsDegradationActive - should be false initially
if rm.IsDegradationActive() {
t.Error("Expected degradation mode to be inactive initially")
}
}
// TestResourceMonitorIsEmergencyStopActive tests the IsEmergencyStopActive method.
func TestResourceMonitorIsEmergencyStopActive(t *testing.T) {
testutil.ResetViperConfig(t, "")
rm := NewResourceMonitor()
defer rm.Close()
// Test initial state
active := rm.IsEmergencyStopActive()
if active {
t.Error("Expected emergency stop to be inactive initially")
}
// The method should return a consistent value on multiple calls
for i := 0; i < 5; i++ {
if rm.IsEmergencyStopActive() != active {
t.Error("IsEmergencyStopActive should return consistent values")
}
}
}
// TestResourceMonitorIsDegradationActive tests the IsDegradationActive method.
func TestResourceMonitorIsDegradationActive(t *testing.T) {
testutil.ResetViperConfig(t, "")
rm := NewResourceMonitor()
defer rm.Close()
// Test initial state
active := rm.IsDegradationActive()
if active {
t.Error("Expected degradation mode to be inactive initially")
}
// The method should return a consistent value on multiple calls
for i := 0; i < 5; i++ {
if rm.IsDegradationActive() != active {
t.Error("IsDegradationActive should return consistent values")
}
}
}
// TestResourceMonitorClose tests the Close method.
func TestResourceMonitorClose(t *testing.T) {
testutil.ResetViperConfig(t, "")
rm := NewResourceMonitor()
// Close should not panic
rm.Close()
// Multiple closes should be safe
rm.Close()
rm.Close()
}

View File

@@ -1,3 +1,4 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import ( import (
@@ -5,9 +6,7 @@ import (
"sync/atomic" "sync/atomic"
"time" "time"
"github.com/sirupsen/logrus" "github.com/ivuorinen/gibidify/shared"
"github.com/ivuorinen/gibidify/gibidiutils"
) )
// ValidateFileProcessing checks if a file can be processed based on resource limits. // ValidateFileProcessing checks if a file can be processed based on resource limits.
@@ -21,12 +20,12 @@ func (rm *ResourceMonitor) ValidateFileProcessing(filePath string, fileSize int6
// Check if emergency stop is active // Check if emergency stop is active
if rm.emergencyStopRequested { if rm.emergencyStopRequested {
return gibidiutils.NewStructuredError( return shared.NewStructuredError(
gibidiutils.ErrorTypeValidation, shared.ErrorTypeValidation,
gibidiutils.CodeResourceLimitMemory, shared.CodeResourceLimitMemory,
"processing stopped due to emergency memory condition", "processing stopped due to emergency memory condition",
filePath, filePath,
map[string]interface{}{ map[string]any{
"emergency_stop_active": true, "emergency_stop_active": true,
}, },
) )
@@ -35,12 +34,12 @@ func (rm *ResourceMonitor) ValidateFileProcessing(filePath string, fileSize int6
// Check file count limit // Check file count limit
currentFiles := atomic.LoadInt64(&rm.filesProcessed) currentFiles := atomic.LoadInt64(&rm.filesProcessed)
if int(currentFiles) >= rm.maxFiles { if int(currentFiles) >= rm.maxFiles {
return gibidiutils.NewStructuredError( return shared.NewStructuredError(
gibidiutils.ErrorTypeValidation, shared.ErrorTypeValidation,
gibidiutils.CodeResourceLimitFiles, shared.CodeResourceLimitFiles,
"maximum file count limit exceeded", "maximum file count limit exceeded",
filePath, filePath,
map[string]interface{}{ map[string]any{
"current_files": currentFiles, "current_files": currentFiles,
"max_files": rm.maxFiles, "max_files": rm.maxFiles,
}, },
@@ -50,12 +49,12 @@ func (rm *ResourceMonitor) ValidateFileProcessing(filePath string, fileSize int6
// Check total size limit // Check total size limit
currentTotalSize := atomic.LoadInt64(&rm.totalSizeProcessed) currentTotalSize := atomic.LoadInt64(&rm.totalSizeProcessed)
if currentTotalSize+fileSize > rm.maxTotalSize { if currentTotalSize+fileSize > rm.maxTotalSize {
return gibidiutils.NewStructuredError( return shared.NewStructuredError(
gibidiutils.ErrorTypeValidation, shared.ErrorTypeValidation,
gibidiutils.CodeResourceLimitTotalSize, shared.CodeResourceLimitTotalSize,
"maximum total size limit would be exceeded", "maximum total size limit would be exceeded",
filePath, filePath,
map[string]interface{}{ map[string]any{
"current_total_size": currentTotalSize, "current_total_size": currentTotalSize,
"file_size": fileSize, "file_size": fileSize,
"max_total_size": rm.maxTotalSize, "max_total_size": rm.maxTotalSize,
@@ -65,12 +64,12 @@ func (rm *ResourceMonitor) ValidateFileProcessing(filePath string, fileSize int6
// Check overall timeout // Check overall timeout
if time.Since(rm.startTime) > rm.overallTimeout { if time.Since(rm.startTime) > rm.overallTimeout {
return gibidiutils.NewStructuredError( return shared.NewStructuredError(
gibidiutils.ErrorTypeValidation, shared.ErrorTypeValidation,
gibidiutils.CodeResourceLimitTimeout, shared.CodeResourceLimitTimeout,
"overall processing timeout exceeded", "overall processing timeout exceeded",
filePath, filePath,
map[string]interface{}{ map[string]any{
"processing_duration": time.Since(rm.startTime), "processing_duration": time.Since(rm.startTime),
"overall_timeout": rm.overallTimeout, "overall_timeout": rm.overallTimeout,
}, },
@@ -88,60 +87,93 @@ func (rm *ResourceMonitor) CheckHardMemoryLimit() error {
var m runtime.MemStats var m runtime.MemStats
runtime.ReadMemStats(&m) runtime.ReadMemStats(&m)
currentMemory := gibidiutils.SafeUint64ToInt64WithDefault(m.Alloc, 0) currentMemory := shared.SafeUint64ToInt64WithDefault(m.Alloc, 0)
if currentMemory > rm.hardMemoryLimitBytes { if currentMemory <= rm.hardMemoryLimitBytes {
rm.mu.Lock() return nil
defer rm.mu.Unlock()
// Log violation if not already logged
violationKey := "hard_memory_limit"
if !rm.violationLogged[violationKey] {
logrus.Errorf("Hard memory limit exceeded: %dMB > %dMB",
currentMemory/1024/1024, rm.hardMemoryLimitMB)
rm.violationLogged[violationKey] = true
}
if rm.enableGracefulDegr {
// Force garbage collection
runtime.GC()
// Check again after GC
runtime.ReadMemStats(&m)
currentMemory = gibidiutils.SafeUint64ToInt64WithDefault(m.Alloc, 0)
if currentMemory > rm.hardMemoryLimitBytes {
// Still over limit, activate emergency stop
rm.emergencyStopRequested = true
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeResourceLimitMemory,
"hard memory limit exceeded, emergency stop activated",
"",
map[string]interface{}{
"current_memory_mb": currentMemory / 1024 / 1024,
"limit_mb": rm.hardMemoryLimitMB,
"emergency_stop": true,
},
)
}
// Memory freed by GC, continue with degradation
rm.degradationActive = true
logrus.Info("Memory freed by garbage collection, continuing with degradation mode")
} else {
// No graceful degradation, hard stop
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeResourceLimitMemory,
"hard memory limit exceeded",
"",
map[string]interface{}{
"current_memory_mb": currentMemory / 1024 / 1024,
"limit_mb": rm.hardMemoryLimitMB,
},
)
}
} }
return rm.handleMemoryLimitExceeded(currentMemory)
}
// handleMemoryLimitExceeded handles the case when hard memory limit is exceeded.
func (rm *ResourceMonitor) handleMemoryLimitExceeded(currentMemory int64) error {
rm.mu.Lock()
defer rm.mu.Unlock()
rm.logMemoryViolation(currentMemory)
if !rm.enableGracefulDegr {
return rm.createHardMemoryLimitError(currentMemory, false)
}
return rm.tryGracefulRecovery(currentMemory)
}
// logMemoryViolation logs memory limit violation if not already logged.
func (rm *ResourceMonitor) logMemoryViolation(currentMemory int64) {
violationKey := "hard_memory_limit"
// Ensure map is initialized
if rm.violationLogged == nil {
rm.violationLogged = make(map[string]bool)
}
if rm.violationLogged[violationKey] {
return
}
logger := shared.GetLogger()
logger.Errorf("Hard memory limit exceeded: %dMB > %dMB",
currentMemory/int64(shared.BytesPerMB), rm.hardMemoryLimitMB)
rm.violationLogged[violationKey] = true
}
// tryGracefulRecovery attempts graceful recovery by forcing GC.
func (rm *ResourceMonitor) tryGracefulRecovery(_ int64) error {
// Force garbage collection
runtime.GC()
// Check again after GC
var m runtime.MemStats
runtime.ReadMemStats(&m)
newMemory := shared.SafeUint64ToInt64WithDefault(m.Alloc, 0)
if newMemory > rm.hardMemoryLimitBytes {
// Still over limit, activate emergency stop
rm.emergencyStopRequested = true
return rm.createHardMemoryLimitError(newMemory, true)
}
// Memory freed by GC, continue with degradation
rm.degradationActive = true
logger := shared.GetLogger()
logger.Info("Memory freed by garbage collection, continuing with degradation mode")
return nil return nil
} }
// createHardMemoryLimitError creates a structured error for memory limit exceeded.
func (rm *ResourceMonitor) createHardMemoryLimitError(currentMemory int64, emergencyStop bool) error {
message := "hard memory limit exceeded"
if emergencyStop {
message = "hard memory limit exceeded, emergency stop activated"
}
context := map[string]any{
"current_memory_mb": currentMemory / int64(shared.BytesPerMB),
"limit_mb": rm.hardMemoryLimitMB,
}
if emergencyStop {
context["emergency_stop"] = true
}
return shared.NewStructuredError(
shared.ErrorTypeValidation,
shared.CodeResourceLimitMemory,
message,
"",
context,
)
}

View File

@@ -2,19 +2,46 @@ package fileproc
import ( import (
"errors" "errors"
"strings"
"testing" "testing"
"github.com/spf13/viper" "github.com/spf13/viper"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
"github.com/ivuorinen/gibidify/testutil" "github.com/ivuorinen/gibidify/testutil"
) )
func TestResourceMonitor_FileCountLimit(t *testing.T) { // assertStructuredError verifies that an error is a StructuredError with the expected code.
func assertStructuredError(t *testing.T, err error, expectedCode string) {
t.Helper()
structErr := &shared.StructuredError{}
ok := errors.As(err, &structErr)
if !ok {
t.Errorf("Expected StructuredError, got %T", err)
} else if structErr.Code != expectedCode {
t.Errorf("Expected error code %s, got %s", expectedCode, structErr.Code)
}
}
// validateMemoryLimitError validates that an error is a proper memory limit StructuredError.
func validateMemoryLimitError(t *testing.T, err error) {
t.Helper()
structErr := &shared.StructuredError{}
if errors.As(err, &structErr) {
if structErr.Code != shared.CodeResourceLimitMemory {
t.Errorf("Expected memory limit error code, got %s", structErr.Code)
}
} else {
t.Errorf("Expected StructuredError, got %T", err)
}
}
func TestResourceMonitorFileCountLimit(t *testing.T) {
testutil.ResetViperConfig(t, "") testutil.ResetViperConfig(t, "")
// Set a very low file count limit for testing // Set a very low file count limit for testing
viper.Set("resourceLimits.enabled", true) viper.Set(shared.TestCfgResourceLimitsEnabled, true)
viper.Set("resourceLimits.maxFiles", 2) viper.Set("resourceLimits.maxFiles", 2)
rm := NewResourceMonitor() rm := NewResourceMonitor()
@@ -41,20 +68,14 @@ func TestResourceMonitor_FileCountLimit(t *testing.T) {
} }
// Verify it's the correct error type // Verify it's the correct error type
var structErr *gibidiutils.StructuredError assertStructuredError(t, err, shared.CodeResourceLimitFiles)
ok := errors.As(err, &structErr)
if !ok {
t.Errorf("Expected StructuredError, got %T", err)
} else if structErr.Code != gibidiutils.CodeResourceLimitFiles {
t.Errorf("Expected error code %s, got %s", gibidiutils.CodeResourceLimitFiles, structErr.Code)
}
} }
func TestResourceMonitor_TotalSizeLimit(t *testing.T) { func TestResourceMonitorTotalSizeLimit(t *testing.T) {
testutil.ResetViperConfig(t, "") testutil.ResetViperConfig(t, "")
// Set a low total size limit for testing (1KB) // Set a low total size limit for testing (1KB)
viper.Set("resourceLimits.enabled", true) viper.Set(shared.TestCfgResourceLimitsEnabled, true)
viper.Set("resourceLimits.maxTotalSize", 1024) viper.Set("resourceLimits.maxTotalSize", 1024)
rm := NewResourceMonitor() rm := NewResourceMonitor()
@@ -81,11 +102,103 @@ func TestResourceMonitor_TotalSizeLimit(t *testing.T) {
} }
// Verify it's the correct error type // Verify it's the correct error type
var structErr *gibidiutils.StructuredError assertStructuredError(t, err, shared.CodeResourceLimitTotalSize)
ok := errors.As(err, &structErr) }
if !ok {
t.Errorf("Expected StructuredError, got %T", err) // TestResourceMonitor_MemoryLimitExceeded tests memory limit violation scenarios.
} else if structErr.Code != gibidiutils.CodeResourceLimitTotalSize { func TestResourceMonitorMemoryLimitExceeded(t *testing.T) {
t.Errorf("Expected error code %s, got %s", gibidiutils.CodeResourceLimitTotalSize, structErr.Code) testutil.ResetViperConfig(t, "")
// Set very low memory limit to try to force violations
viper.Set(shared.TestCfgResourceLimitsEnabled, true)
viper.Set("resourceLimits.hardMemoryLimitMB", 0.001) // 1KB - extremely low
rm := NewResourceMonitor()
defer rm.Close()
// Allocate large buffer to increase memory usage before check
largeBuffer := make([]byte, 10*1024*1024) // 10MB allocation
_ = largeBuffer[0] // Use the buffer to prevent optimization
// Check hard memory limit - might trigger if actual memory is high enough
err := rm.CheckHardMemoryLimit()
// Note: This test might not always fail since it depends on actual runtime memory
// But if it does fail, verify it's the correct error type
if err != nil {
validateMemoryLimitError(t, err)
t.Log("Successfully triggered memory limit violation")
} else {
t.Log("Memory limit check passed - actual memory usage may be within limits")
}
}
// TestResourceMonitor_MemoryLimitHandling tests the memory violation detection.
func TestResourceMonitorMemoryLimitHandling(t *testing.T) {
testutil.ResetViperConfig(t, "")
// Enable resource limits with very small hard limit
viper.Set(shared.TestCfgResourceLimitsEnabled, true)
viper.Set("resourceLimits.hardMemoryLimitMB", 0.0001) // Very tiny limit
viper.Set("resourceLimits.enableGracefulDegradation", true)
rm := NewResourceMonitor()
defer rm.Close()
// Allocate more memory to increase chances of triggering limit
buffers := make([][]byte, 0, 100) // Pre-allocate capacity
for i := 0; i < 100; i++ {
buffer := make([]byte, 1024*1024) // 1MB each
buffers = append(buffers, buffer)
_ = buffer[0] // Use buffer
_ = buffers // Use the slice to prevent unused variable warning
// Check periodically
if i%10 == 0 {
err := rm.CheckHardMemoryLimit()
if err != nil {
// Successfully triggered memory limit
if !strings.Contains(err.Error(), "memory limit") {
t.Errorf("Expected error message to mention memory limit, got: %v", err)
}
t.Log("Successfully triggered memory limit handling")
return
}
}
}
t.Log("Could not trigger memory limit - actual memory usage may be lower than limit")
}
// TestResourceMonitorGracefulRecovery tests graceful recovery attempts.
func TestResourceMonitorGracefulRecovery(t *testing.T) {
testutil.ResetViperConfig(t, "")
// Set memory limits that will trigger recovery
viper.Set(shared.TestCfgResourceLimitsEnabled, true)
rm := NewResourceMonitor()
defer rm.Close()
// Force a deterministic 1-byte hard memory limit to trigger recovery
rm.hardMemoryLimitBytes = 1
// Process multiple files to accumulate memory usage
for i := 0; i < 3; i++ {
filePath := "/tmp/test" + string(rune('1'+i)) + ".txt"
fileSize := int64(400) // Each file is 400 bytes
// First few might pass, but eventually should trigger recovery mechanisms
err := rm.ValidateFileProcessing(filePath, fileSize)
if err != nil {
// Once we hit the limit, test that the error is appropriate
if !strings.Contains(err.Error(), "resource") && !strings.Contains(err.Error(), "limit") {
t.Errorf("Expected resource limit error, got: %v", err)
}
break
}
rm.RecordFileProcessed(fileSize)
} }
} }

View File

@@ -5,7 +5,7 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// Walker defines an interface for scanning directories. // Walker defines an interface for scanning directories.
@@ -30,13 +30,16 @@ func NewProdWalker() *ProdWalker {
// Walk scans the given root directory recursively and returns a slice of file paths // Walk scans the given root directory recursively and returns a slice of file paths
// that are not ignored based on .gitignore/.ignore files, the configuration, or the default binary/image filter. // that are not ignored based on .gitignore/.ignore files, the configuration, or the default binary/image filter.
func (w *ProdWalker) Walk(root string) ([]string, error) { func (w *ProdWalker) Walk(root string) ([]string, error) {
absRoot, err := gibidiutils.GetAbsolutePath(root) absRoot, err := shared.AbsolutePath(root)
if err != nil { if err != nil {
return nil, gibidiutils.WrapError( return nil, shared.WrapError(
err, gibidiutils.ErrorTypeFileSystem, gibidiutils.CodeFSPathResolution, err,
shared.ErrorTypeFileSystem,
shared.CodeFSPathResolution,
"failed to resolve root path", "failed to resolve root path",
).WithFilePath(root) ).WithFilePath(root)
} }
return w.walkDir(absRoot, []ignoreRule{}) return w.walkDir(absRoot, []ignoreRule{})
} }
@@ -50,8 +53,10 @@ func (w *ProdWalker) walkDir(currentDir string, parentRules []ignoreRule) ([]str
entries, err := os.ReadDir(currentDir) entries, err := os.ReadDir(currentDir)
if err != nil { if err != nil {
return nil, gibidiutils.WrapError( return nil, shared.WrapError(
err, gibidiutils.ErrorTypeFileSystem, gibidiutils.CodeFSAccess, err,
shared.ErrorTypeFileSystem,
shared.CodeFSAccess,
"failed to read directory", "failed to read directory",
).WithFilePath(currentDir) ).WithFilePath(currentDir)
} }
@@ -69,8 +74,10 @@ func (w *ProdWalker) walkDir(currentDir string, parentRules []ignoreRule) ([]str
if entry.IsDir() { if entry.IsDir() {
subFiles, err := w.walkDir(fullPath, rules) subFiles, err := w.walkDir(fullPath, rules)
if err != nil { if err != nil {
return nil, gibidiutils.WrapError( return nil, shared.WrapError(
err, gibidiutils.ErrorTypeProcessing, gibidiutils.CodeProcessingTraversal, err,
shared.ErrorTypeProcessing,
shared.CodeProcessingTraversal,
"failed to traverse subdirectory", "failed to traverse subdirectory",
).WithFilePath(fullPath) ).WithFilePath(fullPath)
} }

View File

@@ -61,8 +61,6 @@ func TestProdWalkerBinaryCheck(t *testing.T) {
// Reset FileTypeRegistry to ensure clean state // Reset FileTypeRegistry to ensure clean state
fileproc.ResetRegistryForTesting() fileproc.ResetRegistryForTesting()
// Ensure cleanup runs even if test fails
t.Cleanup(fileproc.ResetRegistryForTesting)
// Run walker // Run walker
w := fileproc.NewProdWalker() w := fileproc.NewProdWalker()

View File

@@ -2,103 +2,66 @@
package fileproc package fileproc
import ( import (
"fmt"
"os" "os"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// WriterConfig holds configuration for the writer. // startFormatWriter handles generic writer orchestration for any format.
type WriterConfig struct { // This eliminates code duplication across format-specific writer functions.
Format string // Uses the FormatWriter interface defined in formats.go.
Prefix string func startFormatWriter(
Suffix string outFile *os.File,
} writeCh <-chan WriteRequest,
done chan<- struct{},
prefix, suffix string,
writerFactory func(*os.File) FormatWriter,
) {
defer close(done)
// Validate checks if the WriterConfig is valid. writer := writerFactory(outFile)
func (c WriterConfig) Validate() error {
if c.Format == "" { // Start writing
return gibidiutils.NewStructuredError( if err := writer.Start(prefix, suffix); err != nil {
gibidiutils.ErrorTypeValidation, shared.LogError("Failed to start writer", err)
gibidiutils.CodeValidationFormat,
"format cannot be empty", return
"",
nil,
)
} }
switch c.Format { // Process files
case "markdown", "json", "yaml": for req := range writeCh {
return nil if err := writer.WriteFile(req); err != nil {
default: shared.LogError("Failed to write file", err)
context := map[string]any{
"format": c.Format,
} }
return gibidiutils.NewStructuredError( }
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationFormat, // Close writer
fmt.Sprintf("unsupported format: %s", c.Format), if err := writer.Close(); err != nil {
"", shared.LogError("Failed to close writer", err)
context,
)
} }
} }
// StartWriter writes the output in the specified format with memory optimization. // StartWriter writes the output in the specified format with memory optimization.
func StartWriter(outFile *os.File, writeCh <-chan WriteRequest, done chan<- struct{}, config WriterConfig) { func StartWriter(outFile *os.File, writeCh <-chan WriteRequest, done chan<- struct{}, format, prefix, suffix string) {
// Validate config switch format {
if err := config.Validate(); err != nil { case shared.FormatMarkdown:
gibidiutils.LogError("Invalid writer configuration", err) startMarkdownWriter(outFile, writeCh, done, prefix, suffix)
close(done) case shared.FormatJSON:
return startJSONWriter(outFile, writeCh, done, prefix, suffix)
} case shared.FormatYAML:
startYAMLWriter(outFile, writeCh, done, prefix, suffix)
// Validate outFile is not nil
if outFile == nil {
err := gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeIO,
gibidiutils.CodeIOFileWrite,
"output file is nil",
"",
nil,
)
gibidiutils.LogError("Failed to write output", err)
close(done)
return
}
// Validate outFile is accessible
if _, err := outFile.Stat(); err != nil {
structErr := gibidiutils.WrapError(
err,
gibidiutils.ErrorTypeIO,
gibidiutils.CodeIOFileWrite,
"failed to stat output file",
)
gibidiutils.LogError("Failed to validate output file", structErr)
close(done)
return
}
switch config.Format {
case "markdown":
startMarkdownWriter(outFile, writeCh, done, config.Prefix, config.Suffix)
case "json":
startJSONWriter(outFile, writeCh, done, config.Prefix, config.Suffix)
case "yaml":
startYAMLWriter(outFile, writeCh, done, config.Prefix, config.Suffix)
default: default:
context := map[string]interface{}{ context := map[string]any{
"format": config.Format, "format": format,
} }
err := gibidiutils.NewStructuredError( err := shared.NewStructuredError(
gibidiutils.ErrorTypeValidation, shared.ErrorTypeValidation,
gibidiutils.CodeValidationFormat, shared.CodeValidationFormat,
fmt.Sprintf("unsupported format: %s", config.Format), "unsupported format: "+format,
"", "",
context, context,
) )
gibidiutils.LogError("Failed to encode output", err) shared.LogError("Failed to encode output", err)
close(done) close(done)
} }
} }

View File

@@ -2,17 +2,23 @@ package fileproc_test
import ( import (
"encoding/json" "encoding/json"
"errors"
"fmt"
"io"
"os" "os"
"path/filepath"
"strings" "strings"
"sync" "sync"
"testing" "testing"
"time"
"gopkg.in/yaml.v3" "gopkg.in/yaml.v3"
"github.com/ivuorinen/gibidify/fileproc" "github.com/ivuorinen/gibidify/fileproc"
"github.com/ivuorinen/gibidify/shared"
) )
func TestStartWriter_Formats(t *testing.T) { func TestStartWriterFormats(t *testing.T) {
// Define table-driven test cases // Define table-driven test cases
tests := []struct { tests := []struct {
name string name string
@@ -26,15 +32,17 @@ func TestStartWriter_Formats(t *testing.T) {
} }
for _, tc := range tests { for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) { t.Run(
data := runWriterTest(t, tc.format) tc.name, func(t *testing.T) {
if tc.expectError { data := runWriterTest(t, tc.format)
verifyErrorOutput(t, data) if tc.expectError {
} else { verifyErrorOutput(t, data)
verifyValidOutput(t, data, tc.format) } else {
verifyPrefixSuffix(t, data) verifyValidOutput(t, data, tc.format)
} verifyPrefixSuffix(t, data)
}) }
},
)
} }
} }
@@ -43,7 +51,7 @@ func runWriterTest(t *testing.T, format string) []byte {
t.Helper() t.Helper()
outFile, err := os.CreateTemp(t.TempDir(), "gibidify_test_output") outFile, err := os.CreateTemp(t.TempDir(), "gibidify_test_output")
if err != nil { if err != nil {
t.Fatalf("Failed to create temp file: %v", err) t.Fatalf(shared.TestMsgFailedToCreateFile, err)
} }
defer func() { defer func() {
if closeErr := outFile.Close(); closeErr != nil { if closeErr := outFile.Close(); closeErr != nil {
@@ -59,25 +67,23 @@ func runWriterTest(t *testing.T, format string) []byte {
doneCh := make(chan struct{}) doneCh := make(chan struct{})
// Write a couple of sample requests // Write a couple of sample requests
writeCh <- fileproc.WriteRequest{Path: "sample.go", Content: "package main"} writeCh <- fileproc.WriteRequest{Path: "sample.go", Content: shared.LiteralPackageMain}
writeCh <- fileproc.WriteRequest{Path: "example.py", Content: "def foo(): pass"} writeCh <- fileproc.WriteRequest{Path: "example.py", Content: "def foo(): pass"}
close(writeCh) close(writeCh)
// Start the writer // Start the writer
var wg sync.WaitGroup var wg sync.WaitGroup
wg.Add(1) wg.Go(func() {
go func() { fileproc.StartWriter(outFile, writeCh, doneCh, format, "PREFIX", "SUFFIX")
defer wg.Done() })
fileproc.StartWriter(outFile, writeCh, doneCh, fileproc.WriterConfig{
Format: format,
Prefix: "PREFIX",
Suffix: "SUFFIX",
})
}()
// Wait until writer signals completion // Wait until writer signals completion
wg.Wait() wg.Wait()
<-doneCh // make sure all writes finished select {
case <-doneCh: // make sure all writes finished
case <-time.After(3 * time.Second):
t.Fatal(shared.TestMsgTimeoutWriterCompletion)
}
// Read output // Read output
data, err := os.ReadFile(outFile.Name()) data, err := os.ReadFile(outFile.Name())
@@ -115,6 +121,11 @@ func verifyValidOutput(t *testing.T, data []byte, format string) {
if !strings.Contains(content, "```") { if !strings.Contains(content, "```") {
t.Error("Expected markdown code fences not found") t.Error("Expected markdown code fences not found")
} }
default:
// Unknown format - basic validation that we have content
if len(content) == 0 {
t.Errorf("Unexpected format %s with empty content", format)
}
} }
} }
@@ -129,3 +140,490 @@ func verifyPrefixSuffix(t *testing.T, data []byte) {
t.Errorf("Missing suffix in output: %s", data) t.Errorf("Missing suffix in output: %s", data)
} }
} }
// verifyPrefixSuffixWith checks that output contains expected custom prefix and suffix.
func verifyPrefixSuffixWith(t *testing.T, data []byte, expectedPrefix, expectedSuffix string) {
t.Helper()
content := string(data)
if !strings.Contains(content, expectedPrefix) {
t.Errorf("Missing prefix '%s' in output: %s", expectedPrefix, data)
}
if !strings.Contains(content, expectedSuffix) {
t.Errorf("Missing suffix '%s' in output: %s", expectedSuffix, data)
}
}
// TestStartWriterStreamingFormats tests streaming functionality in all writers.
func TestStartWriterStreamingFormats(t *testing.T) {
tests := []struct {
name string
format string
content string
}{
{"JSON streaming", "json", strings.Repeat("line\n", 1000)},
{"YAML streaming", "yaml", strings.Repeat("data: value\n", 1000)},
{"Markdown streaming", "markdown", strings.Repeat("# Header\nContent\n", 1000)},
}
for _, tc := range tests {
t.Run(
tc.name, func(t *testing.T) {
data := runStreamingWriterTest(t, tc.format, tc.content)
// Verify output is not empty
if len(data) == 0 {
t.Error("Expected streaming output but got empty result")
}
// Format-specific validation
verifyValidOutput(t, data, tc.format)
verifyPrefixSuffixWith(t, data, "STREAM_PREFIX", "STREAM_SUFFIX")
// Verify content was written
content := string(data)
if !strings.Contains(content, shared.TestFileStreamTest) {
t.Error("Expected file path in streaming output")
}
},
)
}
}
// runStreamingWriterTest executes the writer with streaming content.
func runStreamingWriterTest(t *testing.T, format, content string) []byte {
t.Helper()
// Create temp file with content for streaming
contentFile, err := os.CreateTemp(t.TempDir(), "content_*.txt")
if err != nil {
t.Fatalf("Failed to create content file: %v", err)
}
defer func() {
if err := os.Remove(contentFile.Name()); err != nil {
t.Logf("Failed to remove content file: %v", err)
}
}()
if _, err := contentFile.WriteString(content); err != nil {
t.Fatalf("Failed to write content file: %v", err)
}
if err := contentFile.Close(); err != nil {
t.Fatalf("Failed to close content file: %v", err)
}
// Create output file
outFile, err := os.CreateTemp(t.TempDir(), "gibidify_stream_test_output")
if err != nil {
t.Fatalf(shared.TestMsgFailedToCreateFile, err)
}
defer func() {
if closeErr := outFile.Close(); closeErr != nil {
t.Errorf("close temp file: %v", closeErr)
}
if removeErr := os.Remove(outFile.Name()); removeErr != nil {
t.Errorf("remove temp file: %v", removeErr)
}
}()
// Prepare channels with streaming request
writeCh := make(chan fileproc.WriteRequest, 1)
doneCh := make(chan struct{})
// Create reader for streaming
reader, err := os.Open(contentFile.Name())
if err != nil {
t.Fatalf("Failed to open content file for reading: %v", err)
}
defer func() {
if err := reader.Close(); err != nil {
t.Logf("Failed to close reader: %v", err)
}
}()
// Write streaming request
writeCh <- fileproc.WriteRequest{
Path: shared.TestFileStreamTest,
Content: "", // Empty for streaming
IsStream: true,
Reader: reader,
}
close(writeCh)
// Start the writer
var wg sync.WaitGroup
wg.Go(func() {
fileproc.StartWriter(outFile, writeCh, doneCh, format, "STREAM_PREFIX", "STREAM_SUFFIX")
})
// Wait until writer signals completion
wg.Wait()
select {
case <-doneCh:
case <-time.After(3 * time.Second):
t.Fatal(shared.TestMsgTimeoutWriterCompletion)
}
// Read output
data, err := os.ReadFile(outFile.Name())
if err != nil {
t.Fatalf("Error reading output file: %v", err)
}
return data
}
// setupReadOnlyFile creates a read-only file for error testing.
func setupReadOnlyFile(t *testing.T) (*os.File, chan fileproc.WriteRequest, chan struct{}) {
t.Helper()
outPath := filepath.Join(t.TempDir(), "readonly_out")
outFile, err := os.Create(outPath)
if err != nil {
t.Fatalf(shared.TestMsgFailedToCreateFile, err)
}
// Close writable FD and reopen as read-only so writes will fail
_ = outFile.Close()
outFile, err = os.OpenFile(outPath, os.O_RDONLY, 0)
if err != nil {
t.Fatalf("Failed to reopen as read-only: %v", err)
}
writeCh := make(chan fileproc.WriteRequest, 1)
doneCh := make(chan struct{})
writeCh <- fileproc.WriteRequest{
Path: shared.TestFileGo,
Content: shared.LiteralPackageMain,
}
close(writeCh)
return outFile, writeCh, doneCh
}
// setupStreamingError creates a streaming request with a failing reader.
func setupStreamingError(t *testing.T) (*os.File, chan fileproc.WriteRequest, chan struct{}) {
t.Helper()
outFile, err := os.CreateTemp(t.TempDir(), "yaml_stream_*")
if err != nil {
t.Fatalf(shared.TestMsgFailedToCreateFile, err)
}
writeCh := make(chan fileproc.WriteRequest, 1)
doneCh := make(chan struct{})
pr, pw := io.Pipe()
if err := pw.CloseWithError(errors.New("simulated stream error")); err != nil {
t.Fatalf("failed to set pipe error: %v", err)
}
writeCh <- fileproc.WriteRequest{
Path: "stream_fail.yaml",
Content: "", // Empty for streaming
IsStream: true,
Reader: pr,
}
close(writeCh)
return outFile, writeCh, doneCh
}
// setupSpecialCharacters creates requests with special characters.
func setupSpecialCharacters(t *testing.T) (*os.File, chan fileproc.WriteRequest, chan struct{}) {
t.Helper()
outFile, err := os.CreateTemp(t.TempDir(), "markdown_special_*")
if err != nil {
t.Fatalf(shared.TestMsgFailedToCreateFile, err)
}
writeCh := make(chan fileproc.WriteRequest, 2)
doneCh := make(chan struct{})
writeCh <- fileproc.WriteRequest{
Path: "special\ncharacters.md",
Content: "Content with\x00null bytes and\ttabs",
}
writeCh <- fileproc.WriteRequest{
Path: "empty.md",
Content: "",
}
close(writeCh)
return outFile, writeCh, doneCh
}
// runErrorHandlingTest runs a single error handling test.
func runErrorHandlingTest(
t *testing.T,
outFile *os.File,
writeCh chan fileproc.WriteRequest,
doneCh chan struct{},
format string,
expectEmpty bool,
) {
t.Helper()
defer func() {
if err := os.Remove(outFile.Name()); err != nil {
t.Logf("Failed to remove temp file: %v", err)
}
}()
defer func() {
if err := outFile.Close(); err != nil {
t.Logf("Failed to close temp file: %v", err)
}
}()
var wg sync.WaitGroup
wg.Go(func() {
fileproc.StartWriter(outFile, writeCh, doneCh, format, "PREFIX", "SUFFIX")
})
wg.Wait()
// Wait for doneCh with timeout to prevent test hangs
select {
case <-doneCh:
case <-time.After(3 * time.Second):
t.Fatal(shared.TestMsgTimeoutWriterCompletion)
}
// Read output file and verify based on expectation
data, err := os.ReadFile(outFile.Name())
if err != nil {
t.Fatalf("Failed to read output file: %v", err)
}
if expectEmpty && len(data) != 0 {
t.Errorf("expected empty output on error, got %d bytes", len(data))
}
if !expectEmpty && len(data) == 0 {
t.Error("expected non-empty output, got empty")
}
}
// TestStartWriterErrorHandling tests error scenarios in writers.
func TestStartWriterErrorHandling(t *testing.T) {
tests := []struct {
name string
format string
setupError func(t *testing.T) (*os.File, chan fileproc.WriteRequest, chan struct{})
expectEmptyOutput bool
}{
{
name: "JSON writer with read-only file",
format: "json",
setupError: setupReadOnlyFile,
expectEmptyOutput: true,
},
{
name: "YAML writer with streaming error",
format: "yaml",
setupError: setupStreamingError,
expectEmptyOutput: false, // Partial writes are acceptable before streaming errors
},
{
name: "Markdown writer with special characters",
format: "markdown",
setupError: setupSpecialCharacters,
expectEmptyOutput: false,
},
}
for _, tc := range tests {
t.Run(
tc.name, func(t *testing.T) {
outFile, writeCh, doneCh := tc.setupError(t)
runErrorHandlingTest(t, outFile, writeCh, doneCh, tc.format, tc.expectEmptyOutput)
},
)
}
}
// setupCloseTest sets up files and channels for close testing.
func setupCloseTest(t *testing.T) (*os.File, chan fileproc.WriteRequest, chan struct{}) {
t.Helper()
outFile, err := os.CreateTemp(t.TempDir(), "close_test_*")
if err != nil {
t.Fatalf(shared.TestMsgFailedToCreateFile, err)
}
writeCh := make(chan fileproc.WriteRequest, 5)
doneCh := make(chan struct{})
for i := 0; i < 5; i++ {
writeCh <- fileproc.WriteRequest{
Path: fmt.Sprintf("file%d.txt", i),
Content: fmt.Sprintf("Content %d", i),
}
}
close(writeCh)
return outFile, writeCh, doneCh
}
// runCloseTest executes writer and validates output.
func runCloseTest(
t *testing.T,
outFile *os.File,
writeCh chan fileproc.WriteRequest,
doneCh chan struct{},
format string,
) {
t.Helper()
defer func() {
if err := os.Remove(outFile.Name()); err != nil {
t.Logf("Failed to remove temp file: %v", err)
}
}()
defer func() {
if err := outFile.Close(); err != nil {
t.Logf("Failed to close temp file: %v", err)
}
}()
var wg sync.WaitGroup
wg.Go(func() {
fileproc.StartWriter(outFile, writeCh, doneCh, format, "TEST_PREFIX", "TEST_SUFFIX")
})
wg.Wait()
select {
case <-doneCh:
case <-time.After(3 * time.Second):
t.Fatal(shared.TestMsgTimeoutWriterCompletion)
}
data, err := os.ReadFile(outFile.Name())
if err != nil {
t.Fatalf("Failed to read output file: %v", err)
}
if len(data) == 0 {
t.Error("Expected non-empty output file")
}
verifyPrefixSuffixWith(t, data, "TEST_PREFIX", "TEST_SUFFIX")
}
// TestStartWriterWriterCloseErrors tests error handling during writer close operations.
func TestStartWriterWriterCloseErrors(t *testing.T) {
tests := []struct {
name string
format string
}{
{"JSON close handling", "json"},
{"YAML close handling", "yaml"},
{"Markdown close handling", "markdown"},
}
for _, tc := range tests {
t.Run(
tc.name, func(t *testing.T) {
outFile, writeCh, doneCh := setupCloseTest(t)
runCloseTest(t, outFile, writeCh, doneCh, tc.format)
},
)
}
}
// Benchmarks for writer performance
// BenchmarkStartWriter benchmarks basic writer operations across formats.
func BenchmarkStartWriter(b *testing.B) {
formats := []string{"json", "yaml", "markdown"}
for _, format := range formats {
b.Run(format, func(b *testing.B) {
for b.Loop() {
outFile, err := os.CreateTemp(b.TempDir(), "bench_output_*")
if err != nil {
b.Fatalf("Failed to create temp file: %v", err)
}
writeCh := make(chan fileproc.WriteRequest, 2)
doneCh := make(chan struct{})
writeCh <- fileproc.WriteRequest{Path: "sample.go", Content: shared.LiteralPackageMain}
writeCh <- fileproc.WriteRequest{Path: "example.py", Content: "def foo(): pass"}
close(writeCh)
fileproc.StartWriter(outFile, writeCh, doneCh, format, "PREFIX", "SUFFIX")
<-doneCh
_ = outFile.Close()
}
})
}
}
// benchStreamingIteration runs a single streaming benchmark iteration.
func benchStreamingIteration(b *testing.B, format, content string) {
b.Helper()
contentFile := createBenchContentFile(b, content)
defer func() { _ = os.Remove(contentFile) }()
reader, err := os.Open(contentFile)
if err != nil {
b.Fatalf("Failed to open content file: %v", err)
}
defer func() { _ = reader.Close() }()
outFile, err := os.CreateTemp(b.TempDir(), "bench_stream_output_*")
if err != nil {
b.Fatalf("Failed to create output file: %v", err)
}
defer func() { _ = outFile.Close() }()
writeCh := make(chan fileproc.WriteRequest, 1)
doneCh := make(chan struct{})
writeCh <- fileproc.WriteRequest{
Path: shared.TestFileStreamTest,
Content: "",
IsStream: true,
Reader: reader,
}
close(writeCh)
fileproc.StartWriter(outFile, writeCh, doneCh, format, "PREFIX", "SUFFIX")
<-doneCh
}
// createBenchContentFile creates a temp file with content for benchmarks.
func createBenchContentFile(b *testing.B, content string) string {
b.Helper()
contentFile, err := os.CreateTemp(b.TempDir(), "content_*")
if err != nil {
b.Fatalf("Failed to create content file: %v", err)
}
if _, err := contentFile.WriteString(content); err != nil {
b.Fatalf("Failed to write content: %v", err)
}
if err := contentFile.Close(); err != nil {
b.Fatalf("Failed to close content file: %v", err)
}
return contentFile.Name()
}
// BenchmarkStartWriterStreaming benchmarks streaming writer operations across formats.
func BenchmarkStartWriterStreaming(b *testing.B) {
formats := []string{"json", "yaml", "markdown"}
content := strings.Repeat("line content\n", 1000)
for _, format := range formats {
b.Run(format, func(b *testing.B) {
for b.Loop() {
benchStreamingIteration(b, format, content)
}
})
}
}

View File

@@ -1,14 +1,12 @@
// Package fileproc handles file processing, collection, and output formatting.
package fileproc package fileproc
import ( import (
"bufio"
"fmt" "fmt"
"io"
"os" "os"
"path/filepath"
"strings" "strings"
"github.com/ivuorinen/gibidify/gibidiutils" "github.com/ivuorinen/gibidify/shared"
) )
// YAMLWriter handles YAML format output with streaming support. // YAMLWriter handles YAML format output with streaming support.
@@ -21,152 +19,18 @@ func NewYAMLWriter(outFile *os.File) *YAMLWriter {
return &YAMLWriter{outFile: outFile} return &YAMLWriter{outFile: outFile}
} }
const (
maxPathLength = 4096 // Maximum total path length
maxFilenameLength = 255 // Maximum individual filename component length
)
// validatePathComponents validates individual path components for security issues.
func validatePathComponents(trimmed, cleaned string, components []string) error {
for i, component := range components {
// Reject path components that are exactly ".." (path traversal)
if component == ".." {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationPath,
"path traversal not allowed",
trimmed,
map[string]any{
"path": trimmed,
"cleaned": cleaned,
"invalid_component": component,
"component_index": i,
},
)
}
// Reject empty components (e.g., from "foo//bar")
if component == "" && i > 0 && i < len(components)-1 {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationPath,
"path contains empty component",
trimmed,
map[string]any{
"path": trimmed,
"cleaned": cleaned,
"component_index": i,
},
)
}
// Enforce maximum filename length for each component
if len(component) > maxFilenameLength {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationPath,
"path component exceeds maximum length",
trimmed,
map[string]any{
"component": component,
"component_length": len(component),
"max_length": maxFilenameLength,
"component_index": i,
},
)
}
}
return nil
}
// validatePath validates and sanitizes a file path for safe output.
// It rejects absolute paths, path traversal attempts, empty paths, and overly long paths.
func validatePath(path string) error {
// Reject empty paths
trimmed := strings.TrimSpace(path)
if trimmed == "" {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationRequired,
"file path cannot be empty",
"",
nil,
)
}
// Enforce maximum path length to prevent resource abuse
if len(trimmed) > maxPathLength {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationPath,
"path exceeds maximum length",
trimmed,
map[string]any{
"path_length": len(trimmed),
"max_length": maxPathLength,
},
)
}
// Reject absolute paths
if filepath.IsAbs(trimmed) {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationPath,
"absolute paths are not allowed",
trimmed,
map[string]any{"path": trimmed},
)
}
// Validate original trimmed path components before cleaning
origComponents := strings.Split(filepath.ToSlash(trimmed), "/")
for _, comp := range origComponents {
if comp == "" || comp == "." || comp == ".." {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationPath,
"invalid or traversal path component in original path",
trimmed,
map[string]any{"path": trimmed, "component": comp},
)
}
}
// Clean the path to normalize it
cleaned := filepath.Clean(trimmed)
// After cleaning, ensure it's still relative and doesn't start with /
if filepath.IsAbs(cleaned) || strings.HasPrefix(cleaned, "/") {
return gibidiutils.NewStructuredError(
gibidiutils.ErrorTypeValidation,
gibidiutils.CodeValidationPath,
"path must be relative",
trimmed,
map[string]any{"path": trimmed, "cleaned": cleaned},
)
}
// Split into components and validate each one
// Use ToSlash to normalize for cross-platform validation
components := strings.Split(filepath.ToSlash(cleaned), "/")
return validatePathComponents(trimmed, cleaned, components)
}
// Start writes the YAML header. // Start writes the YAML header.
func (w *YAMLWriter) Start(prefix, suffix string) error { func (w *YAMLWriter) Start(prefix, suffix string) error {
// Write YAML header // Write YAML header
if _, err := fmt.Fprintf( if _, err := fmt.Fprintf(
w.outFile, "prefix: %s\nsuffix: %s\nfiles:\n", w.outFile,
gibidiutils.EscapeForYAML(prefix), gibidiutils.EscapeForYAML(suffix), "prefix: %s\nsuffix: %s\nfiles:\n",
shared.EscapeForYAML(prefix),
shared.EscapeForYAML(suffix),
); err != nil { ); err != nil {
return gibidiutils.WrapError( return shared.WrapError(err, shared.ErrorTypeIO, shared.CodeIOWrite, "failed to write YAML header")
err,
gibidiutils.ErrorTypeIO,
gibidiutils.CodeIOWrite,
"failed to write YAML header",
)
} }
return nil return nil
} }
@@ -175,6 +39,7 @@ func (w *YAMLWriter) WriteFile(req WriteRequest) error {
if req.IsStream { if req.IsStream {
return w.writeStreaming(req) return w.writeStreaming(req)
} }
return w.writeInline(req) return w.writeInline(req)
} }
@@ -185,45 +50,39 @@ func (w *YAMLWriter) Close() error {
// writeStreaming writes a large file as YAML in streaming chunks. // writeStreaming writes a large file as YAML in streaming chunks.
func (w *YAMLWriter) writeStreaming(req WriteRequest) error { func (w *YAMLWriter) writeStreaming(req WriteRequest) error {
// Validate path before using it defer shared.SafeCloseReader(req.Reader, req.Path)
if err := validatePath(req.Path); err != nil {
return err
}
// Check for nil reader
if req.Reader == nil {
return gibidiutils.WrapError(
nil, gibidiutils.ErrorTypeValidation, gibidiutils.CodeValidationRequired,
"nil reader in write request",
).WithFilePath(req.Path)
}
defer gibidiutils.SafeCloseReader(req.Reader, req.Path)
language := detectLanguage(req.Path) language := detectLanguage(req.Path)
// Write YAML file entry start // Write YAML file entry start
if _, err := fmt.Fprintf( if _, err := fmt.Fprintf(
w.outFile, " - path: %s\n language: %s\n content: |\n", w.outFile,
gibidiutils.EscapeForYAML(req.Path), language, shared.YAMLFmtFileEntry,
shared.EscapeForYAML(req.Path),
language,
); err != nil { ); err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOWrite, err,
shared.ErrorTypeIO,
shared.CodeIOWrite,
"failed to write YAML file start", "failed to write YAML file start",
).WithFilePath(req.Path) ).WithFilePath(req.Path)
} }
// Stream content with YAML indentation // Stream content with YAML indentation
return w.streamYAMLContent(req.Reader, req.Path) if err := shared.StreamLines(
req.Reader, w.outFile, req.Path, func(line string) string {
return " " + line
},
); err != nil {
return shared.WrapError(err, shared.ErrorTypeIO, shared.CodeIOWrite, "streaming YAML content")
}
return nil
} }
// writeInline writes a small file directly as YAML. // writeInline writes a small file directly as YAML.
func (w *YAMLWriter) writeInline(req WriteRequest) error { func (w *YAMLWriter) writeInline(req WriteRequest) error {
// Validate path before using it
if err := validatePath(req.Path); err != nil {
return err
}
language := detectLanguage(req.Path) language := detectLanguage(req.Path)
fileData := FileData{ fileData := FileData{
Path: req.Path, Path: req.Path,
@@ -233,11 +92,15 @@ func (w *YAMLWriter) writeInline(req WriteRequest) error {
// Write YAML entry // Write YAML entry
if _, err := fmt.Fprintf( if _, err := fmt.Fprintf(
w.outFile, " - path: %s\n language: %s\n content: |\n", w.outFile,
gibidiutils.EscapeForYAML(fileData.Path), fileData.Language, shared.YAMLFmtFileEntry,
shared.EscapeForYAML(fileData.Path),
fileData.Language,
); err != nil { ); err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOWrite, err,
shared.ErrorTypeIO,
shared.CodeIOWrite,
"failed to write YAML entry start", "failed to write YAML entry start",
).WithFilePath(req.Path) ).WithFilePath(req.Path)
} }
@@ -246,8 +109,10 @@ func (w *YAMLWriter) writeInline(req WriteRequest) error {
lines := strings.Split(fileData.Content, "\n") lines := strings.Split(fileData.Content, "\n")
for _, line := range lines { for _, line := range lines {
if _, err := fmt.Fprintf(w.outFile, " %s\n", line); err != nil { if _, err := fmt.Fprintf(w.outFile, " %s\n", line); err != nil {
return gibidiutils.WrapError( return shared.WrapError(
err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOWrite, err,
shared.ErrorTypeIO,
shared.CodeIOWrite,
"failed to write YAML content line", "failed to write YAML content line",
).WithFilePath(req.Path) ).WithFilePath(req.Path)
} }
@@ -256,53 +121,9 @@ func (w *YAMLWriter) writeInline(req WriteRequest) error {
return nil return nil
} }
// streamYAMLContent streams content with YAML indentation.
func (w *YAMLWriter) streamYAMLContent(reader io.Reader, path string) error {
scanner := bufio.NewScanner(reader)
// Increase buffer size to handle long lines (up to 10MB per line)
buf := make([]byte, 0, 64*1024)
scanner.Buffer(buf, 10*1024*1024)
for scanner.Scan() {
line := scanner.Text()
if _, err := fmt.Fprintf(w.outFile, " %s\n", line); err != nil {
return gibidiutils.WrapError(
err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOWrite,
"failed to write YAML line",
).WithFilePath(path)
}
}
if err := scanner.Err(); err != nil {
return gibidiutils.WrapError(
err, gibidiutils.ErrorTypeIO, gibidiutils.CodeIOFileRead,
"failed to scan YAML content",
).WithFilePath(path)
}
return nil
}
// startYAMLWriter handles YAML format output with streaming support. // startYAMLWriter handles YAML format output with streaming support.
func startYAMLWriter(outFile *os.File, writeCh <-chan WriteRequest, done chan<- struct{}, prefix, suffix string) { func startYAMLWriter(outFile *os.File, writeCh <-chan WriteRequest, done chan<- struct{}, prefix, suffix string) {
defer close(done) startFormatWriter(outFile, writeCh, done, prefix, suffix, func(f *os.File) FormatWriter {
return NewYAMLWriter(f)
writer := NewYAMLWriter(outFile) })
// Start writing
if err := writer.Start(prefix, suffix); err != nil {
gibidiutils.LogError("Failed to write YAML header", err)
return
}
// Process files
for req := range writeCh {
if err := writer.WriteFile(req); err != nil {
gibidiutils.LogError("Failed to write YAML file", err)
}
}
// Close writer
if err := writer.Close(); err != nil {
gibidiutils.LogError("Failed to write YAML end", err)
}
} }

View File

@@ -1,367 +0,0 @@
package gibidiutils
import (
"errors"
"testing"
"github.com/stretchr/testify/assert"
)
func TestErrorTypeString(t *testing.T) {
tests := []struct {
name string
errType ErrorType
expected string
}{
{
name: "CLI error type",
errType: ErrorTypeCLI,
expected: "CLI",
},
{
name: "FileSystem error type",
errType: ErrorTypeFileSystem,
expected: "FileSystem",
},
{
name: "Processing error type",
errType: ErrorTypeProcessing,
expected: "Processing",
},
{
name: "Configuration error type",
errType: ErrorTypeConfiguration,
expected: "Configuration",
},
{
name: "IO error type",
errType: ErrorTypeIO,
expected: "IO",
},
{
name: "Validation error type",
errType: ErrorTypeValidation,
expected: "Validation",
},
{
name: "Unknown error type",
errType: ErrorTypeUnknown,
expected: "Unknown",
},
{
name: "Invalid error type",
errType: ErrorType(999),
expected: "Unknown",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := tt.errType.String()
assert.Equal(t, tt.expected, result)
})
}
}
func TestStructuredErrorMethods(t *testing.T) {
t.Run("Error method", func(t *testing.T) {
err := &StructuredError{
Type: ErrorTypeValidation,
Code: CodeValidationRequired,
Message: "field is required",
}
expected := "Validation [REQUIRED]: field is required"
assert.Equal(t, expected, err.Error())
})
t.Run("Error method with context", func(t *testing.T) {
err := &StructuredError{
Type: ErrorTypeFileSystem,
Code: CodeFSNotFound,
Message: testErrFileNotFound,
Context: map[string]interface{}{
"path": "/test/file.txt",
},
}
errStr := err.Error()
assert.Contains(t, errStr, "FileSystem")
assert.Contains(t, errStr, "NOT_FOUND")
assert.Contains(t, errStr, testErrFileNotFound)
assert.Contains(t, errStr, "/test/file.txt")
assert.Contains(t, errStr, "path")
})
t.Run("Unwrap method", func(t *testing.T) {
innerErr := errors.New("inner error")
err := &StructuredError{
Type: ErrorTypeIO,
Code: CodeIOFileWrite,
Message: testErrWriteFailed,
Cause: innerErr,
}
assert.Equal(t, innerErr, err.Unwrap())
})
t.Run("Unwrap with nil cause", func(t *testing.T) {
err := &StructuredError{
Type: ErrorTypeIO,
Code: CodeIOFileWrite,
Message: testErrWriteFailed,
}
assert.Nil(t, err.Unwrap())
})
}
func TestWithContextMethods(t *testing.T) {
t.Run("WithContext", func(t *testing.T) {
err := &StructuredError{
Type: ErrorTypeValidation,
Code: CodeValidationFormat,
Message: testErrInvalidFormat,
}
err = err.WithContext("format", "xml")
err = err.WithContext("expected", "json")
assert.NotNil(t, err.Context)
assert.Equal(t, "xml", err.Context["format"])
assert.Equal(t, "json", err.Context["expected"])
})
t.Run("WithFilePath", func(t *testing.T) {
err := &StructuredError{
Type: ErrorTypeFileSystem,
Code: CodeFSPermission,
Message: "permission denied",
}
err = err.WithFilePath("/etc/passwd")
assert.Equal(t, "/etc/passwd", err.FilePath)
})
t.Run("WithLine", func(t *testing.T) {
err := &StructuredError{
Type: ErrorTypeProcessing,
Code: CodeProcessingFileRead,
Message: "read error",
}
err = err.WithLine(42)
assert.Equal(t, 42, err.Line)
})
}
func TestNewStructuredError(t *testing.T) {
tests := []struct {
name string
errType ErrorType
code string
message string
filePath string
context map[string]interface{}
}{
{
name: "basic error",
errType: ErrorTypeValidation,
code: CodeValidationRequired,
message: "field is required",
filePath: "",
context: nil,
},
{
name: "error with file path",
errType: ErrorTypeFileSystem,
code: CodeFSNotFound,
message: testErrFileNotFound,
filePath: "/test/missing.txt",
context: nil,
},
{
name: "error with context",
errType: ErrorTypeIO,
code: CodeIOFileWrite,
message: testErrWriteFailed,
context: map[string]interface{}{
"size": 1024,
"error": "disk full",
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := NewStructuredError(tt.errType, tt.code, tt.message, tt.filePath, tt.context)
assert.NotNil(t, err)
assert.Equal(t, tt.errType, err.Type)
assert.Equal(t, tt.code, err.Code)
assert.Equal(t, tt.message, err.Message)
assert.Equal(t, tt.filePath, err.FilePath)
assert.Equal(t, tt.context, err.Context)
})
}
}
func TestNewStructuredErrorf(t *testing.T) {
err := NewStructuredErrorf(
ErrorTypeValidation,
CodeValidationSize,
"file size %d exceeds limit %d",
2048, 1024,
)
assert.NotNil(t, err)
assert.Equal(t, ErrorTypeValidation, err.Type)
assert.Equal(t, CodeValidationSize, err.Code)
assert.Equal(t, "file size 2048 exceeds limit 1024", err.Message)
}
func TestWrapError(t *testing.T) {
innerErr := errors.New("original error")
wrappedErr := WrapError(
innerErr,
ErrorTypeProcessing,
CodeProcessingFileRead,
"failed to process file",
)
assert.NotNil(t, wrappedErr)
assert.Equal(t, ErrorTypeProcessing, wrappedErr.Type)
assert.Equal(t, CodeProcessingFileRead, wrappedErr.Code)
assert.Equal(t, "failed to process file", wrappedErr.Message)
assert.Equal(t, innerErr, wrappedErr.Cause)
}
func TestWrapErrorf(t *testing.T) {
innerErr := errors.New("original error")
wrappedErr := WrapErrorf(
innerErr,
ErrorTypeIO,
CodeIOFileCreate,
"failed to create %s in %s",
"output.txt", "/tmp",
)
assert.NotNil(t, wrappedErr)
assert.Equal(t, ErrorTypeIO, wrappedErr.Type)
assert.Equal(t, CodeIOFileCreate, wrappedErr.Code)
assert.Equal(t, "failed to create output.txt in /tmp", wrappedErr.Message)
assert.Equal(t, innerErr, wrappedErr.Cause)
}
func TestSpecificErrorConstructors(t *testing.T) {
t.Run("NewMissingSourceError", func(t *testing.T) {
err := NewMissingSourceError()
assert.NotNil(t, err)
assert.Equal(t, ErrorTypeCLI, err.Type)
assert.Equal(t, CodeCLIMissingSource, err.Code)
assert.Contains(t, err.Message, "source")
})
t.Run("NewFileSystemError", func(t *testing.T) {
err := NewFileSystemError(CodeFSPermission, "access denied")
assert.NotNil(t, err)
assert.Equal(t, ErrorTypeFileSystem, err.Type)
assert.Equal(t, CodeFSPermission, err.Code)
assert.Equal(t, "access denied", err.Message)
})
t.Run("NewProcessingError", func(t *testing.T) {
err := NewProcessingError(CodeProcessingCollection, "collection failed")
assert.NotNil(t, err)
assert.Equal(t, ErrorTypeProcessing, err.Type)
assert.Equal(t, CodeProcessingCollection, err.Code)
assert.Equal(t, "collection failed", err.Message)
})
t.Run("NewIOError", func(t *testing.T) {
err := NewIOError(CodeIOFileWrite, testErrWriteFailed)
assert.NotNil(t, err)
assert.Equal(t, ErrorTypeIO, err.Type)
assert.Equal(t, CodeIOFileWrite, err.Code)
assert.Equal(t, testErrWriteFailed, err.Message)
})
t.Run("NewValidationError", func(t *testing.T) {
err := NewValidationError(CodeValidationFormat, testErrInvalidFormat)
assert.NotNil(t, err)
assert.Equal(t, ErrorTypeValidation, err.Type)
assert.Equal(t, CodeValidationFormat, err.Code)
assert.Equal(t, testErrInvalidFormat, err.Message)
})
}
// TestLogErrorf is already covered in errors_test.go
func TestStructuredErrorChaining(t *testing.T) {
// Test method chaining
err := NewStructuredError(
ErrorTypeFileSystem,
CodeFSNotFound,
testErrFileNotFound,
"",
nil,
).WithFilePath("/test.txt").WithLine(10).WithContext("operation", "read")
assert.Equal(t, "/test.txt", err.FilePath)
assert.Equal(t, 10, err.Line)
assert.Equal(t, "read", err.Context["operation"])
}
func TestErrorCodes(t *testing.T) {
// Test that all error codes are defined
codes := []string{
CodeCLIMissingSource,
CodeCLIInvalidArgs,
CodeFSPathResolution,
CodeFSPermission,
CodeFSNotFound,
CodeFSAccess,
CodeProcessingFileRead,
CodeProcessingCollection,
CodeProcessingTraversal,
CodeProcessingEncode,
CodeConfigValidation,
CodeConfigMissing,
CodeIOFileCreate,
CodeIOFileWrite,
CodeIOEncoding,
CodeIOWrite,
CodeIOFileRead,
CodeIOClose,
CodeValidationRequired,
CodeValidationFormat,
CodeValidationSize,
CodeValidationPath,
CodeResourceLimitFiles,
CodeResourceLimitTotalSize,
CodeResourceLimitMemory,
CodeResourceLimitTimeout,
}
// All codes should be non-empty strings
for _, code := range codes {
assert.NotEmpty(t, code, "Error code should not be empty")
assert.NotEqual(t, "", code, "Error code should be defined")
}
}
func TestErrorUnwrapChain(t *testing.T) {
// Test unwrapping through multiple levels
innermost := errors.New("innermost error")
middle := WrapError(innermost, ErrorTypeIO, CodeIOFileRead, "read failed")
outer := WrapError(middle, ErrorTypeProcessing, CodeProcessingFileRead, "processing failed")
// Test unwrapping
assert.Equal(t, middle, outer.Unwrap())
assert.Equal(t, innermost, middle.Unwrap())
// innermost is a plain error, doesn't have Unwrap() method
// No need to test it
// Test error chain messages
assert.Contains(t, outer.Error(), "Processing")
assert.Contains(t, middle.Error(), "IO")
}

View File

@@ -1,243 +0,0 @@
// Package gibidiutils provides common utility functions for gibidify.
package gibidiutils
import (
"bytes"
"errors"
"fmt"
"strings"
"testing"
"github.com/sirupsen/logrus"
)
// captureLogOutput captures logrus output for testing
func captureLogOutput(f func()) string {
var buf bytes.Buffer
logrus.SetOutput(&buf)
defer logrus.SetOutput(logrus.StandardLogger().Out)
f()
return buf.String()
}
func TestLogError(t *testing.T) {
tests := []struct {
name string
operation string
err error
args []any
wantLog string
wantEmpty bool
}{
{
name: "nil error should not log",
operation: "test operation",
err: nil,
args: nil,
wantEmpty: true,
},
{
name: "basic error logging",
operation: "failed to read file",
err: errors.New("permission denied"),
args: nil,
wantLog: "failed to read file: permission denied",
},
{
name: "error with formatting args",
operation: "failed to process file %s",
err: errors.New("file too large"),
args: []any{"test.txt"},
wantLog: "failed to process file test.txt: file too large",
},
{
name: "error with multiple formatting args",
operation: "failed to copy from %s to %s",
err: errors.New("disk full"),
args: []any{"source.txt", "dest.txt"},
wantLog: "failed to copy from source.txt to dest.txt: disk full",
},
{
name: "wrapped error",
operation: "database operation failed",
err: fmt.Errorf("connection error: %w", errors.New("timeout")),
args: nil,
wantLog: "database operation failed: connection error: timeout",
},
{
name: "empty operation string",
operation: "",
err: errors.New("some error"),
args: nil,
wantLog: ": some error",
},
{
name: "operation with percentage sign",
operation: "processing 50% complete",
err: errors.New("interrupted"),
args: nil,
wantLog: "processing 50% complete: interrupted",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
output := captureLogOutput(func() {
LogError(tt.operation, tt.err, tt.args...)
})
if tt.wantEmpty {
if output != "" {
t.Errorf("LogError() logged output when error was nil: %q", output)
}
return
}
if !strings.Contains(output, tt.wantLog) {
t.Errorf("LogError() output = %q, want to contain %q", output, tt.wantLog)
}
// Verify it's logged at ERROR level
if !strings.Contains(output, "level=error") {
t.Errorf("LogError() should log at ERROR level, got: %q", output)
}
})
}
}
func TestLogErrorf(t *testing.T) {
tests := []struct {
name string
err error
format string
args []any
wantLog string
wantEmpty bool
}{
{
name: "nil error should not log",
err: nil,
format: "operation %s failed",
args: []any{"test"},
wantEmpty: true,
},
{
name: "basic formatted error",
err: errors.New("not found"),
format: "file %s not found",
args: []any{"config.yaml"},
wantLog: "file config.yaml not found: not found",
},
{
name: "multiple format arguments",
err: errors.New("invalid range"),
format: "value %d is not between %d and %d",
args: []any{150, 0, 100},
wantLog: "value 150 is not between 0 and 100: invalid range",
},
{
name: "no format arguments",
err: errors.New("generic error"),
format: "operation failed",
args: nil,
wantLog: "operation failed: generic error",
},
{
name: "format with different types",
err: errors.New("type mismatch"),
format: "expected %s but got %d",
args: []any{"string", 42},
wantLog: "expected string but got 42: type mismatch",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
output := captureLogOutput(func() {
LogErrorf(tt.err, tt.format, tt.args...)
})
if tt.wantEmpty {
if output != "" {
t.Errorf("LogErrorf() logged output when error was nil: %q", output)
}
return
}
if !strings.Contains(output, tt.wantLog) {
t.Errorf("LogErrorf() output = %q, want to contain %q", output, tt.wantLog)
}
// Verify it's logged at ERROR level
if !strings.Contains(output, "level=error") {
t.Errorf("LogErrorf() should log at ERROR level, got: %q", output)
}
})
}
}
func TestLogErrorConcurrency(_ *testing.T) {
// Test that LogError is safe for concurrent use
done := make(chan bool)
for i := 0; i < 10; i++ {
go func(n int) {
LogError("concurrent operation", fmt.Errorf("error %d", n))
done <- true
}(i)
}
// Wait for all goroutines to complete
for i := 0; i < 10; i++ {
<-done
}
}
func TestLogErrorfConcurrency(_ *testing.T) {
// Test that LogErrorf is safe for concurrent use
done := make(chan bool)
for i := 0; i < 10; i++ {
go func(n int) {
LogErrorf(fmt.Errorf("error %d", n), "concurrent operation %d", n)
done <- true
}(i)
}
// Wait for all goroutines to complete
for i := 0; i < 10; i++ {
<-done
}
}
// BenchmarkLogError benchmarks the LogError function
func BenchmarkLogError(b *testing.B) {
err := errors.New("benchmark error")
// Disable output during benchmark
logrus.SetOutput(bytes.NewBuffer(nil))
defer logrus.SetOutput(logrus.StandardLogger().Out)
b.ResetTimer()
for i := 0; i < b.N; i++ {
LogError("benchmark operation", err)
}
}
// BenchmarkLogErrorf benchmarks the LogErrorf function
func BenchmarkLogErrorf(b *testing.B) {
err := errors.New("benchmark error")
// Disable output during benchmark
logrus.SetOutput(bytes.NewBuffer(nil))
defer logrus.SetOutput(logrus.StandardLogger().Out)
b.ResetTimer()
for i := 0; i < b.N; i++ {
LogErrorf(err, "benchmark operation %d", i)
}
}
// BenchmarkLogErrorNil benchmarks LogError with nil error (no-op case)
func BenchmarkLogErrorNil(b *testing.B) {
b.ResetTimer()
for i := 0; i < b.N; i++ {
LogError("benchmark operation", nil)
}
}

View File

@@ -1,10 +0,0 @@
package gibidiutils
// Unicode icons and symbols for CLI UI and test output.
const (
IconSuccess = "✓" // U+2713
IconError = "✗" // U+2717
IconWarning = "⚠" // U+26A0
IconBullet = "•" // U+2022
IconInfo = "" // U+2139 FE0F
)

View File

@@ -1,311 +0,0 @@
// Package gibidiutils provides common utility functions for gibidify.
package gibidiutils
import (
"fmt"
"os"
"path/filepath"
"strings"
)
// EscapeForMarkdown sanitizes a string for safe use in Markdown code-fence and header lines.
// It replaces backticks with backslash-escaped backticks and removes/collapses newlines.
func EscapeForMarkdown(s string) string {
// Escape backticks
safe := strings.ReplaceAll(s, "`", "\\`")
// Remove newlines (collapse to space)
safe = strings.ReplaceAll(safe, "\n", " ")
safe = strings.ReplaceAll(safe, "\r", " ")
return safe
}
// GetAbsolutePath returns the absolute path for the given path.
// It wraps filepath.Abs with consistent error handling.
func GetAbsolutePath(path string) (string, error) {
abs, err := filepath.Abs(path)
if err != nil {
return "", fmt.Errorf("failed to get absolute path for %s: %w", path, err)
}
return abs, nil
}
// GetBaseName returns the base name for the given path, handling special cases.
func GetBaseName(absPath string) string {
baseName := filepath.Base(absPath)
if baseName == "." || baseName == "" {
return "output"
}
return baseName
}
// checkPathTraversal checks for path traversal patterns and returns an error if found.
func checkPathTraversal(path, context string) error {
// Normalize separators without cleaning (to preserve ..)
normalized := filepath.ToSlash(path)
// Split into components
components := strings.Split(normalized, "/")
// Check each component for exact ".." match
for _, component := range components {
if component == ".." {
return NewStructuredError(
ErrorTypeValidation,
CodeValidationPath,
fmt.Sprintf("path traversal attempt detected in %s", context),
path,
map[string]interface{}{
"original_path": path,
},
)
}
}
return nil
}
// cleanAndResolveAbsPath cleans a path and resolves it to an absolute path.
func cleanAndResolveAbsPath(path, context string) (string, error) {
cleaned := filepath.Clean(path)
abs, err := filepath.Abs(cleaned)
if err != nil {
return "", NewStructuredError(
ErrorTypeFileSystem,
CodeFSPathResolution,
fmt.Sprintf("cannot resolve %s", context),
path,
map[string]interface{}{
"error": err.Error(),
},
)
}
return abs, nil
}
// evalSymlinksOrStructuredError wraps filepath.EvalSymlinks with structured error handling.
func evalSymlinksOrStructuredError(path, context, original string) (string, error) {
eval, err := filepath.EvalSymlinks(path)
if err != nil {
return "", NewStructuredError(
ErrorTypeValidation,
CodeValidationPath,
fmt.Sprintf("cannot resolve symlinks for %s", context),
original,
map[string]interface{}{
"resolved_path": path,
"context": context,
"error": err.Error(),
},
)
}
return eval, nil
}
// validateWorkingDirectoryBoundary checks if the given absolute path escapes the working directory.
func validateWorkingDirectoryBoundary(abs, path string) error {
cwd, err := os.Getwd()
if err != nil {
return NewStructuredError(
ErrorTypeFileSystem,
CodeFSPathResolution,
"cannot get current working directory",
path,
map[string]interface{}{
"error": err.Error(),
},
)
}
cwdAbs, err := filepath.Abs(cwd)
if err != nil {
return NewStructuredError(
ErrorTypeFileSystem,
CodeFSPathResolution,
"cannot resolve current working directory",
path,
map[string]interface{}{
"error": err.Error(),
},
)
}
absEval, err := evalSymlinksOrStructuredError(abs, "source path", path)
if err != nil {
return err
}
cwdEval, err := evalSymlinksOrStructuredError(cwdAbs, "working directory", path)
if err != nil {
return err
}
rel, err := filepath.Rel(cwdEval, absEval)
if err != nil {
return NewStructuredError(
ErrorTypeValidation,
CodeValidationPath,
"cannot determine relative path",
path,
map[string]interface{}{
"resolved_path": absEval,
"working_dir": cwdEval,
"error": err.Error(),
},
)
}
if rel == ".." || strings.HasPrefix(rel, ".."+string(os.PathSeparator)) {
return NewStructuredError(
ErrorTypeValidation,
CodeValidationPath,
"source path attempts to access directories outside current working directory",
path,
map[string]interface{}{
"resolved_path": absEval,
"working_dir": cwdEval,
"relative_path": rel,
},
)
}
return nil
}
// ValidateSourcePath validates a source directory path for security.
// It ensures the path exists, is a directory, and doesn't contain path traversal attempts.
//
//revive:disable-next-line:function-length
func ValidateSourcePath(path string) error {
if path == "" {
return NewValidationError(CodeValidationRequired, "source path is required")
}
// Check for path traversal patterns before cleaning
if err := checkPathTraversal(path, "source path"); err != nil {
return err
}
// Clean and get absolute path
abs, err := cleanAndResolveAbsPath(path, "source path")
if err != nil {
return err
}
cleaned := filepath.Clean(path)
// Ensure the resolved path is within or below the current working directory for relative paths
if !filepath.IsAbs(path) {
if err := validateWorkingDirectoryBoundary(abs, path); err != nil {
return err
}
}
// Check if path exists and is a directory
info, err := os.Stat(cleaned)
if err != nil {
if os.IsNotExist(err) {
return NewFileSystemError(CodeFSNotFound, "source directory does not exist").WithFilePath(path)
}
return NewStructuredError(
ErrorTypeFileSystem,
CodeFSAccess,
"cannot access source directory",
path,
map[string]interface{}{
"error": err.Error(),
},
)
}
if !info.IsDir() {
return NewStructuredError(
ErrorTypeValidation,
CodeValidationPath,
"source path must be a directory",
path,
map[string]interface{}{
"is_file": true,
},
)
}
return nil
}
// ValidateDestinationPath validates a destination file path for security.
// It ensures the path doesn't contain path traversal attempts and the parent directory exists.
func ValidateDestinationPath(path string) error {
if path == "" {
return NewValidationError(CodeValidationRequired, "destination path is required")
}
// Check for path traversal patterns before cleaning
if err := checkPathTraversal(path, "destination path"); err != nil {
return err
}
// Get absolute path to ensure it's not trying to escape current working directory
abs, err := cleanAndResolveAbsPath(path, "destination path")
if err != nil {
return err
}
// Ensure the destination is not a directory
if info, err := os.Stat(abs); err == nil && info.IsDir() {
return NewStructuredError(
ErrorTypeValidation,
CodeValidationPath,
"destination cannot be a directory",
path,
map[string]interface{}{
"is_directory": true,
},
)
}
// Check if parent directory exists and is writable
parentDir := filepath.Dir(abs)
if parentInfo, err := os.Stat(parentDir); err != nil {
if os.IsNotExist(err) {
return NewStructuredError(
ErrorTypeFileSystem,
CodeFSNotFound,
"destination parent directory does not exist",
path,
map[string]interface{}{
"parent_dir": parentDir,
},
)
}
return NewStructuredError(
ErrorTypeFileSystem,
CodeFSAccess,
"cannot access destination parent directory",
path,
map[string]interface{}{
"parent_dir": parentDir,
"error": err.Error(),
},
)
} else if !parentInfo.IsDir() {
return NewStructuredError(
ErrorTypeValidation,
CodeValidationPath,
"destination parent is not a directory",
path,
map[string]interface{}{
"parent_dir": parentDir,
},
)
}
return nil
}
// ValidateConfigPath validates a configuration file path for security.
// It ensures the path doesn't contain path traversal attempts.
func ValidateConfigPath(path string) error {
if path == "" {
return nil // Empty path is allowed for config
}
// Check for path traversal patterns before cleaning
return checkPathTraversal(path, "config path")
}

View File

@@ -1,368 +0,0 @@
package gibidiutils
import (
"errors"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestGetBaseName(t *testing.T) {
tests := []struct {
name string
absPath string
expected string
}{
{
name: "normal path",
absPath: "/home/user/project",
expected: "project",
},
{
name: "path with trailing slash",
absPath: "/home/user/project/",
expected: "project",
},
{
name: "root path",
absPath: "/",
expected: "/",
},
{
name: "current directory",
absPath: ".",
expected: "output",
},
{
name: testEmptyPath,
absPath: "",
expected: "output",
},
{
name: "file path",
absPath: "/home/user/file.txt",
expected: "file.txt",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := GetBaseName(tt.absPath)
assert.Equal(t, tt.expected, result)
})
}
}
func TestValidateSourcePath(t *testing.T) {
// Create a temp directory for testing
tempDir := t.TempDir()
tempFile := filepath.Join(tempDir, "test.txt")
require.NoError(t, os.WriteFile(tempFile, []byte("test"), 0o600))
tests := []struct {
name string
path string
expectedError string
}{
{
name: testEmptyPath,
path: "",
expectedError: "source path is required",
},
{
name: testPathTraversalAttempt,
path: "../../../etc/passwd",
expectedError: testPathTraversalDetected,
},
{
name: "path with double dots",
path: "/home/../etc/passwd",
expectedError: testPathTraversalDetected,
},
{
name: "non-existent path",
path: "/definitely/does/not/exist",
expectedError: "does not exist",
},
{
name: "file instead of directory",
path: tempFile,
expectedError: "must be a directory",
},
{
name: "valid directory",
path: tempDir,
expectedError: "",
},
{
name: "valid relative path",
path: ".",
expectedError: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateSourcePath(tt.path)
if tt.expectedError != "" {
assert.Error(t, err)
assert.Contains(t, err.Error(), tt.expectedError)
// Check if it's a StructuredError
var structErr *StructuredError
if errors.As(err, &structErr) {
assert.NotEmpty(t, structErr.Code)
assert.NotEqual(t, ErrorTypeUnknown, structErr.Type)
}
} else {
assert.NoError(t, err)
}
})
}
}
func TestValidateDestinationPath(t *testing.T) {
tempDir := t.TempDir()
tests := []struct {
name string
path string
expectedError string
}{
{
name: testEmptyPath,
path: "",
expectedError: "destination path is required",
},
{
name: testPathTraversalAttempt,
path: "../../etc/passwd",
expectedError: testPathTraversalDetected,
},
{
name: "absolute path traversal",
path: "/home/../../../etc/passwd",
expectedError: testPathTraversalDetected,
},
{
name: "valid new file",
path: filepath.Join(tempDir, "newfile.txt"),
expectedError: "",
},
{
name: "valid relative path",
path: "output.txt",
expectedError: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateDestinationPath(tt.path)
if tt.expectedError != "" {
assert.Error(t, err)
assert.Contains(t, err.Error(), tt.expectedError)
} else {
assert.NoError(t, err)
}
})
}
}
func TestValidateConfigPath(t *testing.T) {
tempDir := t.TempDir()
validConfig := filepath.Join(tempDir, "config.yaml")
require.NoError(t, os.WriteFile(validConfig, []byte("key: value"), 0o600))
tests := []struct {
name string
path string
expectedError string
}{
{
name: testEmptyPath,
path: "",
expectedError: "", // Empty config path is allowed
},
{
name: testPathTraversalAttempt,
path: "../../../etc/config.yaml",
expectedError: testPathTraversalDetected,
},
// ValidateConfigPath doesn't check if file exists or is regular file
// It only checks for path traversal
{
name: "valid config file",
path: validConfig,
expectedError: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateConfigPath(tt.path)
if tt.expectedError != "" {
assert.Error(t, err)
assert.Contains(t, err.Error(), tt.expectedError)
} else {
assert.NoError(t, err)
}
})
}
}
// TestGetAbsolutePath is already covered in paths_test.go
func TestValidationErrorTypes(t *testing.T) {
t.Run("source path validation errors", func(t *testing.T) {
// Test empty source
err := ValidateSourcePath("")
assert.Error(t, err)
var structErrEmptyPath *StructuredError
if errors.As(err, &structErrEmptyPath) {
assert.Equal(t, ErrorTypeValidation, structErrEmptyPath.Type)
assert.Equal(t, CodeValidationRequired, structErrEmptyPath.Code)
}
// Test path traversal
err = ValidateSourcePath("../../../etc")
assert.Error(t, err)
var structErrTraversal *StructuredError
if errors.As(err, &structErrTraversal) {
assert.Equal(t, ErrorTypeValidation, structErrTraversal.Type)
assert.Equal(t, CodeValidationPath, structErrTraversal.Code)
}
})
t.Run("destination path validation errors", func(t *testing.T) {
// Test empty destination
err := ValidateDestinationPath("")
assert.Error(t, err)
var structErrEmptyDest *StructuredError
if errors.As(err, &structErrEmptyDest) {
assert.Equal(t, ErrorTypeValidation, structErrEmptyDest.Type)
assert.Equal(t, CodeValidationRequired, structErrEmptyDest.Code)
}
})
t.Run("config path validation errors", func(t *testing.T) {
// Test path traversal in config
err := ValidateConfigPath("../../etc/config.yaml")
assert.Error(t, err)
var structErrTraversalInConfig *StructuredError
if errors.As(err, &structErrTraversalInConfig) {
assert.Equal(t, ErrorTypeValidation, structErrTraversalInConfig.Type)
assert.Equal(t, CodeValidationPath, structErrTraversalInConfig.Code)
}
})
}
func TestPathSecurityChecks(t *testing.T) {
// Test various path traversal attempts
traversalPaths := []string{
"../etc/passwd",
"../../root/.ssh/id_rsa",
"/home/../../../etc/shadow",
"./../../sensitive/data",
"foo/../../../bar",
}
for _, path := range traversalPaths {
t.Run("source_"+path, func(t *testing.T) {
err := ValidateSourcePath(path)
assert.Error(t, err)
assert.Contains(t, err.Error(), testPathTraversal)
})
t.Run("dest_"+path, func(t *testing.T) {
err := ValidateDestinationPath(path)
assert.Error(t, err)
assert.Contains(t, err.Error(), testPathTraversal)
})
t.Run("config_"+path, func(t *testing.T) {
err := ValidateConfigPath(path)
assert.Error(t, err)
assert.Contains(t, err.Error(), testPathTraversal)
})
}
}
func TestSpecialPaths(t *testing.T) {
t.Run("GetBaseName with special paths", func(t *testing.T) {
specialPaths := map[string]string{
"/": "/",
"": "output",
".": "output",
"..": "..",
"/.": "output", // filepath.Base("/.") returns "." which matches the output condition
"/..": "..",
"//": "/",
"///": "/",
}
for path, expected := range specialPaths {
result := GetBaseName(path)
assert.Equal(t, expected, result, "Path: %s", path)
}
})
}
func TestPathNormalization(t *testing.T) {
tempDir := t.TempDir()
t.Run("source path normalization", func(t *testing.T) {
// Create nested directory
nestedDir := filepath.Join(tempDir, "a", "b", "c")
require.NoError(t, os.MkdirAll(nestedDir, 0o750))
// Test path with redundant separators
redundantPath := tempDir + string(
os.PathSeparator,
) + string(
os.PathSeparator,
) + "a" + string(
os.PathSeparator,
) + "b" + string(
os.PathSeparator,
) + "c"
err := ValidateSourcePath(redundantPath)
assert.NoError(t, err)
})
}
func TestPathValidationConcurrency(t *testing.T) {
tempDir := t.TempDir()
// Test concurrent path validation
paths := []string{
tempDir,
".",
"/tmp",
}
errChan := make(chan error, len(paths)*2)
for _, path := range paths {
go func(p string) {
errChan <- ValidateSourcePath(p)
}(path)
go func(p string) {
errChan <- ValidateDestinationPath(p + "/output.txt")
}(path)
}
// Collect results
for i := 0; i < len(paths)*2; i++ {
<-errChan
}
// No assertions needed - test passes if no panic/race
}

Some files were not shown because too many files have changed in this diff Show More