ivuorinen/gibidify
1
0
Fork 0
mirror of https://github.com/ivuorinen/gibidify.git synced 2026-02-06 14:46:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: modernize workflows, security scanning, and linting configuration (#50)

Browse Source 
* build: update Go 1.25, CI workflows, and build tooling

- Upgrade to Go 1.25
- Add benchmark targets to Makefile
- Implement parallel gosec execution
- Lock tool versions for reproducibility
- Add shellcheck directives to scripts
- Update CI workflows with improved caching

* refactor: migrate from golangci-lint to revive

- Replace golangci-lint with revive for linting
- Configure comprehensive revive rules
- Fix all EditorConfig violations
- Add yamllint and yamlfmt support
- Remove deprecated .golangci.yml

* refactor: rename utils to shared and deduplicate code

- Rename utils package to shared
- Add shared constants package
- Deduplicate constants across packages
- Address CodeRabbit review feedback

* fix: resolve SonarQube issues and add safety guards

- Fix all 73 SonarQube OPEN issues
- Add nil guards for resourceMonitor, backpressure, metricsCollector
- Implement io.Closer for headerFileReader
- Propagate errors from processing helpers
- Add metrics and templates packages
- Improve error handling across codebase

* test: improve test infrastructure and coverage

- Add benchmarks for cli, fileproc, metrics
- Improve test coverage for cli, fileproc, config
- Refactor tests with helper functions
- Add shared test constants
- Fix test function naming conventions
- Reduce cognitive complexity in benchmark tests

* docs: update documentation and configuration examples

- Update CLAUDE.md with current project state
- Refresh README with new features
- Add usage and configuration examples
- Add SonarQube project configuration
- Consolidate config.example.yaml

* fix: resolve shellcheck warnings in scripts

- Use ./*.go instead of *.go to prevent dash-prefixed filenames
  from being interpreted as options (SC2035)
- Remove unreachable return statement after exit (SC2317)
- Remove obsolete gibidiutils/ directory reference

* chore(deps): upgrade go dependencies

* chore(lint): megalinter fixes

* fix: improve test coverage and fix file descriptor leaks

- Add defer r.Close() to fix pipe file descriptor leaks in benchmark tests
- Refactor TestProcessorConfigureFileTypes with helper functions and assertions
- Refactor TestProcessorLogFinalStats with output capture and keyword verification
- Use shared constants instead of literal strings (TestFilePNG, FormatMarkdown, etc.)
- Reduce cognitive complexity by extracting helper functions

* fix: align test comments with function names

Remove underscores from test comments to match actual function names:
- benchmark/benchmark_test.go (2 fixes)
- fileproc/filetypes_config_test.go (4 fixes)
- fileproc/filetypes_registry_test.go (6 fixes)
- fileproc/processor_test.go (6 fixes)
- fileproc/resource_monitor_types_test.go (4 fixes)
- fileproc/writer_test.go (3 fixes)

* fix: various test improvements and bug fixes

- Remove duplicate maxCacheSize check in filetypes_registry_test.go
- Shorten long comment in processor_test.go to stay under 120 chars
- Remove flaky time.Sleep in collector_test.go, use >= 0 assertion
- Close pipe reader in benchmark_test.go to fix file descriptor leak
- Use ContinueOnError in flags_test.go to match ResetFlags behavior
- Add nil check for p.ui in processor_workers.go before UpdateProgress
- Fix resource_monitor_validation_test.go by setting hardMemoryLimitBytes directly

* chore(yaml): add missing document start markers

Add --- document start to YAML files to satisfy yamllint:
- .github/workflows/codeql.yml
- .github/workflows/build-test-publish.yml
- .github/workflows/security.yml
- .github/actions/setup/action.yml

* fix: guard nil resourceMonitor and fix test deadlock

- Guard resourceMonitor before CreateFileProcessingContext call
- Add ui.UpdateProgress on emergency stop and path error returns
- Fix potential deadlock in TestProcessFile using wg.Go with defer close
This commit is contained in:
Ismo Vuorinen
2025-12-10 19:07:11 +02:00
committed by GitHub
parent ea4a39a360
commit 95b7ef6dd3
149 changed files with 22990 additions and 8976 deletions
Download Patch File Download Diff File Expand all files Collapse all files

219
examples/basic-usage.md Normal file
View File

@@ -0,0 +1,219 @@
# Basic Usage Examples
This directory contains practical examples of how to use gibidify for various use cases.
## Simple Code Aggregation
The most basic use case - aggregate all code files from a project into a single output:
```bash
# Aggregate all files from current directory to markdown
gibidify -source . -format markdown -destination output.md
# Aggregate specific directory to JSON
gibidify -source ./src -format json -destination code-dump.json
# Aggregate with custom worker count
gibidify -source ./project -format yaml -destination project.yaml -concurrency 8
```
## With Configuration File
For repeatable processing with custom settings:
1. Copy the configuration example:
```bash
cp config.example.yaml ~/.config/gibidify/config.yaml
```
2. Edit the configuration file to your needs, then run:
```bash
gibidify -source ./my-project
```
## Output Formats
### JSON Output
Best for programmatic processing and data analysis:
```bash
gibidify -source ./src -format json -destination api-code.json
```
Example JSON structure:
```json
{
"files": [
{
"path": "src/main.go",
"content": "package main...",
"language": "go",
"size": 1024
}
],
"metadata": {
"total_files": 15,
"total_size": 45678,
"processing_time": "1.2s"
}
}
```
### Markdown Output
Great for documentation and code reviews:
```bash
gibidify -source ./src -format markdown -destination code-review.md
```
### YAML Output
Structured and human-readable:
```bash
gibidify -source ./config -format yaml -destination config-dump.yaml
```
## Advanced Usage Examples
### Large Codebase Processing
For processing large projects with performance optimizations:
```bash
gibidify -source ./large-project \
-format json \
-destination large-output.json \
-concurrency 16 \
--verbose
```
### Memory-Conscious Processing
For systems with limited memory:
```bash
gibidify -source ./project \
-format markdown \
-destination output.md \
-concurrency 4
```
### Filtered Processing
Process only specific file types (when configured):
```bash
# Configure file patterns in config.yaml
filePatterns:
- "*.go"
- "*.py"
- "*.js"
# Then run
gibidify -source ./mixed-project -destination filtered.json
```
### CI/CD Integration
For automated documentation generation:
```bash
# In your CI pipeline
gibidify -source . \
-format markdown \
-destination docs/codebase.md \
--no-colors \
--no-progress \
-concurrency 2
```
## Error Handling
### Graceful Failure Handling
The tool handles common issues gracefully:
```bash
# This will fail gracefully if source doesn't exist
gibidify -source ./nonexistent -destination out.json
# This will warn about permission issues but continue
gibidify -source ./restricted-dir -destination out.md --verbose
```
### Resource Limits
Configure resource limits in your config file:
```yaml
resourceLimits:
enabled: true
maxFiles: 5000
maxTotalSize: 1073741824 # 1GB
fileProcessingTimeoutSec: 30
overallTimeoutSec: 1800 # 30 minutes
hardMemoryLimitMB: 512
```
## Performance Tips
1. **Adjust Concurrency**: Start with number of CPU cores, adjust based on I/O vs CPU bound work
2. **Use Appropriate Format**: JSON is fastest, Markdown has more overhead
3. **Configure File Limits**: Set reasonable limits in config.yaml for your use case
4. **Monitor Memory**: Use `--verbose` to see memory usage during processing
5. **Use Progress Indicators**: Enable progress bars for long-running operations
## Integration Examples
### With Git Hooks
Create a pre-commit hook to generate code documentation:
```bash
#!/bin/sh
# .git/hooks/pre-commit
gibidify -source . -format markdown -destination docs/current-code.md
git add docs/current-code.md
```
### With Make
Add to your Makefile:
```makefile
.PHONY: code-dump
code-dump:
gibidify -source ./src -format json -destination dist/codebase.json
.PHONY: docs
docs:
gibidify -source . -format markdown -destination docs/codebase.md
```
### Docker Usage
```dockerfile
FROM golang:1.25-alpine
RUN go install github.com/ivuorinen/gibidify@latest
WORKDIR /workspace
COPY . .
RUN gibidify -source . -format json -destination /output/codebase.json
```
## Common Use Cases
### 1. Code Review Preparation
```bash
gibidify -source ./feature-branch -format markdown -destination review.md
```
### 2. AI Code Analysis
```bash
gibidify -source ./src -format json -destination ai-input.json
```
### 3. Documentation Generation
```bash
gibidify -source ./lib -format markdown -destination api-docs.md
```
### 4. Backup Creation
```bash
gibidify -source ./project -format yaml -destination backup-$(date +%Y%m%d).yaml
```
### 5. Code Migration Prep
```bash
gibidify -source ./legacy-code -format json -destination migration-analysis.json
```

469
examples/configuration-examples.md Normal file
View File

@@ -0,0 +1,469 @@
# Configuration Examples
This document provides practical configuration examples for different use cases.
## Basic Configuration
Create `~/.config/gibidify/config.yaml`:
```yaml
# Basic setup for most projects
fileSizeLimit: 5242880 # 5MB per file
maxConcurrency: 8
ignoreDirectories:
- vendor
- node_modules
- .git
- dist
- target
# Enable file type detection
fileTypes:
enabled: true
```
## Development Environment Configuration
Optimized for active development with fast feedback:
```yaml
# ~/.config/gibidify/config.yaml
fileSizeLimit: 1048576 # 1MB - smaller files for faster processing
ignoreDirectories:
- vendor
- node_modules
- .git
- dist
- build
- tmp
- cache
- .vscode
- .idea
# Conservative resource limits for development
resourceLimits:
enabled: true
maxFiles: 1000
maxTotalSize: 104857600 # 100MB
fileProcessingTimeoutSec: 10
overallTimeoutSec: 300 # 5 minutes
maxConcurrentReads: 4
hardMemoryLimitMB: 256
# Fast backpressure for responsive development
backpressure:
enabled: true
maxPendingFiles: 500
maxPendingWrites: 50
maxMemoryUsage: 52428800 # 50MB
memoryCheckInterval: 100
# Simple output for quick reviews
output:
metadata:
includeStats: true
includeTimestamp: true
```
## Production/CI Configuration
High-performance setup for automated processing:
```yaml
# Production configuration
fileSizeLimit: 10485760 # 10MB per file
maxConcurrency: 16
ignoreDirectories:
- vendor
- node_modules
- .git
- dist
- build
- target
- tmp
- cache
- coverage
- .nyc_output
- __pycache__
# High-performance resource limits
resourceLimits:
enabled: true
maxFiles: 50000
maxTotalSize: 10737418240 # 10GB
fileProcessingTimeoutSec: 60
overallTimeoutSec: 7200 # 2 hours
maxConcurrentReads: 20
hardMemoryLimitMB: 2048
# High-throughput backpressure
backpressure:
enabled: true
maxPendingFiles: 5000
maxPendingWrites: 500
maxMemoryUsage: 1073741824 # 1GB
memoryCheckInterval: 1000
# Comprehensive output for analysis
output:
metadata:
includeStats: true
includeTimestamp: true
includeFileCount: true
includeSourcePath: true
includeFileTypes: true
includeProcessingTime: true
includeTotalSize: true
includeMetrics: true
```
## Security-Focused Configuration
Restrictive settings for untrusted input:
```yaml
# Security-first configuration
fileSizeLimit: 1048576 # 1MB maximum
ignoreDirectories:
- "**/.*" # All hidden directories
- vendor
- node_modules
- tmp
- temp
- cache
# Strict resource limits
resourceLimits:
enabled: true
maxFiles: 100 # Very restrictive
maxTotalSize: 10485760 # 10MB total
fileProcessingTimeoutSec: 5
overallTimeoutSec: 60 # 1 minute max
maxConcurrentReads: 2
rateLimitFilesPerSec: 10 # Rate limiting enabled
hardMemoryLimitMB: 128 # Low memory limit
# Conservative backpressure
backpressure:
enabled: true
maxPendingFiles: 50
maxPendingWrites: 10
maxMemoryUsage: 10485760 # 10MB
memoryCheckInterval: 10 # Frequent checks
# Minimal file type detection
fileTypes:
enabled: true
# Disable potentially risky file types
disabledLanguageExtensions:
- .bat
- .cmd
- .ps1
- .sh
disabledBinaryExtensions:
- .exe
- .dll
- .so
```
## Language-Specific Configuration
### Go Projects
```yaml
fileSizeLimit: 5242880
ignoreDirectories:
- vendor
- .git
- bin
- pkg
fileTypes:
enabled: true
customLanguages:
.mod: go-mod
.sum: go-sum
filePatterns:
- "*.go"
- "go.mod"
- "go.sum"
- "*.md"
```
### JavaScript/Node.js Projects
```yaml
fileSizeLimit: 2097152 # 2MB
ignoreDirectories:
- node_modules
- .git
- dist
- build
- coverage
- .nyc_output
fileTypes:
enabled: true
customLanguages:
.vue: vue
.svelte: svelte
.astro: astro
filePatterns:
- "*.js"
- "*.ts"
- "*.jsx"
- "*.tsx"
- "*.vue"
- "*.json"
- "*.md"
```
### Python Projects
```yaml
fileSizeLimit: 5242880
ignoreDirectories:
- .git
- __pycache__
- .pytest_cache
- venv
- env
- .env
- dist
- build
- .tox
fileTypes:
enabled: true
customLanguages:
.pyi: python-interface
.ipynb: jupyter-notebook
filePatterns:
- "*.py"
- "*.pyi"
- "requirements*.txt"
- "*.toml"
- "*.cfg"
- "*.ini"
- "*.md"
```
## Output Format Configurations
### Detailed Markdown Output
```yaml
output:
template: "detailed"
metadata:
includeStats: true
includeTimestamp: true
includeFileCount: true
includeSourcePath: true
includeFileTypes: true
includeProcessingTime: true
markdown:
useCodeBlocks: true
includeLanguage: true
headerLevel: 2
tableOfContents: true
syntaxHighlighting: true
lineNumbers: true
maxLineLength: 120
variables:
project_name: "My Project"
author: "Development Team"
version: "1.0.0"
```
### Compact JSON Output
```yaml
output:
template: "minimal"
metadata:
includeStats: true
includeFileCount: true
```
### Custom Template Output
```yaml
output:
template: "custom"
custom:
header: |
# {{ .ProjectName }} Code Dump
Generated: {{ .Timestamp }}
Total Files: {{ .FileCount }}
footer: |
---
Processing completed in {{ .ProcessingTime }}
fileHeader: |
## {{ .Path }}
Language: {{ .Language }} | Size: {{ .Size }} bytes
fileFooter: ""
variables:
project_name: "Custom Project"
```
## Environment-Specific Configurations
### Docker Container
```yaml
# Optimized for containerized environments
fileSizeLimit: 5242880
maxConcurrency: 4 # Conservative for containers
resourceLimits:
enabled: true
hardMemoryLimitMB: 512
maxFiles: 5000
overallTimeoutSec: 1800
backpressure:
enabled: true
maxMemoryUsage: 268435456 # 256MB
```
### GitHub Actions
```yaml
# CI/CD optimized configuration
fileSizeLimit: 2097152
maxConcurrency: 2 # Conservative for shared runners
ignoreDirectories:
- .git
- .github
- node_modules
- vendor
- dist
- build
resourceLimits:
enabled: true
maxFiles: 2000
overallTimeoutSec: 900 # 15 minutes
hardMemoryLimitMB: 1024
```
### Local Development
```yaml
# Developer-friendly settings
fileSizeLimit: 10485760 # 10MB
maxConcurrency: 8
# Show progress and verbose output
output:
metadata:
includeStats: true
includeTimestamp: true
includeProcessingTime: true
includeMetrics: true
markdown:
useCodeBlocks: true
includeLanguage: true
syntaxHighlighting: true
```
## Template Examples
### Custom API Documentation Template
```yaml
output:
template: "custom"
custom:
header: |
# {{ .Variables.api_name }} API Documentation
Version: {{ .Variables.version }}
Generated: {{ .Timestamp }}
## Overview
This document contains the complete source code for the {{ .Variables.api_name }} API.
## Statistics
- Total Files: {{ .FileCount }}
- Total Size: {{ .TotalSize | formatSize }}
- Processing Time: {{ .ProcessingTime }}
---
fileHeader: |
### {{ .Path }}
**Type:** {{ .Language }}
**Size:** {{ .Size | formatSize }}
```{{ .Language }}
fileFooter: |
```
---
footer: |
## Summary
Documentation generated with [gibidify](https://github.com/ivuorinen/gibidify)
variables:
api_name: "My API"
version: "v1.2.3"
```
### Code Review Template
```yaml
output:
template: "custom"
custom:
header: |
# Code Review: {{ .Variables.pr_title }}
**PR Number:** #{{ .Variables.pr_number }}
**Author:** {{ .Variables.author }}
**Date:** {{ .Timestamp }}
## Files Changed ({{ .FileCount }})
fileHeader: |
## 📄 {{ .Path }}
<details>
<summary>{{ .Language | upper }} • {{ .Size | formatSize }}</summary>
```{{ .Language }}
fileFooter: |
```
</details>
footer: |
---
**Review Summary:**
- Files reviewed: {{ .FileCount }}
- Total size: {{ .TotalSize | formatSize }}
- Generated in: {{ .ProcessingTime }}
variables:
pr_title: "Feature Implementation"
pr_number: "123"
author: "developer@example.com"
```