gibidify/README.md

gibidify

gibidify is a CLI application written in Go that scans a source directory recursively and aggregates code files into a single text file. The output file is designed for use with LLM models and includes a prefix, file sections with separators, and a suffix.

Features

• Recursive directory scanning with smart file filtering
• Configurable file type detection - add/remove extensions and languages
• Multiple output formats - markdown, JSON, YAML
• Memory-optimized processing - streaming for large files, intelligent back-pressure
• Concurrent processing with configurable worker pools
• Comprehensive configuration via YAML with validation
• Production-ready with structured error handling and benchmarking
• Modular architecture - clean, focused codebase (92 files, ~21.5K lines) with ~63ns registry lookups
• Enhanced CLI experience - progress bars, colored output, helpful error messages
• Cross-platform with Docker support
• Advanced template system - 4 built-in templates (default, minimal, detailed, compact) with custom template support, variable substitution, and YAML-based configuration
• Comprehensive metrics and profiling - real-time processing statistics, performance analysis, memory usage tracking, and automated recommendations

Installation

Clone the repository and build the application:

git clone https://github.com/ivuorinen/gibidify.git
cd gibidify
go build -o gibidify .

Usage

./gibidify \
  -source <source_directory> \
  -destination <output_file> \
  -format markdown|json|yaml \
  -concurrency <num_workers> \
  --prefix="..." \
  --suffix="..." \
  --no-colors \
  --no-progress \
  --verbose \
  --log-level debug

Flags:

• -source: directory to scan.
• -destination: output file path (optional; defaults to <source>.<format>).
• -format: output format (markdown, json, or yaml).
• -concurrency: number of concurrent workers.
• --prefix / --suffix: optional text blocks.
• --no-colors: disable colored terminal output.
• --no-progress: disable progress bars.
• --verbose: enable verbose output and detailed logging.
• --log-level: set log level (default: warn; accepted values: debug, info, warn, error).

Docker

A Docker image can be built using the provided Dockerfile:

docker build -t ghcr.io/ivuorinen/gibidify:<tag> .

Run the Docker container:

docker run --rm \
  -v $(pwd):/workspace \
  -v $HOME/.config/gibidify:/config \
  ghcr.io/ivuorinen/gibidify:<tag> \
  -source /workspace/your_source_directory \
  -destination /workspace/output.txt \
  --prefix="Your prefix text" \
  --suffix="Your suffix text"

Configuration

gibidify supports a YAML configuration file. Place it at:

• $XDG_CONFIG_HOME/gibidify/config.yaml or
• $HOME/.config/gibidify/config.yaml or
• in the folder you run the application from.

Example configuration:

fileSizeLimit: 5242880  # 5 MB
ignoreDirectories:
  - vendor
  - node_modules
  - .git
  - dist
  - build
  - target

# FileType customization
fileTypes:
  enabled: true
  # Add custom file extensions
  customImageExtensions:
    - .webp
    - .avif
  customBinaryExtensions:
    - .custom
  customLanguages:
    .zig: zig
    .odin: odin
    .v: vlang
  # Disable default extensions
  disabledImageExtensions:
    - .bmp
  disabledBinaryExtensions:
    - .exe
  disabledLanguageExtensions:
    - .bat

# Memory optimization (back-pressure management)
backpressure:
  enabled: true
  maxPendingFiles: 1000      # Max files in file channel buffer
  maxPendingWrites: 100      # Max writes in write channel buffer
  maxMemoryUsage: 104857600  # 100MB max memory usage
  memoryCheckInterval: 1000  # Check memory every 1000 files

# Output and template customization
output:
  # Template selection: default, minimal, detailed, compact, or custom
  # Templates control output structure and formatting
  template: "default"
  # Metadata options
  metadata:
    includeStats: true
    includeTimestamp: true
    includeFileCount: true
    includeSourcePath: true
    includeMetrics: true
  # Markdown-specific options
  markdown:
    useCodeBlocks: true
    includeLanguage: true
    headerLevel: 2
    tableOfContents: false
    useCollapsible: false
    syntaxHighlighting: true
    lineNumbers: false
  # Custom template variables
  variables:
    project_name: "My Project"
    author: "Developer Name"
    version: "1.0.0"

See config.example.yaml for a comprehensive configuration example.

License

This project is licensed under the MIT License.