mirror of
https://github.com/ivuorinen/gibidify.git
synced 2026-01-26 03:24:05 +00:00
Ismo Vuorinen
95b7ef6dd3
* build: update Go 1.25, CI workflows, and build tooling - Upgrade to Go 1.25 - Add benchmark targets to Makefile - Implement parallel gosec execution - Lock tool versions for reproducibility - Add shellcheck directives to scripts - Update CI workflows with improved caching * refactor: migrate from golangci-lint to revive - Replace golangci-lint with revive for linting - Configure comprehensive revive rules - Fix all EditorConfig violations - Add yamllint and yamlfmt support - Remove deprecated .golangci.yml * refactor: rename utils to shared and deduplicate code - Rename utils package to shared - Add shared constants package - Deduplicate constants across packages - Address CodeRabbit review feedback * fix: resolve SonarQube issues and add safety guards - Fix all 73 SonarQube OPEN issues - Add nil guards for resourceMonitor, backpressure, metricsCollector - Implement io.Closer for headerFileReader - Propagate errors from processing helpers - Add metrics and templates packages - Improve error handling across codebase * test: improve test infrastructure and coverage - Add benchmarks for cli, fileproc, metrics - Improve test coverage for cli, fileproc, config - Refactor tests with helper functions - Add shared test constants - Fix test function naming conventions - Reduce cognitive complexity in benchmark tests * docs: update documentation and configuration examples - Update CLAUDE.md with current project state - Refresh README with new features - Add usage and configuration examples - Add SonarQube project configuration - Consolidate config.example.yaml * fix: resolve shellcheck warnings in scripts - Use ./*.go instead of *.go to prevent dash-prefixed filenames from being interpreted as options (SC2035) - Remove unreachable return statement after exit (SC2317) - Remove obsolete gibidiutils/ directory reference * chore(deps): upgrade go dependencies * chore(lint): megalinter fixes * fix: improve test coverage and fix file descriptor leaks - Add defer r.Close() to fix pipe file descriptor leaks in benchmark tests - Refactor TestProcessorConfigureFileTypes with helper functions and assertions - Refactor TestProcessorLogFinalStats with output capture and keyword verification - Use shared constants instead of literal strings (TestFilePNG, FormatMarkdown, etc.) - Reduce cognitive complexity by extracting helper functions * fix: align test comments with function names Remove underscores from test comments to match actual function names: - benchmark/benchmark_test.go (2 fixes) - fileproc/filetypes_config_test.go (4 fixes) - fileproc/filetypes_registry_test.go (6 fixes) - fileproc/processor_test.go (6 fixes) - fileproc/resource_monitor_types_test.go (4 fixes) - fileproc/writer_test.go (3 fixes) * fix: various test improvements and bug fixes - Remove duplicate maxCacheSize check in filetypes_registry_test.go - Shorten long comment in processor_test.go to stay under 120 chars - Remove flaky time.Sleep in collector_test.go, use >= 0 assertion - Close pipe reader in benchmark_test.go to fix file descriptor leak - Use ContinueOnError in flags_test.go to match ResetFlags behavior - Add nil check for p.ui in processor_workers.go before UpdateProgress - Fix resource_monitor_validation_test.go by setting hardMemoryLimitBytes directly * chore(yaml): add missing document start markers Add --- document start to YAML files to satisfy yamllint: - .github/workflows/codeql.yml - .github/workflows/build-test-publish.yml - .github/workflows/security.yml - .github/actions/setup/action.yml * fix: guard nil resourceMonitor and fix test deadlock - Guard resourceMonitor before CreateFileProcessingContext call - Add ui.UpdateProgress on emergency stop and path error returns - Fix potential deadlock in TestProcessFile using wg.Go with defer close
131 lines
6.6 KiB
Markdown
131 lines
6.6 KiB
Markdown
# TODO: gibidify
|
|
|
|
Prioritized improvements by impact/effort.
|
|
|
|
## ✅ Completed
|
|
|
|
**Core**: Config validation, structured errors, benchmarking, linting (revive: 0 issues) ✅
|
|
**Architecture**: Modularization (92 files, ~21.5K lines), CLI (progress/colors), security (path validation, resource limits, scanning) ✅
|
|
|
|
## 🚀 Critical Priorities
|
|
|
|
### Testing Coverage (URGENT)
|
|
- [x] **CLI module testing** (0% → 83.8%) - COMPLETED ✅
|
|
- [x] cli/flags_test.go - Flag parsing and validation ✅
|
|
- [x] cli/errors_test.go - Error formatting and structured errors ✅
|
|
- [x] cli/ui_test.go - UI components, colors, progress bars ✅
|
|
- [x] cli/processor_test.go - Processing workflow integration ✅
|
|
- [x] **Utils module testing** (7.4% → 90.0%) - COMPLETED ✅
|
|
- [x] utils/writers_test.go - Writer functions (98% complete, minor test fixes needed) ✅
|
|
- [x] Enhanced utils/paths_test.go - Security and edge cases ✅
|
|
- [x] Enhanced utils/errors_test.go - StructuredError system ✅
|
|
- [x] **Testutil module testing** (45.1% → 73.7%) - COMPLETED ✅
|
|
- [x] testutil/utility_test.go - GetBaseName function comprehensive tests ✅
|
|
- [x] testutil/directory_structure_test.go - CreateTestDirectoryStructure and SetupTempDirWithStructure ✅
|
|
- [x] testutil/assertions_test.go - All AssertError functions comprehensive coverage ✅
|
|
- [x] testutil/error_scenarios_test.go - Edge cases and performance benchmarks ✅
|
|
- [x] **Main module testing** (41% → 50.0%) - COMPLETED ✅
|
|
- [x] **Fileproc module improvement** (66% → 74.5%) - COMPLETED ✅
|
|
|
|
### ✅ Metrics & Profiling - COMPLETED
|
|
- [x] **Comprehensive metrics collection system** with processing statistics ✅
|
|
- [x] File processing metrics (processed, skipped, errors) ✅
|
|
- [x] Size metrics (total, average, largest, smallest file sizes) ✅
|
|
- [x] Performance metrics (files/sec, bytes/sec, processing time) ✅
|
|
- [x] Memory and resource tracking (peak memory, current memory, goroutine count) ✅
|
|
- [x] Format-specific metrics and error breakdown ✅
|
|
- [x] Phase timing (collection, processing, writing, finalize) ✅
|
|
- [x] Concurrency tracking and recommendations ✅
|
|
- [x] **Performance measurements and reporting** ✅
|
|
- [x] Real-time progress reporting in CLI ✅
|
|
- [x] Verbose mode with detailed statistics ✅
|
|
- [x] Final comprehensive profiling reports ✅
|
|
- [x] Performance recommendations based on metrics ✅
|
|
- [x] **Structured logging integration** with centralized logging service ✅
|
|
- [x] Configurable log levels (debug, info, warn, error) ✅
|
|
- [x] Context-aware logging with structured data ✅
|
|
- [x] Metrics data integration in log output ✅
|
|
|
|
### ✅ Output Customization - COMPLETED
|
|
- [x] **Template system for output formatting** ✅
|
|
- [x] Builtin templates: default, minimal, detailed, compact ✅
|
|
- [x] Custom template support with variables ✅
|
|
- [x] Template functions for formatting (formatSize, basename, etc.) ✅
|
|
- [x] Header/footer and file header/footer customization ✅
|
|
- [x] **Configurable markdown options** ✅
|
|
- [x] Code block controls (syntax highlighting, line numbers) ✅
|
|
- [x] Header levels and table of contents ✅
|
|
- [x] Collapsible sections for space efficiency ✅
|
|
- [x] Line length limits and long file folding ✅
|
|
- [x] Custom CSS support ✅
|
|
- [x] **Metadata integration in outputs** ✅
|
|
- [x] Configurable metadata inclusion (stats, timestamp, file counts) ✅
|
|
- [x] Processing metrics in output (performance, memory usage) ✅
|
|
- [x] File type breakdown and error summaries ✅
|
|
- [x] Source path and processing time information ✅
|
|
- [x] **Enhanced configuration system** ✅
|
|
- [x] Template selection and customization options ✅
|
|
- [x] Metadata control flags ✅
|
|
- [x] Markdown formatting preferences ✅
|
|
- [x] Custom template variables support ✅
|
|
|
|
### Documentation
|
|
- [ ] API docs, user guides
|
|
|
|
## Guidelines
|
|
|
|
**Before**: `make lint-fix && make lint` (0 issues), >80% coverage
|
|
**Priorities**: Testing → Security → UX → Extensions
|
|
|
|
## Status (2025-08-23 - Phase 3 Feature Implementation Complete)
|
|
|
|
**Health: 10/10** - Advanced metrics & profiling system and comprehensive output customization implemented
|
|
|
|
**Stats**: 92 files (~21.5K lines), 77.9% overall coverage achieved
|
|
- CLI: 83.8% ✅, Utils: 90.0% ✅, Config: 77.0% ✅, Testutil: 73.7% ✅, Fileproc: 74.5% ✅, Main: 50.0% ✅, Metrics: 96.0% ✅, Templates: 87.3% ✅, Benchmark: 64.7% ✅
|
|
|
|
**Completed Today**:
|
|
- ✅ **Phase 1**: Consolidated duplicate code patterns
|
|
- Writer closeReader → utils.SafeCloseReader
|
|
- Custom yamlQuoteString → utils.EscapeForYAML
|
|
- Streaming patterns → utils.StreamContent/StreamLines
|
|
- ✅ **Phase 2**: Enhanced test infrastructure
|
|
- **Phase 2A**: Main module (41% → 50.0%) - Complete integration testing
|
|
- **Phase 2B**: Fileproc module (66% → 74.5%) - Streaming and backpressure testing
|
|
- **Phase 2C**: Testutil module (45.1% → 73.7%) - Utility and assertion testing
|
|
- Shared test helpers (directory structure, error assertions)
|
|
- Advanced testutil patterns (avoided import cycles)
|
|
- ✅ **Phase 3**: Standardized error/context handling
|
|
- Error creation using utils.WrapError family
|
|
- Centralized context cancellation patterns
|
|
- ✅ **Phase 4**: Documentation updates
|
|
|
|
**Impact**: Eliminated code duplication, enhanced maintainability, achieved comprehensive test coverage across all major modules
|
|
|
|
**Completed This Session**:
|
|
- ✅ **Phase 3A**: Advanced Metrics & Profiling System
|
|
- Comprehensive processing statistics collection (files, sizes, performance)
|
|
- Real-time progress reporting with detailed metrics
|
|
- Phase timing tracking (collection, processing, writing, finalize)
|
|
- Memory and resource usage monitoring
|
|
- Format-specific metrics and error breakdown
|
|
- Performance recommendations engine
|
|
- Structured logging integration
|
|
- ✅ **Phase 3B**: Output Customization Features
|
|
- Template system with 4 builtin templates (default, minimal, detailed, compact)
|
|
- Custom template support with variable substitution
|
|
- Configurable markdown options (code blocks, TOC, collapsible sections)
|
|
- Metadata integration with selective inclusion controls
|
|
- Enhanced configuration system for all customization options
|
|
- ✅ **Phase 3C**: Comprehensive Testing & Integration
|
|
- Full test coverage for metrics and templates packages
|
|
- Integration with existing CLI processor workflow
|
|
- Deadlock-free concurrent metrics collection
|
|
- Configuration system extensions
|
|
|
|
**Impact**: Added powerful analytics and customization capabilities while maintaining high code quality and test coverage
|
|
|
|
**Next Session**:
|
|
- Phase 4: Enhanced documentation and user guides
|
|
- Optional: Advanced features (watch mode, incremental processing, etc.)
|