File System Indexer CLI

Improved prompt:

Design and implement a high-speed file system indexer and search engine in the Go programming language, targeted at large-scale enterprise environments with millions of files and directories. Your primary objective is to create a system that can efficiently index and search file metadata and content across complex, nested directory structures.

Key requirements and constraints:

1. Directory Traversal: Implement recursive directory traversal with a configurable maximum depth of up to 20 levels. The system must efficiently handle nested directories without exceeding available memory.

2. Metadata Extraction: Develop a module to extract and store detailed file metadata including:

   • File size
   • Creation, modification, and access timestamps
   • File permissions and ownership
   • File type (regular, directory, symbolic link, etc.)

3. Content Indexing: Create an indexing engine that can:

   • Parse and index file contents efficiently, supporting common file formats like text, PDF, and Microsoft Office documents
   • Implement optional full-text search functionality
   • Handle binary files by extracting relevant metadata (e.g., file type, size)

4. Query Syntax: Develop an advanced query syntax that supports:

   • Boolean operators (AND, OR, NOT)
   • Wildcard matching (* to match any sequence of characters)
   • File metadata queries (e.g., size:>10MB, modified:>2022-01-01)

5. Incremental Indexing: Implement a mechanism to efficiently update the index when files are added, modified, or deleted, minimizing the need for full rebuilds.

6. Export Formats: Add functionality to export search results and metadata in both JSON and CSV formats, with customizable field selection.

7. Search Result Highlighting: Provide textual highlighting of matched search terms within the content of indexed files.

8. Duplicate File Detection: Develop a module using checksums (e.g., SHA-256 or MD5) to identify and report duplicate files based on content similarity.

9. Performance Statistics: Implement real-time monitoring of system performance including indexing speed, search latency, and memory usage.

10. Concurrent Processing: Optimize the system for multi-core utilization by implementing parallel processing in both indexing and search operations.

The final system should provide a command-line interface and a well-documented API for integration with other enterprise tools. It should be highly configurable through environment variables, configuration files, or flags.

Please provide a detailed architecture overview, key algorithm choices, and sample code snippets for the most complex components. Assume the AI has advanced knowledge of Go programming and file system internals.