Rewriting the Silver Searcher in Rust

I rewrote the core search path of The Silver Searcher in Rust and benchmarked that rewrite against the original tool. On the only measured performance workload, rust-ag was roughly 2x faster than ag.

That was the honest headline, but not the whole story. rg and ugrep were still faster, the performance data covered only 1 of 38 registered scenarios, and the overall claim gate still returned fail because the reproducibility bundle validation step was not finished. So this was a promising first result, not a victory lap.

The result in one view

The measured workload was the literal-simple scenario: search for "foo" across the repository working tree.

The key comparison is ag versus rust-ag. That pair was checked for output parity. The cross-tool numbers provide context but are not the same kind of comparison because rg and ugrep make different default choices about traversal and ignore handling.

What was rewritten

The Rust rewrite replaces ag's 23 C source files (~6,098 lines) with 6 Rust modules (~3,128 lines) -- 51% fewer lines for equivalent search functionality. The architectural decisions were deliberate:

The single-threaded design was a deliberate scope cut. ag's pthreads pool adds complexity but the benchmark runs both tools with --workers=1 for fairness, so the comparison measures search + I/O, not thread scheduling.

The search core

The heart of the rewrite is search_text_multiline(), which faithfully reproduces ag's print state machine. ag walks the input byte by byte, tracking whether the current position is inside a match region, to determine which lines to emit as matches versus context:

fn search_text_multiline(
    text: &str, re: &Regex, opts: &Opts,
) -> Vec<Match> {
    // First pass: collect match regions, limited by max_count
    let mut regions = Vec::new();
    let mut search_start = 0;
    while let Some(m) = re.find_at(text, search_start) {
        regions.push(MatchRegion {
            start: m.start(), end: m.end(),
        });
        if regions.len() >= opts.max_count { break; }
        if m.start() == m.end() {
            search_start = next_char_boundary(text, m.end());
        } else {
            search_start = m.end();
        }
    }
 
    // Second pass: simulate ag's byte-by-byte print state machine
    // to determine which lines are matches vs context
    // ...
}

The two-pass approach separates regex matching from line attribution. The first pass collects all match regions using the regex crate's find_at for anchored searching at each position. The second pass walks the byte stream to map those regions onto line boundaries -- the same state machine ag uses in its print_file_matches function.

The ignore engine

ag's ignore handling is one of its trickiest subsystems. The Rust rewrite uses a stack-based IgnoreEngine that pushes a new rule scope when entering a directory and pops it when leaving:

This prevents scope leakage -- a .gitignore rule in src/ should not affect files in tests/. ag's C implementation uses the same conceptual approach but with a different data structure. The stack-based design makes the Rust version easier to verify: each directory's ignore rules are an isolated frame, and the frame lifetime matches the directory traversal lifetime.

The ignore engine handles .gitignore, .hgignore, .ignore, --ignore patterns, hidden file rules, and VCS directory skipping. 24 parity tests verify that rust-ag and ag agree on which files to include or exclude.

Correctness

Before looking at speed, I checked whether the rewrite still behaves like ag. The test suite includes 214 Rust tests covering the full search surface:

The parity tests run rust-ag and ag on identical inputs and assert matching stdout, stderr, and exit codes. The 38 declared scenarios in manifests/scenarios.json define the full test surface across literal search, regex search, CLI behavior, edge cases, and exit code verification.

Where the 2x comes from

The regex engine swap is the primary contributor. ag uses PCRE (Perl-Compatible Regular Expressions), a backtracking engine written in C. rust-ag uses the regex crate, which is built on a Thompson NFA with lazy DFA optimization.

For the literal-simple scenario (search for "foo"), the regex crate detects the literal pattern and uses Aho-Corasick or memchr for the search. This is a fast-path that PCRE does not have for this pattern class -- PCRE compiles the pattern into bytecode and interprets it, even when the pattern contains no metacharacters. The difference is significant at scale: scanning a working tree involves matching the pattern against thousands of files, and the per-match overhead compounds.

The I/O path also differs. ag uses mmap for file access, which avoids explicit read syscalls but involves page fault handling and TLB pressure. rust-ag uses std::fs::read_to_string, which does a buffered read syscall. For the file sizes in this working tree (mostly small source files), the buffered read is competitive with mmap because the kernel's readahead heuristic effectively prefetches the data.

That said, both rg and ugrep are still faster. rg uses the same regex crate but adds parallelism, a custom directory walker, and a memory-mapped searcher. ugrep uses a SIMD-accelerated DFA with its own parallel traversal. rust-ag's single-threaded, non-SIMD search is a baseline that validates the regex engine advantage, not the ceiling of what Rust can do.

Speedup across run types

The 2x result was not a one-off. Across local, nightly, and manual runs, rust-ag stayed between 1.96x and 2.03x faster than ag.

The ranking stayed stable: ugrep > rg > rust-ag > ag.

Limitations

1. One of 38 scenarios. Only literal-simple has measured performance data. A regex-heavy, output-heavy, or large-file workload could change the ranking shape.

2. Three samples per tool. Each tool has 3 measured samples after 1 warmup iteration. That is enough for a first read but not enough to compute tight confidence intervals.

3. Single machine. Everything ran on one Apple M4, 16 GiB RAM, macOS Darwin 25.3.0. Linux, x86_64, or a different filesystem could move the numbers.

4. Claim gate still failed. The parity checks, threshold checks, and cross-run agreement passed individually, but the full gate stayed red because the reproducibility bundle validation step is incomplete.

5. Single-threaded rewrite. rust-ag accepts but ignores --workers / --parallel flags. The comparison is fair (both use --workers=1) but does not reflect ag's multi-threaded potential.

6. No committed evidence artifacts. The benchmark infrastructure generates evidence at runtime but does not commit timing results. Numbers are reproducible on-demand via run_matrix.py, not from stored artifacts.

How this was built

The implementation was built using Factory mission mode. The mission system planned the rewrite across milestones (search core, ignore engine, CLI surface, parity tests, benchmark harness), ran worker sessions for each feature, and executed scrutiny and parity validators after every step.

The test-to-source ratio is 1.6:1 by line count (4,923 test lines vs 3,128 source lines across 14 Rust files). An additional 7 Python test scripts in scripts/parity/ validate manifest integrity, fixture hashes, and stderr parity, providing a second layer of verification outside the Rust test harness.

Reproducibility

git clone --branch mission/rust-rewrite-benchmark-publication \
  https://github.com/sagaragas/the_silver_searcher.git
cd the_silver_searcher
 
# Build both
./build.sh              # C ag
cargo build --workspace # Rust ag
 
# Run parity tests
python3 tests/edge-cases/setup_fixtures.py
cargo test --workspace -- --test-threads=5
python3 -m pytest scripts/parity -v
 
# Run benchmark scenarios
python3 scripts/parity/run_matrix.py --target rust --group smoke
python3 scripts/parity/validate_outputs.py --run latest

The 38 scenarios are defined in manifests/scenarios.json with cross-linked corpus hashes (manifests/corpus.json, 126 files with SHA-256) and query patterns (manifests/queries.json, 13 patterns). The manifest integrity chain ensures that changes to the corpus or queries invalidate the scenario hashes.

Source

• Published repository branch
• Rust rewrite workspace
• Parity tooling
• Scenario and fixture manifests