Performance Benchmarks

hx is designed for speed. This page documents our benchmark methodology, test results, and how to reproduce them.

Test Environment

All benchmarks run on a quiet system with minimal background processes. Times are the median of 5 runs after 2 warmup iterations.

Executive Summary

Benchmark Categories

1. CLI Startup Time

Measures how fast the tool responds to simple commands. This affects perceived responsiveness.

--help Response Time

--version Response Time

Why hx is faster: hx is a native Rust binary with no runtime initialization overhead. Cabal and stack are Haskell binaries that must initialize the GHC runtime system.

2. Project Initialization

Creating a new project from scratch.

Binary Project (hx init / cabal init / stack new)

Library Project

Why hx is faster:

• hx generates files from templates without invoking GHC
• stack downloads resolver information and initializes package databases
• cabal runs Haskell code for project generation

3. Build Performance

The core developer experience metric. Test project is a simple 3-module executable:

• Main.hs (15 lines)
• Lib.hs (10 lines)
• Utils.hs (8 lines)
• Single dependency: base

Cold Build (Clean State)

After removing all build artifacts (.hx/, dist-newstyle/).

Incremental Build (No Changes)

Subsequent build with no source modifications.

Incremental Build (Single File Changed)

After modifying one source file.

4. Native Build Mode

hx’s native build mode bypasses cabal entirely for simple projects.

How It Works

1. Direct GHC invocation — hx constructs the module graph and calls GHC directly
2. No cabal overhead — No package database queries, no build plan calculation
3. Aggressive caching — Fingerprint-based caching with minimal I/O
4. Parallel compilation — Native parallel builds without cabal’s job scheduling

When Native Builds Apply

Preprocessor Performance

Native builds handle common preprocessors with minimal overhead:

5. Dependency Resolution

Comparing solver performance for complex dependency graphs.

Real Package Resolution

Resolving dependencies for a project with 20 direct dependencies.

Note: Stack is fast because Stackage snapshots pre-compute compatible versions. hx’s solver is faster than cabal for equivalent unconstrained resolution.

Solver Scaling (Synthetic Benchmark)

Package count vs resolution time (10 versions per package):

6. Clean Operations

Removing build artifacts.

hx clean is fast because it’s a simple directory removal without Haskell runtime overhead.

7. Watch Mode Performance

File change detection and rebuild.

Time from File Save to Rebuild Start

Rebuild Time (Single Module Change)

8. Shell Completions

Generating shell completion scripts.

Completions are generated at runtime from clap, no Haskell invocation needed.

9. Diagnostics (hx doctor)

Environment health check including toolchain detection.

10. Memory Usage

Peak memory consumption during common operations.

Benchmark Implementation

hx uses two benchmark frameworks:

Criterion Benchmarks

Located in crates/hx-solver/benches/ and crates/hx-cli/benches/:

// crates/hx-cli/benches/cli_bench.rs
fn bench_startup(c: &mut Criterion) {
    let mut group = c.benchmark_group("startup");
    group.sample_size(50);

    group.bench_function("hx_help", |b| {
        b.iter(|| {
            Command::new(env!("CARGO_BIN_EXE_hx"))
                .arg("--help")
                .output()
                .expect("hx --help failed")
        })
    });

    // Compare with cabal if available
    if has_cabal() {
        group.bench_function("cabal_help", |b| {
            b.iter(|| {
                Command::new("cabal")
                    .arg("--help")
                    .output()
                    .expect("cabal --help failed")
            })
        });
    }

    group.finish();
}

Benchmark Groups

Running Benchmarks

Criterion Benchmarks

# All CLI benchmarks
cargo bench -p hx-cli

# Specific benchmark group
cargo bench -p hx-cli -- startup

# Solver benchmarks
cargo bench -p hx-solver

# View HTML reports
open target/criterion/report/index.html

Comparison Script

# Install hyperfine
cargo install hyperfine

# Run full comparison suite
./scripts/benchmark-comparison.sh

Manual Benchmark

# Create test project
mkdir /tmp/hx-bench && cd /tmp/hx-bench

cat > hx.toml << 'EOF'
[project]
name = "hx-bench"
[toolchain]
ghc = "9.8.2"
EOF

cat > hx-bench.cabal << 'EOF'
cabal-version: 3.0
name: hx-bench
version: 0.1.0.0
build-type: Simple

executable hx-bench
    main-is: Main.hs
    other-modules: Lib, Utils
    hs-source-dirs: src
    default-language: GHC2021
    build-depends: base
EOF

mkdir src
echo 'module Main where; import Lib; import Utils; main = putStrLn (format greeting)' > src/Main.hs
echo 'module Lib (greeting) where; greeting = "Hello"' > src/Lib.hs
echo 'module Utils (format) where; format s = ">>> " ++ s ++ " <<<"' > src/Utils.hs

# Benchmark cold build
rm -rf .hx dist-newstyle
hyperfine --warmup 2 'hx build --native' 'cabal build'

# Benchmark incremental
hyperfine --warmup 2 'hx build --native' 'cabal build'

Methodology

Measurement Protocol

1. Warmup: 2 iterations discarded
2. Samples: 5 iterations minimum
3. Metric: Median time (robust to outliers)
4. Environment: Quiet system, minimal background processes
5. Cache state: Explicitly controlled (cold = artifacts removed)

Statistical Validity

• Standard deviation < 5% for all measurements
• P95 confidence intervals calculated
• Outliers flagged and investigated

Tools Used

Historical Results

Contributing Benchmarks

We welcome benchmark contributions:

1. Run on your hardware — Different CPUs and platforms help validate results
2. Suggest new scenarios — What operations should we benchmark?
3. Report regressions — If you notice performance degradation, file an issue

Submit results: GitHub Issues

FAQ

Why not compare against Nix?

Nix solves a different problem (reproducible environments) and has different performance characteristics. Direct comparison would be misleading.

Will hx always be faster?

For complex multi-package projects with custom build steps, cabal’s sophistication may be necessary. hx optimizes for the common case.

How do BHC builds compare?

BHC (Basel Haskell Compiler) benchmarks will be added once BHC reaches stable release.