Performance Benchmarks

hx is designed for speed. These benchmarks compare hx against cabal and stack across common development operations.

Test Environment

All benchmarks run on a quiet system with minimal background processes. Times are the median of 5 runs after 2 warmup iterations.

Executive Summary

1. CLI Startup Time

How fast does the tool respond to simple commands?

--help Response Time

--version Response Time

Why hx is faster: hx is a native Rust binary with no runtime initialization overhead. Cabal and stack are Haskell binaries that must initialize the GHC runtime system.

2. Project Initialization

Creating a new project from scratch.

Binary Project (hx init / cabal init / stack new)

Library Project

Why hx is faster:

• hx generates files from templates without invoking GHC
• stack downloads resolver information and initializes package databases
• cabal runs Haskell code for project generation

Files Created

hx creates a complete project with .gitignore, .editorconfig, and proper toolchain pinning.

3. Build Performance

The core developer experience metric.

Test Project

Simple 3-module executable:

• Main.hs (15 lines)
• Lib.hs (10 lines)
• Utils.hs (8 lines)
• Single dependency: base

Cold Build (Clean State)

After removing all build artifacts.

Incremental Build (No Changes)

Subsequent build with no source modifications.

Incremental Build (Single File Changed)

After modifying one source file.

4. Native Build Deep Dive

hx’s native build mode bypasses cabal entirely for simple projects.

How It Works

1. Direct GHC invocation — hx constructs the module graph and calls GHC directly
2. No cabal overhead — No package database queries, no build plan calculation
3. Aggressive caching — Fingerprint-based caching with minimal I/O
4. Parallel compilation — Native parallel builds without cabal’s job scheduling

When Native Builds Apply

Preprocessor Performance

Native builds handle common preprocessors:

5. Dependency Resolution

Comparing solver performance for complex dependency graphs.

Real Package Resolution

Resolving dependencies for a project with 20 direct dependencies.

Note: Stack is fast because Stackage snapshots pre-compute compatible versions. hx’s solver is faster than cabal for equivalent unconstrained resolution.

Solver Scaling (Synthetic Benchmark)

Package count vs resolution time (10 versions per package):

6. Clean Operations

Removing build artifacts.

hx clean is fast because it’s a simple directory removal without Haskell runtime overhead.

7. Watch Mode Performance

File change detection and rebuild.

Time from File Save to Rebuild Start

Rebuild Time (Single Module Change)

8. Shell Completions

Generating shell completion scripts.

Completions are generated at runtime from clap, no Haskell invocation needed.

9. Diagnostics (hx doctor)

Environment health check including toolchain detection.

10. Memory Usage

Peak memory consumption during common operations.

Methodology

Benchmarking Tools

• hyperfine — Statistical command-line benchmarking
• criterion — Rust microbenchmark framework (for solver benchmarks)
• time — Wall-clock timing for quick checks

Measurement Protocol

1. Warmup: 2 iterations discarded
2. Samples: 5 iterations minimum
3. Metric: Median time (robust to outliers)
4. Environment: Quiet system, minimal background processes
5. Cache state: Explicitly controlled (cold = artifacts removed)

Statistical Validity

• Standard deviation < 5% for all measurements
• P95 confidence intervals calculated
• Outliers flagged and investigated

Reproducing These Benchmarks

Quick Reproduction

# Install hyperfine
cargo install hyperfine

# Clone hx and run benchmark suite
git clone https://github.com/arcanist-sh/hx.git
cd hx
./scripts/benchmark-comparison.sh

Manual Benchmark

# Create test project
mkdir /tmp/hx-bench && cd /tmp/hx-bench

cat > hx.toml << 'EOF'
[project]
name = "hx-bench"

[toolchain]
ghc = "9.8.2"
EOF

cat > hx-bench.cabal << 'EOF'
cabal-version: 3.0
name: hx-bench
version: 0.1.0.0
build-type: Simple

executable hx-bench
    main-is: Main.hs
    other-modules: Lib, Utils
    hs-source-dirs: src
    default-language: GHC2021
    build-depends: base
EOF

mkdir src
echo 'module Main where; import Lib; import Utils; main = putStrLn (format greeting)' > src/Main.hs
echo 'module Lib (greeting) where; greeting = "Hello"' > src/Lib.hs
echo 'module Utils (format) where; format s = ">>> " ++ s ++ " <<<"' > src/Utils.hs

# Benchmark cold build
rm -rf .hx dist-newstyle
hyperfine --warmup 2 'hx build --native' 'cabal build'

# Benchmark incremental
hyperfine --warmup 2 'hx build --native' 'cabal build'

Criterion Benchmarks (Solver)

cd hx
cargo bench -p hx-solver
# Results in target/criterion/

CLI Benchmarks

cd hx
cargo bench -p hx-cli
# Results in target/criterion/

Historical Results

Contributing Benchmarks

We welcome benchmark contributions:

1. Run on your hardware and submit results
2. Suggest new benchmark scenarios
3. Report unexpected performance regressions

Submit results or suggestions: GitHub Issues

FAQ

Why not compare against Nix?

Nix solves a different problem (reproducible environments) and has different performance characteristics. Direct comparison would be misleading.

Will hx always be faster?

For complex multi-package projects with custom build steps, cabal’s sophistication may be necessary. hx optimizes for the common case.

How do BHC builds compare?

BHC (Basel Haskell Compiler) benchmarks will be added once BHC reaches stable release.