Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Benchmarks

This page lists where bunsen measures performance, and the two things that make a measurement honest: a real backend, and knowing whether a number is cold or warm. It quotes no numbers; they depend on the hardware, and the benchmarks print their own.

What there is

The criterion benches live in crates/public/bunsen/benches:

BenchMeasuresRuns on
contractsshape-contract unpacks, asserts and periodic asserts (cost)the host
drop_block2D DropBlock over a small image batchthe CPU backend, always
mathmaybe_iroot against a floating-point referencethe host
lbmthe D2Q9 lattice-Boltzmann collision, streaming and update steps, in f32 and f64performance_device()
whisper_decodethe Whisper encoder and decode loop at base.en’s shapes, with random weights, cold and warmperformance_device()

Outside the benches:

  • silero-bench runs a mono WAV file through Silero VAD’s streaming context on performance_device(), prints the per-chunk speech probabilities, and optionally checks them against an expected JSON file.
  • conway’s benchmark subcommand reports the sustained steps per second of the 2D or 3D Game of Life on the device --device selects. It measures the tensor kernels behind the simulation kits.
  • bunsen-asr’s transcribe logs, at -v, each file’s audio length, transcription time and realtime factor, and the totals over the set. Its README runs it over a speech dataset.

Cold and warm

The CubeCL backends autotune their kernels, and the tuning is keyed on shapes. The first call at a new shape runs the tuner; later calls at that shape use what it found. A cold call and a warm one can differ by more than the optimization you are about to try, so measure both before deciding what to optimize, and know which one a number is.

  • Criterion warms up before it measures, so its numbers are warm. whisper_decode times each case’s first call once and prints it as cold.
  • conway benchmark leaves a warmup fraction of its steps out of the timing.
  • bunsen-asr sizes every push to the stream driver’s grain, so the front end and the autotuned kernels behind it see one shape, not a drift of remainders.
  • A benchmark whose shapes change every iteration measures the tuner, not the kernel.

Always pass a backend

A benchmark on performance_device measures whatever bunsen’s backend feature selected, and without one that is the CPU. It doesn’t fail; it reports CPU numbers (Testing and backends). Pass the backend explicitly: --features wgpu, or another backend, for bunsen’s own benches, and --features bunsen/wgpu for a binary that reaches performance_device() through bunsen, such as silero-bench. The examples and bunsen-asr, conway among them, pick their device at run time with --device; a backend is selectable only when the binary was built with its feature, and without one auto is the CPU. The examples’ READMEs and the whisper_decode bench’s docs give exact commands.

Benches build optimized already. Run the binaries with --release too: the work is inside burn’s kernels, and an unoptimized build measures the wrong thing.