Measuring

Every chapter in this part has ended by telling you to measure. This is how, and more importantly, how to get a number you can trust.

Why this is harder than it looks

Four things will give you a wrong number, and the first two will give you one that is wrong by an order of magnitude.

The compile happens on the first call. compile_kernel shells out to teenyc and caches the result. Time that call and you have measured the compiler. Compile outside the timing loop, always.

Launches are asynchronous — but this API already handles it. cuLaunchKernel returns before the kernel has run, so in CUDA generally a timing loop that launches and stops the clock measures the cost of asking, not of doing.

Device::launch calls cuCtxSynchronize immediately after launching, in both its typed and arg-packed paths. Its comment says why, and it is not about timing: synchronising there makes a GPU-side fault surface as a CUDA error code rather than as a SIGSEGV somewhere later.

So a loop around device.launch measures real kernel time, and you do not need your own barrier. Worth knowing in both directions — it also means you cannot overlap launches or hide latency behind the host, because every launch waits.

The first run is never representative. Caches are cold, clocks have not boosted, memory is not resident. Warm up.

Clocks move. GPUs throttle when hot and boost when idle. A benchmark run immediately after another one is not measuring the same machine.

criterion, which this tree uses, handles warm-up and repetition and gives you a distribution rather than a single number. It does not handle the first two — the compile and the synchronisation are yours.

The harness in this tree

kernels/teeny-kernels/benches/conv2d_bn_silu.rs is the pattern to copy. Its structure:

Compile once, outside the loop.

let kernel = Conv2dBnSiluForward::new(kh, kw, stride_h, stride_w, pad_h, pad_w, 1, BLOCK_OW_SCALAR);
let ptx = std::fs::read(compile_kernel(&kernel, target, false)?)?;
let program = testing::load_program_from_ptx::<Conv2dBnSiluForward>(&ptx)?;

Note force: false. The cache is wanted here — you are not benchmarking compilation.

Allocate and fill once, outside the loop.

let mut x_buf = device.buffer::<f32>(shape.nb * shape.c_in * shape.hh * shape.ww)?;
x_buf.to_device(&shape.x_host())?;

Host-to-device copies are slow and are not what you are measuring.

Only the launch is inside.

group.bench_function(format!("scalar/{}", shape.label), |b| {
    b.iter(|| { device.launch(&program, &cfg, (...)) })
});

Use deterministic inputs. The bench generates them arithmetically:

fn x_host(&self) -> Vec<f32> {
    (0..self.nb * self.c_in * self.hh * self.ww)
        .map(|i| (i as f32 % 17.0 - 8.0) * 0.1)
        .collect()
}

Not random. Two runs get identical data, and any difference between them is the kernel.

What to compare against

A number alone means nothing. 142 µs is neither good nor bad.

Against the alternative you would otherwise ship. The fused kernel against the three unfused ones. This is the comparison that decides whether the work was worth it.

Against the other implementations of the same thing. The conv bench times three kernels across shapes chosen to straddle the dispatch thresholds — so the measurement answers “does the lowering still pick the right one?”, not just “how fast is this?”.

Against the hardware’s limit. For a memory-bound kernel, divide the bytes moved by the elapsed time and compare with the card’s peak bandwidth. At 80% you are close to done. At 15% something is wrong, and Chapter 17 is where to look. This is the most useful single check available, and it needs no reference implementation.

Choosing shapes

Benchmark the shapes you run, not round numbers.

Powers of two are the friendliest case: no ragged tail, no masked lanes, tiles that divide evenly. A kernel benchmarked only at 1024×1024 can be much worse at 1000×1000, and 1000 is the realistic one.

The conv bench picks shapes deliberately either side of the dispatch thresholds, which is the right instinct: benchmark where behaviour changes, not where it is comfortable.

Running it

cargo bench -p teeny-kernels --features cuda,training --bench conv2d_bn_silu

On Blackwell you may need the PTX-version workaround from Chapter 4:

TEENYC_PTX_VERSION=87 cargo bench -p teeny-kernels --features cuda,training --bench conv2d_bn_silu

criterion writes an HTML report and, on a second run, compares against the previous one — which makes “did my change help?” a question it answers directly.

Recording a result

A measurement without its context is not reproducible. Record:

• The card, by name and compute capability.
• The shapes.
• The block sizes and tile shapes.
• The date. Driver and toolchain versions move.

A worked result

The conv bench, run on an RTX 5070 (sm_120), CUDA 13.3, driver 610.43.02. Criterion means; the kernel the lowering actually picks for each shape is marked ✓.

Shape	scalar	tiled	gemm
1×1, c_out=8, 32×32	11.8 µs ✓	13.3 µs	12.7 µs
1×1, c_out=16, 32×32	16.7 µs	13.2 µs ✓	12.4 µs
1×1, c_out=32, 40×40	42.4 µs	14.5 µs	12.8 µs ✓
3×3, c_out=32, 40×40	871.4 µs	92.0 µs ✓	n/a

Read it as the bench’s author intended — as a check on whether the dispatch thresholds from Chapter 12 still hold.

Three of the four are right. At c_out=8 the scalar kernel wins and is chosen. At c_out=32 the GEMM kernel wins and is chosen, 3.3× faster than scalar. On the 3×3 convolution, where GEMM does not apply, tiled beats scalar by 9.5× — the single biggest number here, and a good illustration of why the naive kernel from Chapter 11 is not the one you ship.

One is not. At c_out=16 the lowering picks the tiled kernel at 13.2 µs, but the GEMM kernel does it in 12.4 µs — about 6% faster, with non-overlapping confidence intervals. The GEMM threshold is 32; on this card it could come down to 16.

That is a real finding from one bench run, and it is worth being careful with it: 6% on one card at one shape is not enough to move a threshold that has to serve every card. What it justifies is measuring the same point on the other targets, which is exactly the argument a hand-picked constant needs to survive.

Recording a result

The table above has everything a measurement needs: the card, the shapes, what was compared, and what was chosen. Add the date when you record one — driver and toolchain versions move, and these numbers came from a specific pairing.

An invented number is worse than a blank one. A blank prompts someone to measure; an invented number gets quoted.

Profiling

criterion tells you a kernel is slow. It does not tell you why.

For that you need a profiler — NVIDIA’s Nsight Compute reports achieved bandwidth, occupancy, warp stall reasons and instruction mix per kernel, which is the level at which “why” gets answered.

Nothing in this SDK integrates with it, so this book cannot teach reading one. It has been used on these kernels, though, and the result is a good example of what a profiler tells you that a stopwatch does not:

Profiling YOLO26n’s conv layers showed occupancy as low as 8% on deep layers with small spatial extent and many output channels. Theoretical occupancy was 100% in every case — so it was not register or shared-memory pressure. It was purely a grid-size problem: the fixed tile size produced as few as 40 thread blocks, which cannot fill the machine no matter how good the kernel is.

That distinction — achieved 8%, theoretical 100% — is the whole value of the tool. A benchmark would have told you the layer was slow. Only the profiler told you it was slow because there was not enough work to go around, which points at the tile size rather than at the kernel body.

The fix was sm_count, described in Chapter 16.

What you can do without a profiler, and should do first:

1. Compute achieved bandwidth by hand. Bytes moved ÷ time. Compare with the card’s specification.
2. Read the MLIR. Chapter 9. Count the loads and stores; check nothing is loaded twice.
3. Vary one thing at a time. Block size, tile shape, layout. The measurement tells you which mattered.

That covers most of what a first profiler session would have told you.

Next: the numbers themselves.