KernelArc · Multi-Agent Autoresearch
One GPU, multiple agents, 1st place on the NVIDIA leaderboard
A single developer with one GPU climbed to 1st place — first with a single optimization agent, then with a multi-agent autoresearch system that reused every win, avoided repeated dead ends, and broke a wall the single-agent run could not cross.
Speed up a core AI operation on the world's fastest GPU.
The task: L1/030_attention_output_projection_with_residual — a large matrix multiplication plus a residual addition on NVIDIA's Blackwell B200.
The benchmark scores against a speed-of-light lower bound — the closer you get, the higher the score.
This isn't a single optimization problem — it's 16 simultaneous ones. The final score is the mean across all input shapes, and the evaluation runs on a different machine than your development GPU. Two constraints make this particularly hard:
Every shape must be fast
The score is the mean across all 16 input sizes — no room to specialize on a subset.
The judge machine is different
What runs fastest locally may run slowest on the evaluation server. Software version differences change which GPU library routines get selected.
Five phases from defaults to the podium.
Each phase built on the last — what worked was kept, what failed was recorded. The breakthroughs came from realizing what to stop doing as much as what to try next.
Trust the defaults
Standard cuBLASLt with default settings plus a separate CUDA kernel for the residual addition. Slow — vendor defaults are not tuned for every workload shape.
Memorize what works
Profiled all 16 shapes, discovered each preferred a different algorithm variant. Hard-coded per-shape winners. Fused the residual into the GEMM epilogue — the separate kernel disappeared entirely.
Learn on the spot — runtime autotuning
Hard-coded indices broke on the judge's machine. Rewrote to run a compact tournament during warmup: cold-cache timing, dual fused/split strategy, hidden configuration exploration, and accuracy filtering.
Exhaust the single-agent search
Library policy tuning, exhaustive tile/stage/split-K search, CUTLASS SM100 kernels, cuDNN, Triton, CUDA graphs, hand-written TMA experiments, FP8 precision. No stable gain past ~0.479 SOL.
Multi-agent autoresearch — KernelArc
Strategy-specialized agents across optimization tracks sharing dynamic memory. First broke NVIDIA's baseline to reach 3rd, then eliminated dead-end strategies and invested in measurement fidelity to climb to 1st.
The wall was not hardware — it was a search-process wall.
The single-agent run exposed a ceiling at 0.479 SOL. Multi-agent coordination with dynamic memory broke through it — not by finding a new algorithm, but by changing how the search itself worked.
A single agent hit 0.479 SOL and could not move past it — no matter how many approaches it tried.
Hard-coded optimizations broke on the judge machine (15 of 16 shapes failed), so runtime autotuning made the solution portable. But even with portability solved, exhaustive search across libraries, custom kernels, and reduced precision found no stable gain.
| Approach tried | Outcome |
|---|---|
| Runtime autotuning (cold-cache, dual strategy, hidden configs) | Portable — reached 0.479 |
| Exhaustive library search (thousands of variants) | No stable gain past 0.479 |
| CUTLASS SM100 / cuDNN / Triton / CUDA graphs | Slower or N/A |
| Hand-written TMA kernels + FP8 precision | Correctness issues |
The false lesson
The agent concluded that ~0.036 ms was a hardware ceiling. The leaderboard proved otherwise — #1 ran faster on the same B200 class. The wall was not hardware; it was a search-process wall.
Multiple specialized agents with shared dynamic memory broke the ceiling by correcting each other's false conclusions.
KernelArc launched agents across orthogonal optimization tracks — library policy, fusion, scheduling, memory behavior, precision — all reading and writing to a shared memory that accumulates wins, records specific traps, and corrects outdated lessons.
| Dynamic memory behavior | Effect |
|---|---|
| Win by one agent | Becomes starting point for others |
| Specific trap recorded | Prevents repetition across agents |
| False trap corrected (e.g. "ceiling" disproved) | Unblocks stuck search paths |
| Best variant always preserved | Every round starts from the best known state |
The "0.036 ms ceiling" was explicitly marked wrong once the leaderboard showed a faster result. Dynamic memory doesn't just accumulate — it corrects.
Not a hand-written GEMM — a better use of the vendor library, measured more faithfully.
The final breakthrough was not a new matrix multiply — it was realizing that fewer strategies measured more faithfully beats many strategies measured roughly. Eliminating dead ends freed budget for deeper benchmarking of survivors — the score moved from 0.506 to 0.511, from 3rd to 1st place on one B200.
Note on evaluation versions: All SOL scores in this post were measured under Evaluation Stack v1.0. NVIDIA has since upgraded to Evaluation Stack v1.1 — scores between versions are not directly comparable due to changes in the evaluation service. Under v1.1, our submission scores 0.481 (latency 0.037 ms) and retains 1st place. The leaderboard screenshot above reflects v1.1 results. Historical v1.0 results remain available on the leaderboard via version filter.