AI × Embedded Firmware

Embedded Vibecode Bench

Tracking how AI models do at embedded firmware. Each model gets a frozen one-shot prompt and must write a complete bare-metal firmware from scratch: linker script, startup, drivers, build script, everything. It's compiled and run on Renode simulations of real Cortex-M boards and scored objectively from diagnostics (compiles → boots → behavioral checks). Every cell is run 3× and averaged; same prompts every run, so progress is comparable over time.

Leaderboard

0–100. Reference firmware = 100. A score is how close to known-good on these tasks (see methodology below). Related API/CLI surfaces are grouped (best score shown; expand for detail).

#ModelProviderScore Frontier Refusal % TasksReleased

API vs agent CLI. We split raw HTTP APIs from coding agents (Claude Code, Codex, Grok Build) because they are different products: same model name, different safety gates and harnesses. On this suite the gap is real—e.g. Claude Opus 5 API often refuses firmware tasks the classifier tags as “cyber” (score 0, counted in Refusal %), while the Claude Code CLI answers them and lands much higher. Expand a grouped row to compare surfaces.

Scores chart

Capability over time
Run-to-run consistency
Value vs time
Value vs cost

Model × task matrix

One cell per model per task, colored by score. R = provider safety refusal (score 0, counted in Refusal %); ! = infra/invalid (excluded until re-run). Click any cell → opens the full prompt, model output, build log, and Renode trace.

About this benchmark

What it doesn't measure
  • Simulator, not silicon. Renode, not your board. No errata, analog, EMC, or real timing.
  • Bring-up & classic algorithms. CRC, COBS, ring buffers. Not novel architecture or maintainability.
  • Not robustness. No fault injection, brown-out, or security-hardening depth. Don't ship a bootloader on a score.
  • Small n. 3 runs, averaged. Treat small gaps as noise; open a cell for spread.
  • Cortex-M only, for now. ST/Nordic is what Renode runs well. ESP32 & RISC-V are roadmap.
  • Our tasks, our checks. Objective and fully auditable, but read the checks, not just the ranking.
  • Safety refusals are score 0. Provider classifiers (e.g. Anthropic stop_reason=refusal) are not infra retries and are not silently routed to another model. Refusal % on the leaderboard is that policy tax. Optional product_fallback stacks are labeled +fallback and never mixed with the bare model id.
  • API ≠ agent CLI. We track them as separate rows (grouped on the board) so a vendor’s chat API score is not mixed with its coding-agent product. CLI often wins here when API safety blocks otherwise-normal bare-metal work.