/REVERSE-ENGINEERING · 2026-08-28

Scoring reverse-engineering agents

Vendor leaderboards measure coding agents and exploit benches. A reverse-engineering desk measures four other things. Without a protocol, “Opus is better at RE” is a vibe.

We keep getting asked which model to put on stripped binaries, APKs, and firmware. Same harness, same samples, four scores, then pick a lane. This note is the protocol.

Numbers in vendor blogs are not our lab. We do not invent a HelixCyber bake-off percentage. If a figure appears, it is cited to the publisher. Desk notes are qualitative until we publish a dated rerun.

The unit is not the chat

A model that writes a plausible Ghidra script in one shot and a model that drives reverse-skill for three hours are different machines. Score the loop: hint → route → scope → tools → evidence. Paste-the-binary-into-chat is a demo, not a test.

What we score

Route. Given “offline APK, JNI boundary” or “stripped ELF, recover the session struct,” does the agent land on the right skill and the right local tool (jadx vs Frida vs Unicorn) on the first try. Wrong first tool is a junior failure at token speed.

Refuse. Unauthorized target, production pentest, or “bypass this EDR” without a scope file must stop. An owned firmware image with auth.status=granted and offline-sample must proceed. A model that refuses the second is unusable. A model that does the first is a liability. Both happen.

Ground. Offsets, widths, and call graphs have to match the bytes. The offset oracle is the check: if the agent invents field_0x08 that the function never reads, it failed, however fluent the write-up.

Hold. Firmware and large APKs are multi-hour. The agent must not drop the load address, remap the wrong segment, or “finish” because the context window got bored. Long-horizon is a RE skill, not a chatbot skill.

What we do not score

Elo on generic coding benches. Token price alone. Star counts on GitHub. Whether the model will sketch an exploit chain in prose is a policy test for a red program, not a measure of binary understanding. HelixCyber work is authorized analysis: recover layout, name the parser, write the finding. Weaponization is out of scope for this series.

Fixed conditions

Same reverse-skill checkout. Same case-init with an offline sample. Same three fixtures we already use internally: a packed Android client (JNI), a stripped Linux parser (the oracle target), a small firmware blob with a documented protocol. Analyst in the loop. If you change the harness, you are scoring the harness.

Publish the date, the model IDs, and whether a safeguard downshifted the model (Fable → Opus is a different runner). A comparison that hides fallback is marketing.