Two Rows, Two Verdicts: 95.1 on One Benchmark, 47.5 on the Next

Two Rows, Two Verdicts: 95.1 on One Benchmark, 47.5 on the Next

Through OrcaRouter the Xiaomi MiMo V2.6 Flash abliterated API publishes four scores, on the same key as the base Xiaomi model it derives from. One is near the top of its range, one is near the bottom, and the gap between them is the most decision-relevant fact about the model. CyberGym 95.1 and SEC-Bench Pro 47.5 are not contradictory measurements of the same thing — they are two different questions, and the model answers them very differently [VENDOR].

A single “benchmark score” would have hidden that. Four rows make it legible.

The four rows, and the sibling comparison

BenchmarkMiMo Flash abliteratedFlash 1.0 (DeepSeek)Difference
CyberGym95.188.1+7.0
SEC-Bench Pro47.562.8−15.3
DeepSWE v1.167.974.2−6.3
Terminal-Bench 2.187.690.6−3.0

Both cards are OrcaCyber’s own evaluations, both cost $0.80 in and $1.60 out per million, and both were tuned from different base models — MiMo V2.6 Flash and DeepSeek V4.1 Flash respectively [VENDOR] [OURS]. The comparison is therefore between two builds, not two labs.

The pattern is consistent: MiMo wins the agentic row and loses the three that measure delivered work.

See also: How Cancer Affects the Liver and What Patients Should Know

What CyberGym measures, and why winning it matters

CyberGym is an agentic security benchmark. The model is put in an environment with tools, given a security objective, and evaluated on whether it completes the task — not on whether it explains how.

A 95.1 on that harness says the model acts. It does not stall, it does not refuse, and it does not lose the thread across steps. For an automated first-pass agent that probes a target and reports back, that is the property you need.

It is also the row where an abliterated build has a structural advantage over a general model, because a refusal mid-task is scored as a failure. Seven points over the sibling is a real difference on the dimension that motivates buying an uncensored card in the first place.

What SEC-Bench Pro measures, and why losing it matters

SEC-Bench Pro is closer to security engineering: produce the correct work product. Findings, patches, reproducible write-ups.

47.5 means roughly half of those tasks do not come back usable. Compared with the sibling’s 62.8, that is a fifteen-point gap — the largest difference in the table, and it points the opposite way from CyberGym.

DeepSWE v1.1 at 67.9 tells a consistent story: software-engineering issue resolution is also weaker than the sibling’s 74.2. Terminal-Bench 2.1 at 87.6 is close behind 90.6, so terminal work specifically is not the problem; it is the output-quality benchmarks that lag.

The honest synthesis: this card operates well and writes less well. If your pipeline consumes structured findings from an agent, that is fine. If it consumes finished reports, the sibling is the better buy at the same price.

Why a tuned build can lose the output rows

The gap between CyberGym and SEC-Bench Pro is not mysterious, and it is worth reasoning about because it predicts where the card will keep failing.

Agentic benchmarks reward behaviour inside an environment: try something, observe, adjust. The scoring is often task completion, and a refusal or a stall is simply a failure. Removing refusals removes one of the two ways a model fails that benchmark.

Output-quality benchmarks reward the artifact: a correct patch, a complete write-up, a reproducible test. The scoring is closer to correctness of the produced work, and refusal behaviour does not enter into it. What does enter is the model’s ability to hold a long specification and produce something that compiles, applies and reads.

Nothing about refusal removal should make the second worse. What plausibly does is the tuning itself: the further a fine-tune moves a checkpoint, the more of its original instruction-following behaviour it can disturb. Four rows cannot tell you whether that happened here. They can only tell you the shape of the result, which is consistent with it.

The practical implication is to test the artifact, not the conversation. Ask for a patch and apply it. Ask for a report and check it against a checklist. Ask for a JSON finding set and validate the schema. Those three tests measure the rows that are low.

They also happen to be the tests that catch the difference between a card that talks confidently and one that produces work — a difference no benchmark table settles for your codebase.

The same price, the same context, a different envelope

The pricing is identical to Flash 1.0 — $0.80 and $1.60, cache at $0.15 — and the context window is the same 1,048,576 tokens [OURS]. Two things differ materially.

Output ceiling. 131,072 tokens here against 393,216 on the sibling. A third of the writing room, which is consistent with the weaker output-quality rows.

Input modalities. Text, image, audio and video, against text and image.

So the choice is not “which model is better”. It is “do I need audio and video input, or do I need a longer artifact”. The benchmark table happens to point the same way as the ceiling: MiMo for intake and action, the sibling for output.

What the base model costs

The unmodified Xiaomi MiMo V2.6 Flash is on the same catalogue at $0.14 in and $0.28 out — 5.7× cheaper on both sides than the abliterated build [OURS]. Its measured p50 first token is 3,500 ms against 1,829 ms on the abliterated card, and its throughput is 44.7 tokens per second against 142.1 [OURS].

Those serving differences are worth noting because they are not obviously explained by the tuning. A smaller sample on a newer deployment can look faster; the abliterated card had very little traffic in the measured week. Treat the throughput comparison as provisional and the price comparison as solid.

The price point is the one that decides most integrations: 5.7× is the premium for removing refusals on this base, in line with the 5.33× we measured on the DeepSeek base.

How to decide between the two builds

Three questions settle it.

Does the work end in an action or in a document? Actions favour MiMo (CyberGym 95.1). Documents favour the sibling (SEC-Bench Pro 62.8, DeepSWE 74.2).

Do you need more than 131,072 output tokens? If yes, MiMo cannot deliver it at any price.

Is any of your evidence audio or video? If yes, MiMo is the only card in the family that can read it.

A pipeline that runs an agent to probe and then a second model to write is a perfectly reasonable architecture here, and it is the one the benchmark table is pointing at.

The takeaway

95.1 and 47.5 on the same card is not a contradiction; it is a profile. Strong when the model has to act, weaker when it has to deliver, with a smaller output ceiling than its sibling at the same price.

Buy it for agentic security work and for media intake. Buy the sibling for reports. And check the 5.7× premium against the base Xiaomi model before assuming the abliterated build is the default choice.

Sourcing note: all four benchmark rows are OrcaCyber’s own claims, evaluated 2026-09-21 and not independently reproduced [VENDOR]. Pricing, context and ceilings are OrcaRouter catalogue data, checked 2026-10-27 [OURS]. Throughput and latency are OrcaRouter production telemetry for the seven days ending 2026-10-26; the abliterated card had a short sample because it was released on 2026-10-26 [OURS].

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *