Twenty hours with Hy3

Recently tried out the new hy3-295B-A21B. Writing up the testing process I went through (and will keep doing so with future models). Wrote this up with some help from the work logs and the clankers (positive, affectionate affect).

What is this model auditioning for?#

I watch for model releases and look for things to slot into the local setup. I like to think of a low, medium, high slot for my local setup. Medium is currently taken by Deepseek v4 flash which runs tp=2 on 2x 6000s. Was hoping this model would be competitive but its a bit too big to run tp=2 so I tested it tp=4. I tend to use my medium model for agentic data retrieval (e.g. querying my session history for the logs to support this article). Lots of tool calls, concurrent streams etc… Was hoping Hy3 would be competitive.

My go to benchmark for this is something called estonia: a long-context retrieval eval: thirty of the same question fired concurrently against the same ~134K-token document, where scoring requires actually finding concrete answers in the haystack. DeepSeek v4 flash goes 30/30 on it, consistently. I also run lavd a short-context reasoning eval — ten items, long chains of thought, scored on the final answer — as another quick check.

First run: the trap#

First on the box was the community 4-bit quant, kodelow/Hy3-NVFP4-W4A16 — ~181GB of weights, fits tp=4 with room for KV cache. The first estonia run came back and it was bad. Unimpressive score, and slow — single-digit tokens per second per stream, which for a 21B-active MoE on this hardware made no sense. My honest first reaction was that I’d messed up the serving config, and my second was to start suspecting the quant.

The run had two real issues:

1. Thinking was off. Hy3’s chat template defaults to no thinking
2. Prefill was strangled. The batched-token limit was at a default that chopped the 134K-token prompt into 33 chunks and squeezed the speculative-decoding slots. Raising it to 16K chunks got prefill to ~2,900 tok/s

The overnight A/B#

With thinking forced on and prefill fixed, the score improved but not that much. Next suspect: the community 4-bit quant itself. Tencent publishes an official FP8, tencent/Hy3-FP8 — 300GB, barely fits, but fits. I pulled it down and had an agent run both quants through both benchmarks overnight, same settings, results ready in the morning:

The surprise: the 4-bit outscored the official FP8 by five questions, while running nearly twice as fast the other way. Thirty items at temp 0.9 means real error bars, but I think the result is real, and the mechanism is the key thing to understand about quantizing MoE models: not every expert tolerates compression equally. The official FP8 quantizes everything uniformly — every expert, every layer, activations included. A uniform scheme degrades the compression-sensitive experts along with the tolerant ones, and a degraded expert is functionally a pruned one: still routed to, no longer doing its job. The W4A16 spends its bits selectively and scores higher with half the nominal precision. “Official” doesn’t mean “carefully quantized.”

The morning puzzler#

Reports online had the exact same hardware — four RTX PRO 6000s — serving Hy3 FP8 at 170 tok/s. I was staring at runs averaging 2.4 tok/s per stream. When I actually measured single-stream I got 102–127 tok/s, inside the online band — those are short-context, single-stream numbers. The workloads I care about are in the opposite corner: concurrent agents large context, and there Hy3 collapses. ~60 tok/s aggregate at 10 streams, ~70 at 30 — flat; adding concurrency added nothing. Short-context concurrency scaled fine (~232 tok/s on lavd), so it wasn’t a serving bug.

The explanation was sitting in the model config all along. Hy3 uses dense grouped-query attention — 80 layers, 8 KV heads at width 128, no sliding window, no sparse tricks — which is 320KB of KV cache per token. A single 134K stream carries 43GB of KV, and every decode step reads it. The cards weren’t compute-bound; they were drowning in cache reads. Back-of-envelope bandwidth math put the ceiling for my workload around 165 tok/s aggregate, and I was 2× under that — plausibly the speculative decoder double-dipping KV reads. MTP-off and FlashInfer were untested levers, so headroom existed. But even perfectly tuned, the best possible Hy3 at this workload lands well under what a smaller-cache model does casually. The 170 tok/s was real and so was my 60 — different corners of the envelope.

The pivot#

By mid-morning of day two the picture was: 25/30 at best on estonia, an architectural wall you can’t config-flag away, and an incumbent that’s smaller, faster, better at the job, and leaves two cards free. There were levers left to pull and a quant rematch worth running, but that same morning a community quant of GLM-5.2 appeared — the full 753B, no pruning, sized for exactly this class of four-card box, with published benchmark diffs within noise of full precision. I stopped the Hy3 reload mid-restart and pivoted hard. By early afternoon GLM-5.2 was serving on the same port, and when I later ran it through the same context-and-concurrency matrix it held throughput flat out to 200K context — the exact scaling Hy3 couldn’t do. That model is the next post.

Hy3 served production traffic for about twenty hours. Three days later I swept the last references to it out of the fleet config. My verbatim verdict in the logs: “that model sucked.” The fairer version: real model, real strengths — short-context single-stream among them — and architecturally mismatched with long-context concurrent retrieval, the only job I was hiring for.

What I’m keeping#

1. Benchmarks that match your use case are the whole game. The leaderboard scores said Hy3 was a frontier model; thirty questions against a 134K-token document said it couldn’t do my job. Having a bench that encodes what I actually need is what made this a twenty-hour decision instead of a two-week one.
2. Architecture sets long-context speed, and it varies a lot between models. KV bytes per token is different for every model and it’s right there in the config. Hy3’s dense attention means 43GB of cache per long stream, and no serving flag fixes that. Cheap to check before downloading next time.
3. Evaluate for the slot, not the leaderboard. The model that wins your workload beats the model that wins the feed.