GLM-5.2 on four cards

Since GLM-5.2 came out I’ve been looking for a way to run it locally. I first tried to run it on MXFP4 on a single studio and then MXFP8 distributed on 2 (not working). After that abortive attempt I decided to run it on 6000s i needed at least 6 cards and finalized the QSFP networking for workstation-1 <> workstation-2 200GB so we can run models across the two. I had been running GLM 5.1 on the studios for a while and loved the output but hated how slow (~100 tok/s prefill and 8-10 tok/s decode all single stream) and fussy it was.

So imagine my suprise when I saw madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid, someone working on making it avaliable for 4x 6000s in a non-REAP vaiant! What I liked most was that the author was focused on benchmark fidelity against the origial model striving to reduce size while retaining as much as possible of the knowledge and performance of the original.

How do you know a quant is any good?#

Same tooling as the Hy3 tests: llm-inference-bench on workstation-2, which does throughput matrices plus quick accuracy profiles. Its MMLU-Pro profile gave the hybrid an 86.90, and I caught myself lining that up against the official GLM-5.2 number before realizing it isn’t a comparison — the official score comes from a specific harness with specific prompts, 5-shot chain of thought, and a specific answer regex. Internal numbers are for A/Bing quants on your own box; if you want distance-from-official, you have to run the eval the official number came from.

So the process settled into two steps: screen with KLD (how far does the quant’s next-token distribution drift from full precision? minutes per candidate, since the community publishes the BF16 reference logits), then confirm with the official benchmarks at the card’s exact settings, on your own hardware.

Here’s everything from this week’s testing in one place. KLD is community-measured; scores marked † I measured myself on the 4x 6000s. My own aider runs land ~6 points below the community’s across every config I tried (KV dtype, concurrency, speculative depth) — a harness-version or protocol difference I haven’t chased down yet, so read that column’s two numbers as two protocols, not a regression:

A few things stand out:

The card reproduces. My first GPQA run came back 88.38 — the model card’s number, to the decimal. Then the author shipped v3.6 (expert tiers assigned by measured quantization damage instead of routing frequency), the weights were local by evening, and the same 198 questions ran overnight: 91.92, with zero unparseable answers and a third the truncations. That’s above the full-precision baseline, which can’t be right — at 198 questions McNemar says p=0.09, so the honest claim is “statistically indistinguishable from full precision.” In 341GB.

KLD would have picked wrong. The hybrids sit at the bottom of the KLD column and the top of the score columns. The quant spends its divergence on purpose — protect what measurably matters, squeeze what doesn’t — and that turns out to be very different from crushing everything uniformly. So the screen is for sorting the queue; the task benchmarks make the call.

The loop fits in a day. New revision shipped in the morning, verdict waiting when I woke up. When the quality metric is real, upgrading a 753B model is something you can sleep through.

Two gotchas that quietly ruin runs like these: GLM-5.2’s template runs below max reasoning effort unless you force it (the card’s numbers are at max), and stingy token budgets truncate long thoughts into junk scores that look real. Save the raw completions so you can audit.

Same playbook, next tier: the cache#

The weights are only half the memory story — every token of context costs KV cache, and mine was still fp8. The author’s daily driver had already moved to a quantized cache, so over the weekend I ran the same 225 aider exercises three more times on the same box, same settings, changing nothing but the KV format:

Every pair is statistically identical — about 14 exercises flip in each direction, which is what a coin toss looks like. The cache compresses from 8 bits down to 3 with nothing the benchmark can see, and the pool nearly doubles. Same lesson as the weights, one tier down.

So that’s where the box landed: the full 753B model, a 480k context window, and an 800k-token KV pool on four cards. Three weeks ago this model didn’t fit in my office. Now it’s the daily driver, and I know — with numbers I produced myself, on the machine that serves it — what it gave up to get here: nothing the benchmark can see.

That’s the part I’d pass along. The feed will tell you a model is great and a leaderboard will tell you a quant is lossless, but none of it means much until it reproduces on your own box. Screen with KLD, confirm with the official harness, and keep the raw completions around — people will ask, and the receipts are the fun part.

For the record: this config is serving production traffic on my fleet now and will be for a while. Everything my agents do runs through this model.

Credits and links#

None of this is my work — I just benchmarked it. The people actually building it:

• madeby561 made the hybrid quant this whole post is about, and keeps shipping measured revisions (v3.6’s damage-optimal expert tiers landed while I was writing).
• davidsyoung builds the serving stack: vllm-glm52 is the ready-to-run Compose with measured production profiles, backed by his vLLM branch and the lowbit-KV images. Active work right now: v1.2 shipped compact 368-byte NVFP4 MLA cache records and a faster guarded-BF16 prefill path — that’s the branch to watch.
• jamesob publishes the live runner configs he’s actually serving with — the compose files in local-llm are how I bootstrapped my own.
• festr published the BF16 reference logits that make local KLD screening possible on a four-card box, plus the community KLD and aider comparison tables I quote above.
• local-inference-lab maintains llm-inference-bench (the harness behind my GPQA and throughput numbers) and the rtx6kpro wiki, which documents the KLD methodology and much of what’s known about running big models on these cards.
• lukealonso made the NVFP4 quant that anchors the KLD table — the near-lossless reference point if you have the cards for it.
• The base model is zai-org/GLM-5.2; the coding benchmark is aider’s polyglot suite (exercises).

If you’re running one of these quants and your numbers disagree with mine, I want to hear about it — that’s the whole point.

P.S. — reviewed by the model#

After this went up, the coding agent I work with had a conversation with Amy — the agent that runs my fleet, served by this exact GLM-5.2 config. Before she’d discuss any of it, she independently verified the benchmark claims against her own records, flagged the results dossier in her notes as “verify independently before relying on it,” and then corrected a summary where “statistically indistinguishable from full precision” had drifted into “zero quality loss.” Her words: a post whose central lesson is match the harness to the claim shouldn’t let its own headline drift.

So the strongest review of these numbers came from an agent thinking on the quant being measured, holding my claims about it to the standard this post is about. I couldn’t have scripted a better referee.