← Back to blog

Inference · AMD · November 19, 2026

The world's fastest FP8 GLM-5.3-Flash endpoint, powered by AMD MI355X

We're pushing inference latency down so you can build for moments that can't wait.


Fused draft and verify kernels on AMD Instinct MI355X, with 8-bit GLM-5.3-Flash throughput rising from 304 to 639 tokens per second

When inference happens in milliseconds, you can build things that weren't possible before. A customer recently asked Momento to generate headlines from football plays with latency SLOs of 250 ms p50 and 500 ms p99 for short outputs.

We started with Gemma for its smaller footprint, but GLM-5.3-Flash gave us more useful results for the task. It was also slower. For a headline to land within 250 ms, the entire response has to arrive within that budget.

We doubled token throughput from 304 to 639 tok/s on a coding-session replay using four AMD Instinct MI355X GPUs. Both builds ran SGLang with the original FP8 weights and the same draft length. These decode measurements do not establish end-to-end latency for headline generation.

Decode throughput tok/s

Original FP8 weights, four MI355X GPUs. Each run replayed one coding-agent session for 15 minutes using AgentX on NVIDIA's AIPerf, with high reasoning effort. Median per-response output rate, excluding time to first token.

Our FP8 build measured 639 tok/s during decoding on the coding-session replay. In a separate whole-request test that includes time to first token, it measured 508 tok/s. The comparison below uses whole-request output speed.

We tested from AWS with one request at a time and took the median of three passes. The other figures come from Artificial Analysis's November 19 snapshot of hosted GLM-5.3-Flash endpoints.

GLM-5.3-Flash · Whole-request output speed
Endpointtok/s
Momento · FP8508
Inco FAST457
Zyphra304
Parasail255
Bitdeer AI234
Together AI203
Fireworks150
Baseten122
Selected endpoints, excluding names explicitly labeled FP4. Other providers' weight precision is not independently verified.
Published figures: rounded five-day averages as of November 19, 2026. Momento: median of three passes that day. See the full leaderboard.

Make the target faster

Fusing the target model's layers increased throughput from 304 to 499 tok/s. It is a general decode optimization that also accelerates the target's verification pass when speculation is enabled.

AMD's open-source monokernel project fuses each layer into a persistent kernel on each GPU, including the MoE block. Inter-GPU communication happens inside the kernel rather than through separate communication launches. The target forward pass takes 46 main launches, including the output tail.

Upstream monokernel required 4-bit expert weights for GLM-5.3-Flash. We added an FP8 expert path to run the model's original weights inside the fused kernel. It reads the weights and their scale factors directly. We tuned prefetch distance and skipped expert-weight reads for padding rows.

Making speculative decoding pay off

Speculative decoding lets a lightweight draft head propose several tokens for the full model to verify in one pass. Accepted tokens share the cost of that pass. GLM-5.3-Flash includes a multi-token prediction head for drafting, but the benefit depends on the cost of drafting and verification as well as acceptance.

Speculation was already enabled in our baseline. With target fusion in place, we optimized drafting and verification to move from 499 to 639 tok/s. The draft length and weight precision stayed unchanged.

  1. Bring the draft onto fused kernels 499 → 569 tok/s

    Target fusion accelerated verification, but drafting still used the original kernels. We moved the draft head onto fused kernels to reduce the cost of generating proposals.

    The fused draft block reuses the target model's attention and output kernels while preserving access to the same committed context.

    This build also replaced the verify step's full-vocabulary sort for top-p sampling with a cutoff search. The measured gain includes both changes.

  2. Cut the verification bottleneck 569 → 639 tok/s

    With drafting faster, verification remained the dominant cost. We focused next on synchronization overhead and the sampling cutoff search.

    Reducing synchronization. Verification processes one row per token position, with one more row per request than the draft length. Stateful layers introduce dependencies between those rows. The original kernel checked input readiness per row and used barriers between state updates. We moved the readiness check to the sequence level and removed the intermediate barriers.

    The kernel retains intermediate state at each position so it can commit the accepted prefix. Where threads previously waited for one thread to compute and store that state, they now recompute it locally. The extra arithmetic costs less than the synchronization it replaces.

    We also removed some work that wasn't needed. With three or four requests in flight, the expert block used to scan every expert to work out which rows go to which experts. It now looks only at the experts the rows selected.

    None of this changes the computed values. Every commit computes each output using the same operations in the same order as before. We compared digests of the outputs and of every piece of saved state to check for differences. They matched for every combination of requests and rows we tested. We then timed the kernels on their own on four GPUs, using the same mix of attention layers a real step runs. Median launch time fell from 88 to 83 microseconds with one request and from 142 to 128 with two, across seven kernel-only trials using synthetic weights and routing.

    Tightening the top-p search. Our sampler finds the probability cutoff without sorting the full vocabulary. Previously, every row started with the same search bounds, so the slowest row determined launch latency. Initializing bounds from each row's values cut our slowest recorded launch from 410 to 180 microseconds.

Matched profiles of the 569 and 639 tok/s builds show verification ran 7.2% faster and accounted for 63% of the reduction in step time. It remained about 70% of the step.

Verification tuning, before and after

Whole step

Before
5.625 ms
After
5.178 ms

Target verification

Before
3.905 ms
After
3.624 ms
Before: the 569 tok/s build. After: the 639 tok/s build. Medians from six profiled windows of about 200 steps per build, across two fresh server starts. Same fixed prompt, FP8 weights, four MI355X GPUs and one request in flight. These are separate from the AgentX replay that produced the throughput figures. All bars share a time scale; lower is better. Profiling adds about 7% overhead to both builds.
Profiling setup and build details

Both builds used TP4, the same approximately 2,200-token prompt, three draft steps and four draft tokens, temperature 1.0, top-p 0.95, and rejection sampling. Each server start contributed three windows. Unprofiled median step times were 5.263 ms and 4.847 ms.

Verification share was 69.3% before and 69.7% after, with overlapping ranges. Absolute verification times did not overlap across the six windows, and both server starts agreed. Sampling times had overlapping ranges, so this experiment does not establish a sampling speedup. Generated text was not seeded.

Before: kernel 2692ee1, with no extra switches. After: kernel 28f0890, with SGLANG_OPT_SPEC_PACKED_VERIFY_BCAST=1, SGLANG_OPT_SPEC_VERIFY_REUSE_STAGED_INPUTS=1, and SGLANG_OPT_SPEC_RELAY_IN_PLACE=1. Both used SGLang 6be0d75ae0 and image lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261111.

Verification covers the target forward pass, including communication inside the fused kernels. Separate result broadcasts are counted under sampling; separate drafting all-reduces are counted under drafting. Categories do not overlap within a window. Independently computed component medians need not sum to the median whole-step time.

The next bottlenecks

Each decode step still runs a couple of hundred small operations around the fused kernels. We're combining the work that builds tables and prepares inputs. Higher draft acceptance offers another way to get more from each verification pass, provided the better draft doesn't cost more than it saves.

The next test is holding that speed with more requests in flight. We're also exploring sharing conversation state over RDMA so another server can pick up a conversation without recomputing its history. Faster kernels are only part of the latency budget. Reducing network latency matters too, because the customer waits for the whole request.

How fast is fast enough? Let your evals decide.

The fastest endpoint in the full November 19 Artificial Analysis snapshot was labeled FP4 and measured 863 tok/s in whole-request output speed. It was the only published endpoint above our FP8 result of 508 tok/s. Quantization is another way to push speed further, but the quality tradeoff needs to be measured on your workload.

Momento offers both 4-bit and 8-bit GLM-5.3-Flash options. Our 4-bit build measured 698 tokens per second on the same coding-session test, compared with 639 for the original FP8 weights used here.

For the football headlines that started this work, we now generate high-quality headlines at 250 ms p50, with p99 reliably under 500 ms, measured end to end in the headline demo. We met our quality and latency SLO targets for the customer. Those are the results that matter for their application, separate from the coding-session throughput benchmarks above.

That result makes us want to push further. At this latency, generated headlines can become part of the live experience. We're pursuing more workflows that become possible when inference keeps pace with the moment.

Try the fastest FP8 GLM-5.3-Flash