Tencent Hy4 Architecture: Why 770B Isn’t the Point

A 770-billion-parameter model sounds like a brute-force scaling story. The Tencent Hy4 architecture tells a more interesting one: its backbone activates 49 billion parameters for each token, then keeps narrowing the work through sparse attention, reused token indexes, and speculative decoding. The headline is capacity; the design is selectivity.

“Active parameters” is useful shorthand and a terrible invoice. It omits routing, memory traffic, attention, and communication—the charges that arrive after the multiplication count. Hy4 exposes unusually detailed architecture for a preview, but its end-to-end efficiency still needs independent measurement.

Tencent Hy4 architecture at a glance

Tencent’s official Hy4 repository describes a 78-layer backbone with a one-million-token context window. The first feed-forward layer is dense; the other 77 use a mixture-of-experts design. Four identity Hyper-Connection residual streams and a separate multi-token-prediction layer complete the main stack.

Component	Hy4 preview specification	Why it matters
Backbone	770B total; 49B active per token	Separates stored capacity from routed work
Transformer depth	78 layers	One dense FFN, then 77 MoE layers
Experts per MoE layer	256 routed + 1 shared; top 8 routed	Only a subset processes each token
Attention	Gated DSA; top 2,048 positions	Narrows core attention at long context
IndexCache	21 full-index + 57 shared-index layers	Reuses token-selection results
Residual path	4 iHC streams	Expands information flow between layers
MTP layer	10B total; 0.7B active	Supports speculative decoding

The 770B figure excludes the 10B multi-token-prediction layer, which helps explain why some listings round the entire package to roughly 780B. The published Hy4 weights and configuration use an Apache 2.0 license. Total parameters, active parameters, checkpoint size, and inference throughput are four different measurements; collapsing them into one number hides the engineering choices.

MoE stores 770B parameters but routes a fraction

A dense transformer sends every token through every feed-forward parameter in a layer. A mixture-of-experts layer instead uses a router to choose a small group of specialized feed-forward networks. Hy4 gives each of its 77 MoE layers 256 routed experts plus one shared expert, then selects eight routed experts for each token.

Across the backbone, Tencent reports 49B active parameters per token—about 6.4% of the 770B stored total. That ratio is the model’s basic capacity bargain. More experts can hold more patterns, while any one token visits only a fraction of them.

The bargain has fine print. All experts must remain available in memory, the router must distribute tokens without overloading a few destinations, and expert traffic can cross accelerator boundaries. Attention, embeddings, shared components, and orchestration also continue to run, so “49B active” does not make Hy4 operationally equivalent to a dense 49B model.

Parameter count is therefore a map of different costs: total count drives storage, while active count approximates the arithmetic path. Neither captures memory bandwidth or interconnect overhead. Hy4 reduces multiplication; it does not repeal systems engineering.

Gated DSA narrows the million-token problem

Ordinary dense self-attention compares each query token with every eligible earlier token. As the sequence grows, that all-to-all pattern becomes the painful part: doubling sequence length can roughly quadruple the score-computation work. A million-token window is therefore a workload challenge, not merely a large input field.

Hy4 uses Gated DeepSeek Sparse Attention, based on the mechanism described in the DeepSeek Sparse Attention paper. A lightweight indexer scores candidate history positions, selects 2,048 of them, and sends that smaller set into the more expensive core-attention calculation. At the stated context ceiling, that selected set is roughly one-fifth of one percent of the available positions.

For core attention, this changes the long-sequence shape from approximately O(L²) to O(Lk), where k is 2,048. It does not make the complete attention system linear for free. The indexer still has to inspect the sequence to decide what deserves attention, and a bad selection can discard useful context before the stronger attention operation sees it.

The gate adds another learned control over the sparse result. Conceptually, Hy4 is separating retrieval from detailed comparison: first find a small candidate set, then spend attention capacity on it. The next bottleneck is obvious—why repeat nearly the same search in every layer?

IndexCache reuses the search across layers

The IndexCache paper reports that adjacent sparse-attention layers often select heavily overlapping token sets. IndexCache designates some layers as “full,” where the indexer performs the search, and others as “shared,” where the layer reuses the nearest earlier full layer’s selection. It is memoization for the question “which old tokens matter here?”

Hy4’s shipped configuration contains 21 full-index layers and 57 shared-index layers. In other words, roughly 73% of its layers reuse an index instead of calculating a new one. That percentage is derived from the current configuration, not a Tencent performance claim, and it reveals how aggressively the preview applies cross-layer reuse.

The original paper reports prefill and decoding gains on separate DSA model experiments, including a 30B model. Those numbers are evidence that the method can work, not benchmarks for Hy4. Hy4-specific throughput depends on its much larger MoE backbone, four-stream residual path, kernels, batch shape, parallel layout, and hardware.

There is also a quality tradeoff to test. Reusing an index assumes that neighboring layers need similar evidence, but successive layers transform what the token represents. The architecture is betting that selection changes slowly enough for reuse to save more than it loses.

Four residual streams and one predictive layer

A standard residual connection carries one main stream through the network and adds each block’s update. The original Hyper-Connections paper generalizes that path into multiple parallel residual streams whose connections can be mixed across depth. Hy4 uses an identity variant, abbreviated iHC, with four streams.

Four streams may improve information and gradient flow through a deep model. The Hy4 repository does not quantify iHC’s isolated contribution, however, so the feature deserves no solo credit for the model’s results. Treat it as a disclosed design choice, not a measured source of Hy4’s performance.

The multi-token-prediction layer attacks a different cost. It has 10B total parameters, 0.7B active, and proposes additional future tokens that a serving engine can verify together. Tencent’s vLLM example enables three speculative tokens, which can reduce serial decoding steps when the proposals are accepted.

MoE reduces feed-forward work, sparse attention reduces detailed comparisons, IndexCache reduces repeated selection, and MTP reduces serial generation. That systems perspective complements our hardware-side architecture analysis: model design and accelerator design are now chasing the same scarce resources from opposite directions.

Benchmarks meet deployment reality

Tencent reports a blind internal evaluation covering 203 engineering tasks and 163 company experts. Hy4 averaged 2.99, compared with 2.92 for GLM-5.3 and 2.94 for Kimi K3. Against those models, Tencent reports win/tie/loss shares of 46.8%/12.8%/40.4% and 51.2%/7.9%/40.9%, respectively.

This is more informative than a single cherry-picked prompt, but it remains a vendor-designed internal evaluation. Task selection, serving settings, reviewer population, and score aggregation all influence the result. Our guide to reading AI coding benchmarks explains why a narrow score advantage should become an evaluation hypothesis, not a procurement conclusion.

The repository also documents two preview limitations: Hy4 can spend longer reasoning than necessary and can over-verify work it has already completed. Both behaviors can consume the efficiency won elsewhere in the stack. A model that performs fewer mathematical operations but generates a needlessly long reasoning trace may still deliver poor time-to-answer economics.

Deployment makes the scale concrete. Hugging Face lists roughly 1.56TB for the complete checkpoint, while Tencent’s official vLLM and SGLang recipes use eight-way tensor parallelism. Actual memory needs vary with precision, cache size, batch load, and runtime overhead, but this is not a casual workstation model.

Open weights still matter: operators can inspect the configuration, run private infrastructure, and tune the serving stack. They do not remove the capital and engineering needed to keep hundreds of billions of parameters available. For context on how weights, quantization, and KV cache affect smaller deployments, see our practical VRAM sizing guide.

The architecture bet still needs a receipt

Can Hy4’s selective mechanisms preserve their theoretical savings after routing, indexing, distributed communication, and memory traffic are counted? The decisive measurement is not parameters owned or even parameters active. It is quality per GPU-hour on production-shaped workloads, alongside latency and memory.

The next scale axis may be how rarely a model must use each expensive component. Watch for reproducible third-party million-token serving tests—or Tencent’s first post-preview checkpoint with end-to-end efficiency data. That evidence will show whether Hy4’s 770B headline is a distraction or the capacity reserve behind a genuinely better compute bargain.