Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment

EMNLP 2026 Main Conference

Abstract

Retraining-free MoE compression reduces deployment memory by pruning or merging experts, but often treats the compressed checkpoint as the final artifact. We argue that this view is incomplete: compressed MoE checkpoints are better understood as compressed initializations that benefit from a tiny post-compression adjustment stage. Across two MoE LLM backbones, four pruning/merging methods, three expert-retention ratios, and 28 benchmarks, we compare LM fine-tuning and teacher-based KD under matched small-data budgets and measured GPU costs. Using only 3,000 C4 examples and a single epoch of adjustment, Full FT recovers 37.3% of the original-to-compressed performance gap on average. Moreover, LM fine-tuning is more cost-effective than standard token-level KD, and full-parameter adjustment gives the strongest cost--recovery trade-off among the tested scopes. These results suggest that retraining-free compression should be paired with small post-compression adjustment to recover a substantial portion of the performance lost during compression.

Repository layout

.
├── common/
│   └── README.md
├── qwen3/
│   ├── compression_qwen3/
│   ├── PCA_c4/
│   └── PCA_math/
├── gemma4/
│   ├── compression_gemma4/
│   ├── PCA_c4/
│   └── PCA_math/
└── additional_experiments/
    ├── qwen3/
    └── gemma4/

PCA denotes post-compression adjustment in this repository. The two backbones and their compressor pairs are:

Backbone	Expert pruning	Expert merging
Qwen3-30B-A3B-Instruct-2507	REAP	HC-SMoE
gemma-4-26B-A4B-it	AIMER	M-SMoE

Experiment flow

1. Obtain the original backbone checkpoint, then create the 50%, 62.5%, or 75% expert-retention checkpoints using the scripts under qwen3/compression_qwen3/ or gemma4/compression_gemma4/.
2. Run post-compression adjustment from each method directory under PCA_c4/All_strategy_*_c4/. These are the main 3,000-example C4 runs.
3. Set MAX_CALIB_SAMPLES=1024 in the same C4 launchers for the C4 sample-count robustness study. A duplicate PCA_c4_1024 source tree is intentionally not included because it has no operational difference beyond that default sample count.
4. Run the OpenR1-Math-220k robustness experiments from qwen3/PCA_math/.../HC-SMoE_math/ and gemma4/PCA_math/.../AIMER_math/, which are the two combinations reported in the paper.
5. Run downstream evaluation through the original lm-harness_results_* launchers. The six behavioral-proxy evaluation pipelines are retained under additional_experiments/.

Every method directory contains its own README and environment file where one was available in the experiment snapshot. Start with common/README.md for datasets and the lm-evaluation-harness version.

Adjustment settings

The paper compares 13 configurations: causal-LM fine-tuning and token-level KD at router-only, router plus 8/16/50 selected experts, router plus all experts, and full-model scopes, together with direct router-logit matching. Main runs use the first 3,000 non-empty C4 examples for one epoch with per-device batch size 2, gradient accumulation 4, maximum sequence length 512, learning rate 5e-5, zero weight decay, warmup ratio 0.03, gradient clipping at 1.0, KD temperature 1.0, and seed 42. Every adjustment job uses two NVIDIA H200 GPUs.

Model checkpoints are generated by compression and adjustment scripts but are not part of this repository. Experimental results, measurements, logs, and generated tables are also excluded. Cache, temporary, log, mirror, result, and checkpoint paths are ignored by Git. Use the environment variables documented beside each launcher to point to local models, the C4 file, the evaluation harness, and conda installation.

Evaluation

The reported downstream suite contains 28 benchmark rows across general knowledge, mathematics, code, chain-of-thought, and multiple-choice categories. The evaluation environment was Python 3.12.12, PyTorch 2.10.0+cu128, Transformers 5.8.1, vLLM 0.19.1, and lm-evaluation-harness 0.4.13.dev0.

The additional behavioral proxies are IFEval, TruthfulQA-MC1, ToxiGen, WMDP, CrowS-Pairs English, and WinoGender. They are reported independently rather than combined into a scalar “safety score.” See additional_experiments/README.md.

Third-party code and data

Except where otherwise noted, the authors' original code is licensed under the Apache License 2.0. REAP and AIMER files retain their Apache-2.0 license files; HC-SMoE and the UNITES-derived M-SMoE files retain their MIT license files. Third-party components remain under their respective licenses; see THIRD_PARTY_NOTICES.md.

The C4 shard, OpenR1-Math-220k, model checkpoints, and lm-evaluation-harness checkout are not vendored.

Citation

@inproceedings{hyeon2026beyond,
  title     = {Beyond Retraining-Free {MoE} Compression: A Cost-Normalized Study of Post-Compression Adjustment},
  author    = {Sieun Hyeon and Jaeyoung Do},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year      = {2026}
}