Published in ICLR, 2025
This paper introduces FlexPrefill, a flexible sparse pre-filling mechanism for large language models that dynamically adjusts attention patterns in real-time, improving speed and accuracy in long-sequence inference compared to prior sparse attention methods.
Published in arxiv, 2026
This paper introduces MiniMax Sparse Attention, a blockwise sparse attention mechanism built on GQA that preserves model quality while enabling efficient 1M-context inference with large reductions in attention compute and wall-clock latency.
Published in arxiv, 2026
This paper introduces the MiniMax-M2 series, a Mixture-of-Experts LLM family for agentic deployment that combines agent-driven data pipelines, scalable RL, and self-evolving training to deliver frontier-tier performance with only 9.8B activated parameters per token.
Published in arxiv, 2025
This paper comprehensively investigates model merging in pre-training, showing that merging constant-learning-rate checkpoints on dense/MoE architectures (millions to 100B+ params) improves performance, predicts annealing, boosts efficiency, reduces costs, and provides ablation-driven insights.
Published in arxiv, 2024
This paper proposes Fira, a new training framework for Large Language Models that achieves full-rank training performance while maintaining low-rank memory efficiency, outperforming existing approaches in pre-training and fine-tuning experiments.