hiroppy's site

お問い合わせ

hiroppy's site

LMSYS Blog

lmsys.org/

Articles

12月24日 07:01

Last updated

SpecBundle & SpecForge v0.2: Production-Ready Speculative Decoding Models and Framework

<h2><a id="tldr" class="anchor" href="#tldr" aria-hidden="true"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 1...

LMSYS Blog2025-12-23

librarytool

Power Up Diffusion LLMs: Day‑0 Support for LLaDA 2.0

<h2><a id="tldr" class="anchor" href="#tldr" aria-hidden="true"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 1...

LMSYS Blog2025-12-19

librarytool

Mini-SGLang: Efficient Inference Engine in a Nutshell

<p>We're excited to introduce <strong>Mini-SGLang</strong>, a lightweight yet high-performance inference framework for Large Language Models (LLMs). Derived ...

LMSYS Blog2025-12-17

apitool

SGLang Day-0 Support for MiMo-V2-Flash Model

<h2><a id="introduction" class="anchor" href="#introduction" aria-hidden="true"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1...

LMSYS Blog2025-12-16

platform

SGLang Adds Day-0 Support for the Highly Efficient, Open Nemotron 3 Nano Hybrid MoE Model

<p>We are excited to announce that SGLang supports the latest highly efficient NVIDIA Nemotron 3 Nano model on Day 0!</p> <p>Nemotron 3 Nano, part of the new...

LMSYS Blog2025-12-15

librarytool

Let Tensors Fly — Accelerating Large Model Weight Loading with R-Fork

<h2><a id="tldr" class="anchor" href="#tldr" aria-hidden="true"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 1...

LMSYS Blog2025-12-10

frameworktool

Support FSDP2 as A Training Backend for Miles

<blockquote> <p><strong>TL;DR:</strong></p> <p><strong>We have added FSDP to <a href="https://github.com/radixark/miles">Miles</a> as a more flexible trainin...

LMSYS Blog2025-12-03

frameworktool

Boost SGLang Inference: Native NVIDIA Model Optimizer Integration for Seamless Quantization and Deployment

<p>(Updated on Dec 2)</p> <p>We are thrilled to announce a major new feature in SGLang: native support for <a href="https://github.com/NVIDIA/TensorRT-Model-...

LMSYS Blog2025-12-02

librarytool

From research to production: Accelerate OSS LLM with EAGLE-3 on Vertex

<p><strong>TL;DR:</strong> Speculative decoding boosts LLM inference, but traditional methods require a separate, inefficient draft model. Vertex AI utilizes...

LMSYS Blog2025-12-01

apitool

'Unified FP8: Moving Beyond Mixed Precision for Stable and Accelerated MoE RL'

<blockquote> <p>TL;DR: We have implemented fully FP8-based sampling and training in RL. Experiments show that for MoE models, the larger the model, the more ...

LMSYS Blog2025-11-25

librarytool

'Unified FP8: Moving Beyond Mixed Precision for Stable and Accelerated MoE RL'

<blockquote> <p>TL;DR: We have implemented fully FP8-based sampling and training in RL. Experiments show that for MoE models, the larger the model, the more ...

LMSYS Blog2025-11-24

tool

LMSYS Fellowship Program

<p>We're proud to launch the LMSYS Fellowship Program!</p> <p>This year, the program will provide funding to full-time PhD students in the United States who ...

LMSYS Blog2025-11-23

tool

'Introducing Miles — RL Framework To Fire Up Large-Scale MoE Training'

<blockquote> <p><em>A journey of a thousand miles is made one small step at a time.</em></p> </blockquote> <p>We're excited to introduce Miles, an enterprise...

LMSYS Blog2025-11-19

librarytool

🚀 AutoRound Meets SGLang: Enabling Quantized Model Inference with AutoRound

<h2><a id="overview" class="anchor" href="#overview" aria-hidden="true"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbo...

LMSYS Blog2025-11-14

librarytool

SGLang Diffusion: Accelerating Video and Image Generation

<p>We are excited to introduce SGLang Diffusion, which brings SGLang's state-of-the-art performance to accelerate image and video generation for diffusion mo...

LMSYS Blog2025-11-07

apilibrarytool

'No Free Lunch: Deconstruct Efficient Attention with MiniMax M2'

<p>We are excited to announce day-one support for the new flagship model, MiniMax M2, on SGLang. The MiniMax M2 redefines efficiency for agents: it is a comp...

LMSYS Blog2025-11-04

platform

'No Free Lunch: Deconstruct Efficient Attention with MiniMax M2'

<p>We are excited to announce day-one support for the new flagship model, MiniMax M2, on SGLang. The MiniMax M2 redefines efficiency for agents: it is a comp...

LMSYS Blog2025-11-03

tool

Optimizing GPT-OSS on NVIDIA DGX Spark: Getting the Most Out of Your Spark

<p>We’ve got some exciting updates about the <strong>NVIDIA DGX Spark</strong>! In the week following the official launch, we collaborated closely with NVIDI...

LMSYS Blog2025-11-03

apitool

SGLang-Jax: An Open-Source Solution for Native TPU Inference

<p>We're excited to introduce SGLang-Jax, a state-of-the-art open-source inference engine built entirely on Jax and XLA. It leverages SGLang's high-performan...

LMSYS Blog2025-10-29

librarytool

Accelerating Hybrid Inference in SGLang with KTransformers CPU Kernels

<h2><a id="background-hybrid-inference-for-sparse-moe-models" class="anchor" href="#background-hybrid-inference-for-sparse-moe-models" aria-hidden="true"><sv...

LMSYS Blog2025-10-22

frameworklibrarytool

NVIDIA and SGLang Accelerating SemiAnalysis InferenceMAX and GB200 Together

<p>The SGLang and NVIDIA teams have a strong track record of collaboration, consistently delivering inference optimizations and system-level improvements to ...

LMSYS Blog2025-10-14

apiframeworktool

NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

<p>Thanks to NVIDIA’s early access program, we are thrilled to get our hands on the NVIDIA DGX™ Spark. It’s quite an unconventional system, as NVIDIA rarely ...

LMSYS Blog2025-10-13

frameworktool

SGLang Day 0 Support for DeepSeek-V3.2 with Sparse Attention

<p>We are excited to announce that <strong>SGLang supports DeepSeek-V3.2 on Day 0</strong>! According to the DeepSeek <a href="https://github.com/deepseek-ai...

LMSYS Blog2025-09-29

librarytool

PD-Multiplexing: Unlocking High-Goodput LLM Serving with GreenContext

<p>This post highlights our initial efforts to support <strong>a new serving paradigm, PD-Multiplexing, in</strong> <strong>SGLang.</strong> It is designed t...

LMSYS Blog2025-09-28

librarytool

Together with SGLang: Best Practices for Serving DeepSeek-R1 on H20-96G

<h2><a id="introduction" class="anchor" href="#introduction" aria-hidden="true"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1...

LMSYS Blog2025-09-26

frameworktool

Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part II): 3.8x Prefill, 4.8x Decode Throughput

<p>The GB200 NVL72 is one of the most powerful hardware for deep learning. In this blog post, we share our progress to optimize the inference performance of ...

LMSYS Blog2025-09-25

librarytool

Enabling Deterministic Inference for SGLang

<p>This post highlights our initial efforts to achieve deterministic inference in SGLang. By integrating batch invariant kernels released by Thinking Machine...

LMSYS Blog2025-09-22

apitool

Optimizing FP4 Mixed-Precision Inference on AMD GPUs

<p>Haohui Mai (CausalFlow.ai), Lei Zhang (AMD)</p> <h2><a id="introduction" class="anchor" href="#introduction" aria-hidden="true"><svg aria-hidden="true" cl...

LMSYS Blog2025-09-21

librarytool

SGLang HiCache: Fast Hierarchical KV Caching with Your Favorite Storage Backends

<h2><a id="from-the-community" class="anchor" href="#from-the-community" aria-hidden="true"><svg aria-hidden="true" class="octicon octicon-link" height="16" ...

LMSYS Blog2025-09-10

librarytool

LongCat-Flash: Deploying Meituan's Agentic Model with SGLang

<h3><a id="1-introduction-deploying-meituans-agentic-open-source-moe-model" class="anchor" href="#1-introduction-deploying-meituans-agentic-open-source-moe-m...

LMSYS Blog2025-09-01

librarytool

Finetune and deploy GPT-OSS in MXFP4: ModelOpt+SGLang

<p>GPT-OSS, the first open-source model family from OpenAI's lab since GPT-2, demonstrates strong math, coding, and general capabilities even when compared w...

LMSYS Blog2025-08-28

librarytool

SGLang for gpt-oss: From Day 0 Support to Enhanced Performance

<p>We are excited to announce a major update for SGLang, focusing on deep performance optimizations and new features for the recently released openai/gpt-oss...

LMSYS Blog2025-08-27

librarytool

GLM-4.5 Meets SGLang: Reasoning, Coding, and Agentic Abilities

<p>Today, we are excited to introduce our latest flagship models <a href="https://huggingface.co/zai-org/GLM-4.5">GLM-4.5</a> and <a href="https://huggingfac...

LMSYS Blog2025-07-31

librarytool

SpecForge: Accelerating Speculative Decoding Training for SGLang

<p>Speculative decoding is a powerful technique for accelerating Large Language Model (LLM) inference. In this blog post, we are excited to announce the open...

LMSYS Blog2025-07-25

frameworktool

Deploying Kimi K2 with PD Disaggregation and Large-Scale Expert Parallelism on 128 H200 GPUs

<h2><a id="1️⃣-introduction-deploying-the-most-advanced-open-source-moe-model" class="anchor" href="#1️⃣-introduction-deploying-the-most-advanced-open-source...

LMSYS Blog2025-07-20

frameworktool

Accelerating SGLang with Multiple Token Prediction

<h2><a id="tldr" class="anchor" href="#tldr" aria-hidden="true"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 1...

LMSYS Blog2025-07-17

librarytool

How to support new VLMs into SGLang: A Case Study with NVILA

<p>The world of LLMs is evolving at a remarkable pace, with Visual Language Models (VLMs) at the forefront of this revolution. These models power application...

LMSYS Blog2025-07-16

apicloudtool

Cost Effective Deployment of DeepSeek R1 with Intel® Xeon® 6 CPU on SGLang

<p>The impressive performance of DeepSeek R1 marked a rise of giant Mixture of Experts (MoE) models in Large Language Models (LLM). However, its massive mode...

LMSYS Blog2025-07-14

librarytool

slime: An SGLang-Native Post-Training Framework for RL Scaling

<h2><a id="vision-that-drives-slime" class="anchor" href="#vision-that-drives-slime" aria-hidden="true"><svg aria-hidden="true" class="octicon octicon-link" ...

LMSYS Blog2025-07-09

frameworktool

OME: Revolutionizing LLM Infrastructure with Model-Driven Architecture

<h2><a id="the-tale-of-two-teams-why-model-serving-is-broken" class="anchor" href="#the-tale-of-two-teams-why-model-serving-is-broken" aria-hidden="true"><sv...

LMSYS Blog2025-07-08

cloudplatformtool

Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part I): 2.7x Higher Decoding Throughput

<p>The GB200 NVL72 is the world's most advanced hardware for AI training and inference. In this blog post, we're excited to share early results from running ...

LMSYS Blog2025-06-16

librarytool

Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs

<p>DeepSeek is a popular open-source large language model (LLM) praised for its strong performance. However, its large size and unique architecture, which us...

LMSYS Blog2025-05-05

LMSYS Blog

SpecBundle & SpecForge v0.2: Production-Ready Speculative Decoding Models and Framework

Power Up Diffusion LLMs: Day‑0 Support for LLaDA 2.0

Mini-SGLang: Efficient Inference Engine in a Nutshell

SGLang Day-0 Support for MiMo-V2-Flash Model

SGLang Adds Day-0 Support for the Highly Efficient, Open Nemotron 3 Nano Hybrid MoE Model

Let Tensors Fly — Accelerating Large Model Weight Loading with R-Fork

Support FSDP2 as A Training Backend for Miles

Boost SGLang Inference: Native NVIDIA Model Optimizer Integration for Seamless Quantization and Deployment

From research to production: Accelerate OSS LLM with EAGLE-3 on Vertex

'Unified FP8: Moving Beyond Mixed Precision for Stable and Accelerated MoE RL'

'Unified FP8: Moving Beyond Mixed Precision for Stable and Accelerated MoE RL'

LMSYS Fellowship Program

'Introducing Miles — RL Framework To Fire Up Large-Scale MoE Training'

🚀 AutoRound Meets SGLang: Enabling Quantized Model Inference with AutoRound

SGLang Diffusion: Accelerating Video and Image Generation

'No Free Lunch: Deconstruct Efficient Attention with MiniMax M2'

'No Free Lunch: Deconstruct Efficient Attention with MiniMax M2'

Optimizing GPT-OSS on NVIDIA DGX Spark: Getting the Most Out of Your Spark

SGLang-Jax: An Open-Source Solution for Native TPU Inference

Accelerating Hybrid Inference in SGLang with KTransformers CPU Kernels

NVIDIA and SGLang Accelerating SemiAnalysis InferenceMAX and GB200 Together

NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

SGLang Day 0 Support for DeepSeek-V3.2 with Sparse Attention

PD-Multiplexing: Unlocking High-Goodput LLM Serving with GreenContext

Together with SGLang: Best Practices for Serving DeepSeek-R1 on H20-96G

Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part II): 3.8x Prefill, 4.8x Decode Throughput

Enabling Deterministic Inference for SGLang

Optimizing FP4 Mixed-Precision Inference on AMD GPUs

SGLang HiCache: Fast Hierarchical KV Caching with Your Favorite Storage Backends

LongCat-Flash: Deploying Meituan's Agentic Model with SGLang

Finetune and deploy GPT-OSS in MXFP4: ModelOpt+SGLang

SGLang for gpt-oss: From Day 0 Support to Enhanced Performance

GLM-4.5 Meets SGLang: Reasoning, Coding, and Agentic Abilities

SpecForge: Accelerating Speculative Decoding Training for SGLang

Deploying Kimi K2 with PD Disaggregation and Large-Scale Expert Parallelism on 128 H200 GPUs

Accelerating SGLang with Multiple Token Prediction

How to support new VLMs into SGLang: A Case Study with NVILA

Cost Effective Deployment of DeepSeek R1 with Intel® Xeon® 6 CPU on SGLang

slime: An SGLang-Native Post-Training Framework for RL Scaling

OME: Revolutionizing LLM Infrastructure with Model-Driven Architecture

Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part I): 2.7x Higher Decoding Throughput

Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs

Power Up Diffusion LLMs: Day‑0 Support for LLaDA 2.0