生产级模型 API 的性能观察

                                                                                                                                   Throughput metrics for models.

‍

                                                                                                                                Latency metrics for different models.

GPT‑OSS‑20B delivered substantially higher throughput, achieving roughly 2.5× the request and token throughput of both Gemma 3 and Llama 4. It also streamed responses faster, sustaining approximately 121 tokens per second per user—more than double the other models.

Gemma 3 27B‑it and Llama 4 Scout 17B‑16E showed similar streaming characteristics, but with significantly longer time‑to‑first‑token (TTFT). Gemma 3, in particular, exhibited the highest TTFT, indicating that prefill computation was a primary bottleneck under these conditions.

Short prompt workloads emphasize token generation efficiency over prefill latency. Under this regime, GPT‑OSS‑20B’s kernel and attention optimizations translated directly into a more interactive, responsive experience.

Concurrency Ramping and Saturation Behavior

Baseline performance captures only part of the picture. In production, systems must absorb traffic bursts and sustain concurrency without degrading user experience. To evaluate this, prompts filtered from the ShareGPT dataset were sent to single-replica endpoints with progressively increasing maximum concurrency. The same prompt set was used for consistency, and endpoints were reset between runs to eliminate caching effects.

As concurrency increases, the serving stack must handle overlapping workloads, larger batches, and growing contention for GPU memory and compute. The point at which requests begin to fail, latency rises, and throughput plateaus or declines marks the threshold where user experience degrades and additional scaling is required.

Concurrency ramping used identical inputs (1–63 tokens) and a maximum output length of 100 tokens across GPT-OSS-20B, Gemma 3 27B, and LLaMa 4 Scout 17B-16E Instruct. The results are shown in the graphs below.

‍

Figure 1 shows GPT-OSS outperforming Gemma and LLaMa at concurrencies below 1600. Beyond this point, both Gemma and GPT exhibit degraded response times, with rising instability and batch loss. Although Gemma 27B and GPT-OSS-20B scale sharply at low concurrency, both are single-GPU models, which likely limits their ability to sustain higher concurrent loads relative to LLaMa.

LLaMa, by contrast, is a larger Mixture-of-Experts model spanning four GPUs. This leads to slower initial scaling due to routing and synchronization overhead, but supports acceptable performance and request success rates up to roughly 4000 concurrent in-flight requests. LLaMa peaks early (around 500 concurrent requests) before gradually declining in token throughput, unlike Gemma’s flatter plateau or GPT’s later peak before instability.

From a production perspective, GPT-OSS offers the best balance of throughput, scalability, and stability. Gemma saturates early with the lowest token throughput, while LLaMa trades efficiency for greater concurrency tolerance.

GPT shows the steepest initial scaling, increasing from ~35 req/s at concurrency 32 to over 100 req/s at a maximum concurrency of 256. Throughput then continues to climb, reaching a stable plateau of ~130–136 req/s between ~800 and ~1400 concurrency. The plateau is roughly twice as fast as Gemma’s, indicating superior utilisation of GPU resources and more efficient scheduling. The fluctuations present at high concurrency are due to batches and requests being lost as the service loses stability.

For the Gemma model, request throughput increases from ~8 req/s at concurrency 16 to ~49 req/s at 200, driven by improved batching efficiency. Throughput then rises gradually, reaching a flat plateau of ~56–59 req/s between 400 and ~1400 concurrency, indicating early saturation and limited benefit from additional inflight requests.

Minor fluctuations appear at higher concurrency, including a dip around 1500 concurrency (~53 req/s) caused by request loss. Beyond ~1600 concurrency, throughput becomes unstable and request success rates drop sharply, signaling unreliable operation.

The LLaMA model shows slower scaling at low concurrency, increasing from ~9 req/s at concurrency 16 to a peak of ~70 req/s at ~512 concurrency. Unlike GPT and Gemma, LLaMA reaches its maximum throughput earlier, after which throughput gradually declines. Throughput decreases slowly from ~70 req/s at 512 to ~62–64 req/s in the 1000–2500 concurrency range. A significant drop occurs at 3250 concurrent requests corresponding to one of the runs at that concurrency losing a large batch of requests.