{ "collected_at": "2026-09-14", "methodology_note": "Aggregate output-token throughput (max-batch) is the number that determines $/M tokens for a saturated server. Single-stream and interactivity-constrained numbers are labeled. Where a source reports per-GPU throughput, gpu_count reflects the source's unit; scaling to a full 8-GPU node is flagged as derived in notes. Every tokens_per_sec_total value traces to the source_url; null means the source exists but publishes no usable tok/s figure.", "throughput": [ { "model": "Llama 3.3 70B Instruct", "precision": "FP8", "gpu_config": "2xH100 SXM (TP2)", "gpu_count": 2, "accelerator_canonical": "H100-SXM", "tokens_per_sec_total": 6092.28, "throughput_mode": "max-batch", "framework": "TensorRT-LLM v0.21", "batch_context": "128 in / 128 out, infinite request rate (max load)", "source_url": "https://nvidia.github.io/TensorRT-LLM/performance/perf-overview.html", "source_date": "2025", "notes": "Official NVIDIA perf overview. Also 5,892.94 tok/s at 128in/2048out. An 8xH100 node running 4 independent TP2 replicas would be ~24.4k tok/s (derived, not published). Short-context figure; longer ISL reduces this." }, { "model": "Llama 3.3 70B Instruct", "precision": "FP8", "gpu_config": "2xH200 SXM (TP2)", "gpu_count": 2, "accelerator_canonical": "H200-SXM", "tokens_per_sec_total": 7467.36, "throughput_mode": "max-batch", "framework": "TensorRT-LLM v0.21", "batch_context": "128 in / 2048 out, infinite request rate (max load)", "source_url": "https://nvidia.github.io/TensorRT-LLM/performance/perf-overview.html", "source_date": "2025", "notes": "6,327.98 tok/s at 128/128. H200's HBM capacity advantage shows at longer output lengths. 8-GPU node as 4x TP2 replicas ~29.9k tok/s (derived)." }, { "model": "Llama 3.3 70B Instruct", "precision": "FP4 (NVFP4)", "gpu_config": "1xB200 (TP1)", "gpu_count": 1, "accelerator_canonical": "B200", "tokens_per_sec_total": 10613.84, "throughput_mode": "max-batch", "framework": "TensorRT-LLM v0.21", "batch_context": "128 in / 128 out, infinite request rate (max load)", "source_url": "https://nvidia.github.io/TensorRT-LLM/performance/perf-overview.html", "source_date": "2025", "notes": "70B fits on ONE B200 at FP4. 9,445.51 tok/s at 128/2048. 8xB200 node = 8 replicas ~85k tok/s (derived). FP4 vs FP8 is not apples-to-apples on quality." }, { "model": "Llama 3.3 70B", "precision": "FP4 (NVFP4)", "gpu_config": "B200 (per-GPU, InferenceMAX)", "gpu_count": 1, "accelerator_canonical": "B200", "tokens_per_sec_total": 10000, "throughput_mode": "max-batch", "framework": "TensorRT-LLM", "batch_context": "at 50 tok/s/user interactivity constraint (InferenceMAX)", "source_url": "https://blogs.nvidia.com/blog/blackwell-inferencemax-benchmark-results/", "source_date": "2025-10-09", "notes": "NVIDIA blog on SemiAnalysis InferenceMAX: 'over 10,000 tok/s per GPU at 50 TPS/user — 4x higher per-GPU throughput than H200' (implies H200 ~2,500 tok/s/GPU under same constraint, derived)." }, { "model": "Llama 3(.3) 70B", "precision": "FP8", "gpu_config": "8xH100 SXM (per-GPU plateau, InferenceMAX)", "gpu_count": 1, "accelerator_canonical": "H100-SXM", "tokens_per_sec_total": 900, "throughput_mode": "max-batch", "framework": "vLLM / TensorRT-LLM (InferenceMAX harness)", "batch_context": "1k in / 4k out reasoning-like workload; per-GPU plateau", "source_url": "https://newsletter.semianalysis.com/p/inferencemax-open-source-inference", "source_date": "2025-10-09", "notes": "SemiAnalysis InferenceMAX: H100 'quickly plateaus around 900 tok/s per GPU' on long-output workloads — i.e. ~7.2k tok/s per 8-GPU node (derived). Much lower than the 128/128 TRT number because long output = KV-cache bound. Use this for realistic chat/reasoning traffic, the TRT 128/128 number for short-context upper bound." }, { "model": "Llama 3.3 70B Instruct", "precision": "BF16", "gpu_config": "4xH100 SXM5 (NIM default TP4)", "gpu_count": 4, "accelerator_canonical": "H100-SXM", "tokens_per_sec_total": 7000, "throughput_mode": "max-batch", "framework": "NVIDIA NIM (TensorRT-LLM)", "batch_context": "200 in / 200 out, 500 concurrent users", "source_url": "https://blog.silexdata.com/blog/evaluating-llama-33-70b-inference-h100-a100/", "source_date": "2025-05-18", "notes": "Independent writeup (Silex Data). ~7,000 tok/s total across 4 GPUs at saturation, <5s TTFT to 500 sessions. Doubling to 8xH100 (2 replicas) ~14k tok/s (derived)." }, { "model": "Llama 3.3 70B Instruct", "precision": "BF16", "gpu_config": "4xA100 80GB PCIe (NIM TP4)", "gpu_count": 4, "accelerator_canonical": "A100-SXM", "tokens_per_sec_total": 570, "throughput_mode": "max-batch", "framework": "NVIDIA NIM (TensorRT-LLM)", "batch_context": "200 in / 200 out, saturated at ~50 concurrent users", "source_url": "https://blog.silexdata.com/blog/evaluating-llama-33-70b-inference-h100-a100/", "source_date": "2025-05-18", "notes": "CAVEAT: source hardware is A100 80GB PCIe, not SXM (canonical enum forced; treat as lower bound for SXM w/ NVLink). Same test as the 4xH100 row: H100 showed 12-14x advantage. No credible 8xA100-SXM node-level Llama-3.x-70B total-throughput publication found — this is the closest sourced observation." }, { "model": "Llama 3.1 70B Instruct", "precision": "BF16", "gpu_config": "4xMI300X (TP4)", "gpu_count": 4, "accelerator_canonical": "MI300X", "tokens_per_sec_total": null, "throughput_mode": "max-batch", "framework": "vLLM v0.6.2 (ROCm)", "batch_context": "ShareGPT, QPS sweep to saturation", "source_url": "https://vllm.ai/blog/2024-10-23-vllm-serving-amd", "source_date": "2024-10-23", "notes": "GAP: official vLLM/AMD blog publishes 5.76 req/s (ShareGPT, optimized settings; 1.8x TGI) but no tok/s figure. No credible absolute tok/s node-level publication found for Llama-3.x-70B on MI300X; SemiAnalysis (Dec 2024 + InferenceMAX) says MI300X competitive with H200-vLLM only at low interactivity. AMD ROCm 7.0 docs describe the Llama 3.3 70B FP8 vLLM benchmark recipe but publish no results (https://rocm.docs.amd.com/en/docs-7.0-rc1/preview/benchmark-docker/inference-vllm-llama-3.3-70b-fp8.html)." }, { "model": "Llama 2 70B (MLPerf closed)", "precision": "FP8", "gpu_config": "8xH200 SXM (700W)", "gpu_count": 8, "accelerator_canonical": "H200-SXM", "tokens_per_sec_total": 34864, "throughput_mode": "max-batch", "framework": "TensorRT-LLM (MLPerf Inference v4.1 offline)", "batch_context": "MLPerf Llama2-70B offline scenario; server scenario 32,790 tok/s", "source_url": "https://developer.nvidia.com/blog/nvidia-blackwell-platform-sets-new-llm-inference-records-in-mlperf-inference-v4-1/", "source_date": "2024-08-28", "notes": "ADJACENT MODEL: Llama 2 70B, not 3.x — same size class, audited MLPerf result. Best node-level 70B-class anchor for 8xH200." }, { "model": "Llama 2 70B (MLPerf closed)", "precision": "FP4", "gpu_config": "1xB200", "gpu_count": 1, "accelerator_canonical": "B200", "tokens_per_sec_total": 11264, "throughput_mode": "max-batch", "framework": "TensorRT-LLM (MLPerf Inference v4.1 offline, preview)", "batch_context": "MLPerf Llama2-70B offline; server 10,756 tok/s", "source_url": "https://developer.nvidia.com/blog/nvidia-blackwell-platform-sets-new-llm-inference-records-in-mlperf-inference-v4-1/", "source_date": "2024-08-28", "notes": "ADJACENT MODEL (Llama 2 70B). Audited MLPerf. Consistent with TRT-LLM v0.21 Llama 3.3 70B FP4 B200 number (10.6k) — two independent confirmations of ~10-11k tok/s per B200 for 70B-class FP4." }, { "model": "Mixtral 8x7B (MLPerf closed)", "precision": "FP8", "gpu_config": "8xH100 SXM", "gpu_count": 8, "accelerator_canonical": "H100-SXM", "tokens_per_sec_total": 52416, "throughput_mode": "max-batch", "framework": "TensorRT-LLM (MLPerf Inference v4.1 offline)", "batch_context": "MLPerf Mixtral-8x7B offline scenario; server 50,796 tok/s", "source_url": "https://developer.nvidia.com/blog/nvidia-blackwell-platform-sets-new-llm-inference-records-in-mlperf-inference-v4-1/", "source_date": "2024-08-28", "notes": "Fills the 'Mixtral-class on 8xH100' cell with an audited number. MoE: only ~13B active params, hence ~8x the 70B-dense throughput." }, { "model": "Mixtral 8x7B (MLPerf closed)", "precision": "FP8", "gpu_config": "8xH200 SXM", "gpu_count": 8, "accelerator_canonical": "H200-SXM", "tokens_per_sec_total": 59022, "throughput_mode": "max-batch", "framework": "TensorRT-LLM (MLPerf Inference v4.1 offline)", "batch_context": "MLPerf Mixtral-8x7B offline; server 57,177 tok/s", "source_url": "https://developer.nvidia.com/blog/nvidia-blackwell-platform-sets-new-llm-inference-records-in-mlperf-inference-v4-1/", "source_date": "2024-08-28", "notes": "Audited MLPerf result." }, { "model": "Llama 3.1 8B Instruct", "precision": "FP8", "gpu_config": "1xH100 SXM", "gpu_count": 1, "accelerator_canonical": "H100-SXM", "tokens_per_sec_total": 26401.48, "throughput_mode": "max-batch", "framework": "TensorRT-LLM v0.21", "batch_context": "128 in / 128 out, infinite request rate (max load)", "source_url": "https://nvidia.github.io/TensorRT-LLM/performance/perf-overview.html", "source_date": "2025", "notes": "21,413.21 tok/s at 128in/2048out. Upper bound (short context, FP8, TRT). vLLM BF16 realistic-traffic numbers are several times lower." }, { "model": "Llama 3.1 8B Instruct", "precision": "FP8", "gpu_config": "1xH200 SXM", "gpu_count": 1, "accelerator_canonical": "H200-SXM", "tokens_per_sec_total": 27027.8, "throughput_mode": "max-batch", "framework": "TensorRT-LLM v0.21", "batch_context": "128 in / 128 out, infinite request rate (max load)", "source_url": "https://nvidia.github.io/TensorRT-LLM/performance/perf-overview.html", "source_date": "2025", "notes": "23,102.25 tok/s at 128/2048. 8B barely benefits from H200 over H100 (compute-bound at short context)." }, { "model": "Llama 3 8B", "precision": "FP16", "gpu_config": "1xA100 80GB", "gpu_count": 1, "accelerator_canonical": "A100-SXM", "tokens_per_sec_total": 2400, "throughput_mode": "max-batch", "framework": "vLLM", "batch_context": "100 concurrent users, databricks-dolly-15k prompts (BentoCloud gpu.a100.1x80)", "source_url": "https://www.bentoml.com/blog/benchmarking-llm-inference-backends", "source_date": "2024-06-05", "notes": "BentoML cross-backend benchmark: vLLM 2,300-2,500 tok/s (midpoint recorded); LMDeploy hit ~4,000 tok/s on same hardware. Llama 3 (not 3.1) but identical architecture/size. Older, conservative." }, { "model": "Llama 3.1 8B", "precision": "unspecified (FP16 assumed)", "gpu_config": "1xL40S", "gpu_count": 1, "accelerator_canonical": "L40S", "tokens_per_sec_total": 325, "throughput_mode": "max-batch", "framework": "unspecified (vLLM implied)", "batch_context": "batch size 8; batch 1 = 43.8 tok/s (single-stream)", "source_url": "https://www.spheron.network/blog/best-nvidia-gpus-for-llms/", "source_date": "2026-09-06", "notes": "WEAK SOURCE: GPU-cloud marketing aggregator, framework/context not disclosed, and batch-8 is far from saturation — a saturated L40S should do considerably better. No first-party vLLM/NVIDIA L40S Llama-3.1-8B total-throughput publication found (NIM perf docs page has been removed/moved). Treat as indicative floor only." }, { "model": "DeepSeek-R1-Distill-Llama-8B (Llama 3.1 8B architecture)", "precision": "FP16", "gpu_config": "1xRTX 4090 24GB", "gpu_count": 1, "accelerator_canonical": "RTX4090", "tokens_per_sec_total": 2769.3, "throughput_mode": "max-batch", "framework": "vLLM", "batch_context": "100 in / 600 out, 300 requests, offline batch; online serving 2,699.72 tok/s", "source_url": "https://www.databasemart.com/blog/vllm-gpu-benchmark-rtx4090", "source_date": "2026-08-25", "notes": "Hosting-provider benchmark; model is the R1 distill onto the Llama 3.1 8B base, so architecturally identical to Llama 3.1 8B. Best available proxy — no first-party Llama-3.1-8B-on-4090 vLLM publication found; secondary sources scatter 100-2,550 tok/s depending on batch/quant." }, { "model": "DeepSeek-R1-0528 671B", "precision": "FP8 (native)", "gpu_config": "8xH200 SXM (TP8 + DP attention)", "gpu_count": 8, "accelerator_canonical": "H200-SXM", "tokens_per_sec_total": 19024.33, "throughput_mode": "max-batch", "framework": "SGLang v0.5.3", "batch_context": "2K-token prompts, high concurrency; 20,448 tok/s at 128-token prompts; 16,097 at 4K; 16,092 at 32K; vLLM v0.11 baseline 17,691 at 2K", "source_url": "https://docs.gpustack.ai/2.0/performance-lab/deepseek-r1/h200/", "source_date": "2025-2026 (GPUStack 2.0 performance lab, undated)", "notes": "Single-node 8xH200. Best-tuned config: --tp 8 --enable-dp-attention. ~2.4k output tok/s per H200 at realistic prompt lengths." }, { "model": "DeepSeek-V3 671B (37B active)", "precision": "FP8 (native)", "gpu_config": "H200 multi-node wide-EP (CoreWeave, InfiniBand CX-7)", "gpu_count": 1, "accelerator_canonical": "H200-SXM", "tokens_per_sec_total": 2200, "throughput_mode": "max-batch", "framework": "vLLM v0.11 (Wide-EP, DBO, EPLB, P/D disaggregation)", "batch_context": "sustained decode throughput per GPU, production-like multi-node deployment", "source_url": "https://vllm.ai/blog/2025-12-17-large-scale-serving", "source_date": "2025-12-17", "notes": "Official vLLM blog. Per-GPU decode figure (2.2k tok/s/H200, up from ~1.5k earlier in 2025). Cross-checks GPUStack single-node number (~2.4k/GPU). 8-GPU-equivalent ~17.6k tok/s (derived)." }, { "model": "DeepSeek-R1 671B", "precision": "FP4 (NVFP4)", "gpu_config": "8xB200 (DGX B200)", "gpu_count": 8, "accelerator_canonical": "B200", "tokens_per_sec_total": 30000, "throughput_mode": "max-batch", "framework": "TensorRT-LLM v0.17 (early Blackwell stack)", "batch_context": "1,024 in / 2,048 out; also >250 tok/s/user single-user mode", "source_url": "https://developer.nvidia.com/blog/nvidia-blackwell-delivers-world-record-deepseek-r1-inference-performance/", "source_date": "2025-03-18", "notes": "'Over 30,000 tok/s' system max throughput = ~3.75k/GPU. Vendor number, FP4 vs FP8 caveat. Software has improved substantially since (see InferenceX row)." }, { "model": "DeepSeek-R1-0528 671B", "precision": "FP8", "gpu_config": "B200 (per-chip, InferenceX)", "gpu_count": 1, "accelerator_canonical": "B200", "tokens_per_sec_total": 4852, "throughput_mode": "max-batch", "framework": "InferenceX harness (TRT-LLM/vLLM/SGLang best-of)", "batch_context": "8K in / 1K out, at 50 tok/s/user interactivity constraint; 1,461 at 89 tok/s/user; 1,051 at 128", "source_url": "https://inferencex.semianalysis.com/compare/deepseek-r1-b200-vs-h200", "source_date": "2026 (live dashboard, continuously updated)", "notes": "SemiAnalysis InferenceX (ex-InferenceMAX) open benchmark. tok/s/chip = total throughput per GPU subject to the latency constraint. B200 = 3x H200 at 50 tok/s/user." }, { "model": "DeepSeek-R1-0528 671B", "precision": "FP8", "gpu_config": "H200 (per-chip, InferenceX)", "gpu_count": 1, "accelerator_canonical": "H200-SXM", "tokens_per_sec_total": 1591, "throughput_mode": "max-batch", "framework": "InferenceX harness (TRT-LLM/vLLM/SGLang best-of)", "batch_context": "8K in / 1K out, at 50 tok/s/user interactivity constraint; 819 at 89 tok/s/user; 625 at 128", "source_url": "https://inferencex.semianalysis.com/compare/deepseek-r1-b200-vs-h200", "source_date": "2026 (live dashboard, continuously updated)", "notes": "Lower than GPUStack/vLLM ~2.2-2.4k per-GPU figures because of the 8K-token input length and 50 tok/s/user interactivity floor." }, { "model": "DeepSeek-R1 671B", "precision": "FP8", "gpu_config": "8xMI300X", "gpu_count": 8, "accelerator_canonical": "MI300X", "tokens_per_sec_total": 21224.6, "throughput_mode": "max-batch", "framework": "Moreh MoAI (expert parallelism; SGLang-compatible serving)", "batch_context": "decode throughput at highest concurrency tested", "source_url": "https://moreh.io/technical-report/21k-output-tokens-per-second-deepseek-inference-on-amd-instinct-mi300x-gpus-with-expert-parallelism-251113/", "source_date": "2025-11-13", "notes": "Vendor (Moreh) optimized stack, not vanilla SGLang — they separately claim 1.5x vanilla SGLang on MI300X. Shows MI300X parity with 8xH100-class DeepSeek serving when software is tuned. Cross-check: dstack (Mar 2025, early software) got only 4,574 tok/s online with vLLM on same hardware." }, { "model": "DeepSeek-R1 671B", "precision": "FP8", "gpu_config": "8xMI300X", "gpu_count": 8, "accelerator_canonical": "MI300X", "tokens_per_sec_total": 4574, "throughput_mode": "max-batch", "framework": "vLLM (ROCm, early 2025)", "batch_context": "3,200 in / 800-6,400 out sweep, concurrency 4-128; best online result", "source_url": "https://dstack.ai/blog/h200-mi300x-deepskeek-benchmark/", "source_date": "2025-03-18", "notes": "Independent (dstack). Same study: 8xH200 SGLang offline 6,311 tok/s, H200 TRT-LLM online 4,176 tok/s. Early-2025 software on both sides — conservative floor; Moreh row shows the tuned ceiling. Illustrates 4-5x software-dependent spread on identical MI300X hardware." } ], "api_prices": [ { "model": "Llama 3.3 70B Instruct", "provider": "Together AI (serverless)", "usd_per_m_output_tokens": 1.04, "usd_per_m_input_tokens": 1.04, "source_url": "https://www.together.ai/pricing" }, { "model": "Llama 3.1 8B Instruct", "provider": "Together AI (serverless)", "usd_per_m_output_tokens": 0.14, "usd_per_m_input_tokens": 0.14, "source_url": "https://www.together.ai/pricing" }, { "model": "Llama 3.3 70B Instruct Turbo", "provider": "DeepInfra", "usd_per_m_output_tokens": 0.32, "usd_per_m_input_tokens": 0.1, "source_url": "https://deepinfra.com/pricing" }, { "model": "Llama 3.1 8B Instruct Turbo", "provider": "DeepInfra", "usd_per_m_output_tokens": 0.04, "usd_per_m_input_tokens": 0.02, "source_url": "https://deepinfra.com/pricing" }, { "model": "DeepSeek-R1-0528", "provider": "DeepInfra", "usd_per_m_output_tokens": 2.15, "usd_per_m_input_tokens": 0.5, "source_url": "https://deepinfra.com/pricing" }, { "model": "DeepSeek-V3", "provider": "DeepInfra", "usd_per_m_output_tokens": 0.89, "usd_per_m_input_tokens": 0.32, "source_url": "https://deepinfra.com/pricing" }, { "model": "Generic dense 16B+ tier (covers Llama-70B class)", "provider": "Fireworks AI (serverless, catch-all tier)", "usd_per_m_output_tokens": 0.9, "usd_per_m_input_tokens": 0.9, "source_url": "https://docs.fireworks.ai/serverless/pricing" } ] }