NVIDIA H200 NVL (2x) - gpt-oss-120b

November 5, 2025 at 05:18 PM

Dataset: reference (v1.0)

NVIDIA NVIDIA H200 NVL openai gpt-oss-120b

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

3,166.06

Peak generation speed

Best Input TPS

11,929.37

Peak prefill speed

Best Energy Efficiency

0.01 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

357.63 ms

Lowest latency

Best E2E (P95)

716.58 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	2,048	64x	3,166.06★	231.42	0.07	2,794.33	4,892.25	34,597.65	100.0%
128	512	1x	139.50	46.25	0.57	767.40	767.40	2,637.14	100.0%
128	512	2x	282.08	86.00	0.29	620.30	756.09	2,899.47	100.0%
128	512	4x	452.54	145.68	0.22	825.13	958.61	3,477.27	100.0%
128	512	8x	728.63	243.19	0.15	1,046.76	1,494.41	4,202.05	100.0%
128	512	16x	1,056.09	390.94	0.12	1,688.32	2,952.67	5,214.76	100.0%
128	512	32x	1,397.37	503.56	0.09	3,343.76	4,820.62	8,093.25	100.0%
128	512	64x	2,251.33	874.39	0.07	3,291.33	6,009.23	9,095.05	98.4%
128	512	128x	2,492.75	942.62	0.07	6,188.37	12,018.29	16,337.42	100.0%
128	512	256x	2,286.84	848.73	0.07	15,823.03	28,945.04	34,137.99	99.6%
128	1,024	1x	172.52	22.90	0.66	553.01	553.01	5,326.20	100.0%
128	1,024	2x	317.55	43.19	0.35	639.32	713.64	5,785.12	100.0%
128	1,024	4x	514.89	74.90	0.24	1,032.72	1,380.33	6,774.34	100.0%
128	1,024	8x	802.35	124.30	0.18	1,206.45	2,009.11	8,242.68	100.0%
128	1,024	16x	1,267.33	196.48	0.13	1,592.59	2,526.73	10,444.02	100.0%
128	1,024	32x	1,898.29	312.30	0.10	1,847.92	3,358.79	13,031.04	100.0%
128	1,024	64x	2,988.10	457.78	0.07	2,318.95	4,563.89	17,244.77	100.0%
128	1,024	128x	3,035.63	474.81	0.07	10,231.29	20,242.73	32,968.62	100.0%
128	1,024	256x	3,040.37	476.34	0.07	25,402.61	50,855.47	64,192.32	100.0%
128	1,024	512x	2,948.89	469.45	0.07	56,805.81	114,072.15	126,681.55	100.0%
128	2,048	1x	169.85	13.15	0.62	978.15	978.15	9,270.99	100.0%
128	2,048	2x	246.91	22.90	0.45	661.51	903.18	10,610.19	100.0%
128	2,048	4x	542.97	37.44	0.26	892.07	1,456.73	13,560.76	100.0%
128	2,048	8x	839.08	61.60	0.19	1,116.59	2,041.58	16,643.81	100.0%
128	2,048	16x	1,313.39	97.27	0.14	1,442.19	3,477.07	21,150.40	100.0%
128	2,048	32x	2,021.59	153.77	0.10	1,797.66	2,948.48	26,551.93	100.0%
128	2,048	128x	3,163.76	232.05	0.07	17,907.02	38,204.59	68,597.57	100.0%
512	128	1x	99.02	691.77	0.17	357.63	357.63	716.58	100.0%
512	128	2x	71.16	1,264.29	0.07	631.63	673.00	782.80	100.0%
512	512	1x	165.01	182.29	0.33	359.44	359.44	2,720.90	100.0%
512	512	2x	283.03	340.52	0.18	594.56	706.28	2,918.81	100.0%
512	512	4x	499.71	572.34	0.13	597.56	714.99	3,449.06	100.0%
512	512	8x	746.37	939.80	0.09	1,055.14	1,673.32	4,183.24	100.0%
512	512	16x	1,179.50	1,509.03	0.07	1,331.72	2,255.26	5,221.01	100.0%
512	512	32x	1,700.76	2,318.56	0.05	1,849.61	2,908.99	6,783.01	100.0%
512	512	64x	1,973.21	2,772.51	0.04	5,224.63	7,396.21	11,309.15	100.0%
512	512	128x	2,462.84	3,514.96	0.04	6,316.75	12,005.54	16,870.40	100.0%
512	512	256x	2,566.01	3,527.03	0.04	13,555.72	26,755.81	32,251.16	100.0%
512	512	512x	2,383.92	3,331.80	0.04	31,117.04	62,341.32	68,201.75	100.0%
512	1,024	1x	176.77	92.19	0.40	421.03	421.03	5,379.04	100.0%
512	1,024	2x	285.41	151.54	0.27	1,281.18	1,381.54	6,557.91	100.0%
512	1,024	4x	464.46	247.50	0.19	1,736.84	1,901.19	7,982.94	100.0%
512	1,024	8x	733.05	456.28	0.14	1,356.45	2,029.47	8,644.40	100.0%
512	1,024	16x	1,172.79	760.96	0.10	1,295.31	2,387.09	10,380.17	100.0%
512	1,024	32x	1,900.10	1,195.31	0.07	1,741.47	3,161.63	13,228.61	100.0%
512	1,024	64x	2,957.82	1,804.23	0.05	2,154.10	4,199.27	17,148.13	100.0%
512	1,024	128x	3,000.67	1,857.81	0.05	9,632.09	19,761.48	32,959.14	100.0%
512	1,024	256x	2,954.27	1,773.83	0.05	25,528.73	53,707.94	66,922.13	100.0%
512	2,048	1x	182.85	70.20	0.45	407.32	407.32	7,057.89	100.0%
512	2,048	2x	332.59	85.64	0.31	670.65	802.25	11,606.98	100.0%
512	2,048	4x	565.75	144.99	0.21	823.05	1,086.72	13,630.64	100.0%
512	2,048	8x	763.77	238.28	0.17	1,039.03	1,818.08	16,554.54	100.0%
512	2,048	16x	1,263.76	377.76	0.12	1,216.89	2,264.92	20,957.71	100.0%
512	2,048	32x	2,026.90	587.32	0.09	1,744.28	3,304.92	27,023.75	100.0%
512	2,048	64x	3,134.69	906.92	0.06	2,063.75	3,609.42	34,676.29	100.0%
512	2,048	128x	3,003.04	886.40	0.06	19,262.61	40,598.16	70,035.44	100.0%
1,024	512	1x	166.23	367.51	0.21	405.06	405.06	2,652.73	100.0%
1,024	512	2x	197.96	663.50	0.13	1,300.94	1,454.04	2,938.66	100.0%
1,024	512	4x	343.37	1,121.34	0.09	1,520.08	1,985.17	3,482.97	100.0%
1,024	512	8x	664.23	1,790.40	0.06	1,356.84	2,139.20	4,354.32	100.0%
1,024	512	16x	1,032.86	2,884.34	0.05	1,825.23	2,989.49	5,404.87	100.0%
1,024	512	32x	1,311.85	3,547.07	0.04	4,061.34	5,308.46	8,823.33	100.0%
1,024	512	64x	2,246.52	6,627.05	0.03	3,463.64	6,357.58	9,321.21	100.0%
1,024	512	128x	2,298.11	6,477.09	0.03	7,050.46	13,584.02	18,158.37	100.0%
1,024	512	256x	2,214.29	6,181.30	0.03	15,761.71	30,923.46	36,499.18	100.0%
1,024	1,024	1x	144.68	183.44	0.40	1,351.32	1,351.32	5,314.42	100.0%
1,024	1,024	2x	305.72	331.69	0.20	730.52	816.79	5,883.94	100.0%
1,024	1,024	4x	445.82	536.88	0.14	1,899.53	2,621.89	7,270.81	100.0%
1,024	1,024	8x	810.81	899.21	0.10	1,464.95	2,701.46	8,691.47	100.0%
1,024	1,024	16x	1,306.11	1,473.41	0.07	1,363.74	2,651.95	10,617.69	100.0%
1,024	1,024	32x	2,008.35	2,362.75	0.05	2,239.77	4,233.58	13,193.59	100.0%
1,024	1,024	64x	2,955.19	3,568.21	0.04	2,477.71	4,171.07	17,296.41	100.0%
1,024	1,024	128x	3,028.37	3,646.65	0.04	10,114.91	19,759.11	33,295.32	100.0%
1,024	1,024	256x	3,011.90	3,591.26	0.04	25,548.28	51,316.16	64,949.26	100.0%
1,024	2,048	1x	167.82	92.08	0.46	1,410.68	1,410.68	10,587.50	100.0%
1,024	2,048	2x	337.20	169.21	0.25	625.29	741.48	11,537.62	100.0%
1,024	2,048	4x	547.77	287.26	0.18	1,241.87	1,637.10	13,598.42	100.0%
1,024	2,048	8x	889.23	471.04	0.13	1,466.98	2,561.77	16,602.01	100.0%
1,024	2,048	16x	1,354.83	738.96	0.10	1,681.72	2,918.49	21,190.79	100.0%
1,024	2,048	32x	2,120.64	1,162.63	0.07	2,138.18	4,072.99	26,896.69	100.0%
1,024	2,048	64x	2,971.46	1,682.55	0.05	4,190.01	6,090.61	37,166.37	100.0%
1,024	2,048	128x	3,058.12	1,747.26	0.05	18,814.37	39,916.83	70,691.92	100.0%
1,024	2,048	256x	3,041.67	1,701.32	0.05	53,040.41	112,879.99	143,143.94	100.0%
2,048	128	1x	22.27	1,327.94	0.06	1,315.19	1,315.19	1,480.55	100.0%
2,048	128	2x	67.46	4,637.87	0.01	682.93	731.36	842.35	100.0%
2,048	128	4x	46.08	3,852.94	0.03	859.10	908.50	1,008.71	50.0%
2,048	512	1x	160.07	727.54	0.14	448.77	448.77	2,704.97	100.0%
2,048	512	2x	167.17	1,307.64	0.08	1,594.84	2,179.02	2,989.95	100.0%
2,048	512	4x	416.36	2,249.35	0.05	1,031.49	1,438.50	3,477.33	100.0%
2,048	512	8x	505.06	3,598.16	0.04	1,441.67	2,355.75	4,330.46	100.0%
2,048	512	16x	581.24	3,604.63	0.04	4,958.51	5,733.85	8,630.60	100.0%
2,048	512	32x	1,203.31	7,439.31	0.02	3,653.32	5,218.25	8,301.14	100.0%
2,048	512	64x	2,075.49	11,929.37	0.02	4,275.16	6,441.14	10,292.47	100.0%
2,048	512	128x	2,035.00	11,873.45	0.02	7,956.69	15,248.35	19,847.94	99.2%
2,048	1,024	1x	169.57	369.16	0.20	643.96	643.96	5,329.50	100.0%
2,048	1,024	2x	299.84	677.68	0.13	933.11	1,248.12	5,778.19	100.0%
2,048	1,024	4x	496.85	1,146.89	0.09	824.25	964.35	6,823.11	100.0%
2,048	1,024	8x	635.69	1,879.44	0.07	1,130.86	1,665.57	8,314.42	100.0%
2,048	1,024	16x	1,140.41	2,945.76	0.05	1,385.48	2,663.91	10,566.95	100.0%
2,048	1,024	32x	1,703.38	4,575.98	0.03	2,118.62	4,059.98	13,594.79	100.0%
2,048	1,024	64x	2,740.73	7,064.69	0.02	2,924.97	5,461.09	17,544.33	100.0%
2,048	1,024	128x	2,880.05	7,116.62	0.02	10,296.59	21,638.17	34,077.40	100.0%
2,048	1,024	256x	2,616.20	6,460.15	0.03	28,170.45	59,963.84	72,883.81	100.0%
2,048	2,048	1x	176.15	185.49	0.32	962.39	962.39	10,607.08	100.0%
2,048	2,048	2x	336.50	338.57	0.19	623.64	691.53	11,570.42	100.0%
2,048	2,048	4x	443.50	593.68	0.14	985.30	1,477.49	13,198.39	100.0%
2,048	2,048	8x	659.95	1,001.60	0.10	1,138.36	1,831.02	15,605.57	100.0%
2,048	2,048	16x	959.61	1,457.37	0.08	1,324.57	2,263.83	21,396.93	100.0%
2,048	2,048	32x	1,736.78	2,315.51	0.05	2,302.80	3,895.19	26,949.05	100.0%
2,048	2,048	64x	2,713.97	3,373.62	0.04	3,129.56	7,626.58	36,870.71	100.0%
2,048	2,048	128x	3,044.21	3,510.80	0.04	18,074.97	40,007.85	70,377.16	100.0%
2,048	2,048	256x	3,039.96	3,569.92	0.04	49,029.01	107,612.06	135,564.76	100.0%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H200 NVL

GPU Count2

GPU Memory (Total)280 GB

GPU Driver580.95.05

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)600 W

CPU ModelIntel(R) Xeon(R) 6960P

RAM2,267 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.11.0

OSUbuntu

OS Version22.04.5 LTS (Jammy Jellyfish)

Kernel Version5.15.0-88-generic

Python Version3.10.12

Model Configuration

Provideropenai

Model Namegpt-oss-120b

QuantizationMXFP4

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model LengthUnknown

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.80

Temperature0.70

Top-P1.00

Top-K-1