NVIDIA H100 80GB HBM3 (8x) - gpt-oss-120b

November 13, 2025 at 08:07 AM

Dataset: reference (v1.0)

NVIDIA NVIDIA H100 80GB HBM3 openai gpt-oss-120b

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

18,672.47

Peak generation speed

Best Input TPS

50,200.55

Peak prefill speed

Best Energy Efficiency

0.02 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

1,364.94 ms

Lowest latency

Best E2E (P95)

3,219.48 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
2,048	2,048	1024x	18,672.47★	21,871.30	0.03	14,591.20	21,056.79	82,870.74	100.0%
128	512	1x	31.22	9.43	7.11	3,077.02	3,077.02	12,940.58	100.0%
128	512	2x	53.01	18.74	4.11	4,068.08	4,750.31	13,319.07	100.0%
128	512	4x	112.84	38.63	2.29	3,718.67	6,181.44	13,144.89	100.0%
128	512	8x	242.34	78.24	1.22	2,956.45	5,256.42	13,112.60	100.0%
128	512	16x	439.37	155.77	0.72	3,818.20	8,686.77	13,221.55	100.0%
128	512	32x	808.63	307.13	0.42	3,928.24	7,642.88	13,322.80	100.0%
128	512	64x	1,707.19	601.86	0.24	3,750.13	7,458.30	13,479.67	100.0%
128	512	128x	3,187.28	1,191.72	0.15	4,158.98	8,525.86	13,579.68	100.0%
128	512	256x	6,142.45	2,267.84	0.09	4,280.16	8,383.73	14,241.42	100.0%
128	512	512x	10,677.49	4,090.53	0.06	4,829.05	9,994.34	15,223.58	99.6%
128	512	1024x	10,809.45	4,060.51	0.06	5,443.74	11,304.59	17,904.88	99.8%
128	1,024	1x	36.35	4.75	7.18	2,201.79	2,201.79	25,695.08	100.0%
128	1,024	2x	73.26	9.75	3.83	2,136.36	2,293.36	25,622.29	100.0%
128	1,024	4x	123.22	19.25	2.30	5,443.93	9,159.02	26,376.03	100.0%
128	1,024	8x	237.82	38.73	1.26	3,619.85	6,536.87	26,499.87	100.0%
128	1,024	16x	534.53	77.73	0.70	3,125.70	6,381.25	26,521.72	100.0%
128	1,024	32x	983.72	153.51	0.42	3,856.45	7,724.00	26,684.32	100.0%
128	1,024	64x	1,952.67	301.73	0.25	4,120.38	8,139.09	26,952.46	100.0%
128	1,024	128x	3,870.79	601.06	0.15	4,139.06	8,618.73	27,093.94	100.0%
128	1,024	256x	7,393.02	1,152.03	0.09	4,144.45	8,731.05	28,085.83	100.0%
128	1,024	512x	13,109.68	2,088.96	0.06	4,616.80	10,064.19	29,988.59	100.0%
128	1,024	1024x	13,387.14	2,113.53	0.05	5,436.52	12,043.20	34,013.07	100.0%
128	2,048	1x	37.28	2.41	7.37	4,010.73	4,010.73	50,586.54	100.0%
128	2,048	2x	68.77	4.80	4.03	2,748.56	3,062.12	51,777.39	100.0%
128	2,048	4x	134.44	9.87	2.27	4,301.92	7,133.85	51,483.21	100.0%
128	2,048	8x	257.55	19.33	1.29	3,158.81	4,844.99	53,120.02	100.0%
128	2,048	16x	551.39	38.66	0.72	3,067.25	4,870.34	53,325.36	100.0%
128	2,048	32x	973.88	75.73	0.45	3,704.26	8,225.97	54,121.08	100.0%
128	2,048	64x	2,070.60	150.22	0.26	4,172.48	9,610.59	54,204.33	100.0%
128	2,048	128x	4,070.38	298.51	0.16	3,830.38	7,843.86	54,691.97	100.0%
128	2,048	256x	6,340.65	475.44	0.11	4,549.79	10,167.82	68,610.58	100.0%
128	2,048	512x	11,752.38	891.82	0.07	4,908.31	10,435.49	72,379.55	100.0%
128	2,048	1024x	17,803.15	1,334.70	0.05	5,347.72	11,446.77	71,066.57	100.0%
512	128	1x	16.19	131.67	1.96	2,005.77	2,005.77	3,767.62	100.0%
512	128	2x	32.03	303.54	0.88	1,967.31	2,170.30	3,274.48	100.0%
512	128	4x	50.70	453.88	0.64	1,895.56	2,774.46	3,272.19	75.0%
512	512	1x	34.61	38.75	4.03	1,766.49	1,766.49	12,800.64	100.0%
512	512	2x	61.54	78.10	2.29	2,975.75	3,133.87	12,731.75	100.0%
512	512	4x	109.72	131.13	1.34	2,588.61	3,403.07	14,794.88	100.0%
512	512	8x	192.46	254.00	0.80	3,747.11	5,588.54	15,533.13	100.0%
512	512	16x	372.09	502.63	0.46	3,367.26	6,258.86	15,778.67	100.0%
512	512	32x	737.70	994.89	0.25	3,625.61	7,696.62	16,020.80	100.0%
512	512	64x	1,368.49	1,957.62	0.15	4,236.01	7,802.63	16,233.07	100.0%
512	512	128x	2,681.41	3,878.39	0.09	4,337.84	7,787.74	16,329.10	100.0%
512	512	256x	5,378.37	7,453.19	0.06	4,332.63	7,673.25	16,932.95	100.0%
512	512	512x	9,927.11	13,726.87	0.04	5,187.63	9,023.13	18,224.20	100.0%
512	512	1024x	12,940.84	17,877.99	0.03	6,822.27	11,975.33	20,922.65	100.0%
512	1,024	1x	37.26	19.31	5.22	1,702.51	1,702.51	25,680.21	100.0%
512	1,024	2x	69.56	38.07	2.96	2,940.78	3,953.92	26,110.94	100.0%
512	1,024	4x	142.88	75.11	1.60	2,079.75	3,173.83	26,324.82	100.0%
512	1,024	8x	266.26	148.87	0.89	3,586.80	4,818.39	26,529.32	100.0%
512	1,024	16x	498.32	296.25	0.53	2,641.61	3,766.36	26,768.24	100.0%
512	1,024	32x	957.37	590.10	0.31	3,795.71	7,229.33	27,022.98	100.0%
512	1,024	64x	1,887.02	1,161.15	0.18	4,183.92	7,757.14	27,406.28	100.0%
512	1,024	128x	3,714.66	2,287.07	0.11	3,705.88	6,989.96	27,717.40	100.0%
512	1,024	256x	7,292.38	4,391.45	0.07	4,088.79	7,593.77	28,818.02	100.0%
512	1,024	512x	13,667.35	8,185.37	0.05	4,648.08	8,559.81	30,679.13	100.0%
512	1,024	1024x	18,422.95	11,014.94	0.03	6,126.06	11,300.22	36,522.04	100.0%
512	2,048	1x	38.72	9.71	6.07	1,709.28	1,709.28	51,105.28	100.0%
512	2,048	2x	77.82	19.73	3.27	1,962.13	2,203.40	50,435.72	100.0%
512	2,048	4x	125.69	37.14	2.06	2,457.71	4,063.75	53,245.15	100.0%
512	2,048	8x	260.50	73.64	1.10	3,561.26	4,693.07	53,634.29	100.0%
512	2,048	16x	473.73	148.82	0.67	2,710.07	5,243.89	53,316.63	100.0%
512	2,048	32x	973.01	296.08	0.37	3,531.13	6,142.36	53,870.40	100.0%
512	2,048	64x	1,865.24	540.15	0.23	3,727.68	7,427.90	58,924.52	100.0%
512	2,048	128x	3,387.61	986.35	0.14	3,834.05	7,080.87	64,393.68	100.0%
512	2,048	256x	6,539.64	1,906.24	0.09	4,132.68	7,975.19	66,452.59	100.0%
512	2,048	512x	12,142.72	3,500.36	0.06	4,849.86	9,100.78	72,115.89	100.0%
512	2,048	1024x	16,881.70	4,859.38	0.04	7,134.87	13,065.30	74,691.45	100.0%
1,024	512	1x	31.90	74.40	2.74	2,459.80	2,459.80	13,103.45	100.0%
1,024	512	2x	62.83	148.09	1.50	2,524.26	2,982.04	13,182.09	100.0%
1,024	512	4x	106.22	289.96	0.83	4,069.58	5,583.48	13,475.73	100.0%
1,024	512	8x	211.45	571.13	0.46	4,021.75	6,156.26	13,710.42	100.0%
1,024	512	16x	424.65	1,144.37	0.27	3,984.15	5,791.10	13,719.03	100.0%
1,024	512	32x	807.81	2,267.30	0.15	4,156.02	7,436.66	13,828.89	100.0%
1,024	512	64x	1,591.82	4,488.01	0.09	4,610.53	8,769.62	13,975.35	100.0%
1,024	512	128x	3,044.34	8,569.54	0.06	4,553.99	7,904.51	14,498.94	99.2%
1,024	512	256x	5,530.89	15,918.78	0.04	5,492.89	9,795.60	15,625.76	99.6%
1,024	512	512x	10,024.74	28,435.84	0.03	6,587.42	10,515.66	17,514.71	100.0%
1,024	512	1024x	11,553.80	32,773.70	0.02	9,307.16	15,200.77	22,178.16	99.7%
1,024	1,024	1x	32.10	37.62	4.22	4,948.67	4,948.67	25,918.23	100.0%
1,024	1,024	2x	71.87	74.90	2.17	2,220.01	2,276.56	26,045.60	100.0%
1,024	1,024	4x	134.92	149.75	1.23	3,701.63	5,062.49	26,097.03	100.0%
1,024	1,024	8x	256.89	291.04	0.71	4,154.29	5,688.76	26,909.33	100.0%
1,024	1,024	16x	507.06	583.44	0.39	4,161.71	6,876.91	26,913.72	100.0%
1,024	1,024	32x	990.09	1,157.32	0.22	4,269.72	8,008.59	27,118.42	100.0%
1,024	1,024	64x	1,855.76	2,283.68	0.13	4,419.38	8,968.40	27,479.85	100.0%
1,024	1,024	128x	3,747.41	4,489.55	0.08	4,306.07	7,584.16	27,911.64	100.0%
1,024	1,024	256x	7,168.29	8,683.47	0.05	4,729.05	8,829.91	28,777.64	100.0%
1,024	1,024	512x	13,469.81	15,866.79	0.04	5,446.98	9,339.39	31,434.87	100.0%
1,024	1,024	1024x	14,863.28	17,687.89	0.03	8,036.43	13,298.05	38,284.75	100.0%
1,024	2,048	1x	35.34	18.71	5.39	5,322.01	5,322.01	52,099.10	100.0%
1,024	2,048	2x	72.16	37.76	2.90	4,578.91	7,086.77	51,699.53	100.0%
1,024	2,048	4x	138.51	72.72	1.66	4,777.69	6,566.19	53,743.72	100.0%
1,024	2,048	8x	286.94	147.61	0.86	3,355.72	4,878.05	53,054.09	100.0%
1,024	2,048	16x	525.35	290.55	0.52	4,461.39	9,856.01	54,063.35	100.0%
1,024	2,048	32x	1,082.02	581.19	0.29	4,019.11	6,924.52	54,034.35	100.0%
1,024	2,048	64x	2,018.03	1,143.42	0.18	3,951.20	7,211.16	54,968.16	100.0%
1,024	2,048	128x	3,970.46	2,266.20	0.11	4,459.57	8,807.74	55,423.77	100.0%
1,024	2,048	256x	7,699.61	4,402.35	0.07	4,934.34	9,009.11	56,988.94	100.0%
1,024	2,048	512x	14,166.95	8,019.96	0.05	6,108.63	10,214.85	62,440.23	100.0%
1,024	2,048	1024x	17,979.88	10,267.40	0.03	9,691.90	16,959.27	76,422.89	100.0%
2,048	128	1x	23.61	611.37	0.46	1,364.94	1,364.94	3,219.48	100.0%
2,048	128	2x	14.23	595.82	0.52	2,099.58	2,099.58	3,246.21	50.0%
2,048	512	1x	32.72	154.79	1.57	2,421.60	2,421.60	12,714.26	100.0%
2,048	512	2x	60.12	290.86	0.85	2,938.37	3,467.49	13,470.95	100.0%
2,048	512	4x	117.28	582.26	0.50	3,099.03	3,863.92	13,455.93	100.0%
2,048	512	8x	180.91	1,175.43	0.29	4,562.60	7,816.41	13,310.00	100.0%
2,048	512	16x	368.04	2,266.74	0.16	4,433.41	7,789.38	13,796.61	100.0%
2,048	512	32x	755.76	4,574.31	0.09	4,516.34	7,800.66	13,647.61	100.0%
2,048	512	64x	1,457.47	8,613.28	0.06	5,041.78	8,266.29	14,480.79	100.0%
2,048	512	128x	2,769.19	16,120.56	0.04	5,645.28	9,260.51	15,452.64	100.0%
2,048	512	256x	5,021.59	29,872.90	0.03	6,876.39	11,113.59	16,646.74	100.0%
2,048	512	512x	8,418.47	50,200.55	0.02	9,143.15	14,351.75	19,613.36	99.6%
2,048	512	1024x	8,273.31	48,936.11	0.02	14,204.02	21,115.78	27,235.20	99.8%
2,048	1,024	1x	32.98	75.81	2.70	4,212.75	4,212.75	25,958.98	100.0%
2,048	1,024	2x	65.15	149.84	1.48	4,425.84	4,497.76	26,137.27	100.0%
2,048	1,024	4x	139.40	297.99	0.75	2,894.42	3,410.41	26,302.99	100.0%
2,048	1,024	8x	200.65	582.33	0.49	3,372.38	5,789.74	26,862.81	100.0%
2,048	1,024	16x	416.84	1,157.91	0.27	4,355.46	8,889.66	26,998.09	100.0%
2,048	1,024	32x	864.25	2,305.10	0.15	4,424.43	7,512.36	27,104.86	100.0%
2,048	1,024	64x	1,772.18	4,513.52	0.09	4,540.30	8,174.91	27,677.38	100.0%
2,048	1,024	128x	3,579.53	8,716.48	0.06	4,997.28	9,111.13	28,644.04	100.0%
2,048	1,024	256x	6,737.22	16,657.76	0.04	6,006.98	10,286.20	29,925.44	100.0%
2,048	1,024	512x	12,109.00	29,635.29	0.03	8,028.69	12,449.47	33,561.80	100.0%
2,048	1,024	1024x	15,343.33	37,646.45	0.02	14,487.42	20,218.99	45,000.76	100.0%
2,048	2,048	1x	38.49	38.86	3.80	2,481.14	2,481.14	50,636.77	100.0%
2,048	2,048	2x	74.67	75.40	1.97	2,784.15	2,795.01	51,974.62	100.0%
2,048	2,048	4x	104.48	151.64	1.26	3,656.24	4,795.75	51,600.04	100.0%
2,048	2,048	8x	192.12	292.94	0.76	3,973.53	6,003.97	53,418.88	100.0%
2,048	2,048	16x	372.27	583.07	0.43	4,096.18	7,922.12	53,626.70	100.0%
2,048	2,048	32x	874.58	1,157.18	0.23	4,369.15	7,449.95	54,042.98	100.0%
2,048	2,048	64x	1,825.17	2,263.66	0.14	5,223.31	9,970.30	55,233.80	100.0%
2,048	2,048	128x	3,846.41	4,423.24	0.08	5,591.07	9,451.80	56,494.14	100.0%
2,048	2,048	256x	7,204.48	8,512.34	0.05	6,826.63	11,870.12	58,669.83	100.0%
2,048	2,048	512x	13,178.61	15,408.00	0.04	9,057.79	12,811.69	64,681.28	100.0%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H100 80GB HBM3

GPU Count8

GPU Memory (Total)632 GB

GPU Driver570.195.03

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)700 W

CPU ModelIntel(R) Xeon(R) Platinum 8480+

RAM1,772 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.11.0

OSUbuntu

OS Version22.04.5 LTS (Jammy Jellyfish)

Kernel Version5.15.0-88-generic

Python Version3.10.12

Model Configuration

Provideropenai

Model Namegpt-oss-120b

QuantizationMXFP4

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length8192

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.95

Temperature0.70

Top-P1.00

Top-K-1