NVIDIA A100 80GB PCIe (2x) - gpt-oss-120b

November 2, 2025 at 09:38 PM

Dataset: reference (v1.0)

NVIDIA NVIDIA A100 80GB PCIe openai gpt-oss-120b

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

1,673.99

Peak generation speed

Best Input TPS

5,556.37

Peak prefill speed

Best Energy Efficiency

0.02 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

629.78 ms

Lowest latency

Best E2E (P95)

1,120.10 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	2,048	128x	1,673.99★	122.73	0.09	35,501.89	77,242.15	133,102.89	100.0%
128	512	1x	97.01	29.44	0.63	890.51	890.51	4,144.33	100.0%
128	512	2x	145.58	52.29	0.43	1,588.76	1,863.27	4,778.02	100.0%
128	512	4x	259.36	91.88	0.29	1,705.54	1,860.53	5,525.68	100.0%
128	512	8x	466.41	150.96	0.21	1,538.85	2,377.28	6,797.46	100.0%
128	512	16x	682.35	223.58	0.15	2,160.11	3,835.78	9,205.48	100.0%
128	512	32x	818.94	332.31	0.13	4,279.09	8,698.73	12,258.35	100.0%
128	512	64x	1,402.02	499.33	0.08	4,558.62	9,242.35	16,208.27	100.0%
128	512	128x	1,361.56	491.66	0.08	12,860.76	24,818.40	33,048.32	100.0%
128	1,024	1x	104.16	14.41	0.72	1,158.86	1,158.86	8,467.29	100.0%
128	1,024	2x	189.67	27.02	0.44	1,352.30	1,552.51	9,250.17	100.0%
128	1,024	4x	332.00	47.11	0.30	1,343.30	2,179.85	10,780.49	100.0%
128	1,024	8x	469.07	75.64	0.23	1,772.41	2,474.91	13,567.52	100.0%
128	1,024	16x	736.41	110.14	0.17	2,775.41	5,215.04	18,709.24	100.0%
128	1,024	32x	1,081.49	168.31	0.12	3,044.72	6,338.49	24,311.39	100.0%
128	1,024	64x	1,598.74	245.27	0.08	4,118.26	8,414.01	33,136.13	100.0%
128	1,024	128x	1,603.33	250.21	0.08	20,361.41	41,048.40	65,168.03	100.0%
128	1,024	256x	1,595.86	245.60	0.08	52,678.93	107,115.77	132,996.02	100.0%
128	2,048	1x	107.36	7.11	0.76	1,700.99	1,700.99	17,156.63	100.0%
128	2,048	2x	198.91	13.26	0.46	1,138.43	1,452.81	18,807.09	100.0%
128	2,048	4x	336.00	23.21	0.32	1,294.08	1,759.94	21,882.58	100.0%
128	2,048	8x	482.89	38.24	0.25	1,559.12	3,177.81	26,848.71	100.0%
128	2,048	16x	756.35	56.89	0.18	2,363.52	5,031.14	36,235.77	100.0%
128	2,048	32x	1,053.26	81.85	0.13	3,157.07	5,647.83	50,054.92	100.0%
128	2,048	64x	1,636.19	120.90	0.09	4,423.45	8,237.24	67,310.04	100.0%
128	2,048	256x	1,581.40	116.21	0.09	42,269.59	94,876.62	144,249.75	55.5%
512	128	1x	52.68	442.86	0.14	652.22	652.22	1,120.10	100.0%
512	128	2x	65.09	809.60	0.10	875.00	910.40	1,226.34	100.0%
512	128	4x	106.32	1,027.80	0.09	914.56	1,164.08	1,435.31	75.0%
512	512	1x	104.24	119.40	0.35	655.39	655.39	4,153.25	100.0%
512	512	2x	185.79	209.12	0.23	677.91	807.30	4,755.29	100.0%
512	512	4x	296.83	358.66	0.16	1,093.10	1,288.95	5,513.48	100.0%
512	512	8x	421.52	575.13	0.12	1,606.02	2,974.99	6,858.98	100.0%
512	512	16x	625.25	868.84	0.09	2,293.25	4,656.01	9,101.71	100.0%
512	512	32x	939.59	1,236.23	0.07	3,649.26	7,060.64	12,848.42	100.0%
512	512	64x	1,300.87	1,810.59	0.05	5,279.44	9,866.53	17,478.52	100.0%
512	512	128x	1,299.85	1,817.13	0.05	13,255.72	25,409.61	34,732.20	100.0%
512	1,024	1x	113.76	59.52	0.50	629.78	629.78	8,332.54	100.0%
512	1,024	2x	193.53	106.86	0.32	1,060.30	1,463.10	9,308.41	100.0%
512	1,024	4x	333.79	180.54	0.22	1,166.12	1,597.00	10,952.96	100.0%
512	1,024	8x	468.81	289.89	0.17	1,728.46	2,403.62	13,619.81	100.0%
512	1,024	16x	717.76	450.86	0.12	1,889.79	3,016.95	17,591.74	100.0%
512	1,024	32x	1,055.95	631.54	0.09	2,793.86	5,540.92	25,233.26	100.0%
512	1,024	64x	1,575.62	945.67	0.06	3,967.58	8,556.05	33,558.36	100.0%
512	1,024	128x	1,605.17	982.60	0.06	19,423.68	39,341.79	64,531.32	100.0%
512	1,024	256x	1,536.00	928.20	0.06	52,437.03	110,998.04	136,274.74	100.0%
512	2,048	1x	112.16	28.31	0.61	762.05	762.05	17,518.00	100.0%
512	2,048	2x	204.44	51.87	0.38	814.92	944.45	19,180.16	100.0%
512	2,048	4x	330.21	89.11	0.27	846.52	968.07	22,192.69	100.0%
512	2,048	8x	527.67	140.30	0.20	1,297.82	2,173.65	28,148.47	100.0%
512	2,048	16x	741.16	221.56	0.15	2,038.45	3,464.48	35,797.81	100.0%
512	2,048	32x	1,000.31	304.76	0.12	2,914.05	5,269.14	52,307.98	100.0%
512	2,048	64x	1,610.78	470.20	0.07	4,275.01	7,930.45	67,644.59	100.0%
512	2,048	128x	1,611.86	477.12	0.07	35,372.80	75,858.06	133,131.39	100.0%
512	2,048	256x	1,573.60	458.87	0.07	42,653.18	86,277.22	144,454.15	57.0%
1,024	512	1x	72.79	223.16	0.28	1,722.88	1,722.88	4,365.84	100.0%
1,024	512	2x	170.06	397.48	0.16	1,004.43	1,365.95	4,910.05	100.0%
1,024	512	4x	248.77	685.31	0.11	1,898.44	3,025.04	5,701.40	100.0%
1,024	512	8x	386.37	1,071.69	0.08	2,455.14	3,469.72	7,297.65	100.0%
1,024	512	16x	606.63	1,563.44	0.06	3,045.61	4,288.09	10,025.92	100.0%
1,024	512	32x	872.38	2,301.00	0.05	4,537.71	7,029.69	13,589.60	100.0%
1,024	512	64x	1,187.44	3,313.88	0.03	7,120.09	10,995.87	18,830.60	100.0%
1,024	512	128x	1,161.49	3,245.05	0.03	15,093.13	29,057.96	38,186.31	99.2%
1,024	1,024	1x	99.33	114.75	0.41	1,510.34	1,510.34	8,496.57	100.0%
1,024	1,024	2x	184.95	204.29	0.25	1,320.20	1,880.54	9,558.84	100.0%
1,024	1,024	4x	308.66	347.81	0.17	1,702.10	2,605.43	11,232.92	100.0%
1,024	1,024	8x	501.11	561.95	0.13	1,997.09	3,233.83	13,918.80	100.0%
1,024	1,024	16x	746.12	837.43	0.09	2,273.10	3,155.25	18,744.13	100.0%
1,024	1,024	32x	1,068.34	1,219.61	0.06	3,374.33	6,317.29	25,714.48	100.0%
1,024	1,024	64x	1,546.70	1,848.22	0.04	4,379.29	8,036.14	33,899.75	100.0%
1,024	1,024	128x	1,584.08	1,867.63	0.04	20,516.92	40,805.89	67,035.27	100.0%
1,024	1,024	256x	1,450.86	1,726.10	0.05	56,948.68	118,993.23	145,167.72	100.0%
1,024	2,048	1x	116.27	57.63	0.50	652.26	652.26	16,916.44	100.0%
1,024	2,048	2x	185.88	100.81	0.33	2,280.96	3,450.40	19,377.51	100.0%
1,024	2,048	4x	333.63	171.58	0.22	1,615.29	2,823.57	22,772.63	100.0%
1,024	2,048	8x	526.30	275.09	0.17	1,792.32	2,983.94	28,456.98	100.0%
1,024	2,048	16x	774.60	413.31	0.12	2,590.49	4,551.81	37,976.54	100.0%
1,024	2,048	32x	1,119.13	598.35	0.09	3,540.20	5,843.81	52,470.92	100.0%
1,024	2,048	64x	1,598.87	899.25	0.06	4,588.26	8,476.35	69,837.39	100.0%
1,024	2,048	128x	1,605.22	915.74	0.06	36,902.18	77,829.31	137,125.22	100.0%
1,024	2,048	256x	1,538.65	873.76	0.06	43,065.29	94,225.70	149,818.97	55.9%
2,048	128	1x	6.28	1,544.74	0.05	1,215.77	1,215.77	1,274.07	100.0%
2,048	128	2x	75.58	2,848.11	0.03	898.58	901.89	1,374.73	100.0%
2,048	128	4x	38.03	3,293.06	0.02	1,530.52	1,602.30	1,782.85	75.0%
2,048	512	1x	94.92	452.31	0.14	870.54	870.54	4,351.26	100.0%
2,048	512	2x	141.65	805.72	0.10	1,606.09	1,801.53	4,860.97	100.0%
2,048	512	4x	239.20	1,429.17	0.06	1,292.40	1,591.66	5,481.96	100.0%
2,048	512	8x	387.15	2,056.23	0.05	2,244.82	2,621.12	7,591.94	100.0%
2,048	512	16x	512.45	2,916.10	0.04	4,088.52	6,929.19	10,666.54	100.0%
2,048	512	32x	656.17	4,004.86	0.03	6,476.64	11,120.40	15,537.00	100.0%
2,048	512	64x	936.36	5,436.38	0.02	10,313.28	13,270.33	22,873.54	100.0%
2,048	512	128x	994.14	5,556.37	0.02	18,248.34	34,828.80	44,698.53	100.0%
2,048	512	256x	951.47	5,536.22	0.02	37,363.94	79,310.16	88,972.19	99.2%
2,048	1,024	1x	104.20	219.08	0.26	836.52	836.52	8,982.26	100.0%
2,048	1,024	2x	185.96	406.92	0.16	1,245.21	1,545.08	9,627.31	100.0%
2,048	1,024	4x	256.75	719.57	0.11	1,632.96	2,424.08	10,891.79	100.0%
2,048	1,024	8x	406.43	1,178.70	0.08	1,609.46	2,156.68	13,268.91	100.0%
2,048	1,024	16x	592.43	1,741.70	0.06	2,449.36	4,231.93	17,932.50	100.0%
2,048	1,024	32x	936.84	2,334.51	0.05	3,274.40	5,885.41	26,739.32	100.0%
2,048	1,024	64x	1,454.50	3,607.18	0.03	5,151.41	9,245.42	34,559.52	100.0%
2,048	1,024	128x	1,492.54	3,634.94	0.03	20,821.46	43,406.09	68,569.22	100.0%
2,048	1,024	256x	1,469.22	3,603.67	0.03	54,776.31	112,557.72	138,402.81	100.0%
2,048	2,048	1x	104.11	106.93	0.41	1,171.99	1,171.99	18,403.09	100.0%
2,048	2,048	2x	196.42	199.47	0.25	1,167.62	1,523.51	19,641.87	100.0%
2,048	2,048	4x	278.52	358.68	0.17	1,147.96	1,447.05	21,851.23	100.0%
2,048	2,048	8x	393.15	611.38	0.12	1,621.84	2,252.75	25,595.87	100.0%
2,048	2,048	16x	614.50	878.48	0.09	2,547.34	4,868.79	35,579.74	100.0%
2,048	2,048	32x	927.70	1,205.77	0.07	4,220.84	8,074.34	51,777.13	100.0%
2,048	2,048	64x	1,490.77	1,790.94	0.05	4,970.44	10,079.02	69,699.68	100.0%
2,048	2,048	128x	1,526.69	1,772.69	0.05	36,320.06	80,568.87	140,875.36	100.0%
2,048	2,048	256x	1,495.35	1,775.60	0.05	43,596.65	93,080.80	151,209.83	57.4%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA A100 80GB PCIe

GPU Count2

GPU Memory (Total)160 GB

GPU Driver570.195.03

CUDA VersionUnknown

Compute Capability8.0

Power Limit (per GPU)300 W

CPU ModelIntel Xeon Processor (Icelake)

RAM31 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.11.0

OSUbuntu

OS Version22.04.5 LTS (Jammy Jellyfish)

Kernel Version5.15.0-88-generic

Python Version3.10.12

Model Configuration

Provideropenai

Model Namegpt-oss-120b

QuantizationMXFP4

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length32768

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.90

Temperature0.70

Top-P1.00

Top-K-1