NVIDIA A100-SXM4-80GB (2x) - gpt-oss-120b Reasoning

December 19, 2025 at 07:06 AM

Dataset: reference (v1.0)

NVIDIA NVIDIA A100-SXM4-80GB openai gpt-oss-120b

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

3,860.39

Peak generation speed

Best Input TPS

15,892.90

Peak prefill speed

Best Energy Efficiency

0.01 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

46.30 ms

Lowest latency

Best E2E (P95)

938.60 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	512	256x	3,860.39★	1,005.73	0.03	441.22	622.23	32,387.74	100.0%
128	128	1x	126.95	126.95	0.30	75.55	75.55	938.60	100.0%
128	128	2x	222.83	228.31	0.15	78.00	81.07	1,093.08	100.0%
128	128	4x	388.98	400.00	0.10	95.52	97.84	1,268.98	100.0%
128	128	8x	613.64	648.36	0.07	127.75	131.18	1,580.34	100.0%
128	128	16x	944.20	1,000.97	0.05	202.13	206.94	2,049.25	100.0%
128	128	32x	1,365.33	1,442.33	0.04	325.17	333.22	2,822.87	100.0%
128	128	64x	1,874.94	1,961.04	0.03	500.98	579.22	4,091.66	100.0%
128	128	128x	2,524.84	2,650.75	0.02	847.58	1,012.87	6,068.41	100.0%
128	128	256x	3,244.61	3,416.99	0.02	1,370.87	1,853.33	9,403.47	100.0%
128	128	512x	3,265.41	3,425.09	0.02	5,340.76	11,095.74	18,549.51	100.0%
128	128	1024x	3,305.24	3,472.76	0.02	13,689.25	28,900.37	36,415.39	100.0%
128	512	1x	136.20	33.04	0.52	46.83	46.83	3,692.92	100.0%
128	512	2x	237.53	59.09	0.31	56.33	59.39	4,229.62	100.0%
128	512	4x	410.28	103.59	0.21	63.76	66.65	4,902.57	100.0%
128	512	8x	655.16	167.21	0.15	76.97	80.51	6,139.23	100.0%
128	512	16x	973.23	254.50	0.11	96.37	100.78	8,093.40	100.0%
128	512	32x	1,393.09	372.13	0.09	120.97	130.52	10,997.03	100.0%
128	512	64x	1,987.21	526.54	0.06	186.05	210.02	15,426.25	100.0%
128	512	128x	2,864.86	743.01	0.04	280.19	341.26	21,916.04	100.0%
128	512	512x	3,809.05	1,003.67	0.03	16,213.08	33,000.36	64,673.26	100.0%
128	1,024	1x	135.71	16.31	0.59	48.38	48.38	7,478.96	100.0%
128	1,024	2x	237.89	29.40	0.37	56.80	59.84	8,502.18	100.0%
128	1,024	4x	390.23	51.83	0.26	64.48	67.60	9,800.21	100.0%
128	1,024	8x	613.17	83.20	0.19	64.46	67.63	12,340.01	100.0%
128	1,024	16x	970.77	123.32	0.14	94.11	98.38	16,716.69	100.0%
128	1,024	32x	1,358.82	179.85	0.10	124.71	134.22	22,781.08	100.0%
128	1,024	64x	1,941.33	255.29	0.08	211.97	231.90	31,882.30	100.0%
128	1,024	128x	2,811.10	371.88	0.05	300.78	365.74	43,916.35	100.0%
128	1,024	256x	3,797.16	502.51	0.04	402.15	532.60	64,988.36	100.0%
128	1,024	512x	3,641.12	486.61	0.04	32,810.80	69,321.69	134,028.57	100.0%
128	2,048	1x	134.95	8.08	0.66	48.00	48.00	15,094.23	100.0%
128	2,048	2x	235.81	16.00	0.41	58.74	61.78	15,611.64	100.0%
128	2,048	4x	393.69	25.38	0.29	63.11	66.35	20,010.37	100.0%
128	2,048	8x	582.56	42.00	0.22	72.24	75.40	24,449.30	100.0%
128	2,048	16x	908.57	60.71	0.16	96.43	100.46	33,966.52	100.0%
128	2,048	32x	1,215.12	86.30	0.13	131.11	141.44	47,498.74	100.0%
128	2,048	64x	1,838.11	125.29	0.09	201.31	229.30	65,020.27	100.0%
128	2,048	128x	2,709.69	183.94	0.06	290.57	351.93	88,907.96	100.0%
128	2,048	256x	3,555.36	245.52	0.04	1,360.96	1,832.19	133,441.07	100.0%
128	2,048	512x	3,174.31	221.06	0.05	11,091.59	101,508.68	168,360.82	59.2%
512	128	1x	118.29	493.04	0.09	107.41	107.41	1,005.84	100.0%
512	128	2x	209.69	876.65	0.08	112.77	116.65	1,134.37	100.0%
512	128	4x	352.85	1,466.27	0.06	178.39	184.36	1,346.42	100.0%
512	128	8x	573.93	2,308.59	0.04	272.38	275.46	1,708.00	100.0%
512	128	16x	818.84	3,366.99	0.03	479.18	483.90	2,345.62	100.0%
512	128	32x	1,147.25	4,719.40	0.02	878.85	887.32	3,357.57	100.0%
512	128	64x	1,523.08	6,206.43	0.02	1,524.55	1,637.10	5,078.37	100.0%
512	128	128x	1,896.68	7,737.45	0.01	2,734.73	3,074.44	8,125.10	100.0%
512	128	256x	2,266.86	9,279.69	0.01	4,302.84	5,959.66	13,523.68	100.0%
512	128	512x	2,265.53	9,256.04	0.01	9,543.48	19,376.02	27,028.40	100.0%
512	512	1x	133.99	132.66	0.34	46.30	46.30	3,738.99	100.0%
512	512	2x	234.99	232.42	0.22	59.19	62.30	4,280.02	100.0%
512	512	4x	403.01	397.19	0.14	72.13	75.40	4,979.53	100.0%
512	512	8x	645.71	635.25	0.10	85.53	88.43	6,214.49	100.0%
512	512	16x	976.38	986.45	0.07	113.57	118.96	8,033.97	100.0%
512	512	32x	1,295.84	1,398.25	0.06	154.94	165.29	11,391.74	100.0%
512	512	64x	2,012.84	2,045.57	0.04	231.47	256.83	15,525.06	100.0%
512	512	128x	2,799.03	2,909.46	0.03	344.98	414.20	21,757.07	100.0%
512	512	256x	3,762.10	3,870.49	0.02	553.10	782.26	32,636.38	100.0%
512	512	512x	3,787.85	3,851.07	0.02	16,408.46	33,594.68	65,467.73	100.0%
512	512	1024x	3,153.80	3,206.28	0.02	53,583.85	122,437.14	152,428.43	96.9%
512	1,024	1x	134.14	65.55	0.47	49.66	49.66	7,566.84	100.0%
512	1,024	2x	234.58	115.15	0.29	61.36	64.40	8,638.41	100.0%
512	1,024	4x	402.78	196.23	0.20	74.36	76.55	10,076.93	100.0%
512	1,024	8x	642.12	312.75	0.14	83.29	88.28	12,622.62	100.0%
512	1,024	16x	924.83	481.88	0.10	111.79	116.50	16,457.77	100.0%
512	1,024	32x	1,239.05	655.91	0.08	881.97	890.59	24,305.89	100.0%
512	1,024	64x	1,801.30	956.45	0.06	1,508.71	1,639.24	33,263.46	100.0%
512	1,024	128x	2,581.83	1,355.97	0.04	2,571.93	3,090.56	46,829.92	100.0%
512	1,024	256x	3,454.12	1,786.03	0.03	4,273.08	5,936.63	71,056.83	100.0%
512	1,024	512x	3,382.72	1,750.81	0.03	36,225.02	79,182.97	144,822.96	100.0%
512	2,048	1x	133.26	32.43	0.57	49.50	49.50	15,292.34	100.0%
512	2,048	2x	233.01	56.92	0.36	62.05	65.28	17,477.25	100.0%
512	2,048	4x	374.57	97.77	0.25	80.59	82.84	20,228.17	100.0%
512	2,048	8x	627.04	153.11	0.18	83.78	87.33	25,793.31	100.0%
512	2,048	16x	819.91	248.33	0.14	113.37	119.04	31,942.70	100.0%
512	2,048	32x	1,155.83	327.26	0.11	150.94	159.34	48,735.44	100.0%
512	2,048	64x	1,832.96	491.97	0.07	231.07	252.49	64,712.24	100.0%
512	2,048	128x	2,565.44	689.53	0.05	2,751.73	3,290.08	92,186.19	100.0%
512	2,048	256x	3,385.66	917.17	0.04	4,271.64	6,021.46	138,534.79	100.0%
512	2,048	512x	3,151.04	848.17	0.04	7,737.66	60,367.90	160,247.65	58.0%
1,024	128	1x	111.95	917.22	0.07	152.81	152.81	1,062.72	100.0%
1,024	128	2x	208.16	1,626.98	0.05	170.85	173.81	1,198.53	100.0%
1,024	128	4x	340.69	2,695.86	0.03	272.70	277.31	1,448.56	100.0%
1,024	128	8x	497.68	4,043.88	0.02	463.12	468.53	1,932.96	100.0%
1,024	128	16x	713.71	5,729.76	0.02	832.09	837.78	2,733.49	100.0%
1,024	128	32x	947.88	7,583.74	0.01	1,570.87	1,611.71	4,118.65	100.0%
1,024	128	64x	1,186.53	9,539.58	0.01	2,841.82	3,046.28	6,543.73	100.0%
1,024	128	128x	1,394.93	11,260.43	0.01	4,803.67	5,882.94	11,077.14	100.0%
1,024	128	256x	1,593.14	12,825.39	0.01	7,870.32	11,593.64	19,426.03	100.0%
1,024	128	512x	1,587.85	12,791.14	0.01	12,816.01	31,147.97	38,792.68	100.0%
1,024	512	1x	132.47	256.78	0.22	49.21	49.21	3,797.09	100.0%
1,024	512	2x	232.71	452.00	0.14	54.97	58.32	4,321.84	100.0%
1,024	512	4x	404.26	785.41	0.09	75.52	77.99	4,975.07	100.0%
1,024	512	8x	644.56	1,254.69	0.07	89.59	92.55	6,240.47	100.0%
1,024	512	16x	976.22	1,906.21	0.05	123.19	128.70	8,231.55	100.0%
1,024	512	32x	1,391.89	2,747.84	0.04	171.12	185.46	11,410.57	100.0%
1,024	512	64x	1,979.73	3,983.79	0.03	253.68	285.72	15,734.10	100.0%
1,024	512	128x	2,760.09	5,576.56	0.02	373.01	448.28	22,422.90	100.0%
1,024	512	256x	3,547.81	7,093.44	0.01	656.88	930.14	35,223.56	100.0%
1,024	512	512x	3,680.26	7,379.23	0.01	17,127.16	34,862.72	67,675.41	100.0%
1,024	512	1024x	2,775.23	5,555.76	0.02	43,380.53	94,320.88	133,188.90	78.2%
1,024	1,024	1x	126.20	121.22	0.36	157.97	157.97	8,042.90	100.0%
1,024	1,024	2x	222.34	214.02	0.22	178.60	181.83	9,127.52	100.0%
1,024	1,024	4x	380.29	366.15	0.15	279.72	286.59	10,672.96	100.0%
1,024	1,024	8x	615.80	594.04	0.11	486.84	489.85	13,181.18	100.0%
1,024	1,024	16x	901.04	871.96	0.08	879.57	889.42	18,003.00	100.0%
1,024	1,024	32x	1,273.94	1,250.58	0.06	1,665.53	1,720.53	25,100.56	100.0%
1,024	1,024	64x	1,758.02	1,778.61	0.04	2,558.08	3,048.85	35,313.91	100.0%
1,024	1,024	128x	2,415.10	2,430.40	0.03	3,908.41	5,962.16	51,664.96	100.0%
1,024	1,024	256x	3,128.63	3,193.61	0.02	8,152.52	11,472.20	78,511.58	100.0%
1,024	1,024	512x	3,148.33	3,171.18	0.02	41,026.76	92,169.96	158,046.71	100.0%
1,024	1,024	1024x	3,239.12	3,270.81	0.02	41,050.11	81,381.71	152,556.53	52.5%
1,024	2,048	1x	128.59	62.25	0.48	47.87	47.87	15,660.94	100.0%
1,024	2,048	2x	231.84	111.09	0.30	58.24	61.33	17,588.05	100.0%
1,024	2,048	4x	397.76	190.64	0.20	111.45	115.69	20,500.76	100.0%
1,024	2,048	8x	618.11	303.48	0.15	89.86	92.69	25,802.00	100.0%
1,024	2,048	16x	885.30	441.23	0.11	834.90	840.40	35,592.21	100.0%
1,024	2,048	32x	1,256.77	626.95	0.08	1,590.44	1,615.03	50,095.20	100.0%
1,024	2,048	64x	1,724.16	911.69	0.06	2,832.80	3,046.43	68,933.17	100.0%
1,024	2,048	128x	2,401.68	1,248.59	0.05	4,871.65	5,849.94	100,548.36	100.0%
1,024	2,048	256x	3,123.03	1,633.16	0.03	7,469.03	11,734.18	153,824.65	100.0%
1,024	2,048	512x	3,021.59	1,588.05	0.03	7,295.97	68,999.29	161,981.96	56.4%
2,048	128	1x	101.46	1,597.40	0.04	260.94	260.94	1,232.34	100.0%
2,048	128	2x	178.10	2,860.58	0.03	272.53	275.94	1,369.10	100.0%
2,048	128	4x	278.94	4,536.46	0.02	477.08	480.42	1,726.96	100.0%
2,048	128	8x	403.42	6,529.83	0.02	860.60	868.12	2,389.58	100.0%
2,048	128	16x	542.23	8,662.70	0.01	1,603.82	1,613.50	3,597.57	100.0%
2,048	128	32x	689.57	10,967.92	0.01	2,706.46	3,047.30	5,683.03	100.0%
2,048	128	64x	810.30	12,994.60	0.01	4,830.59	5,923.32	9,579.24	100.0%
2,048	128	128x	914.95	14,622.35	0.01	7,999.47	11,670.53	17,014.43	100.0%
2,048	128	256x	995.56	15,892.90	0.01	18,581.52	23,173.98	31,306.08	100.0%
2,048	128	512x	995.39	15,873.14	0.01	20,258.12	54,558.71	62,534.32	100.0%
2,048	512	1x	129.17	505.39	0.14	52.90	52.90	3,894.15	100.0%
2,048	512	2x	227.86	887.66	0.09	63.49	66.81	4,413.78	100.0%
2,048	512	4x	355.74	1,550.13	0.06	80.68	83.08	5,055.29	100.0%
2,048	512	8x	555.27	2,544.21	0.04	101.80	108.35	6,145.96	100.0%
2,048	512	16x	875.29	3,780.18	0.03	136.62	145.56	8,261.93	100.0%
2,048	512	32x	1,265.57	5,210.86	0.02	191.77	208.94	11,980.82	100.0%
2,048	512	64x	1,878.81	7,591.35	0.02	282.81	344.75	16,420.94	100.0%
2,048	512	128x	2,655.86	10,661.44	0.01	453.82	625.46	23,349.48	100.0%
2,048	512	256x	2,170.40	8,671.51	0.01	13,507.73	23,245.64	57,511.66	100.0%
2,048	1,024	1x	129.75	251.57	0.23	53.81	53.81	7,822.65	100.0%
2,048	1,024	2x	228.63	441.38	0.15	63.82	67.96	8,878.68	100.0%
2,048	1,024	4x	380.86	760.85	0.10	74.82	77.30	10,300.84	100.0%
2,048	1,024	8x	598.01	1,233.22	0.07	96.32	101.79	12,685.47	100.0%
2,048	1,024	16x	830.71	1,879.87	0.05	141.55	151.66	16,624.68	100.0%
2,048	1,024	32x	1,141.67	2,668.43	0.04	190.39	213.00	23,418.34	100.0%
2,048	1,024	64x	1,796.71	3,721.74	0.03	303.59	341.21	33,563.41	100.0%
2,048	1,024	128x	2,598.53	5,267.68	0.02	464.71	616.50	47,386.31	100.0%
2,048	1,024	256x	3,455.91	7,124.05	0.02	754.83	1,184.42	70,031.23	100.0%
2,048	1,024	512x	2,595.22	5,298.10	0.02	48,676.60	118,887.81	188,455.43	100.0%
2,048	2,048	1x	126.87	122.46	0.37	256.94	256.94	16,070.95	100.0%
2,048	2,048	2x	223.94	215.21	0.24	269.97	273.31	18,207.72	100.0%
2,048	2,048	4x	311.11	385.21	0.16	468.11	471.97	20,347.37	100.0%
2,048	2,048	8x	455.10	639.43	0.11	840.36	844.00	24,472.92	100.0%
2,048	2,048	16x	689.12	934.51	0.09	1,610.05	1,619.69	33,456.39	100.0%
2,048	2,048	32x	1,024.50	1,242.10	0.07	2,950.76	3,037.71	50,345.42	100.0%
2,048	2,048	64x	1,533.86	1,718.28	0.05	5,101.76	5,899.65	72,777.67	100.0%
2,048	2,048	128x	2,256.87	2,371.15	0.04	7,928.55	11,668.50	105,430.51	100.0%
2,048	2,048	256x	2,829.05	3,020.78	0.03	13,717.40	22,985.22	165,359.14	100.0%
2,048	2,048	512x	2,916.56	3,129.20	0.03	8,843.61	63,846.80	175,277.12	58.6%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA A100-SXM4-80GB

GPU Count2

GPU Memory (Total)160 GB

GPU Driver570.195.03

CUDA VersionUnknown

Compute Capability8.0

Power Limit (per GPU)400 W

CPU ModelAMD EPYC-Milan Processor

RAM174 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.11.0

OSUbuntu

OS Version24.04.3 LTS (Noble Numbat)

Kernel Version6.14.0-29-generic

Python Version3.12.3

Model Configuration

Provideropenai

Model Namegpt-oss-120b

QuantizationMXFP4

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length32768

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.90

Temperature0.70

Top-P1.00

Top-K-1