NVIDIA A100-SXM4-80GB (1x) - qwen3-8b

March 9, 2026 at 01:06 AM

Dataset: reference (v1.0)

NVIDIA NVIDIA A100-SXM4-80GB qwen qwen3-8b

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

5,547.95

Peak generation speed

Best Input TPS

42,519.57

Peak prefill speed

Best Energy Efficiency

0.00 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

19.63 ms

Lowest latency

Best E2E (P95)

1,474.61 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	512	512x	5,547.95★	1,388.05	0.01	84.01	125.76	9,123.15	100.0%
128	128	1x	86.84	86.84	0.29	37.70	37.70	1,474.61	100.0%
128	128	2x	171.01	169.00	0.15	22.14	22.63	1,496.49	100.0%
128	128	4x	333.99	332.03	0.08	40.76	45.98	1,531.33	100.0%
128	128	8x	121.90	120.36	0.13	6,912.73	6,914.80	8,397.22	100.0%
128	512	1x	88.17	22.04	0.65	19.63	19.63	5,807.12	100.0%
128	512	2x	171.70	42.42	0.34	22.80	23.17	5,963.30	100.0%
128	512	4x	338.68	84.17	0.17	31.41	35.67	6,044.00	100.0%
128	512	8x	673.67	167.30	0.09	33.81	36.61	6,039.33	100.0%
128	512	16x	1,267.78	320.28	0.05	77.98	100.22	6,372.76	100.0%
128	512	32x	2,302.81	582.48	0.03	103.76	176.08	7,017.05	100.0%
128	512	64x	4,043.79	1,030.61	0.02	149.77	316.77	7,913.26	100.0%
128	512	128x	5,353.03	1,354.83	0.02	184.38	372.09	9,364.86	100.0%
128	512	256x	5,507.07	1,391.06	0.02	89.86	118.18	9,103.71	100.0%
128	512	1024x	5,501.52	1,388.95	0.01	98.46	138.79	9,118.20	100.0%
128	1,024	1x	88.11	11.01	0.82	20.42	20.42	11,621.57	100.0%
128	1,024	2x	171.12	21.14	0.40	29.51	32.01	11,965.66	100.0%
128	1,024	4x	299.27	41.94	0.23	32.27	34.98	12,131.72	100.0%
128	1,024	8x	628.85	81.88	0.11	29.43	31.93	12,342.53	100.0%
128	1,024	16x	1,222.12	157.53	0.06	38.81	43.12	12,962.14	100.0%
128	1,024	32x	2,171.25	279.82	0.04	51.59	61.19	14,637.15	100.0%
128	1,024	64x	3,802.73	485.86	0.02	68.76	90.27	16,842.71	100.0%
128	1,024	128x	4,836.75	628.01	0.02	86.05	124.59	20,263.86	100.0%
128	1,024	256x	4,858.06	626.68	0.02	92.19	124.82	20,319.78	100.0%
128	1,024	512x	4,859.92	625.83	0.02	95.09	136.18	20,349.06	100.0%
128	1,024	1024x	4,857.65	626.35	0.02	94.59	138.94	20,328.76	100.0%
128	2,048	1x	87.53	5.47	0.91	20.02	20.02	23,397.14	100.0%
128	2,048	2x	169.63	10.48	0.45	22.27	22.74	24,145.74	100.0%
128	2,048	4x	273.06	20.71	0.28	30.05	32.07	24,569.88	100.0%
128	2,048	8x	570.18	39.55	0.14	31.14	33.67	25,556.56	100.0%
128	2,048	16x	1,123.76	74.82	0.07	41.14	46.85	27,306.58	100.0%
128	2,048	32x	1,861.67	128.76	0.05	54.65	65.28	31,844.37	100.0%
128	2,048	64x	3,099.51	216.35	0.03	66.87	91.35	37,901.00	100.0%
128	2,048	128x	3,854.18	268.41	0.03	89.22	122.90	47,560.64	100.0%
128	2,048	256x	3,876.65	266.25	0.03	132.84	209.02	47,880.19	100.0%
128	2,048	512x	3,876.41	267.85	0.03	105.25	135.43	47,562.84	100.0%
512	128	1x	86.20	333.33	0.13	45.16	45.16	1,484.97	100.0%
512	128	2x	167.76	649.41	0.07	45.85	46.26	1,525.97	100.0%
512	128	4x	321.20	1,248.43	0.03	88.77	89.77	1,593.06	100.0%
512	128	8x	609.89	2,360.33	0.02	115.87	163.49	1,675.66	100.0%
512	128	16x	1,084.75	4,192.80	0.01	178.45	287.92	1,878.68	100.0%
512	128	32x	1,740.02	6,734.49	0.01	294.88	551.35	2,328.99	100.0%
512	128	64x	2,618.93	10,144.82	0.01	541.83	1,067.42	3,086.36	100.0%
512	128	128x	3,428.88	13,300.03	0.01	493.18	1,228.15	3,659.14	100.0%
512	128	256x	4,760.13	18,463.74	0.00	140.78	187.51	2,608.37	100.0%
512	128	512x	4,760.13	18,463.74	0.00	147.20	197.67	2,610.96	100.0%
512	128	1024x	4,746.01	18,408.97	0.00	145.31	200.10	2,612.14	100.0%
512	512	1x	87.90	84.98	0.44	22.45	22.45	5,825.11	100.0%
512	512	2x	170.61	165.11	0.22	24.49	24.89	6,000.56	100.0%
512	512	4x	333.66	324.21	0.11	38.96	42.10	6,136.61	100.0%
512	512	8x	660.43	638.99	0.06	38.65	42.53	6,198.69	100.0%
512	512	16x	1,235.41	1,193.79	0.03	51.67	58.59	6,619.37	100.0%
512	512	32x	2,145.18	2,079.08	0.02	79.91	94.20	7,605.25	100.0%
512	512	64x	3,654.56	3,557.91	0.01	114.39	149.43	8,862.51	100.0%
512	512	128x	4,683.93	4,542.04	0.01	136.56	167.48	10,838.07	100.0%
512	512	256x	4,670.51	4,542.87	0.01	153.89	209.18	10,843.29	100.0%
512	1,024	1x	87.77	42.43	0.63	23.04	23.04	11,667.05	100.0%
512	1,024	2x	169.89	82.21	0.31	31.81	35.30	12,054.61	100.0%
512	1,024	4x	298.00	161.89	0.17	39.00	41.28	12,290.08	100.0%
512	1,024	8x	643.12	314.85	0.09	41.95	46.87	12,581.71	100.0%
512	1,024	16x	1,185.00	583.22	0.05	49.18	53.84	13,557.86	100.0%
512	1,024	32x	2,014.80	985.82	0.03	81.11	102.83	16,059.95	100.0%
512	1,024	64x	3,302.37	1,658.03	0.02	122.66	150.53	19,084.18	100.0%
512	1,024	128x	4,130.41	2,101.54	0.02	148.45	209.97	23,527.05	100.0%
512	1,024	256x	4,162.51	2,094.10	0.02	164.54	221.34	23,617.98	100.0%
512	1,024	512x	4,155.13	2,096.31	0.02	149.72	207.66	23,595.44	100.0%
512	2,048	1x	87.16	21.07	0.78	22.45	22.45	23,496.22	100.0%
512	2,048	2x	168.49	40.77	0.39	31.02	34.06	24,307.59	100.0%
512	2,048	4x	268.97	80.08	0.23	33.07	37.55	24,846.55	100.0%
512	2,048	8x	560.33	151.21	0.12	37.67	41.13	26,202.42	100.0%
512	2,048	16x	1,078.90	278.57	0.07	49.35	57.79	28,400.97	100.0%
512	2,048	32x	1,757.80	463.44	0.04	81.75	101.90	34,184.05	100.0%
512	2,048	64x	2,735.04	760.29	0.03	125.49	163.11	41,677.79	100.0%
512	2,048	128x	3,389.80	936.42	0.03	197.07	315.65	52,928.36	100.0%
512	2,048	256x	3,404.10	933.48	0.03	159.83	212.99	53,096.28	100.0%
512	2,048	512x	3,397.89	937.94	0.03	141.62	195.08	52,832.96	100.0%
1,024	128	1x	83.82	636.54	0.08	81.75	81.75	1,527.65	100.0%
1,024	128	2x	162.54	1,235.56	0.04	82.79	83.17	1,574.47	100.0%
1,024	128	4x	304.40	2,332.34	0.02	151.62	152.94	1,681.81	100.0%
1,024	128	8x	554.41	4,254.47	0.01	192.91	289.15	1,843.60	100.0%
1,024	128	16x	911.44	7,010.24	0.01	302.77	551.20	2,235.93	100.0%
1,024	128	32x	1,336.81	10,260.77	0.01	559.95	1,053.50	3,041.93	100.0%
1,024	128	64x	1,828.57	14,042.86	0.01	1,008.83	2,068.59	4,434.61	100.0%
1,024	128	128x	2,369.49	18,191.23	0.00	909.39	2,392.50	5,320.06	100.0%
1,024	128	256x	3,834.63	29,439.48	0.00	223.51	327.73	3,253.03	100.0%
1,024	128	512x	3,855.42	29,599.10	0.00	218.96	306.82	3,239.88	100.0%
1,024	128	1024x	3,820.90	29,334.03	0.00	220.70	327.86	3,271.18	100.0%
1,024	512	1x	87.48	166.07	0.30	23.93	23.93	5,853.22	100.0%
1,024	512	2x	169.20	321.55	0.15	28.12	28.44	6,051.57	100.0%
1,024	512	4x	328.31	628.89	0.08	42.33	46.24	6,235.62	100.0%
1,024	512	8x	625.82	1,225.13	0.04	48.48	53.55	6,409.45	100.0%
1,024	512	16x	1,153.17	2,242.28	0.02	67.62	77.92	7,010.57	100.0%
1,024	512	32x	1,897.10	3,688.72	0.02	94.52	123.11	8,495.69	100.0%
1,024	512	64x	3,079.26	6,036.46	0.01	163.16	215.21	10,373.35	100.0%
1,024	512	128x	3,852.27	7,437.30	0.01	216.74	310.70	13,125.36	100.0%
1,024	512	256x	3,844.56	7,447.44	0.01	219.71	308.36	13,107.70	100.0%
1,024	1,024	1x	87.19	108.10	0.41	24.58	24.58	8,980.59	100.0%
1,024	1,024	2x	136.53	162.30	0.27	34.76	38.60	11,752.48	100.0%
1,024	1,024	4x	295.15	314.29	0.13	34.36	35.66	12,479.66	100.0%
1,024	1,024	8x	586.51	600.81	0.07	56.49	70.81	13,074.14	100.0%
1,024	1,024	16x	1,052.48	1,109.61	0.04	66.17	81.05	14,184.65	100.0%
1,024	1,024	32x	1,762.37	1,769.52	0.03	106.04	132.40	17,741.33	100.0%
1,024	1,024	64x	2,830.12	2,849.28	0.02	164.03	225.42	22,024.95	100.0%
1,024	1,024	128x	3,471.25	3,511.74	0.02	222.90	295.82	27,896.97	100.0%
1,024	1,024	256x	3,487.41	3,501.23	0.02	215.45	309.50	27,974.78	100.0%
1,024	1,024	512x	3,473.12	3,505.48	0.02	238.31	320.04	27,950.20	100.0%
1,024	2,048	1x	87.25	185.57	0.28	25.97	25.97	5,226.10	100.0%
1,024	2,048	2x	114.60	100.55	0.39	28.31	28.71	18,702.80	100.0%
1,024	2,048	4x	246.01	157.05	0.20	35.26	38.37	24,965.16	100.0%
1,024	2,048	8x	496.94	297.29	0.11	48.67	58.66	26,105.49	100.0%
1,024	2,048	16x	918.44	544.35	0.06	67.15	83.83	28,914.26	100.0%
1,024	2,048	32x	1,529.16	866.92	0.04	108.04	135.93	36,241.00	100.0%
1,024	2,048	64x	2,403.05	1,336.11	0.03	157.27	205.22	47,028.94	100.0%
1,024	2,048	128x	2,909.65	1,612.32	0.02	313.47	613.40	60,857.59	100.0%
1,024	2,048	256x	2,910.68	1,623.26	0.02	226.45	325.64	60,448.14	100.0%
1,024	2,048	512x	2,938.13	1,643.18	0.02	219.71	302.43	59,714.41	100.0%
1,024	2,048	1024x	2,909.97	1,629.34	0.02	229.72	322.48	60,217.70	100.0%
2,048	128	1x	79.31	1,209.42	0.04	148.05	148.05	1,614.24	100.0%
2,048	128	2x	152.11	2,317.88	0.02	107.29	158.34	1,681.81	100.0%
2,048	128	4x	274.09	4,179.34	0.01	233.72	292.55	1,865.93	100.0%
2,048	128	8x	461.68	7,046.44	0.01	496.99	565.72	2,215.87	100.0%
2,048	128	16x	693.30	10,585.99	0.01	948.97	1,084.16	2,949.78	100.0%
2,048	128	32x	903.40	13,798.19	0.01	1,286.29	2,104.54	4,515.01	100.0%
2,048	128	64x	1,120.81	17,122.45	0.01	1,828.83	4,027.67	7,249.50	100.0%
2,048	128	128x	1,440.79	22,015.98	0.00	1,681.78	4,856.82	8,796.40	100.0%
2,048	128	256x	2,758.03	42,143.93	0.00	387.88	600.76	4,555.05	100.0%
2,048	128	512x	2,782.61	42,519.57	0.00	368.29	563.23	4,511.69	100.0%
2,048	128	1024x	2,781.40	42,501.09	0.00	385.32	570.98	4,515.25	100.0%
2,048	512	1x	86.15	328.45	0.19	33.57	33.57	5,943.47	100.0%
2,048	512	2x	166.15	632.97	0.09	33.90	34.29	6,162.27	100.0%
2,048	512	4x	318.95	1,215.85	0.05	48.39	49.49	6,418.79	100.0%
2,048	512	8x	602.18	2,297.71	0.03	74.14	83.00	6,798.35	100.0%
2,048	512	16x	1,050.26	4,009.10	0.02	111.38	129.99	7,792.68	100.0%
2,048	512	32x	1,574.94	6,034.05	0.01	160.60	205.90	10,344.69	100.0%
2,048	512	64x	2,418.48	9,236.70	0.01	262.27	367.51	13,495.27	100.0%
2,048	512	128x	2,864.12	11,014.19	0.01	387.74	580.76	17,669.75	100.0%
2,048	512	256x	2,859.07	11,016.67	0.01	362.64	537.01	17,638.09	100.0%
2,048	1,024	1x	86.10	164.13	0.33	33.33	33.33	11,892.45	100.0%
2,048	1,024	2x	165.84	315.90	0.16	33.61	34.00	12,347.96	100.0%
2,048	1,024	4x	317.45	605.05	0.09	46.32	47.57	12,901.18	100.0%
2,048	1,024	8x	597.08	1,139.14	0.05	65.37	81.64	13,714.43	100.0%
2,048	1,024	16x	1,033.50	1,972.56	0.03	110.71	126.32	15,843.39	100.0%
2,048	1,024	32x	1,494.97	2,916.32	0.02	175.14	219.40	21,427.49	100.0%
2,048	1,024	64x	2,273.10	4,438.19	0.02	276.12	375.71	28,142.55	100.0%
2,048	1,024	128x	2,672.86	5,257.23	0.01	453.21	826.74	37,109.87	100.0%
2,048	1,024	256x	2,663.27	5,322.47	0.01	392.23	570.57	36,660.50	100.0%
2,048	2,048	1x	85.69	81.67	0.51	30.85	30.85	23,901.04	100.0%
2,048	2,048	2x	160.28	169.72	0.25	34.20	34.68	22,906.78	100.0%
2,048	2,048	4x	276.07	303.56	0.14	48.03	54.97	25,318.85	100.0%
2,048	2,048	8x	553.66	558.16	0.08	72.75	81.82	27,993.72	100.0%
2,048	2,048	16x	947.44	964.56	0.05	93.34	113.13	32,405.61	100.0%
2,048	2,048	32x	1,339.21	1,444.73	0.03	175.49	216.30	43,276.64	100.0%
2,048	2,048	64x	1,967.34	2,105.81	0.03	662.56	1,134.26	59,367.86	100.0%
2,048	2,048	128x	2,216.63	2,424.66	0.02	1,768.51	4,713.15	80,575.73	100.0%
2,048	2,048	256x	2,306.96	2,542.38	0.02	884.50	1,638.60	76,841.94	100.0%
2,048	2,048	512x	2,304.74	2,553.33	0.02	834.35	1,481.76	76,516.07	100.0%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA A100-SXM4-80GB

GPU Count1

GPU Memory (Total)80 GB

GPU Driver570.211.01

CUDA VersionUnknown

Compute Capability8.0

Power Limit (per GPU)400 W

CPU ModelAMD EPYC-Milan Processor

RAM92 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.17.0

OSUbuntu

OS Version24.04.4 LTS (Noble Numbat)

Kernel Version6.14.0-29-generic

Python Version3.12.3

Model Configuration

Providerqwen

Model Nameqwen3-8b

QuantizationFP16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length32768

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.85

Temperature0.70

Top-P1.00

Top-K-1