NVIDIA H20 (8x) - deepseek-v3.1

October 26, 2025 at 03:43 PM

Dataset: reference (v1.0)

NVIDIA NVIDIA H20 deepseek-ai deepseek-v3.1

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

865.08

Peak generation speed

Best Input TPS

4,142.63

Peak prefill speed

Best Energy Efficiency

0.16 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

80.10 ms

Lowest latency

Best E2E (P95)

3,336.74 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	512	64x	865.08★	221.97	0.73	1,707.12	2,089.74	36,704.62	100.0%
128	128	1x	38.36	36.56	5.16	103.10	103.10	3,336.74	100.0%
128	128	2x	69.72	68.08	3.30	203.32	263.71	3,668.95	100.0%
128	128	4x	123.11	122.14	1.88	373.78	408.48	4,156.31	100.0%
128	128	8x	193.43	193.99	1.38	557.53	578.23	5,284.28	100.0%
128	128	16x	310.30	312.58	0.93	884.20	961.51	6,588.80	100.0%
128	128	32x	490.77	491.49	0.66	1,422.50	1,554.96	8,330.51	100.0%
128	128	64x	748.35	746.70	0.44	2,519.37	2,727.05	10,886.10	100.0%
128	128	128x	757.90	759.34	0.46	8,833.42	13,366.23	21,502.16	100.0%
128	128	256x	639.24	641.53	0.59	27,902.38	42,534.03	50,864.66	100.0%
128	512	1x	39.04	9.30	9.25	98.44	98.44	13,112.79	100.0%
128	512	2x	73.73	18.00	5.16	195.83	196.21	13,886.64	100.0%
128	512	4x	132.43	32.85	3.15	279.95	313.87	15,460.48	100.0%
128	512	8x	187.03	52.31	2.45	457.59	476.80	19,626.07	100.0%
128	512	16x	333.11	86.20	1.56	654.23	668.92	23,917.66	100.0%
128	512	32x	536.39	139.52	1.11	1,171.43	1,235.62	29,385.83	100.0%
128	512	128x	854.99	218.77	0.75	21,614.52	40,022.51	74,798.57	100.0%
128	1,024	1x	39.29	4.68	10.74	92.60	92.60	26,063.99	100.0%
128	1,024	2x	74.33	9.07	5.89	152.42	181.77	27,551.31	100.0%
128	1,024	4x	133.57	16.57	3.58	288.32	325.79	30,662.17	100.0%
128	1,024	8x	187.52	26.44	2.83	362.54	383.09	38,834.66	100.0%
128	1,024	16x	326.95	43.63	1.86	560.61	574.21	47,274.32	100.0%
128	1,024	32x	522.29	70.69	1.29	1,012.22	1,150.91	58,002.39	100.0%
128	1,024	64x	671.78	89.01	1.05	1,814.92	2,191.79	89,738.41	100.0%
128	1,024	128x	688.32	90.92	1.02	40,564.10	97,826.75	172,631.43	100.0%
128	1,024	256x	642.11	83.15	1.10	54,685.02	116,474.05	194,610.40	51.6%
128	2,048	1x	39.22	3.17	10.98	99.03	99.03	38,477.66	100.0%
128	2,048	2x	72.97	5.31	6.26	208.09	208.16	46,993.37	100.0%
128	2,048	4x	110.25	8.62	4.39	358.36	395.46	56,777.86	100.0%
128	2,048	8x	155.38	14.55	3.39	426.62	447.65	66,612.60	100.0%
128	2,048	16x	280.45	24.01	2.20	593.00	607.54	85,406.67	100.0%
128	2,048	32x	442.49	38.86	1.55	1,025.12	1,117.65	103,426.89	100.0%
128	2,048	64x	556.77	47.86	1.28	1,808.22	2,123.37	165,245.32	100.0%
128	2,048	128x	577.00	48.13	1.27	34,440.63	123,787.37	224,696.50	68.0%
512	128	1x	37.57	145.58	2.37	157.68	157.68	3,406.39	100.0%
512	128	2x	70.08	272.38	1.43	192.60	247.50	3,650.22	100.0%
512	128	4x	125.55	485.04	0.81	296.43	335.64	4,074.41	100.0%
512	128	8x	196.81	759.18	0.60	476.49	507.53	5,196.96	100.0%
512	128	16x	316.64	1,226.96	0.41	859.80	887.64	6,453.74	100.0%
512	128	32x	485.24	1,892.23	0.30	1,465.44	1,655.34	8,419.86	100.0%
512	128	64x	728.79	2,851.65	0.21	2,153.40	2,890.18	11,148.88	100.0%
512	128	128x	645.86	2,517.64	0.24	10,353.19	16,820.54	25,211.26	100.0%
512	512	1x	37.68	37.83	5.98	94.70	94.70	13,108.58	100.0%
512	512	2x	74.28	72.17	3.31	140.29	140.61	13,784.09	100.0%
512	512	4x	131.20	126.71	2.07	359.87	397.30	15,606.95	100.0%
512	512	8x	198.42	200.55	1.51	406.55	450.63	19,692.28	100.0%
512	512	16x	300.37	329.43	1.07	716.55	843.57	24,082.11	100.0%
512	512	32x	508.37	533.62	0.72	1,259.43	1,364.51	29,896.68	100.0%
512	512	64x	617.95	636.79	0.61	2,110.96	2,991.52	48,435.49	100.0%
512	512	128x	603.66	617.37	0.65	26,760.74	79,798.00	102,714.80	100.0%
512	1,024	1x	38.02	18.92	8.06	155.60	155.60	26,219.42	100.0%
512	1,024	2x	74.16	36.03	4.48	210.19	210.51	27,613.42	100.0%
512	1,024	4x	131.89	64.05	2.76	323.09	390.88	30,879.23	100.0%
512	1,024	8x	191.04	101.36	2.09	485.11	535.08	38,964.41	100.0%
512	1,024	16x	282.71	169.25	1.50	751.49	779.69	46,870.14	100.0%
512	1,024	32x	494.14	273.31	1.00	1,417.83	1,611.24	58,377.01	100.0%
512	1,024	64x	550.52	289.66	0.94	2,197.91	3,111.86	108,686.59	100.0%
512	1,024	128x	547.77	289.30	0.94	36,800.31	112,689.36	170,974.29	80.5%
512	2,048	1x	38.19	12.54	9.15	157.02	157.02	39,539.06	100.0%
512	2,048	2x	73.64	22.13	5.14	196.93	256.27	44,931.91	100.0%
512	2,048	4x	111.54	38.90	3.54	354.39	388.60	50,031.75	100.0%
512	2,048	8x	163.62	62.02	2.61	522.24	552.73	62,369.16	100.0%
512	2,048	16x	254.78	108.68	1.82	785.87	812.36	72,971.12	100.0%
512	2,048	32x	418.75	156.11	1.29	1,504.55	1,582.66	99,496.65	100.0%
512	2,048	64x	484.25	171.69	1.18	2,337.64	3,150.51	184,611.50	100.0%
512	2,048	128x	473.59	167.67	1.20	21,877.01	123,073.62	220,751.90	59.4%
1,024	128	1x	37.71	287.27	1.37	174.15	174.15	3,393.29	100.0%
1,024	128	2x	70.08	534.90	0.81	237.30	262.35	3,650.64	100.0%
1,024	128	4x	122.90	938.31	0.47	377.72	417.64	4,164.36	100.0%
1,024	128	8x	191.83	1,467.40	0.36	592.12	671.75	5,332.45	100.0%
1,024	128	16x	302.51	2,320.68	0.25	1,062.89	1,152.88	6,757.52	100.0%
1,024	128	32x	444.24	3,409.31	0.19	1,740.76	2,285.02	9,207.50	100.0%
1,024	128	64x	432.64	3,339.21	0.20	5,215.87	12,227.75	18,764.46	100.0%
1,024	512	1x	38.97	74.22	4.22	80.10	80.10	13,135.56	100.0%
1,024	512	2x	74.15	141.50	2.25	148.23	173.41	13,807.05	100.0%
1,024	512	4x	132.04	252.01	1.39	283.70	323.16	15,506.74	100.0%
1,024	512	8x	207.71	397.21	1.00	529.64	570.44	19,711.75	100.0%
1,024	512	16x	335.52	643.47	0.69	1,071.20	1,119.85	24,400.92	100.0%
1,024	512	32x	458.75	900.18	0.54	1,890.47	2,436.59	32,410.39	100.0%
1,024	512	64x	485.96	970.35	0.54	13,898.13	36,515.61	64,759.75	100.0%
1,024	512	128x	469.92	939.88	0.57	48,038.71	107,708.22	133,765.44	100.0%
1,024	1,024	1x	39.01	37.14	6.05	172.21	172.21	26,248.09	100.0%
1,024	1,024	2x	74.08	70.78	3.42	225.51	250.76	27,603.81	100.0%
1,024	1,024	4x	132.89	126.82	2.10	400.78	456.97	30,819.08	100.0%
1,024	1,024	8x	209.48	200.37	1.50	579.75	629.04	39,080.69	100.0%
1,024	1,024	16x	334.66	326.13	1.04	1,067.01	1,114.54	48,159.73	100.0%
1,024	1,024	32x	380.19	379.53	0.95	1,844.69	2,181.37	80,417.99	100.0%
1,024	1,024	64x	406.69	405.60	0.93	27,282.72	102,955.61	151,265.35	100.0%
1,024	1,024	128x	393.28	412.79	0.94	33,335.46	99,498.96	158,695.93	53.9%
1,024	2,048	1x	39.06	28.32	6.82	81.14	81.14	34,405.04	100.0%
1,024	2,048	2x	67.64	45.23	4.28	217.56	217.88	42,773.88	100.0%
1,024	2,048	4x	115.92	64.62	2.99	357.55	398.66	59,904.06	100.0%
1,024	2,048	8x	182.59	108.73	2.04	471.12	503.56	70,235.92	100.0%
1,024	2,048	16x	283.72	178.55	1.45	1,001.52	1,045.11	82,937.01	100.0%
1,024	2,048	32x	357.74	240.37	1.22	1,959.25	2,670.35	124,812.05	100.0%
1,024	2,048	64x	362.88	241.56	1.22	27,154.91	107,059.53	208,543.27	85.9%
2,048	128	1x	36.72	564.54	0.78	252.47	252.47	3,485.69	100.0%
2,048	128	2x	68.36	1,046.46	0.41	247.50	345.51	3,741.78	100.0%
2,048	128	4x	118.00	1,806.64	0.27	546.00	586.63	4,337.06	100.0%
2,048	128	8x	173.46	2,747.89	0.21	933.50	1,005.68	5,690.26	100.0%
2,048	128	16x	271.22	4,142.63	0.16	1,312.76	1,851.66	7,538.37	100.0%
2,048	128	32x	247.88	3,822.12	0.18	4,564.77	10,770.82	16,353.72	100.0%
2,048	512	1x	38.43	147.71	2.42	251.93	251.93	13,321.86	100.0%
2,048	512	2x	73.41	280.93	1.37	168.49	195.03	13,947.12	100.0%
2,048	512	4x	103.82	495.95	0.90	523.67	586.06	15,804.41	100.0%
2,048	512	8x	150.07	780.10	0.66	931.54	1,021.64	20,057.95	100.0%
2,048	512	16x	256.25	1,265.52	0.45	1,201.51	1,445.49	24,707.71	100.0%
2,048	512	32x	267.13	1,203.38	0.49	10,219.91	31,497.84	51,962.69	100.0%
2,048	512	64x	311.42	1,321.52	0.47	31,952.51	63,661.88	90,935.40	100.0%
2,048	512	128x	294.56	1,196.67	0.50	51,733.05	97,406.74	125,500.30	64.1%
2,048	1,024	1x	38.69	74.36	4.08	251.45	251.45	26,465.01	100.0%
2,048	1,024	2x	73.73	141.08	2.28	166.47	197.07	27,775.65	100.0%
2,048	1,024	4x	101.37	254.19	1.53	481.09	579.62	30,833.41	100.0%
2,048	1,024	8x	142.86	416.18	1.09	642.94	698.36	37,603.00	100.0%
2,048	1,024	16x	216.06	663.91	0.78	1,486.08	1,840.77	47,103.98	100.0%
2,048	1,024	32x	263.11	645.04	0.77	10,771.07	56,850.42	96,953.15	100.0%
2,048	1,024	64x	302.53	689.14	0.75	56,547.45	116,431.60	163,896.81	92.2%
2,048	1,024	128x	282.72	563.81	0.85	58,456.68	116,387.69	174,626.82	43.8%
2,048	2,048	1x	38.71	52.39	5.07	252.76	252.76	37,537.55	100.0%
2,048	2,048	2x	67.42	97.54	2.95	307.78	308.15	39,790.83	100.0%
2,048	2,048	4x	89.18	184.79	1.90	528.28	591.73	41,958.12	100.0%
2,048	2,048	8x	123.36	280.89	1.43	885.97	976.07	54,011.83	100.0%
2,048	2,048	16x	190.55	438.35	1.01	1,346.98	1,775.66	66,612.97	100.0%
2,048	2,048	32x	249.52	401.77	1.06	24,318.77	82,058.57	151,159.22	100.0%
2,048	2,048	64x	276.66	406.98	1.05	33,184.32	92,523.95	174,079.53	57.8%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H20

GPU Count8

GPU Memory (Total)760 GB

GPU Driver570.195.03

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)500 W

CPU ModelIntel(R) Xeon(R) Platinum 8469C

RAM1,007 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.10.2

OSUbuntu

OS Version24.04.3 LTS (Noble Numbat)

Kernel Version6.8.0-79-generic

Python Version3.12.3

Model Configuration

Providerdeepseek-ai

Model Namedeepseek-v3.1

QuantizationFP8

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length40960

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization90.00

Temperature0.70

Top-P1.00

Top-K-1