NVIDIA H20 (8x) - qwen2.5-vl-72b-instruct

October 24, 2025 at 08:18 PM

Dataset: reference (v1.0)

NVIDIA NVIDIA H20 qwen qwen2.5-vl-72b-instruct

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

2,266.04

Peak generation speed

Best Input TPS

6,375.82

Peak prefill speed

Best Energy Efficiency

0.11 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

61.20 ms

Lowest latency

Best E2E (P95)

3,817.82 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	1,024	512x	2,266.04★	442.25	0.39	65,598.25	108,701.35	130,455.20	100.0%
128	128	1x	23.55	22.45	7.79	1,394.75	1,394.75	5,434.37	100.0%
128	128	2x	46.39	45.31	5.21	1,473.81	1,476.47	5,516.64	100.0%
128	128	4x	91.54	90.83	3.55	581.18	1,354.91	5,323.46	100.0%
128	128	8x	253.40	254.15	1.74	422.37	499.52	4,036.88	100.0%
128	128	16x	470.14	473.82	0.89	700.54	773.79	4,345.82	100.0%
128	128	32x	763.61	764.73	0.57	1,333.19	1,644.32	5,351.03	100.0%
128	128	64x	1,221.12	1,223.22	0.31	2,415.12	2,784.89	6,645.46	100.0%
128	128	128x	1,346.61	1,348.18	0.32	4,019.23	5,075.17	9,280.31	100.0%
128	128	256x	1,043.38	1,048.72	0.41	17,868.05	22,963.80	27,612.93	100.0%
128	512	1x	35.80	11.23	9.97	105.39	105.39	10,839.41	100.0%
128	512	2x	72.06	17.59	7.22	94.95	96.00	14,207.42	100.0%
128	512	4x	141.28	35.04	3.66	182.48	223.62	14,493.97	100.0%
128	512	8x	273.96	70.41	2.97	324.29	373.87	14,571.46	100.0%
128	512	16x	531.91	137.85	1.53	446.76	471.46	14,955.32	100.0%
128	512	32x	990.93	256.50	0.82	724.22	843.88	15,967.51	100.0%
128	512	64x	1,647.98	433.72	0.48	1,391.90	1,838.69	18,746.33	100.0%
128	512	128x	1,864.27	481.21	0.39	4,110.56	6,829.63	23,913.73	100.0%
128	512	256x	1,938.79	502.25	0.38	24,996.47	36,091.95	53,612.15	100.0%
128	512	512x	2,107.20	549.76	0.37	57,241.97	87,590.69	105,477.04	100.0%
128	512	1024x	1,679.03	437.47	0.47	73,670.06	101,952.74	118,963.92	46.8%
128	1,024	1x	35.91	7.34	10.96	107.89	107.89	16,597.63	100.0%
128	1,024	2x	59.59	13.96	8.16	144.77	149.40	17,574.86	100.0%
128	1,024	4x	121.31	26.53	6.29	167.82	195.94	19,073.40	100.0%
128	1,024	8x	209.27	49.21	3.65	289.65	358.51	20,644.90	100.0%
128	1,024	16x	391.12	79.64	2.09	470.67	619.83	23,673.65	100.0%
128	1,024	32x	787.77	161.13	1.05	901.14	987.99	24,236.95	100.0%
128	1,024	64x	1,268.99	248.48	0.65	1,515.41	2,024.09	28,917.27	100.0%
128	1,024	128x	1,860.38	364.80	0.43	4,656.40	5,175.34	37,685.76	100.0%
128	1,024	256x	1,927.46	379.56	0.43	29,113.37	48,436.13	72,912.60	100.0%
128	1,024	1024x	1,674.36	323.99	0.52	70,065.30	102,609.86	125,192.49	37.1%
128	2,048	1x	35.85	10.85	10.11	61.20	61.20	11,210.75	100.0%
128	2,048	2x	64.88	12.84	8.17	143.09	144.25	19,249.71	100.0%
128	2,048	4x	101.36	21.54	5.60	211.69	291.06	22,594.29	100.0%
128	2,048	8x	229.40	46.05	2.72	381.45	498.89	21,942.49	100.0%
128	2,048	16x	414.61	86.14	1.83	548.56	673.59	21,773.51	100.0%
128	2,048	32x	757.80	149.65	1.11	892.34	1,050.20	26,294.02	100.0%
128	2,048	64x	1,469.26	289.11	0.60	1,517.38	2,066.16	27,638.12	100.0%
128	2,048	128x	1,981.47	385.67	0.39	3,843.57	4,631.79	33,194.32	100.0%
128	2,048	256x	2,111.57	425.48	0.39	27,475.81	44,942.42	69,642.57	100.0%
128	2,048	512x	2,156.45	426.20	0.38	64,945.08	104,600.90	128,703.76	100.0%
128	2,048	1024x	1,804.24	352.95	0.46	83,996.13	124,691.44	147,958.12	49.4%
512	128	1x	33.52	129.88	2.66	325.88	325.88	3,817.82	100.0%
512	128	2x	66.61	258.91	1.77	346.27	347.97	3,841.78	100.0%
512	128	4x	108.13	417.74	0.82	980.14	1,179.29	4,732.92	100.0%
512	128	8x	238.36	919.46	0.61	648.69	757.38	4,292.80	100.0%
512	128	16x	380.53	1,474.54	0.48	1,174.76	1,769.65	5,372.53	100.0%
512	128	32x	610.98	2,380.82	0.28	2,291.47	2,941.72	6,692.54	100.0%
512	128	64x	892.47	3,472.06	0.19	3,561.33	5,191.74	9,143.96	100.0%
512	128	128x	935.59	3,641.44	0.16	6,737.28	11,233.53	15,423.70	100.0%
512	128	256x	746.86	2,909.30	0.20	24,075.97	33,508.70	39,690.23	100.0%
512	512	1x	36.13	35.00	6.50	63.21	63.21	14,168.70	100.0%
512	512	2x	71.55	69.53	3.27	102.91	116.18	14,308.74	100.0%
512	512	4x	142.55	137.68	2.32	160.86	175.37	14,360.63	100.0%
512	512	8x	267.42	257.88	1.55	535.39	775.01	15,312.74	100.0%
512	512	16x	504.61	511.37	0.98	582.72	932.17	15,507.95	100.0%
512	512	32x	953.97	975.73	0.52	837.56	973.98	16,336.90	100.0%
512	512	64x	1,584.12	1,580.70	0.31	2,175.83	3,302.41	20,102.32	100.0%
512	512	128x	1,615.58	1,625.29	0.25	5,350.02	6,803.36	25,013.10	100.0%
512	512	256x	1,619.22	1,640.97	0.27	28,392.87	44,822.91	63,618.81	100.0%
512	512	512x	1,895.10	1,899.75	0.26	57,167.87	99,635.04	118,776.31	100.0%
512	512	1024x	1,192.05	1,195.84	0.41	66,228.82	95,015.90	112,203.68	32.3%
512	1,024	1x	35.63	24.08	7.99	314.44	314.44	20,571.93	100.0%
512	1,024	2x	69.42	45.50	5.69	344.61	344.71	21,784.15	100.0%
512	1,024	4x	125.07	85.81	3.85	294.77	459.02	22,773.40	100.0%
512	1,024	8x	237.98	169.80	2.47	533.60	748.88	23,121.28	100.0%
512	1,024	16x	390.91	301.66	1.42	914.45	1,626.27	25,025.58	100.0%
512	1,024	32x	746.45	549.47	0.77	1,878.78	2,713.78	28,166.48	100.0%
512	1,024	64x	1,357.60	991.29	0.43	2,679.89	4,378.41	30,091.22	100.0%
512	1,024	128x	1,707.08	1,206.41	0.32	7,160.77	14,645.61	41,868.15	100.0%
512	1,024	256x	2,000.36	1,406.70	0.30	32,284.66	56,218.71	83,320.77	100.0%
512	1,024	512x	1,989.54	1,396.31	0.29	79,514.32	137,213.71	163,338.10	100.0%
512	2,048	1x	35.72	25.35	7.72	309.97	309.97	19,537.77	100.0%
512	2,048	2x	52.87	41.01	4.95	483.00	607.51	23,650.23	100.0%
512	2,048	4x	134.08	94.48	3.67	353.53	431.67	20,784.64	100.0%
512	2,048	8x	247.68	173.65	2.22	649.68	1,020.52	22,519.09	100.0%
512	2,048	16x	440.40	303.31	1.33	1,113.99	1,689.71	25,985.11	100.0%
512	2,048	32x	768.83	565.47	0.76	1,938.18	2,388.73	27,293.10	100.0%
512	2,048	64x	1,271.40	896.06	0.46	3,639.78	5,861.96	33,486.63	100.0%
512	2,048	128x	1,579.02	1,115.32	0.32	7,399.61	9,653.67	42,614.06	100.0%
512	2,048	256x	1,831.28	1,284.12	0.30	32,746.06	56,328.57	84,825.42	100.0%
512	2,048	512x	2,074.36	1,457.32	0.28	73,991.02	130,375.09	158,316.68	100.0%
512	2,048	1024x	1,861.04	1,297.80	0.31	101,216.77	169,847.02	196,768.11	56.7%
1,024	128	1x	31.35	238.80	1.66	574.93	574.93	4,082.92	100.0%
1,024	128	2x	54.75	417.88	0.92	877.10	1,110.37	4,673.78	100.0%
1,024	128	4x	108.52	828.53	0.73	785.84	1,106.69	4,712.10	100.0%
1,024	128	8x	172.42	1,318.91	0.43	1,507.91	2,298.19	5,934.35	100.0%
1,024	128	16x	310.87	2,384.79	0.30	2,093.76	2,936.75	6,581.54	100.0%
1,024	128	32x	500.18	3,837.71	0.18	3,092.13	4,264.79	8,174.94	100.0%
1,024	128	64x	589.48	4,526.44	0.15	5,681.11	9,270.12	13,837.79	100.0%
1,024	128	128x	566.29	4,348.96	0.15	11,401.87	23,771.15	28,135.42	100.0%
1,024	512	1x	36.07	68.69	4.49	77.56	77.56	14,193.67	100.0%
1,024	512	2x	71.43	136.31	2.23	108.72	122.43	14,332.78	100.0%
1,024	512	4x	141.36	269.81	2.04	151.15	170.14	14,481.36	100.0%
1,024	512	8x	273.18	522.41	1.05	297.17	309.52	14,987.09	100.0%
1,024	512	16x	491.36	942.36	0.68	918.28	1,976.72	16,660.79	100.0%
1,024	512	32x	858.77	1,684.37	0.38	1,671.24	2,752.32	18,638.55	100.0%
1,024	512	64x	1,340.95	2,671.20	0.23	3,523.94	5,824.22	23,511.34	100.0%
1,024	512	128x	1,394.09	2,738.82	0.19	9,225.20	18,308.94	36,602.92	100.0%
1,024	512	256x	1,447.98	2,828.35	0.21	27,970.07	55,301.07	76,774.29	100.0%
1,024	512	512x	1,479.00	2,885.52	0.21	73,771.97	128,909.33	154,267.53	98.0%
1,024	512	1024x	1,360.92	2,652.06	0.24	104,007.19	167,739.89	189,066.84	54.7%
1,024	1,024	1x	35.05	47.20	5.73	576.81	576.81	20,628.30	100.0%
1,024	1,024	2x	61.40	78.52	3.31	894.40	1,137.40	24,612.83	100.0%
1,024	1,024	4x	129.65	175.48	2.64	535.83	666.07	22,023.58	100.0%
1,024	1,024	8x	239.10	292.28	1.58	1,149.88	1,794.93	26,478.78	100.0%
1,024	1,024	16x	418.19	534.86	1.04	1,427.55	1,896.12	27,499.57	100.0%
1,024	1,024	32x	691.90	930.67	0.57	3,036.73	4,736.66	32,075.70	100.0%
1,024	1,024	64x	1,073.41	1,460.81	0.36	5,502.12	11,249.75	39,425.72	100.0%
1,024	1,024	128x	1,453.99	2,022.18	0.25	11,137.87	15,826.23	47,250.81	100.0%
1,024	1,024	256x	1,498.34	2,015.48	0.25	40,107.49	71,511.92	104,656.24	100.0%
1,024	1,024	512x	1,678.60	2,270.32	0.25	81,819.81	148,541.30	176,276.21	85.4%
1,024	2,048	1x	35.18	47.45	5.70	570.28	570.28	20,521.73	100.0%
1,024	2,048	2x	70.66	93.03	4.07	362.42	599.12	20,963.09	100.0%
1,024	2,048	4x	116.25	145.89	2.84	658.17	674.51	26,296.66	100.0%
1,024	2,048	8x	251.72	317.82	1.45	1,336.01	1,807.94	24,551.63	100.0%
1,024	2,048	16x	391.48	530.04	0.98	1,881.40	2,886.14	27,113.53	100.0%
1,024	2,048	32x	693.83	921.83	0.57	3,882.84	5,375.76	31,257.37	100.0%
1,024	2,048	64x	1,146.96	1,624.34	0.33	6,078.41	10,060.38	36,151.84	100.0%
1,024	2,048	128x	1,632.07	2,262.05	0.24	10,878.64	16,824.37	48,484.32	100.0%
1,024	2,048	256x	1,737.80	2,357.26	0.24	34,348.37	64,737.50	96,253.44	100.0%
1,024	2,048	512x	1,653.77	2,251.90	0.25	77,212.19	142,048.42	170,402.56	85.0%
2,048	128	1x	27.71	426.07	0.94	1,104.68	1,104.68	4,618.85	100.0%
2,048	128	2x	54.84	839.55	0.64	1,135.44	1,136.49	4,665.76	100.0%
2,048	128	4x	107.63	1,647.89	0.49	1,223.01	1,226.38	4,754.89	100.0%
2,048	128	8x	174.80	2,671.90	0.29	1,517.12	2,312.33	5,852.28	100.0%
2,048	128	16x	249.94	3,817.55	0.19	3,124.75	4,485.83	8,189.24	100.0%
2,048	128	32x	321.63	4,913.39	0.15	6,026.63	8,912.40	12,720.37	100.0%
2,048	128	64x	417.34	6,375.82	0.11	8,240.50	14,816.49	19,550.82	100.0%
2,048	128	128x	373.19	5,701.66	0.11	17,948.06	32,870.66	42,116.00	100.0%
2,048	512	1x	33.59	129.10	2.83	1,103.33	1,103.33	15,243.11	100.0%
2,048	512	2x	66.78	255.59	1.99	1,131.79	1,133.55	15,331.53	100.0%
2,048	512	4x	119.39	510.32	1.44	394.12	1,015.80	15,203.81	100.0%
2,048	512	8x	198.25	861.47	0.72	1,731.52	3,252.85	18,165.59	100.0%
2,048	512	16x	402.54	1,706.45	0.45	1,949.14	3,437.90	18,321.25	100.0%
2,048	512	32x	676.37	2,786.55	0.26	4,141.37	6,843.47	22,430.62	100.0%
2,048	512	64x	978.61	3,965.12	0.18	7,536.68	14,206.83	31,503.81	100.0%
2,048	512	128x	920.08	3,646.06	0.17	18,491.76	30,298.52	54,407.15	100.0%
2,048	1,024	1x	35.86	119.42	3.07	68.85	68.85	16,451.02	100.0%
2,048	1,024	2x	62.18	163.76	2.76	1,131.51	1,132.79	23,685.61	100.0%
2,048	1,024	4x	115.96	338.43	1.70	660.17	1,199.37	23,043.00	100.0%
2,048	1,024	8x	183.53	631.61	0.98	1,614.73	2,261.12	23,792.17	100.0%
2,048	1,024	16x	297.24	1,002.60	0.64	2,912.87	5,491.42	28,691.18	100.0%
2,048	1,024	32x	637.86	1,849.66	0.39	3,659.86	7,894.38	32,697.61	100.0%
2,048	1,024	64x	978.32	2,757.48	0.25	7,656.84	12,651.81	43,091.86	100.0%
2,048	1,024	128x	1,312.35	3,519.59	0.19	18,688.86	31,101.54	67,450.60	100.0%
2,048	1,024	256x	1,227.86	3,322.58	0.20	53,746.81	102,600.82	140,346.67	100.0%
2,048	2,048	1x	34.18	89.81	3.78	1,105.93	1,105.93	21,884.74	100.0%
2,048	2,048	2x	66.47	191.25	2.49	622.05	1,090.99	20,366.07	100.0%
2,048	2,048	4x	111.80	348.76	1.60	1,185.59	1,209.52	22,096.65	100.0%
2,048	2,048	8x	182.42	583.53	1.16	1,673.04	3,371.18	25,338.17	100.0%
2,048	2,048	16x	308.76	1,138.15	0.64	3,587.71	5,945.37	27,132.91	100.0%
2,048	2,048	32x	494.81	1,489.38	0.44	6,166.68	11,049.64	40,497.53	100.0%
2,048	2,048	64x	953.95	2,666.70	0.25	8,208.46	14,642.50	44,327.75	100.0%
2,048	2,048	128x	1,159.22	3,029.88	0.20	18,416.91	35,383.20	68,343.02	100.0%
2,048	2,048	256x	1,177.67	3,164.96	0.20	55,429.36	103,769.73	141,404.09	100.0%
2,048	2,048	512x	1,237.55	3,338.04	0.21	73,487.70	141,675.30	174,445.80	61.9%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H20

GPU Count8

GPU Memory (Total)760 GB

GPU Driver570.195.03

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)500 W

CPU ModelIntel(R) Xeon(R) Platinum 8469C

RAM1,007 GB

Software Configuration

Inference FrameworkvLLM

Framework Versionv0.9.0

OSUbuntu

OS Version24.04.3 LTS (Noble Numbat)

Kernel Version6.8.0-79-generic

Python Version3.12.3

Model Configuration

Providerqwen

Model Nameqwen2.5-vl-72b-instruct

QuantizationBF16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length32768

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization90.00

Temperature0.70

Top-P1.00

Top-K-1