NVIDIA H200 NVL (2x) - qwen3-coder-30b-a3b-instruct

November 5, 2025 at 03:39 AM

Dataset: reference (v1.0)

NVIDIA NVIDIA H200 NVL qwen qwen3-coder-30b-a3b-instruct

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

5,757.76

Peak generation speed

Best Input TPS

43,900.39

Peak prefill speed

Best Energy Efficiency

0.01 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

82.49 ms

Lowest latency

Best E2E (P95)

954.88 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	512	128x	5,757.76★	1,486.57	0.03	493.77	629.47	10,701.96	100.0%
128	128	1x	134.03	127.75	0.21	128.81	128.81	954.88	100.0%
128	128	2x	208.13	203.25	0.20	253.90	255.45	1,225.81	100.0%
128	128	4x	192.99	191.48	0.19	946.99	1,479.46	2,643.11	100.0%
128	512	1x	143.66	34.23	0.43	82.49	82.49	3,564.10	100.0%
128	512	2x	164.95	40.27	0.39	1,471.82	2,713.41	6,070.86	100.0%
128	512	4x	326.22	82.70	0.25	831.36	1,521.99	6,139.30	100.0%
128	512	8x	750.30	206.14	0.14	279.12	317.20	4,966.96	100.0%
128	512	16x	1,166.52	297.95	0.11	326.97	390.07	6,892.98	100.0%
128	512	32x	1,928.43	490.14	0.07	383.48	434.50	8,329.05	100.0%
128	512	64x	3,335.62	858.95	0.04	449.03	581.70	9,305.04	100.0%
128	512	256x	5,202.77	1,441.04	0.03	4,105.99	8,659.65	18,949.19	100.0%
128	1,024	1x	142.64	16.99	0.54	102.17	102.17	7,175.82	100.0%
128	1,024	2x	190.19	34.06	0.41	98.81	102.14	7,097.67	100.0%
128	1,024	4x	434.45	61.27	0.24	143.88	149.21	8,281.27	100.0%
128	1,024	8x	695.95	111.68	0.17	198.39	214.49	9,187.55	100.0%
128	1,024	16x	982.33	148.77	0.15	260.38	301.03	13,825.84	100.0%
128	1,024	32x	1,884.21	276.96	0.09	351.29	383.78	14,707.82	100.0%
128	1,024	64x	3,198.38	454.90	0.06	404.84	478.11	17,704.76	100.0%
128	1,024	128x	4,061.97	594.61	0.04	625.57	578.05	23,212.89	100.0%
128	1,024	256x	4,502.16	679.27	0.04	8,803.65	20,684.63	44,669.17	100.0%
128	1,024	512x	4,241.25	639.38	0.04	28,303.67	66,828.25	88,046.56	100.0%
128	2,048	1x	148.89	8.87	0.55	105.09	105.09	13,752.45	100.0%
128	2,048	2x	169.85	16.74	0.46	157.95	164.67	14,380.97	100.0%
128	2,048	4x	328.57	36.70	0.30	159.86	185.03	13,478.03	100.0%
128	2,048	8x	428.94	52.39	0.24	213.36	254.03	18,518.08	100.0%
128	2,048	16x	759.77	92.93	0.17	245.32	272.23	19,600.19	100.0%
128	2,048	32x	1,328.57	157.19	0.11	324.03	380.37	24,952.58	100.0%
128	2,048	64x	2,321.36	252.98	0.08	371.72	428.51	31,967.57	100.0%
128	2,048	128x	3,879.42	446.61	0.05	358.25	517.65	34,083.30	100.0%
128	2,048	256x	4,467.94	512.55	0.04	9,849.28	28,187.11	54,516.44	100.0%
128	2,048	512x	4,581.34	530.48	0.04	31,182.19	72,187.27	96,798.13	100.0%
128	2,048	1024x	5,149.59	596.71	0.04	53,646.79	115,093.20	142,290.16	76.9%
512	128	1x	117.86	456.72	0.11	218.04	218.04	1,086.25	100.0%
512	128	2x	195.57	760.12	0.09	286.50	312.26	1,302.18	100.0%
512	128	4x	354.08	1,367.91	0.06	227.99	278.39	1,437.97	100.0%
512	128	8x	552.02	2,129.38	0.04	281.17	343.13	1,847.20	100.0%
512	128	16x	941.18	3,647.06	0.03	390.21	452.74	2,153.83	100.0%
512	128	32x	1,682.30	6,568.31	0.02	405.57	497.65	2,377.00	100.0%
512	128	64x	2,762.20	10,796.07	0.01	442.43	561.71	2,839.28	100.0%
512	128	128x	3,294.45	12,891.05	0.01	654.45	2,023.02	3,987.95	100.0%
512	128	256x	3,888.82	16,613.34	0.01	1,300.91	2,578.64	5,434.03	100.0%
512	128	512x	4,367.02	18,962.24	0.01	2,150.82	3,738.28	6,712.89	100.0%
512	128	1024x	4,088.09	17,959.38	0.01	4,767.59	11,615.30	14,530.24	100.0%
512	512	1x	143.06	139.40	0.29	91.22	91.22	3,557.98	100.0%
512	512	2x	288.53	280.36	0.18	101.45	106.53	3,542.75	100.0%
512	512	4x	501.72	485.28	0.12	141.98	152.62	4,066.35	100.0%
512	512	8x	646.54	624.41	0.12	193.70	223.58	6,312.98	100.0%
512	512	16x	1,313.16	1,301.84	0.06	246.36	262.91	6,079.61	100.0%
512	512	32x	1,827.98	1,819.54	0.05	306.18	336.13	8,696.65	100.0%
512	512	64x	3,344.78	3,286.92	0.03	390.92	474.77	9,467.44	100.0%
512	512	128x	3,882.78	3,858.17	0.02	843.13	687.85	10,671.52	100.0%
512	512	256x	4,807.37	4,894.96	0.02	4,561.61	9,356.02	19,674.74	100.0%
512	512	512x	5,171.42	5,397.51	0.02	12,466.39	25,082.78	34,806.35	100.0%
512	512	1024x	5,633.48	5,767.31	0.02	28,939.52	59,242.97	69,382.11	100.0%
512	1,024	1x	134.46	73.05	0.41	201.43	201.43	6,780.82	100.0%
512	1,024	2x	255.59	127.22	0.25	243.09	250.56	7,804.94	100.0%
512	1,024	4x	416.57	258.87	0.17	206.64	285.68	7,563.01	100.0%
512	1,024	8x	599.56	323.27	0.12	329.48	359.08	12,197.99	100.0%
512	1,024	16x	1,071.83	632.00	0.10	313.57	387.26	12,521.67	100.0%
512	1,024	32x	1,725.19	966.98	0.07	440.19	556.25	16,447.79	100.0%
512	1,024	64x	3,225.17	1,725.87	0.04	414.15	476.47	18,298.14	100.0%
512	1,024	128x	3,260.34	1,760.74	0.04	2,375.87	14,069.77	33,771.17	100.0%
512	1,024	256x	4,120.49	2,280.24	0.03	10,776.55	23,791.45	49,423.24	100.0%
512	1,024	512x	4,397.00	2,422.95	0.03	30,739.23	70,046.52	93,392.58	100.0%
512	1,024	1024x	4,734.18	2,632.87	0.03	56,191.92	117,581.96	137,453.52	81.7%
512	2,048	1x	132.52	32.30	0.51	221.85	221.85	15,354.89	100.0%
512	2,048	2x	239.46	58.53	0.32	206.10	255.97	16,994.49	100.0%
512	2,048	4x	286.12	135.68	0.24	178.19	239.32	13,353.54	100.0%
512	2,048	8x	526.66	266.64	0.16	300.76	364.68	14,611.83	100.0%
512	2,048	16x	719.68	364.22	0.13	316.35	377.35	18,471.23	100.0%
512	2,048	32x	1,354.75	619.97	0.09	362.22	434.45	25,618.40	100.0%
512	2,048	64x	2,243.39	947.10	0.06	417.80	550.07	33,370.05	100.0%
512	2,048	128x	3,951.98	1,639.73	0.04	669.49	695.14	38,446.76	100.0%
512	2,048	256x	4,494.84	1,897.63	0.04	11,854.45	32,588.18	59,365.29	100.0%
512	2,048	512x	4,909.46	2,028.85	0.03	34,662.41	78,679.71	105,345.13	100.0%
512	2,048	1024x	5,160.96	2,172.26	0.03	53,636.04	113,801.72	140,583.88	69.9%
1,024	128	1x	115.52	879.96	0.07	209.67	209.67	1,107.75	100.0%
1,024	128	2x	226.35	1,727.67	0.05	161.52	219.62	1,114.26	100.0%
1,024	128	4x	366.76	2,800.14	0.03	263.01	308.72	1,391.45	100.0%
1,024	128	8x	532.54	4,077.56	0.02	277.53	348.78	1,909.30	100.0%
1,024	128	16x	922.52	7,077.03	0.02	372.12	462.47	2,178.61	100.0%
1,024	128	32x	1,540.46	11,828.00	0.01	495.76	577.76	2,595.46	100.0%
1,024	128	64x	2,428.78	18,704.73	0.01	584.78	875.54	3,235.33	100.0%
1,024	128	128x	3,099.77	23,911.82	0.01	731.42	1,015.32	3,829.35	100.0%
1,024	128	256x	3,577.14	29,234.08	0.01	1,852.57	3,353.46	6,423.26	100.0%
1,024	128	512x	3,557.06	30,096.54	0.01	3,884.77	7,795.67	10,874.68	100.0%
1,024	512	1x	146.12	278.25	0.20	107.98	107.98	3,503.16	100.0%
1,024	512	2x	256.26	488.99	0.12	158.32	160.06	3,992.28	100.0%
1,024	512	4x	426.49	814.04	0.10	155.27	178.93	4,794.60	100.0%
1,024	512	8x	833.93	1,644.55	0.06	211.06	218.86	4,745.31	100.0%
1,024	512	16x	1,141.26	2,253.12	0.05	252.99	283.94	6,960.24	100.0%
1,024	512	32x	1,880.75	3,634.86	0.03	316.00	372.47	8,574.51	100.0%
1,024	512	64x	3,198.99	6,224.42	0.02	400.03	507.93	9,908.29	100.0%
1,024	512	128x	4,101.37	7,986.80	0.02	718.23	666.41	11,139.81	100.0%
1,024	512	256x	4,462.08	8,993.17	0.02	4,993.42	10,084.27	21,066.47	100.0%
1,024	512	512x	5,487.26	11,172.17	0.01	12,874.99	25,033.51	36,201.67	100.0%
1,024	512	1024x	4,756.82	9,676.73	0.02	34,935.52	71,192.37	82,059.91	100.0%
1,024	1,024	1x	129.49	123.65	0.34	1,200.15	1,200.15	7,876.16	100.0%
1,024	1,024	2x	240.92	258.95	0.18	202.15	249.25	7,468.33	100.0%
1,024	1,024	4x	443.48	423.24	0.14	274.50	329.10	9,226.33	100.0%
1,024	1,024	8x	641.59	659.79	0.12	328.28	407.05	11,850.99	100.0%
1,024	1,024	16x	1,057.41	1,056.20	0.08	434.12	542.94	14,844.82	100.0%
1,024	1,024	32x	1,928.26	1,989.81	0.05	459.25	554.10	15,723.36	100.0%
1,024	1,024	64x	2,925.82	3,008.47	0.04	541.50	828.76	20,751.47	100.0%
1,024	1,024	128x	3,947.76	4,191.90	0.03	805.59	1,023.92	26,830.26	100.0%
1,024	1,024	256x	4,056.08	4,219.29	0.03	12,278.30	31,908.89	51,847.36	100.0%
1,024	1,024	512x	4,166.94	4,427.80	0.02	33,657.27	79,396.81	104,335.86	100.0%
1,024	1,024	1024x	4,499.67	4,825.65	0.02	54,729.56	116,126.51	138,196.75	75.4%
1,024	2,048	1x	147.00	101.80	0.35	219.57	219.57	9,568.67	100.0%
1,024	2,048	2x	235.69	188.66	0.22	203.13	251.07	10,220.65	100.0%
1,024	2,048	4x	370.87	263.25	0.18	245.16	302.54	14,109.17	100.0%
1,024	2,048	8x	627.12	467.47	0.13	346.81	381.67	16,367.48	100.0%
1,024	2,048	16x	894.99	639.70	0.12	333.27	414.13	24,506.67	100.0%
1,024	2,048	32x	1,572.60	1,151.17	0.07	516.14	575.65	26,233.92	100.0%
1,024	2,048	64x	2,356.83	1,821.46	0.05	574.82	817.59	34,204.67	100.0%
1,024	2,048	128x	3,560.93	2,785.26	0.03	823.96	936.64	44,479.34	100.0%
1,024	2,048	256x	3,945.15	3,058.42	0.03	15,213.74	41,207.17	76,537.63	100.0%
1,024	2,048	512x	4,226.18	3,320.79	0.03	45,944.78	102,654.18	135,304.97	100.0%
1,024	2,048	1024x	4,711.35	3,741.15	0.03	59,438.33	120,170.37	148,630.57	64.5%
2,048	128	1x	121.90	1,874.29	0.03	227.34	227.34	1,049.48	100.0%
2,048	128	2x	219.93	3,366.84	0.02	199.02	242.67	1,159.23	100.0%
2,048	128	4x	384.67	5,889.56	0.02	246.51	320.70	1,323.11	100.0%
2,048	128	8x	538.38	8,229.23	0.01	352.99	402.96	1,891.24	100.0%
2,048	128	16x	853.51	13,055.51	0.01	435.60	589.71	2,375.97	100.0%
2,048	128	32x	1,412.01	21,591.44	0.01	609.36	833.07	2,855.50	100.0%
2,048	128	64x	1,971.30	30,185.96	0.01	781.25	1,455.89	3,978.74	100.0%
2,048	128	128x	2,845.14	43,900.39	0.01	1,049.35	1,939.87	4,932.09	100.0%
2,048	128	256x	2,776.98	43,837.99	0.01	2,797.10	7,584.13	9,969.20	100.0%
2,048	512	1x	140.81	541.25	0.13	101.89	101.89	3,635.23	100.0%
2,048	512	2x	240.60	920.82	0.08	166.90	172.95	4,249.51	100.0%
2,048	512	4x	391.56	1,778.36	0.06	154.58	175.39	4,398.58	100.0%
2,048	512	8x	580.35	2,377.28	0.05	223.80	250.48	6,558.00	100.0%
2,048	512	16x	1,111.55	4,715.25	0.03	281.96	348.12	6,608.77	100.0%
2,048	512	32x	1,834.33	7,075.88	0.02	329.85	364.80	8,783.25	100.0%
2,048	512	64x	3,109.99	12,214.62	0.01	467.53	532.62	10,095.06	100.0%
2,048	512	128x	3,667.43	14,417.69	0.01	984.19	846.39	12,271.93	100.0%
2,048	512	256x	4,534.58	18,067.49	0.01	5,885.59	11,776.62	23,670.08	100.0%
2,048	512	512x	4,150.05	17,041.99	0.01	16,858.51	35,139.45	47,429.90	100.0%
2,048	1,024	1x	145.81	534.35	0.12	128.40	128.40	3,674.99	100.0%
2,048	1,024	2x	225.07	557.55	0.13	207.25	209.18	6,872.23	100.0%
2,048	1,024	4x	396.93	938.91	0.09	187.26	195.84	8,143.50	100.0%
2,048	1,024	8x	567.84	1,498.80	0.07	270.36	325.08	10,000.47	100.0%
2,048	1,024	16x	930.16	2,398.11	0.05	314.82	370.90	13,017.94	100.0%
2,048	1,024	32x	1,558.39	3,697.67	0.04	533.54	756.79	16,838.96	100.0%
2,048	1,024	64x	2,925.95	6,485.18	0.02	601.40	856.13	19,158.24	100.0%
2,048	1,024	128x	4,031.90	8,539.85	0.02	1,202.36	1,986.87	25,752.55	100.0%
2,048	1,024	256x	4,497.71	9,635.05	0.02	11,806.88	26,556.99	50,803.71	100.0%
2,048	1,024	512x	4,474.32	9,696.82	0.02	31,981.77	71,061.69	92,574.29	100.0%
2,048	2,048	1x	145.00	263.49	0.21	196.13	196.13	7,458.81	100.0%
2,048	2,048	2x	239.08	679.44	0.11	161.35	202.77	5,672.02	100.0%
2,048	2,048	4x	399.09	831.81	0.10	270.05	298.14	9,236.76	100.0%
2,048	2,048	8x	467.93	1,131.99	0.08	332.57	434.22	12,925.77	100.0%
2,048	2,048	16x	688.34	1,478.03	0.07	429.83	562.90	17,377.08	100.0%
2,048	2,048	32x	1,306.11	2,354.81	0.05	601.21	886.41	25,161.69	100.0%
2,048	2,048	64x	2,235.08	3,862.45	0.04	684.83	1,176.03	31,183.75	100.0%
2,048	2,048	128x	3,457.25	5,554.89	0.03	1,112.83	1,760.95	41,830.10	100.0%
2,048	2,048	256x	3,886.37	6,398.25	0.02	13,867.01	37,696.61	68,750.12	100.0%
2,048	2,048	512x	3,851.55	6,393.74	0.02	41,952.86	100,011.70	129,410.34	100.0%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H200 NVL

GPU Count2

GPU Memory (Total)280 GB

GPU Driver580.95.05

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)600 W

CPU ModelIntel(R) Xeon(R) 6960P

RAM2,267 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.11.0

OSUbuntu

OS Version22.04.5 LTS (Jammy Jellyfish)

Kernel Version5.15.0-88-generic

Python Version3.10.12

Model Configuration

Providerqwen

Model Nameqwen3-coder-30b-a3b-instruct

QuantizationFP16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length8192

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.90

Temperature0.70

Top-P1.00

Top-K-1