NVIDIA H200 NVL (2x) - qwen3-30b-a3b

November 7, 2025 at 01:50 AM

Dataset: reference (v1.0)

NVIDIA NVIDIA H200 NVL qwen qwen3-30b-a3b

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

6,124.38

Peak generation speed

Best Input TPS

51,413.77

Peak prefill speed

Best Energy Efficiency

0.00 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

80.05 ms

Lowest latency

Best E2E (P95)

1,029.03 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	512	1024x	6,124.38★	1,619.44	0.03	21,801.48	44,878.08	54,102.09	100.0%
128	128	1x	91.30	87.02	0.36	583.52	583.52	1,401.52	100.0%
128	128	2x	175.34	171.23	0.21	357.86	612.15	1,424.48	100.0%
128	128	4x	193.43	191.92	0.17	1,190.23	1,532.29	2,640.21	100.0%
128	128	8x	573.67	575.35	0.08	294.52	333.60	1,775.38	100.0%
128	128	16x	620.23	624.77	0.09	1,090.83	1,607.70	3,291.78	100.0%
128	128	32x	2,030.75	2,034.72	0.03	365.43	421.70	1,980.77	100.0%
128	128	64x	3,045.39	3,033.48	0.03	391.63	455.27	2,555.71	100.0%
128	128	128x	2,584.91	2,618.96	0.02	515.97	466.33	4,645.84	100.0%
128	512	1x	145.29	34.62	0.42	83.01	83.01	3,523.96	100.0%
128	512	2x	242.31	59.16	0.27	191.52	192.93	4,224.10	100.0%
128	512	4x	437.79	108.59	0.22	144.05	167.11	4,668.53	100.0%
128	512	8x	638.19	162.47	0.18	183.55	201.14	6,303.05	100.0%
128	512	16x	1,128.64	288.77	0.11	236.00	278.18	7,111.92	100.0%
128	512	32x	1,917.19	485.27	0.07	252.10	274.24	8,369.90	100.0%
128	512	64x	3,537.10	885.82	0.05	299.43	358.24	9,011.70	100.0%
128	512	128x	4,207.47	1,065.92	0.04	681.03	480.03	9,924.89	100.0%
128	512	256x	5,160.93	1,355.65	0.03	3,665.93	8,372.80	17,941.22	100.0%
128	512	512x	5,538.10	1,466.77	0.03	9,186.74	19,399.50	28,896.77	100.0%
128	1,024	1x	148.53	17.70	0.51	87.21	87.21	6,891.73	100.0%
128	1,024	2x	270.51	33.02	0.30	125.27	127.26	7,564.85	100.0%
128	1,024	4x	422.44	52.39	0.27	127.51	148.70	9,684.35	100.0%
128	1,024	8x	812.21	107.51	0.16	171.35	182.94	9,537.79	100.0%
128	1,024	16x	1,121.23	145.66	0.14	229.80	284.20	14,120.79	100.0%
128	1,024	32x	1,973.20	252.73	0.09	269.35	308.74	16,129.97	100.0%
128	1,024	64x	3,663.31	471.38	0.05	306.31	369.39	17,075.42	100.0%
128	1,024	128x	3,924.64	500.76	0.04	1,515.52	16,059.46	30,348.55	100.0%
128	1,024	256x	5,962.24	782.64	0.03	7,654.85	16,624.12	35,944.46	100.0%
128	1,024	512x	5,878.08	771.43	0.03	22,537.87	45,301.78	64,370.00	100.0%
128	2,048	1x	147.11	8.76	0.55	87.86	87.86	13,918.64	100.0%
128	2,048	2x	278.26	16.98	0.36	101.35	116.05	14,673.75	100.0%
128	2,048	4x	496.49	31.80	0.24	129.50	140.48	15,817.64	100.0%
128	2,048	8x	584.56	43.64	0.22	180.70	204.83	23,024.68	100.0%
128	2,048	16x	1,232.63	82.83	0.13	220.17	236.08	24,853.65	100.0%
128	2,048	32x	1,745.28	120.36	0.11	599.47	854.98	34,001.95	100.0%
128	2,048	64x	3,185.52	216.29	0.07	315.98	393.25	37,508.12	100.0%
128	2,048	128x	4,775.11	325.37	0.04	795.16	506.82	41,881.24	100.0%
128	2,048	256x	4,779.97	331.57	0.04	18,368.64	39,975.42	81,514.20	100.0%
128	2,048	512x	5,501.68	388.29	0.04	50,010.75	108,425.07	146,593.38	100.0%
128	2,048	1024x	5,634.65	402.10	0.04	58,659.48	120,161.33	154,871.75	56.6%
512	128	1x	117.22	454.21	0.11	190.02	190.02	1,092.12	100.0%
512	128	2x	241.51	938.68	0.06	139.70	192.98	1,052.04	100.0%
512	128	4x	353.84	1,366.97	0.07	251.92	295.25	1,440.48	100.0%
512	128	8x	562.64	2,170.33	0.04	275.59	316.97	1,808.46	100.0%
512	128	16x	963.75	3,736.35	0.03	309.62	360.50	2,104.37	100.0%
512	128	32x	1,710.34	6,681.04	0.02	381.93	453.07	2,322.49	100.0%
512	128	64x	2,957.69	11,526.22	0.01	420.31	585.78	2,655.11	100.0%
512	128	128x	3,248.02	12,630.16	0.01	581.18	2,553.10	4,370.43	100.0%
512	128	256x	4,161.53	17,353.87	0.01	844.13	2,324.46	5,174.83	100.0%
512	128	512x	4,548.13	19,088.77	0.01	1,679.11	3,133.32	5,818.13	100.0%
512	128	1024x	4,393.17	20,312.10	0.01	4,094.47	8,631.40	11,388.98	100.0%
512	512	1x	143.18	138.70	0.32	80.05	80.05	3,575.47	100.0%
512	512	2x	204.19	198.40	0.23	94.86	103.50	4,929.03	100.0%
512	512	4x	421.40	407.00	0.14	143.06	168.11	4,851.35	100.0%
512	512	8x	656.20	632.81	0.11	180.79	205.38	6,230.99	100.0%
512	512	16x	1,162.17	1,125.99	0.08	234.27	267.83	7,023.93	100.0%
512	512	32x	2,306.36	2,297.87	0.04	272.34	300.11	6,887.76	100.0%
512	512	64x	3,506.83	3,427.99	0.03	326.76	359.23	9,098.20	100.0%
512	512	128x	4,933.27	4,865.92	0.02	346.08	404.15	9,649.20	100.0%
512	512	256x	4,993.62	5,042.90	0.02	3,799.91	8,247.80	17,978.17	100.0%
512	512	512x	5,469.17	5,553.63	0.02	10,009.90	19,416.60	28,895.73	100.0%
512	512	1024x	5,554.69	5,577.92	0.02	25,698.70	58,247.67	65,436.90	100.0%
512	1,024	1x	138.15	66.92	0.44	173.55	173.55	7,411.20	100.0%
512	1,024	2x	278.75	135.43	0.26	80.37	84.75	7,312.41	100.0%
512	1,024	4x	447.21	215.96	0.19	204.19	252.79	9,147.75	100.0%
512	1,024	8x	639.00	308.11	0.14	263.09	303.04	12,809.59	100.0%
512	1,024	16x	1,101.01	583.40	0.11	244.83	327.60	13,587.52	100.0%
512	1,024	32x	1,855.75	943.99	0.07	363.79	460.48	16,841.32	100.0%
512	1,024	64x	3,577.98	1,794.28	0.04	382.96	446.71	17,637.94	100.0%
512	1,024	128x	4,728.08	2,340.72	0.03	704.70	499.94	19,940.60	100.0%
512	1,024	256x	5,151.71	2,604.34	0.03	8,997.26	18,963.13	39,167.75	100.0%
512	1,024	512x	5,548.74	2,826.22	0.03	24,527.92	52,039.27	74,252.09	100.0%
512	1,024	1024x	5,722.64	2,922.34	0.03	57,390.37	118,995.24	139,735.32	94.6%
512	2,048	1x	146.31	39.72	0.46	191.85	191.85	12,477.67	100.0%
512	2,048	2x	250.75	68.13	0.33	133.47	178.63	14,398.93	100.0%
512	2,048	4x	394.19	105.04	0.25	188.99	266.54	18,668.21	100.0%
512	2,048	8x	599.27	168.18	0.19	324.60	363.90	23,472.43	100.0%
512	2,048	16x	972.65	290.95	0.13	334.85	368.26	27,249.02	100.0%
512	2,048	32x	1,773.23	502.49	0.09	419.56	519.07	31,709.22	100.0%
512	2,048	64x	3,098.13	846.12	0.06	433.08	568.02	37,541.11	100.0%
512	2,048	128x	5,173.69	1,410.28	0.04	569.53	590.98	41,142.50	100.0%
512	2,048	256x	4,647.68	1,271.63	0.04	18,464.87	43,501.50	84,356.32	100.0%
1,024	128	1x	124.39	947.52	0.06	192.17	192.17	1,029.03	100.0%
1,024	128	2x	242.19	1,848.63	0.04	139.67	188.75	1,045.51	100.0%
1,024	128	4x	366.50	2,798.14	0.04	244.53	291.98	1,390.52	100.0%
1,024	128	8x	743.11	5,684.33	0.02	231.23	261.17	1,367.39	100.0%
1,024	128	16x	941.61	7,223.45	0.01	376.77	473.05	2,153.15	100.0%
1,024	128	32x	1,622.52	12,461.14	0.01	465.61	549.32	2,488.07	100.0%
1,024	128	64x	2,702.78	20,842.94	0.01	458.93	707.01	2,887.47	100.0%
1,024	128	128x	3,191.30	24,536.66	0.01	741.97	2,918.97	4,529.83	100.0%
1,024	128	256x	3,455.20	28,255.89	0.01	1,711.01	4,364.12	6,397.15	100.0%
1,024	128	512x	3,919.66	33,237.72	0.01	3,058.17	5,914.63	9,072.96	100.0%
1,024	128	1024x	4,461.59	36,953.90	0.01	4,988.56	11,282.06	14,100.32	100.0%
1,024	512	1x	137.04	260.97	0.21	165.29	165.29	3,735.81	100.0%
1,024	512	2x	269.26	513.80	0.12	137.38	138.76	3,799.87	100.0%
1,024	512	4x	516.78	986.37	0.08	195.22	224.86	3,955.41	100.0%
1,024	512	8x	653.06	1,248.88	0.08	232.02	299.57	6,252.01	100.0%
1,024	512	16x	1,112.59	2,133.78	0.06	246.13	317.61	7,340.56	100.0%
1,024	512	32x	2,311.41	4,436.34	0.03	342.46	399.26	7,003.86	100.0%
1,024	512	64x	3,408.31	6,550.45	0.02	390.24	567.68	9,456.00	100.0%
1,024	512	128x	3,713.59	7,153.16	0.02	1,254.26	9,618.94	16,841.95	100.0%
1,024	512	256x	4,301.36	8,504.75	0.02	5,193.06	10,015.43	21,849.95	100.0%
1,024	512	512x	5,180.60	10,267.66	0.02	11,659.17	25,085.42	34,795.04	100.0%
1,024	512	1024x	5,402.42	10,752.24	0.01	27,083.07	54,730.29	64,918.02	100.0%
1,024	1,024	1x	148.56	141.45	0.31	202.25	202.25	6,892.44	100.0%
1,024	1,024	2x	268.59	256.26	0.18	230.18	232.57	7,622.23	100.0%
1,024	1,024	4x	443.77	423.51	0.15	229.98	296.12	9,220.87	100.0%
1,024	1,024	8x	671.26	641.84	0.12	240.59	297.89	12,190.18	100.0%
1,024	1,024	16x	1,140.47	1,093.62	0.09	400.57	475.75	14,333.18	100.0%
1,024	1,024	32x	1,951.03	1,888.19	0.05	452.79	577.79	16,579.44	100.0%
1,024	1,024	64x	3,523.36	3,453.42	0.03	490.80	681.25	18,107.45	100.0%
1,024	1,024	128x	3,608.77	3,536.37	0.03	1,668.74	9,540.43	28,320.37	100.0%
1,024	1,024	256x	4,600.08	4,552.31	0.02	10,717.23	20,415.74	41,083.54	100.0%
1,024	1,024	512x	5,258.52	5,268.51	0.02	27,367.01	57,314.21	77,299.10	100.0%
1,024	1,024	1024x	5,599.61	5,598.47	0.02	55,063.58	115,462.51	135,729.81	89.0%
1,024	2,048	1x	145.54	69.29	0.41	183.52	183.52	14,070.26	100.0%
1,024	2,048	2x	257.55	122.86	0.25	210.11	211.66	15,900.72	100.0%
1,024	2,048	4x	502.07	248.52	0.17	200.97	292.06	15,690.43	100.0%
1,024	2,048	8x	643.98	316.54	0.16	283.30	340.46	24,729.41	100.0%
1,024	2,048	16x	1,136.93	566.73	0.10	317.46	387.46	27,679.28	100.0%
1,024	2,048	32x	1,802.38	938.43	0.08	999.48	1,891.13	33,430.48	100.0%
1,024	2,048	64x	3,166.88	1,673.47	0.05	507.88	702.53	37,439.60	100.0%
1,024	2,048	128x	4,579.82	2,419.51	0.03	1,230.11	953.67	43,198.59	100.0%
1,024	2,048	256x	4,616.74	2,445.81	0.03	19,940.03	43,404.17	87,613.48	100.0%
1,024	2,048	512x	4,972.64	2,661.33	0.03	48,631.00	114,271.31	149,219.99	87.7%
2,048	128	1x	124.15	1,908.83	0.04	203.81	203.81	1,030.91	100.0%
2,048	128	2x	232.94	3,565.97	0.02	192.80	211.11	1,091.94	100.0%
2,048	128	4x	417.96	6,399.18	0.02	199.14	225.96	1,221.70	100.0%
2,048	128	8x	539.23	8,242.23	0.01	316.29	355.68	1,891.74	100.0%
2,048	128	16x	871.86	13,316.73	0.01	337.09	541.54	2,312.39	100.0%
2,048	128	32x	1,455.55	22,251.78	0.01	524.30	711.40	2,760.41	100.0%
2,048	128	64x	2,252.54	34,429.44	0.01	730.66	1,218.40	3,544.02	100.0%
2,048	128	128x	2,530.14	38,689.34	0.01	1,106.54	3,225.31	5,488.79	100.0%
2,048	128	256x	2,825.13	43,251.21	0.01	2,680.37	5,569.04	8,997.70	100.0%
2,048	128	512x	3,249.15	51,413.77	0.00	4,780.79	10,467.94	13,726.57	100.0%
2,048	128	1024x	3,156.32	49,914.45	0.01	11,850.90	26,135.58	29,770.75	100.0%
2,048	512	1x	134.63	517.49	0.14	177.93	177.93	3,802.73	100.0%
2,048	512	2x	254.03	972.22	0.08	256.68	257.76	4,027.76	100.0%
2,048	512	4x	402.12	1,539.17	0.06	259.32	307.29	5,089.00	100.0%
2,048	512	8x	620.41	2,468.38	0.05	255.03	330.87	6,326.22	100.0%
2,048	512	16x	1,242.44	4,778.64	0.03	423.18	524.00	6,521.55	100.0%
2,048	512	32x	1,744.35	6,665.10	0.02	521.97	781.46	9,314.58	100.0%
2,048	512	64x	3,136.23	11,989.94	0.01	582.96	1,018.60	10,302.16	100.0%
2,048	512	128x	3,637.05	13,960.18	0.01	1,271.71	1,977.79	13,596.17	100.0%
2,048	512	256x	4,008.36	15,336.18	0.01	8,100.72	16,176.55	27,136.01	100.0%
2,048	512	512x	4,582.71	18,022.62	0.01	14,955.93	31,034.42	42,838.00	100.0%
2,048	512	1024x	4,353.12	16,892.41	0.01	41,756.34	81,539.19	95,689.78	100.0%
2,048	1,024	1x	147.30	283.08	0.20	195.00	195.00	6,950.93	100.0%
2,048	1,024	2x	268.38	513.56	0.14	141.04	186.72	7,581.70	100.0%
2,048	1,024	4x	418.62	875.08	0.10	275.38	342.23	8,945.72	100.0%
2,048	1,024	8x	557.98	1,253.56	0.07	362.89	457.41	12,469.39	100.0%
2,048	1,024	16x	962.68	2,122.62	0.05	459.62	694.99	14,703.88	100.0%
2,048	1,024	32x	1,714.28	3,606.66	0.04	546.90	959.54	17,259.36	100.0%
2,048	1,024	64x	3,057.89	6,229.83	0.02	654.52	1,123.20	19,967.54	100.0%
2,048	1,024	128x	3,480.20	6,892.83	0.02	2,103.63	13,641.02	31,546.38	100.0%
2,048	1,024	256x	4,880.79	9,682.73	0.02	11,641.85	24,412.62	47,193.27	100.0%
2,048	1,024	512x	5,086.68	10,127.12	0.02	33,041.71	68,961.42	90,690.26	100.0%
2,048	1,024	1024x	5,024.20	9,906.06	0.02	61,462.20	119,826.44	141,925.53	75.2%
2,048	2,048	1x	150.33	156.45	0.29	217.06	217.06	12,570.07	100.0%
2,048	2,048	2x	266.25	257.57	0.19	275.99	305.39	15,193.66	100.0%
2,048	2,048	4x	350.67	447.05	0.15	807.31	1,467.48	17,061.78	100.0%
2,048	2,048	8x	658.93	865.09	0.10	330.37	408.67	18,078.46	100.0%
2,048	2,048	16x	882.09	1,201.77	0.08	415.50	543.74	26,000.27	100.0%
2,048	2,048	32x	1,611.79	1,927.72	0.06	530.69	781.89	32,388.14	100.0%
2,048	2,048	64x	2,840.95	3,163.49	0.04	667.42	1,193.94	39,402.31	100.0%
2,048	2,048	128x	3,414.81	3,654.47	0.03	3,339.43	24,196.88	56,045.27	100.0%
2,048	2,048	256x	4,765.42	5,132.70	0.03	21,082.90	46,978.68	92,241.68	100.0%
2,048	2,048	512x	4,707.85	5,106.03	0.03	48,177.25	108,485.30	147,303.83	83.6%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H200 NVL

GPU Count2

GPU Memory (Total)280 GB

GPU Driver580.95.05

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)600 W

CPU ModelIntel(R) Xeon(R) 6960P

RAM2,267 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.11.0

OSUbuntu

OS Version22.04.5 LTS (Jammy Jellyfish)

Kernel Version5.15.0-88-generic

Python Version3.10.12

Model Configuration

Providerqwen

Model Nameqwen3-30b-a3b

QuantizationFP16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model LengthUnknown

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.90

Temperature0.70

Top-P1.00

Top-K-1