NVIDIA H200 NVL (2x) - llama-3.3-70b-instruct

November 5, 2025 at 01:17 AM

Dataset: reference (v1.0)

NVIDIA NVIDIA H200 NVL meta-llama llama-3.3-70b-instruct

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

5,005.29

Peak generation speed

Best Input TPS

11,042.39

Peak prefill speed

Best Energy Efficiency

0.03 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

89.35 ms

Lowest latency

Best E2E (P95)

3,415.33 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	512	512x	5,005.29★	1,324.84	0.05	2,427.87	5,140.84	43,582.32	100.0%
128	128	1x	27.92	26.61	2.68	795.04	795.04	4,583.48	100.0%
128	128	2x	73.08	71.37	1.18	150.24	164.63	3,493.24	100.0%
128	128	4x	142.02	140.92	0.60	192.74	215.24	3,590.97	100.0%
128	128	8x	278.26	279.08	0.32	257.27	289.09	3,654.82	100.0%
128	128	16x	482.30	503.79	0.18	360.08	417.61	4,058.49	100.0%
128	128	32x	752.67	768.60	0.12	442.65	591.24	5,192.43	100.0%
128	128	64x	1,289.49	1,299.86	0.08	541.75	865.63	6,041.39	100.0%
128	128	128x	2,650.37	2,661.09	0.05	788.60	1,412.55	5,696.16	100.0%
128	128	256x	3,398.56	3,431.27	0.04	1,373.25	3,773.89	8,499.89	100.0%
128	128	512x	2,970.73	3,084.06	0.04	4,658.18	7,243.61	19,628.31	100.0%
128	512	1x	37.81	9.01	4.32	89.35	89.35	13,541.46	100.0%
128	512	2x	75.83	18.51	2.17	130.16	139.73	13,494.58	100.0%
128	512	4x	149.26	37.02	1.10	155.39	164.07	13,712.81	100.0%
128	512	8x	289.12	74.51	0.58	238.93	257.77	13,771.64	100.0%
128	512	16x	603.17	152.74	0.29	255.61	268.53	13,452.28	100.0%
128	512	32x	920.12	252.04	0.19	338.18	360.78	16,146.04	100.0%
128	512	64x	1,856.85	484.35	0.11	418.79	508.80	16,556.82	100.0%
128	512	128x	3,275.42	853.07	0.07	632.27	844.45	18,653.78	100.0%
128	512	256x	4,186.70	1,109.88	0.06	697.02	2,597.40	25,169.79	100.0%
128	512	1024x	4,755.61	1,263.01	0.05	19,924.91	47,890.56	86,526.48	100.0%
128	1,024	1x	36.72	5.28	4.95	92.31	92.31	23,067.54	100.0%
128	1,024	2x	63.71	10.95	2.81	133.18	145.18	22,430.66	100.0%
128	1,024	4x	117.27	20.47	1.53	186.52	202.67	24,085.42	100.0%
128	1,024	8x	223.77	37.67	0.82	251.62	276.75	25,891.73	100.0%
128	1,024	16x	457.13	81.44	0.42	314.47	371.98	22,830.40	100.0%
128	1,024	32x	840.78	146.31	0.24	411.31	543.31	26,436.87	100.0%
128	1,024	64x	1,494.27	248.33	0.14	518.99	809.35	32,503.49	100.0%
128	1,024	128x	2,760.55	462.11	0.09	755.12	1,386.43	34,825.00	100.0%
128	1,024	256x	3,914.60	678.59	0.07	931.25	1,602.15	46,388.73	100.0%
128	1,024	512x	4,600.33	798.65	0.06	1,185.02	2,683.50	76,297.81	100.0%
128	1,024	1024x	4,528.41	785.95	0.06	11,352.58	26,843.03	107,604.55	100.0%
128	2,048	1x	37.88	6.28	4.68	91.42	91.42	19,400.24	100.0%
128	2,048	2x	60.08	10.31	2.99	134.36	146.61	23,717.39	100.0%
128	2,048	4x	118.36	20.92	1.52	189.71	202.43	23,676.08	100.0%
128	2,048	8x	213.82	40.51	0.84	258.96	281.88	23,275.86	100.0%
128	2,048	16x	503.08	88.18	0.38	353.34	403.18	22,803.79	100.0%
128	2,048	32x	671.79	121.13	0.30	399.77	515.36	26,463.96	100.0%
128	2,048	64x	1,474.69	252.27	0.15	549.11	865.90	29,670.19	100.0%
128	2,048	128x	2,246.25	382.41	0.11	767.07	1,288.36	35,194.88	100.0%
128	2,048	256x	3,381.87	568.64	0.07	939.70	1,608.96	50,723.91	100.0%
128	2,048	512x	4,195.17	729.80	0.06	1,499.58	2,814.41	77,645.75	100.0%
128	2,048	1024x	4,270.75	736.14	0.06	21,339.36	53,492.77	122,579.06	100.0%
512	128	1x	37.47	145.20	1.00	131.14	131.14	3,415.33	100.0%
512	128	2x	53.82	209.17	0.58	1,451.36	1,452.90	4,753.99	100.0%
512	128	4x	139.02	537.06	0.27	259.85	291.68	3,674.69	100.0%
512	128	8x	267.92	1,033.49	0.14	329.30	434.61	3,797.43	100.0%
512	128	16x	502.82	1,948.44	0.09	494.13	721.57	4,026.33	100.0%
512	128	32x	816.04	3,291.61	0.06	738.66	1,307.29	4,749.16	100.0%
512	128	64x	1,266.51	4,929.62	0.04	1,220.14	2,326.78	6,276.78	100.0%
512	128	128x	1,760.98	6,903.37	0.03	1,862.24	4,230.29	8,836.21	100.0%
512	128	256x	2,035.71	7,948.80	0.03	2,873.79	7,833.28	15,098.84	100.0%
512	128	512x	2,093.83	8,200.93	0.03	4,745.84	14,808.81	26,285.51	100.0%
512	128	1024x	2,157.03	8,418.74	0.03	10,380.22	37,590.13	47,400.88	100.0%
512	512	1x	38.19	38.66	2.68	123.85	123.85	12,801.30	100.0%
512	512	2x	75.37	73.23	1.38	158.62	160.15	13,584.37	100.0%
512	512	4x	147.38	142.34	0.74	252.07	274.96	13,890.65	100.0%
512	512	8x	279.83	278.35	0.38	328.40	441.99	14,168.39	100.0%
512	512	16x	541.17	560.37	0.21	568.60	696.02	14,130.77	100.0%
512	512	32x	959.12	1,037.37	0.12	719.10	1,274.92	15,281.08	100.0%
512	512	64x	1,696.68	1,751.87	0.08	1,067.15	2,296.39	18,000.80	100.0%
512	512	128x	2,700.04	2,823.38	0.06	1,864.28	4,337.15	22,154.65	100.0%
512	512	256x	3,411.42	3,486.46	0.05	2,826.83	7,823.03	35,474.48	100.0%
512	512	512x	3,590.84	3,666.14	0.04	4,569.40	14,453.39	63,376.18	100.0%
512	512	1024x	3,293.07	3,356.23	0.05	43,812.06	94,115.43	145,345.51	99.7%
512	1,024	1x	36.22	24.28	3.35	132.18	132.18	20,401.79	100.0%
512	1,024	2x	58.08	46.01	1.92	197.58	226.31	21,253.58	100.0%
512	1,024	4x	134.28	112.40	0.84	255.14	286.95	17,424.14	100.0%
512	1,024	8x	212.96	144.93	0.61	329.41	435.94	25,094.45	100.0%
512	1,024	16x	413.65	310.78	0.31	605.18	771.92	24,535.04	100.0%
512	1,024	32x	674.40	491.53	0.19	740.83	1,324.67	30,777.87	100.0%
512	1,024	64x	1,296.67	881.48	0.11	1,242.08	2,355.63	35,918.05	100.0%
512	1,024	128x	2,443.69	1,632.67	0.07	1,841.44	4,333.95	38,517.16	100.0%
512	1,024	256x	3,216.14	2,201.34	0.06	2,882.40	7,831.90	56,774.49	100.0%
512	1,024	512x	3,289.63	2,231.17	0.06	4,712.96	15,020.97	103,062.90	100.0%
512	1,024	1024x	3,199.39	2,185.86	0.06	43,269.87	119,174.75	176,237.73	79.2%
512	2,048	1x	33.91	32.47	2.88	132.64	132.64	15,245.17	100.0%
512	2,048	2x	69.60	47.30	1.79	147.85	175.12	20,835.47	100.0%
512	2,048	4x	131.75	98.56	0.93	251.57	283.98	19,943.97	100.0%
512	2,048	8x	209.58	158.32	0.59	355.20	439.76	24,817.30	100.0%
512	2,048	16x	397.11	320.34	0.32	499.62	730.87	22,648.00	100.0%
512	2,048	32x	696.97	492.43	0.21	779.04	1,347.22	27,111.52	100.0%
512	2,048	64x	788.77	532.04	0.18	1,251.45	2,380.14	34,781.22	100.0%
512	2,048	128x	1,953.87	1,318.56	0.09	1,809.66	4,134.34	40,569.76	100.0%
512	2,048	256x	2,982.51	1,995.08	0.06	2,882.33	7,894.19	57,801.13	100.0%
512	2,048	512x	3,007.98	2,023.12	0.06	4,777.11	14,798.78	105,435.54	100.0%
512	2,048	1024x	3,074.73	2,068.80	0.06	43,876.74	118,550.31	178,204.13	80.0%
1,024	128	1x	36.15	275.35	0.57	189.49	189.49	3,540.71	100.0%
1,024	128	2x	71.63	546.73	0.29	220.71	221.92	3,571.25	100.0%
1,024	128	4x	134.03	1,023.30	0.16	350.91	413.11	3,814.30	100.0%
1,024	128	8x	247.88	1,896.15	0.09	584.74	715.59	4,121.60	100.0%
1,024	128	16x	389.90	2,992.57	0.06	1,542.10	1,863.52	5,214.90	100.0%
1,024	128	32x	689.10	5,287.18	0.04	1,140.11	2,303.10	5,845.70	100.0%
1,024	128	64x	948.96	7,329.76	0.03	1,827.68	4,293.72	8,400.83	100.0%
1,024	128	128x	1,197.35	9,203.72	0.03	3,025.68	8,419.82	13,266.40	100.0%
1,024	128	256x	1,308.59	10,106.31	0.03	5,067.11	15,421.40	23,981.27	100.0%
1,024	128	512x	1,254.41	9,644.75	0.03	10,452.63	38,032.41	47,257.35	100.0%
1,024	512	1x	32.57	62.03	2.01	192.21	192.21	15,717.97	100.0%
1,024	512	2x	74.72	142.58	0.94	222.23	224.36	13,702.00	100.0%
1,024	512	4x	144.99	276.74	0.49	343.23	405.11	14,117.09	100.0%
1,024	512	8x	283.36	541.89	0.27	574.24	702.11	14,439.58	100.0%
1,024	512	16x	544.54	1,056.34	0.15	938.37	1,253.85	14,837.07	100.0%
1,024	512	32x	976.17	1,872.55	0.09	1,240.04	2,263.43	16,685.81	100.0%
1,024	512	64x	1,516.78	2,977.14	0.06	1,825.45	4,371.97	20,923.48	100.0%
1,024	512	128x	2,254.67	4,477.24	0.05	2,997.03	8,358.80	27,654.52	100.0%
1,024	512	256x	2,652.84	5,292.04	0.04	5,018.41	15,351.38	46,566.72	100.0%
1,024	512	512x	2,208.24	4,403.71	0.05	18,729.17	82,899.88	108,919.26	100.0%
1,024	1,024	1x	37.21	45.70	2.50	197.47	197.47	21,307.30	100.0%
1,024	1,024	2x	69.12	87.47	1.33	191.88	247.97	22,153.56	100.0%
1,024	1,024	4x	120.64	139.64	0.82	352.89	418.27	27,367.78	100.0%
1,024	1,024	8x	220.76	277.08	0.44	584.42	718.45	27,241.81	100.0%
1,024	1,024	16x	422.16	573.65	0.23	937.53	1,251.88	25,331.13	100.0%
1,024	1,024	32x	742.71	1,028.34	0.14	1,092.03	2,277.18	28,857.04	100.0%
1,024	1,024	64x	1,239.86	1,751.81	0.09	1,799.75	4,324.09	33,495.82	100.0%
1,024	1,024	128x	2,002.23	2,800.45	0.06	2,993.19	8,386.62	44,129.81	100.0%
1,024	1,024	256x	2,634.61	3,612.40	0.05	5,022.06	15,430.19	67,922.29	100.0%
1,024	1,024	512x	2,312.13	3,169.70	0.06	23,316.56	109,238.54	149,854.09	100.0%
1,024	2,048	1x	37.73	44.97	2.55	193.25	193.25	21,654.08	100.0%
1,024	2,048	2x	73.80	96.01	1.23	231.57	245.90	20,292.35	100.0%
1,024	2,048	4x	114.20	139.15	0.81	259.21	293.32	27,554.89	100.0%
1,024	2,048	8x	212.57	249.07	0.47	381.98	457.79	29,761.01	100.0%
1,024	2,048	16x	357.09	461.26	0.28	615.23	990.38	28,212.58	100.0%
1,024	2,048	32x	514.74	638.84	0.20	1,104.98	2,285.09	33,406.86	100.0%
1,024	2,048	64x	1,076.83	1,404.89	0.10	1,753.62	4,248.03	37,716.58	100.0%
1,024	2,048	128x	1,326.67	1,778.96	0.09	2,998.11	8,342.38	43,383.76	100.0%
1,024	2,048	256x	2,179.74	2,964.39	0.06	4,890.72	14,737.14	69,248.52	100.0%
1,024	2,048	512x	2,201.84	2,973.83	0.06	23,117.45	109,174.36	150,616.80	100.0%
1,024	2,048	1024x	1,747.02	2,354.15	0.07	26,081.90	49,347.33	155,676.79	41.6%
2,048	128	1x	33.98	522.43	0.35	396.35	396.35	3,767.08	100.0%
2,048	128	2x	68.61	1,050.39	0.17	252.68	360.49	3,722.55	100.0%
2,048	128	4x	125.18	1,916.63	0.10	538.45	662.00	4,080.61	100.0%
2,048	128	8x	219.13	3,349.45	0.06	774.73	1,200.31	4,647.26	100.0%
2,048	128	16x	353.46	5,443.96	0.04	1,357.33	2,244.19	5,693.25	100.0%
2,048	128	32x	501.59	7,662.50	0.03	1,825.28	4,347.49	8,045.08	100.0%
2,048	128	64x	624.30	9,537.49	0.03	2,979.22	8,482.95	12,887.04	100.0%
2,048	128	128x	721.87	11,042.39	0.03	5,221.11	16,636.89	22,153.17	100.0%
2,048	128	256x	676.48	10,366.11	0.03	10,927.62	38,866.21	46,286.87	100.0%
2,048	512	1x	37.66	144.74	1.18	372.84	372.84	13,596.60	100.0%
2,048	512	2x	73.79	282.41	0.58	257.63	367.45	13,871.15	100.0%
2,048	512	4x	141.68	542.30	0.31	542.82	668.95	14,449.08	100.0%
2,048	512	8x	240.51	1,032.79	0.18	700.49	1,175.35	15,131.33	100.0%
2,048	512	16x	474.02	1,930.21	0.10	1,322.21	2,217.70	16,150.44	100.0%
2,048	512	32x	768.24	3,214.76	0.07	1,815.93	4,368.52	19,348.21	100.0%
2,048	512	64x	1,168.97	4,773.66	0.05	2,990.34	8,496.96	25,973.89	100.0%
2,048	512	128x	1,615.10	6,443.91	0.04	5,186.61	16,614.46	38,315.72	100.0%
2,048	512	256x	1,407.60	5,666.35	0.04	15,135.78	66,343.06	86,245.12	100.0%
2,048	1,024	1x	34.56	125.04	1.29	330.72	330.72	15,711.60	100.0%
2,048	1,024	2x	62.85	177.84	0.90	367.62	382.34	21,664.27	100.0%
2,048	1,024	4x	121.19	332.74	0.48	465.67	616.18	23,207.62	100.0%
2,048	1,024	8x	196.99	503.65	0.31	1,077.80	1,209.99	31,060.05	100.0%
2,048	1,024	16x	351.09	1,059.87	0.16	1,624.96	2,247.09	27,929.83	100.0%
2,048	1,024	32x	560.38	1,683.67	0.11	1,817.18	4,324.11	36,818.86	100.0%
2,048	1,024	64x	1,023.30	3,047.48	0.07	2,978.01	8,366.42	39,153.75	100.0%
2,048	1,024	128x	1,604.62	4,468.88	0.05	5,197.87	16,839.42	53,503.91	100.0%
2,048	1,024	256x	1,564.89	4,367.55	0.05	16,177.46	73,770.45	106,816.38	100.0%
2,048	2,048	1x	38.06	118.51	1.37	332.52	332.52	16,578.68	100.0%
2,048	2,048	2x	71.48	251.46	0.66	245.84	354.24	15,494.76	100.0%
2,048	2,048	4x	75.22	143.56	1.00	527.52	651.59	49,795.42	100.0%
2,048	2,048	8x	195.94	541.74	0.28	585.93	944.75	27,621.33	100.0%
2,048	2,048	16x	295.81	977.75	0.17	1,221.68	1,716.47	31,237.02	100.0%
2,048	2,048	32x	368.85	1,042.50	0.16	1,625.86	3,324.90	38,792.39	100.0%
2,048	2,048	64x	679.14	1,893.87	0.10	2,320.12	6,195.84	38,422.89	100.0%
2,048	2,048	128x	1,131.85	3,012.85	0.07	5,092.48	16,115.27	55,709.06	100.0%
2,048	2,048	256x	1,402.47	3,893.69	0.06	16,574.35	76,412.29	109,099.05	100.0%
2,048	2,048	512x	1,171.67	3,201.60	0.07	46,965.95	116,072.02	156,419.55	59.6%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H200 NVL

GPU Count2

GPU Memory (Total)280 GB

GPU Driver580.95.05

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)600 W

CPU ModelIntel(R) Xeon(R) 6960P

RAM2,267 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.11.0

OSUbuntu

OS Version22.04.5 LTS (Jammy Jellyfish)

Kernel Version5.15.0-88-generic

Python Version3.10.12

Model Configuration

Providermeta-llama

Model Namellama-3.3-70b-instruct

QuantizationFP16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length8192

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.90

Temperature0.70

Top-P1.00

Top-K-1