NVIDIA H200 NVL (2x) - llama-2-70b-hf

November 6, 2025 at 03:39 AM

Dataset: reference (v1.0)

NVIDIA NVIDIA H200 NVL meta-llama llama-2-70b-hf

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

5,012.77

Peak generation speed

Best Input TPS

10,466.05

Peak prefill speed

Best Energy Efficiency

0.03 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

51.52 ms

Lowest latency

Best E2E (P95)

2,070.89 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	512	512x	5,012.77★	1,367.05	0.05	1,207.76	2,328.83	41,556.05	100.0%
128	128	1x	41.29	39.35	2.03	254.43	254.43	3,099.30	100.0%
128	128	2x	84.41	85.09	1.22	65.79	77.08	2,926.04	100.0%
128	128	4x	169.93	168.60	0.56	105.32	118.49	3,006.52	100.0%
128	128	8x	329.47	330.44	0.30	165.28	187.68	3,095.06	100.0%
128	128	16x	621.17	625.72	0.17	238.59	282.77	3,269.81	100.0%
128	128	32x	1,091.21	1,095.07	0.12	328.89	498.57	3,545.87	96.9%
128	128	64x	1,508.75	1,517.87	0.07	1,804.11	1,964.82	5,295.49	100.0%
128	128	128x	1,826.68	1,906.53	0.06	2,185.55	4,272.60	8,152.60	100.0%
128	128	256x	3,318.26	3,473.66	0.04	678.16	1,809.85	7,273.48	99.2%
128	128	512x	3,268.08	3,625.44	0.04	2,074.72	4,791.55	13,227.93	99.8%
128	512	1x	30.89	28.12	2.51	1,341.12	1,341.12	4,314.42	100.0%
128	512	2x	48.35	21.62	3.29	67.03	80.83	11,032.70	100.0%
128	512	4x	175.19	43.46	1.01	81.71	89.46	11,683.49	100.0%
128	512	8x	344.90	86.48	0.52	109.71	123.80	11,868.61	100.0%
128	512	16x	613.53	168.31	0.30	133.93	148.77	12,218.42	100.0%
128	512	32x	1,226.65	328.87	0.17	158.39	181.37	12,388.53	100.0%
128	512	64x	2,000.29	589.84	0.11	150.04	202.66	13,506.85	100.0%
128	512	128x	3,478.11	953.38	0.07	150.51	219.90	15,339.96	100.0%
128	512	256x	4,304.94	1,190.98	0.05	2,206.10	3,406.94	26,662.42	100.0%
128	512	1024x	4,188.17	1,170.55	0.05	14,591.35	49,854.84	81,431.16	99.8%
128	1,024	1x	44.29	7.23	4.37	81.16	81.16	16,843.00	100.0%
128	1,024	2x	88.01	10.74	2.35	69.86	80.94	23,262.81	100.0%
128	1,024	4x	175.65	21.78	1.17	99.87	112.27	23,310.77	100.0%
128	1,024	8x	187.75	38.25	1.03	147.92	166.17	23,433.27	87.5%
128	2,048	1x	44.55	2.65	4.98	52.84	52.84	45,968.56	100.0%
128	2,048	2x	47.92	6.84	4.26	70.60	80.25	34,855.00	100.0%
128	2,048	4x	141.46	10.87	1.55	82.25	90.07	46,727.93	100.0%
128	2,048	8x	192.73	21.83	1.11	111.96	126.94	47,039.46	100.0%
128	2,048	16x	517.36	41.53	0.45	207.18	256.94	49,628.20	100.0%
128	2,048	32x	831.11	76.42	0.28	1,872.69	1,930.93	53,633.40	100.0%
128	2,048	64x	1,804.46	135.56	0.15	3,166.19	3,296.33	60,022.86	100.0%
128	2,048	128x	3,153.17	238.80	0.10	434.66	1,015.35	67,508.88	99.2%
128	2,048	256x	4,168.83	324.51	0.07	695.73	1,796.82	98,546.77	99.6%
128	2,048	512x	3,275.70	253.25	0.09	1,030.68	2,911.88	243,219.45	97.3%
512	128	1x	27.29	105.73	1.14	1,841.03	1,841.03	4,691.30	100.0%
512	128	2x	84.54	328.60	0.44	125.29	137.81	3,024.67	100.0%
512	128	4x	130.69	638.27	0.27	169.69	205.97	3,090.62	100.0%
512	128	8x	309.74	1,194.80	0.15	324.17	365.70	3,296.80	100.0%
512	128	16x	513.78	1,989.83	0.10	490.93	686.56	3,713.16	93.8%
512	128	32x	871.46	3,618.45	0.05	601.48	1,255.48	4,322.93	100.0%
512	128	64x	1,371.96	5,337.46	0.04	992.63	2,248.07	5,820.83	100.0%
512	128	128x	1,279.56	5,285.29	0.04	1,552.74	3,942.16	11,707.64	100.0%
512	512	1x	44.53	43.14	2.60	51.52	51.52	11,496.49	100.0%
512	512	2x	57.64	85.72	1.54	60.62	62.80	11,201.49	100.0%
512	512	4x	150.64	168.92	0.70	87.11	97.61	11,705.67	100.0%
512	512	8x	281.82	331.21	0.37	111.93	123.63	11,916.28	100.0%
512	512	16x	564.11	640.83	0.20	139.41	154.47	12,336.35	100.0%
512	512	32x	1,207.08	1,250.57	0.11	195.14	216.14	12,669.04	100.0%
512	512	64x	2,027.67	2,203.33	0.07	183.36	220.95	14,000.39	98.4%
512	512	128x	3,280.11	3,616.82	0.04	167.74	236.54	16,387.51	100.0%
512	512	256x	3,363.61	3,595.79	0.05	2,556.10	8,444.38	33,661.60	99.6%
512	512	512x	3,525.39	3,738.73	0.04	4,743.98	16,061.81	61,761.94	99.8%
512	512	1024x	3,064.30	3,280.21	0.05	44,466.42	103,102.34	150,797.04	99.7%
512	1,024	1x	43.60	61.26	2.07	116.65	116.65	8,072.31	100.0%
512	1,024	2x	87.39	42.46	1.76	124.19	125.54	23,432.59	100.0%
512	1,024	4x	139.61	83.98	1.05	171.44	205.21	23,541.60	100.0%
512	1,024	8x	299.35	163.75	0.52	327.10	369.45	24,112.35	100.0%
512	1,024	16x	594.20	313.06	0.27	438.19	643.26	25,317.18	100.0%
512	1,024	32x	1,159.26	598.82	0.15	585.89	1,233.75	26,559.08	100.0%
512	1,024	64x	1,762.74	1,050.77	0.11	928.06	2,268.27	30,145.46	100.0%
512	1,024	128x	2,915.04	1,644.47	0.07	1,440.71	4,228.85	38,012.32	99.2%
512	1,024	256x	3,554.22	2,001.52	0.06	2,695.96	10,761.65	61,577.78	99.6%
512	1,024	512x	2,870.08	1,615.53	0.07	5,023.90	19,014.26	147,933.91	99.4%
512	2,048	1x	44.28	10.73	4.21	118.39	118.39	46,245.10	100.0%
512	2,048	2x	87.40	21.23	2.16	106.60	140.04	46,856.96	100.0%
512	2,048	4x	132.11	42.09	1.35	180.40	216.77	46,991.79	100.0%
512	2,048	8x	275.00	82.16	0.67	300.69	371.78	48,057.92	100.0%
512	2,048	16x	511.24	156.74	0.37	419.66	612.90	50,588.87	100.0%
512	2,048	32x	1,041.27	298.19	0.21	586.13	1,248.96	53,440.16	100.0%
512	2,048	64x	1,750.94	503.81	0.13	1,337.95	3,652.75	62,089.30	98.4%
512	2,048	128x	2,768.79	870.62	0.09	1,514.77	4,338.41	72,688.35	100.0%
512	2,048	256x	3,396.65	1,058.94	0.07	2,557.09	8,492.04	114,589.12	100.0%
512	2,048	512x	2,699.66	833.68	0.09	7,431.07	16,795.97	254,305.72	85.0%
1,024	128	1x	40.59	465.62	0.40	185.54	185.54	2,070.89	100.0%
1,024	128	2x	43.19	644.25	0.30	183.03	184.09	2,888.85	100.0%
1,024	128	4x	121.69	1,204.25	0.16	280.14	350.32	3,237.91	100.0%
1,024	128	8x	283.66	2,169.81	0.10	571.54	648.56	3,599.26	100.0%
1,024	128	16x	471.95	3,641.86	0.06	713.23	1,196.15	4,285.64	100.0%
1,024	128	32x	712.91	5,603.96	0.04	946.23	2,329.34	5,493.34	100.0%
1,024	128	64x	922.09	7,386.54	0.04	1,697.68	4,372.09	8,209.11	98.4%
1,024	128	128x	1,132.20	9,154.83	0.03	2,622.74	8,712.84	13,295.58	100.0%
1,024	128	256x	1,208.82	9,723.98	0.03	4,885.67	16,515.44	24,465.96	100.0%
1,024	128	512x	1,214.92	9,762.92	0.03	9,628.29	34,026.05	45,020.51	99.4%
1,024	128	1024x	621.78	4,996.26	0.05	58,609.88	115,659.72	131,302.35	66.8%
1,024	512	1x	43.92	83.63	1.75	181.28	181.28	11,657.09	100.0%
1,024	512	2x	86.71	165.47	0.88	182.69	184.19	11,806.44	100.0%
1,024	512	4x	169.79	324.08	0.46	309.53	378.66	12,055.14	100.0%
1,024	512	8x	299.10	624.59	0.26	571.53	647.28	12,529.25	100.0%
1,024	512	16x	504.96	1,145.78	0.15	678.95	1,213.26	13,663.08	100.0%
1,024	512	32x	967.91	1,989.28	0.09	1,067.44	2,311.64	15,204.91	96.9%
1,024	512	64x	1,509.79	3,267.91	0.06	1,664.97	4,365.66	19,059.42	100.0%
1,024	512	128x	2,194.36	4,682.17	0.05	2,612.00	8,695.84	26,484.00	100.0%
1,024	512	256x	2,513.97	5,356.17	0.04	4,895.48	16,487.47	45,216.76	99.6%
1,024	512	512x	1,949.91	4,140.01	0.05	21,743.88	87,079.55	114,638.42	99.8%
1,024	1,024	1x	38.13	36.31	2.84	3,857.14	3,857.14	26,853.44	100.0%
1,024	1,024	2x	87.27	83.26	1.37	190.83	192.34	23,465.19	100.0%
1,024	1,024	4x	172.40	164.53	0.70	313.04	354.18	23,753.93	100.0%
1,024	1,024	8x	333.16	318.56	0.37	436.93	644.26	24,570.14	100.0%
1,024	1,024	16x	552.28	599.93	0.22	706.81	1,202.26	26,156.05	100.0%
1,024	1,024	32x	1,046.51	1,103.90	0.13	934.26	2,314.32	28,354.53	100.0%
1,024	1,024	64x	1,585.55	1,839.32	0.09	1,581.13	4,475.56	33,445.46	98.4%
1,024	1,024	128x	2,498.18	2,740.97	0.06	2,605.55	8,696.37	45,431.76	100.0%
1,024	1,024	256x	2,933.97	3,246.14	0.05	5,693.61	18,573.72	75,637.54	100.0%
1,024	1,024	512x	2,075.02	2,305.38	0.07	14,184.58	35,204.64	175,177.05	82.4%
1,024	2,048	1x	44.14	21.01	3.59	183.27	183.27	46,393.76	100.0%
1,024	2,048	2x	50.41	42.16	2.52	184.48	185.65	44,357.10	100.0%
1,024	2,048	4x	87.90	83.08	1.38	288.10	358.73	47,036.14	100.0%
1,024	2,048	8x	309.52	160.41	0.52	574.18	653.40	48,810.78	100.0%
1,024	2,048	16x	622.96	301.31	0.27	672.58	1,211.85	52,095.17	100.0%
1,024	2,048	32x	986.70	565.49	0.18	1,166.19	2,302.82	55,490.08	100.0%
1,024	2,048	64x	1,500.00	984.14	0.12	1,594.20	4,389.69	63,721.77	100.0%
1,024	2,048	128x	2,620.50	1,434.56	0.08	2,607.76	8,565.20	86,550.48	99.2%
1,024	2,048	256x	2,422.84	1,469.72	0.08	4,817.84	16,411.85	163,231.99	99.6%
2,048	128	1x	39.78	611.56	0.28	335.50	335.50	3,217.66	100.0%
2,048	128	2x	40.92	1,214.82	0.17	330.33	342.87	3,083.07	100.0%
2,048	128	4x	143.06	2,190.28	0.10	488.39	624.71	3,573.84	100.0%
2,048	128	8x	241.00	3,683.69	0.06	847.66	1,101.91	4,222.63	100.0%
2,048	128	16x	351.32	5,589.89	0.05	1,029.39	2,328.92	5,533.32	100.0%
2,048	128	32x	493.27	7,723.98	0.04	1,640.98	4,526.95	7,973.35	100.0%
2,048	128	64x	577.68	9,411.26	0.03	2,765.57	8,837.49	12,993.79	100.0%
2,048	128	128x	656.16	10,466.05	0.03	4,929.07	17,061.33	23,076.91	99.2%
2,048	128	256x	643.08	10,236.66	0.03	10,713.26	39,646.06	46,844.64	100.0%
2,048	512	1x	43.09	165.61	1.07	322.97	322.97	11,882.74	100.0%
2,048	512	2x	52.52	327.21	0.60	205.74	328.56	11,522.45	100.0%
2,048	512	4x	124.62	635.61	0.31	475.56	610.97	12,323.84	100.0%
2,048	512	8x	295.47	1,177.37	0.17	1,049.69	1,197.50	13,284.38	100.0%
2,048	512	16x	503.49	2,059.32	0.10	1,116.10	2,348.56	15,126.64	100.0%
2,048	512	32x	830.14	3,408.80	0.07	1,705.44	4,520.91	18,225.60	100.0%
2,048	512	64x	1,126.57	4,965.72	0.05	3,611.94	9,189.32	24,995.91	100.0%
2,048	512	128x	1,434.63	6,088.29	0.04	5,967.15	19,243.28	40,500.18	100.0%
2,048	512	256x	1,183.32	5,026.69	0.05	18,927.60	79,123.01	97,646.22	100.0%
2,048	1,024	1x	43.61	83.82	1.79	335.00	335.00	23,477.73	100.0%
2,048	1,024	2x	86.25	165.04	0.93	214.68	337.95	23,740.06	100.0%
2,048	1,024	4x	114.35	325.49	0.53	493.22	628.94	24,073.12	100.0%
2,048	1,024	8x	259.71	617.58	0.27	827.82	1,084.91	25,335.86	100.0%
2,048	1,024	16x	548.29	1,116.54	0.15	1,075.83	2,340.86	27,961.44	100.0%
2,048	1,024	32x	880.90	1,951.96	0.10	1,692.77	4,496.86	31,901.84	100.0%
2,048	1,024	64x	1,290.47	2,727.97	0.07	6,285.79	11,850.79	45,608.21	100.0%
2,048	1,024	128x	1,901.65	4,150.28	0.05	4,995.77	16,567.49	59,580.03	100.0%
2,048	1,024	256x	1,296.59	2,832.23	0.07	8,531.87	28,148.90	141,502.48	84.8%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H200 NVL

GPU Count2

GPU Memory (Total)280 GB

GPU Driver580.95.05

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)600 W

CPU ModelIntel(R) Xeon(R) 6960P

RAM2,267 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.11.0

OSUbuntu

OS Version22.04.5 LTS (Jammy Jellyfish)

Kernel Version5.15.0-88-generic

Python Version3.10.12

Model Configuration

Providermeta-llama

Model Namellama-2-70b-hf

QuantizationFP16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length4096

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.95

Temperature0.70

Top-P1.00

Top-K-1