NVIDIA H200 NVL (2x) - llama-2-70b-hf (50% Max Batch Token)

November 6, 2025 at 05:32 AM

Dataset: reference (v1.0)

NVIDIA NVIDIA H200 NVL meta-llama llama-2-70b-hf

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

4,620.81

Peak generation speed

Best Input TPS

8,844.22

Peak prefill speed

Best Energy Efficiency

0.03 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

51.53 ms

Lowest latency

Best E2E (P95)

417.20 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	512	1024x	4,620.81★	1,285.70	0.05	13,890.51	45,498.05	75,536.60	99.7%
128	128	1x	40.70	38.79	2.06	249.17	249.17	3,144.66	100.0%
128	128	2x	84.58	85.27	1.21	68.94	79.71	2,921.35	100.0%
128	128	4x	132.08	167.33	0.65	98.40	112.04	3,026.15	100.0%
128	128	8x	290.96	330.54	0.32	158.48	181.50	3,098.11	100.0%
128	128	16x	429.53	432.68	0.21	1,713.49	1,756.24	4,744.26	100.0%
128	128	32x	1,062.73	1,102.41	0.10	335.31	509.94	3,505.37	96.9%
128	128	64x	1,873.37	1,929.94	0.06	365.14	622.47	4,079.11	100.0%
128	128	128x	2,806.30	3,015.84	0.05	461.99	1,103.72	5,017.62	100.0%
128	128	256x	2,730.77	3,254.06	0.05	2,215.28	3,906.33	9,279.74	100.0%
128	512	1x	42.79	53.82	1.56	57.54	57.54	2,243.45	100.0%
128	512	2x	53.62	81.25	1.67	63.57	75.82	2,938.26	100.0%
128	512	4x	130.33	42.88	1.31	91.88	101.89	11,841.40	100.0%
128	512	8x	262.65	79.45	0.63	112.31	129.96	12,908.75	100.0%
128	512	16x	460.65	136.19	0.34	159.64	177.10	15,109.66	100.0%
128	512	32x	1,232.45	321.47	0.16	161.22	192.47	12,688.76	100.0%
128	512	64x	2,094.19	570.37	0.10	159.15	224.91	13,812.23	98.4%
128	512	128x	3,606.33	956.10	0.07	153.98	237.20	15,618.61	100.0%
128	512	256x	4,385.66	1,197.76	0.06	545.95	2,438.67	22,224.11	99.6%
128	512	512x	4,586.66	1,263.56	0.05	2,147.84	4,978.44	44,935.51	99.6%
128	1,024	1x	42.73	5.18	4.70	56.32	56.32	23,542.34	100.0%
128	1,024	2x	63.24	10.70	3.11	71.62	73.65	22,697.16	100.0%
128	1,024	4x	138.51	21.43	1.44	108.94	124.79	23,698.66	100.0%
128	1,024	8x	171.09	43.36	1.10	176.07	199.06	23,661.75	100.0%
128	1,024	16x	572.52	82.73	0.37	205.08	248.19	24,895.21	100.0%
128	1,024	32x	1,059.84	160.32	0.22	283.33	425.72	25,503.85	100.0%
128	1,024	64x	2,076.91	288.09	0.12	339.40	566.79	28,165.67	100.0%
128	1,024	128x	3,290.92	499.37	0.08	464.35	1,073.86	32,070.36	99.2%
128	1,024	256x	4,418.04	645.46	0.06	729.18	1,796.83	48,636.28	99.6%
128	1,024	512x	4,462.24	657.55	0.06	1,524.86	3,810.41	92,176.35	99.6%
128	1,024	1024x	3,445.74	507.68	0.07	8,253.12	50,705.05	148,280.87	61.9%
128	2,048	1x	43.54	2.59	5.02	75.11	75.11	47,039.28	100.0%
128	2,048	2x	49.18	5.32	4.24	82.34	93.87	44,914.89	100.0%
128	2,048	4x	125.37	10.77	1.72	95.40	108.35	47,180.83	100.0%
128	2,048	8x	341.78	21.42	0.66	166.10	183.24	47,921.86	100.0%
128	2,048	16x	507.51	41.08	0.45	267.32	300.71	50,194.55	100.0%
128	2,048	32x	1,049.47	79.21	0.24	246.18	404.91	51,690.45	100.0%
128	2,048	64x	1,502.65	137.33	0.17	572.38	736.04	59,289.11	100.0%
128	2,048	128x	3,175.12	239.08	0.10	527.09	1,067.38	68,093.49	100.0%
128	2,048	256x	4,069.91	326.50	0.07	3,921.75	4,456.26	99,645.97	99.6%
128	2,048	512x	3,412.85	270.33	0.09	2,408.95	4,874.55	237,316.31	99.6%
512	128	1x	42.11	163.16	0.87	122.34	122.34	3,040.15	100.0%
512	128	2x	84.57	328.71	0.46	134.54	135.92	3,022.41	100.0%
512	128	4x	162.54	627.94	0.26	186.28	210.92	3,147.19	100.0%
512	128	8x	307.23	1,185.12	0.15	333.18	375.53	3,323.56	100.0%
512	128	16x	494.28	2,112.32	0.09	477.20	665.09	3,721.80	100.0%
512	128	32x	920.24	3,585.94	0.05	639.93	1,269.74	4,370.13	100.0%
512	128	64x	1,327.72	5,309.90	0.04	1,019.72	2,223.40	5,873.71	100.0%
512	128	128x	1,703.84	7,070.54	0.03	1,862.52	4,495.59	8,663.99	99.2%
512	128	256x	1,925.04	7,784.95	0.03	2,968.33	8,556.92	15,231.40	99.6%
512	128	512x	1,924.50	7,897.10	0.03	5,269.31	16,052.70	26,054.14	99.4%
512	512	1x	43.35	52.83	2.39	60.55	60.55	9,364.70	100.0%
512	512	2x	87.49	85.01	1.28	70.46	83.18	11,701.14	100.0%
512	512	4x	171.40	165.54	0.66	83.86	91.57	11,941.06	100.0%
512	512	8x	309.53	329.11	0.36	116.45	130.34	11,992.62	100.0%
512	512	16x	613.37	593.96	0.20	166.61	192.61	12,498.51	93.8%
512	512	32x	1,057.27	1,233.56	0.11	179.87	198.46	12,854.04	100.0%
512	512	64x	1,528.89	1,747.26	0.07	4,259.35	4,367.72	18,181.65	100.0%
512	512	128x	3,372.60	3,663.02	0.05	188.10	281.42	16,684.77	100.0%
512	512	256x	3,334.69	3,555.79	0.05	2,994.70	8,656.51	34,082.19	100.0%
512	1,024	1x	43.26	70.35	1.92	61.79	61.79	7,026.31	100.0%
512	1,024	2x	42.14	40.83	2.66	863.62	906.76	23,198.44	100.0%
512	2,048	1x	43.46	10.52	4.31	51.53	51.53	47,126.23	100.0%
512	2,048	2x	51.70	68.86	1.90	73.49	83.37	13,838.71	100.0%
512	2,048	4x	100.27	41.75	1.65	87.47	95.13	44,550.29	100.0%
512	2,048	8x	321.17	81.98	0.60	113.17	127.38	48,166.78	100.0%
512	2,048	16x	488.09	158.81	0.39	142.52	160.70	49,933.75	100.0%
512	2,048	32x	991.49	299.97	0.21	805.26	861.23	53,165.25	100.0%
512	2,048	64x	1,835.62	540.92	0.13	182.63	247.64	58,736.83	100.0%
512	2,048	128x	2,857.34	851.80	0.09	1,797.36	3,975.00	74,388.41	100.0%
512	2,048	256x	2,982.35	862.35	0.08	2,967.73	8,685.29	141,902.96	100.0%
512	2,048	512x	2,618.11	803.74	0.09	11,511.43	21,940.83	254,397.10	81.8%
1,024	128	1x	9.69	497.45	0.12	1,519.79	1,519.79	1,936.12	100.0%
1,024	128	2x	82.74	631.54	0.28	133.74	194.72	3,085.61	100.0%
1,024	128	4x	152.93	1,167.56	0.16	298.57	369.67	3,339.51	100.0%
1,024	128	8x	202.10	1,680.54	0.11	1,602.00	1,684.22	4,651.79	100.0%
1,024	128	16x	271.02	2,198.27	0.07	2,309.51	4,002.79	7,111.53	100.0%
1,024	128	32x	445.74	3,745.32	0.05	1,904.33	4,833.67	8,042.69	96.9%
1,024	128	64x	830.59	6,837.55	0.04	2,187.68	5,153.33	8,906.10	98.4%
1,024	128	128x	857.53	6,828.21	0.03	4,397.35	10,507.61	17,954.48	99.2%
1,024	128	256x	1,032.84	8,288.91	0.03	10,138.35	21,984.10	29,981.81	99.6%
1,024	128	512x	1,092.93	8,844.22	0.03	14,013.45	43,277.18	52,334.98	99.8%
1,024	128	1024x	685.89	5,606.02	0.05	60,078.96	113,340.03	128,184.25	73.5%
1,024	512	1x	31.02	1,440.18	0.05	183.14	183.14	652.55	100.0%
1,024	512	2x	85.93	163.97	0.90	128.13	188.75	11,909.97	100.0%
1,024	512	4x	156.54	320.38	0.48	291.51	367.23	12,188.99	100.0%
1,024	512	8x	321.52	622.46	0.26	477.74	601.26	12,560.76	100.0%
1,024	512	16x	590.29	1,149.05	0.14	798.79	1,208.26	13,638.44	100.0%
1,024	512	32x	948.33	2,060.79	0.09	1,206.60	2,346.03	15,166.26	100.0%
1,024	512	64x	1,505.77	3,301.70	0.06	1,829.48	4,394.43	18,881.47	100.0%
1,024	512	128x	2,097.30	4,384.79	0.05	3,303.03	9,726.76	28,008.01	99.2%
1,024	512	256x	2,446.92	5,275.40	0.04	5,439.27	18,069.73	46,143.28	100.0%
1,024	512	512x	1,825.26	4,024.35	0.05	24,749.71	93,250.52	119,667.35	99.8%
1,024	1,024	1x	43.08	41.02	2.75	188.65	188.65	23,769.04	100.0%
1,024	1,024	2x	62.24	82.34	1.60	145.88	207.36	23,067.85	100.0%
1,024	1,024	4x	168.14	160.47	0.71	296.28	370.58	24,355.57	100.0%
1,024	1,024	8x	314.69	300.90	0.38	1,786.46	1,863.89	26,015.40	100.0%
1,024	1,024	16x	458.83	526.07	0.23	1,345.14	4,549.04	29,825.89	100.0%
1,024	1,024	32x	933.03	998.82	0.13	3,475.68	5,066.08	31,399.46	100.0%
1,024	1,024	64x	1,419.52	1,772.64	0.09	2,193.38	6,274.14	35,307.02	100.0%
1,024	1,024	128x	2,373.51	2,529.94	0.06	6,542.23	12,501.92	49,579.72	100.0%
1,024	1,024	256x	2,608.18	3,006.57	0.05	10,830.80	24,957.29	81,464.75	100.0%
1,024	1,024	512x	1,978.49	2,256.15	0.07	14,339.67	35,081.08	161,952.43	75.4%
1,024	2,048	1x	42.70	20.33	3.61	978.17	978.17	47,956.82	100.0%
1,024	2,048	2x	87.49	41.74	1.81	132.12	194.24	46,812.05	100.0%
1,024	2,048	4x	170.36	81.29	0.93	327.71	371.53	48,079.55	100.0%
1,024	2,048	8x	267.68	161.02	0.56	478.01	604.04	48,617.19	100.0%
1,024	2,048	16x	595.18	302.06	0.29	687.61	1,214.41	51,970.53	100.0%
1,024	2,048	32x	936.98	559.04	0.18	1,341.59	2,388.55	56,127.24	100.0%
1,024	2,048	64x	1,507.27	913.95	0.12	5,965.82	6,993.19	67,506.85	98.4%
1,024	2,048	128x	2,487.84	1,493.64	0.08	3,249.22	9,338.93	83,896.30	100.0%
1,024	2,048	256x	2,239.68	1,301.36	0.09	5,993.57	19,325.88	183,930.16	99.6%
2,048	128	1x	39.35	604.98	0.29	325.10	325.10	3,252.94	100.0%
2,048	128	2x	29.06	882.86	0.19	1,397.48	1,529.51	4,292.67	100.0%
2,048	512	1x	43.46	167.06	1.10	66.43	66.43	11,779.43	100.0%
2,048	512	2x	59.76	332.65	0.57	82.63	92.13	11,406.95	100.0%
2,048	512	4x	163.03	624.02	0.29	367.43	633.56	12,552.53	100.0%
2,048	512	8x	307.39	1,174.63	0.16	1,063.43	1,213.83	13,315.82	100.0%
2,048	512	16x	510.75	2,068.44	0.10	1,135.89	2,330.28	15,072.88	100.0%
2,048	512	32x	790.78	3,332.60	0.07	2,092.53	4,577.24	18,089.91	96.9%
2,048	512	64x	1,168.50	4,812.39	0.05	3,425.18	9,691.27	25,821.04	100.0%
2,048	512	128x	1,330.64	5,654.14	0.04	11,200.16	23,041.55	44,109.18	100.0%
2,048	512	256x	1,151.44	4,949.83	0.05	20,999.01	79,174.78	99,405.53	100.0%
2,048	1,024	1x	9.07	4,462.59	0.05	347.24	347.24	417.20	100.0%
2,048	1,024	2x	85.97	164.50	0.92	203.37	325.81	23,816.30	100.0%
2,048	1,024	4x	146.63	280.62	0.49	3,667.37	3,813.33	27,924.54	100.0%
2,048	1,024	8x	262.00	612.24	0.28	837.25	1,093.74	25,539.01	100.0%
2,048	1,024	16x	453.08	1,088.72	0.16	2,541.60	2,700.34	28,715.24	100.0%
2,048	1,024	32x	766.32	1,872.46	0.10	4,596.24	5,711.80	33,371.68	100.0%
2,048	1,024	64x	1,211.88	2,868.20	0.07	4,116.15	11,827.31	43,470.76	100.0%
2,048	1,024	128x	1,749.03	4,195.99	0.05	6,169.51	18,528.04	59,352.02	100.0%
2,048	1,024	256x	1,329.91	2,981.27	0.07	22,969.04	113,885.97	152,654.48	91.8%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H200 NVL

GPU Count2

GPU Memory (Total)280 GB

GPU Driver580.95.05

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)600 W

CPU ModelIntel(R) Xeon(R) 6960P

RAM2,267 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.11.0

OSUbuntu

OS Version22.04.5 LTS (Jammy Jellyfish)

Kernel Version5.15.0-88-generic

Python Version3.10.12

Model Configuration

Providermeta-llama

Model Namellama-2-70b-hf

QuantizationFP16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length4096

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.93

Temperature0.70

Top-P1.00

Top-K-1