NVIDIA H100 80GB HBM3 (8x) - llama-3.3-70b-instruct

November 12, 2025 at 10:51 PM

Dataset: reference (v1.0)

NVIDIA NVIDIA H100 80GB HBM3 meta-llama llama-3.3-70b-instruct

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

9,219.60

Peak generation speed

Best Input TPS

16,108.82

Peak prefill speed

Best Energy Efficiency

0.06 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

55.04 ms

Lowest latency

Best E2E (P95)

3,854.59 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	128	512x	9,219.60★	9,292.79	0.07	877.51	1,495.92	6,630.67	100.0%
128	128	1x	27.14	25.87	7.34	616.82	616.82	4,716.43	100.0%
128	128	2x	53.38	52.13	3.73	662.87	683.50	4,777.17	100.0%
128	128	4x	104.83	104.01	3.04	359.70	676.35	4,830.10	100.0%
128	128	8x	219.08	219.73	1.54	207.89	457.97	4,484.58	100.0%
128	128	16x	503.57	507.25	0.90	133.26	168.88	4,057.14	100.0%
128	128	32x	967.17	975.97	0.46	227.44	277.78	4,183.64	100.0%
128	128	64x	1,814.70	1,816.04	0.26	305.07	464.36	4,448.13	100.0%
128	128	128x	3,370.36	3,394.61	0.16	389.14	573.84	4,721.16	100.0%
128	128	256x	5,760.75	5,787.55	0.10	548.14	985.32	5,462.03	100.0%
128	128	1024x	8,299.69	8,339.44	0.08	1,541.56	2,938.63	13,544.64	100.0%
128	512	1x	33.67	8.02	10.89	56.86	56.86	15,204.75	100.0%
128	512	2x	66.46	16.23	5.41	79.80	94.63	15,403.82	100.0%
128	512	4x	131.86	32.71	4.89	68.11	95.04	15,507.73	100.0%
128	512	8x	259.28	65.14	3.03	88.18	118.35	15,761.19	100.0%
128	512	16x	521.50	131.43	1.52	123.20	153.86	15,679.69	100.0%
128	512	32x	958.12	254.51	0.84	191.22	221.23	16,103.55	100.0%
128	512	64x	1,897.37	494.90	0.44	282.86	382.69	16,432.84	100.0%
128	512	128x	3,604.06	950.33	0.25	385.16	562.00	17,129.41	100.0%
128	512	256x	6,557.52	1,733.12	0.15	525.88	858.75	18,707.25	100.0%
128	512	512x	7,117.51	1,891.83	0.14	911.75	1,594.85	32,498.21	100.0%
128	512	1024x	8,033.10	2,120.05	0.14	1,673.59	2,731.47	58,919.98	100.0%
128	1,024	1x	33.65	5.44	11.92	57.09	57.09	22,409.64	100.0%
128	1,024	2x	60.12	11.89	8.28	54.15	56.32	20,771.70	100.0%
128	1,024	4x	119.01	20.51	4.50	78.25	96.11	24,723.66	100.0%
128	1,024	8x	234.00	45.39	3.31	95.63	136.01	22,462.01	100.0%
128	1,024	16x	398.14	69.63	2.07	125.71	160.59	27,358.96	100.0%
128	1,024	32x	766.56	128.95	1.11	195.23	263.77	31,386.62	100.0%
128	1,024	64x	1,484.12	251.63	0.61	269.80	366.70	31,761.61	100.0%
128	1,024	128x	2,839.98	477.59	0.33	401.68	605.51	32,988.90	100.0%
128	1,024	256x	5,168.25	900.26	0.20	550.47	965.92	35,482.04	100.0%
128	1,024	512x	5,873.54	1,023.72	0.18	912.37	1,582.81	56,209.75	100.0%
128	1,024	1024x	6,142.03	1,071.42	0.18	1,819.77	2,946.21	105,200.93	100.0%
128	2,048	1x	33.01	5.76	11.71	57.28	57.28	21,146.92	100.0%
128	2,048	2x	57.09	11.89	8.42	53.09	55.04	20,688.16	100.0%
128	2,048	4x	109.27	18.62	5.87	71.59	93.80	26,876.94	100.0%
128	2,048	8x	217.08	39.78	3.48	84.92	114.34	25,684.39	100.0%
128	2,048	16x	428.52	78.50	1.98	130.40	189.85	26,141.75	100.0%
128	2,048	32x	649.58	115.24	1.20	219.03	280.78	27,926.28	100.0%
128	2,048	64x	1,184.07	205.71	0.69	270.08	395.36	31,065.79	100.0%
128	2,048	128x	2,144.93	360.72	0.40	388.44	607.65	34,699.60	100.0%
128	2,048	256x	4,209.53	716.70	0.23	598.54	1,082.84	38,548.04	100.0%
128	2,048	512x	5,061.13	876.03	0.19	924.69	1,575.21	59,001.50	100.0%
128	2,048	1024x	5,740.01	985.74	0.18	1,852.73	3,259.69	116,175.17	100.0%
512	128	1x	33.21	128.70	2.60	87.54	87.54	3,854.59	100.0%
512	128	2x	66.20	257.31	1.76	87.30	87.89	3,865.73	100.0%
512	128	4x	125.40	484.45	0.94	132.98	175.50	4,082.32	100.0%
512	128	8x	246.57	951.12	0.61	157.68	239.36	4,146.05	100.0%
512	128	16x	468.92	1,847.73	0.39	258.22	370.18	4,282.66	100.0%
512	128	32x	893.35	3,481.13	0.22	450.13	639.18	4,575.47	100.0%
512	128	64x	1,486.63	5,834.86	0.15	691.30	1,249.83	5,406.51	100.0%
512	128	128x	2,699.87	10,619.95	0.08	1,139.22	1,671.73	5,913.08	100.0%
512	128	256x	4,124.76	16,108.82	0.06	1,806.47	2,999.87	7,767.85	100.0%
512	128	512x	3,695.08	14,438.36	0.07	5,543.14	12,386.17	17,147.01	100.0%
512	512	1x	33.25	32.21	6.98	87.80	87.80	15,398.42	100.0%
512	512	2x	64.84	65.10	4.77	85.87	86.22	15,234.47	100.0%
512	512	4x	131.78	127.28	3.12	97.97	148.31	15,533.45	100.0%
512	512	8x	228.85	249.72	2.06	146.13	227.08	15,815.47	100.0%
512	512	16x	436.50	485.71	1.08	327.03	523.52	16,329.11	100.0%
512	512	32x	920.37	952.04	0.55	394.88	620.11	16,751.81	100.0%
512	512	64x	1,830.20	1,829.40	0.30	706.26	978.00	17,388.41	100.0%
512	512	128x	3,154.72	3,227.99	0.17	1,138.33	2,014.96	19,639.35	100.0%
512	512	256x	3,547.97	3,642.47	0.16	2,121.71	3,369.52	32,171.43	100.0%
512	512	512x	3,957.52	4,036.88	0.15	11,265.80	27,968.21	56,547.27	100.0%
512	512	1024x	4,368.09	4,466.75	0.14	36,409.04	75,454.53	97,371.69	100.0%
512	1,024	1x	33.50	36.36	6.47	84.96	84.96	13,613.44	100.0%
512	1,024	2x	61.49	48.87	5.57	84.35	87.46	20,159.65	100.0%
512	1,024	4x	115.49	85.14	4.57	85.87	91.69	22,986.75	100.0%
512	1,024	8x	180.16	141.76	2.57	174.81	289.49	26,393.68	100.0%
512	1,024	16x	378.93	292.84	1.37	232.93	321.11	26,221.72	100.0%
512	1,024	32x	677.80	502.65	0.82	491.17	706.77	29,774.20	100.0%
512	1,024	64x	1,362.22	939.59	0.46	676.97	942.19	32,942.25	100.0%
512	1,024	128x	2,621.90	1,796.42	0.25	1,036.12	1,726.87	35,094.53	100.0%
512	1,024	256x	3,258.46	2,202.29	0.21	2,146.77	3,166.82	48,113.29	100.0%
512	1,024	512x	3,681.22	2,515.75	0.19	17,543.14	61,200.32	90,110.80	100.0%
512	1,024	1024x	4,090.53	2,831.21	0.18	50,666.06	113,233.62	143,406.27	89.9%
512	2,048	1x	33.41	28.23	7.34	88.01	88.01	17,540.96	100.0%
512	2,048	2x	49.83	36.35	6.33	68.49	82.94	26,635.25	100.0%
512	2,048	4x	110.57	87.07	3.13	141.73	159.36	22,301.59	100.0%
512	2,048	8x	217.61	139.29	2.67	169.66	294.24	28,110.73	100.0%
512	2,048	16x	394.90	329.54	1.38	229.27	369.06	23,801.13	100.0%
512	2,048	32x	558.72	417.36	0.87	502.08	863.70	29,760.30	100.0%
512	2,048	64x	1,187.43	806.57	0.50	590.49	821.87	34,909.41	100.0%
512	2,048	128x	1,466.45	982.91	0.35	1,066.64	1,625.71	37,676.56	100.0%
512	2,048	256x	2,680.31	1,807.81	0.23	2,268.83	3,533.18	50,096.30	100.0%
512	2,048	512x	3,662.02	2,443.88	0.20	18,751.30	64,216.14	90,261.76	100.0%
512	2,048	1024x	3,849.73	2,597.82	0.19	49,273.89	112,644.43	142,744.40	85.9%
1,024	128	1x	31.91	243.08	1.53	149.10	149.10	4,011.11	100.0%
1,024	128	2x	61.76	471.41	0.79	215.30	272.36	4,142.45	100.0%
1,024	128	4x	123.14	940.12	0.69	135.44	262.27	4,150.01	100.0%
1,024	128	8x	240.09	1,836.58	0.43	247.13	422.80	4,259.39	100.0%
1,024	128	16x	432.62	3,318.76	0.25	435.71	756.94	4,730.09	100.0%
1,024	128	32x	782.98	6,025.12	0.15	794.82	1,155.47	5,204.63	100.0%
1,024	128	64x	1,401.78	10,763.86	0.09	1,320.81	1,695.90	5,826.27	100.0%
1,024	128	128x	2,032.34	15,712.29	0.07	1,985.68	3,402.58	7,942.36	100.0%
1,024	128	256x	1,930.60	14,850.81	0.07	5,387.39	12,384.11	16,714.51	100.0%
1,024	512	1x	33.44	63.69	4.72	142.18	142.18	15,308.66	100.0%
1,024	512	2x	64.99	124.02	2.37	214.86	269.84	15,754.21	100.0%
1,024	512	4x	128.91	246.05	2.17	156.18	260.04	15,880.51	100.0%
1,024	512	8x	251.01	480.02	1.12	262.74	446.44	16,312.54	100.0%
1,024	512	16x	488.09	940.33	0.71	488.38	762.51	16,698.61	100.0%
1,024	512	32x	916.24	1,802.94	0.39	791.99	1,135.09	17,417.22	100.0%
1,024	512	64x	1,670.65	3,369.80	0.22	1,506.05	1,952.10	18,632.41	100.0%
1,024	512	128x	2,043.73	4,060.93	0.16	2,151.41	3,706.60	23,699.74	100.0%
1,024	512	256x	2,119.26	4,219.93	0.16	10,483.08	27,542.21	46,053.44	100.0%
1,024	512	512x	2,735.23	5,452.98	0.14	30,823.95	71,882.56	91,419.17	100.0%
1,024	512	1024x	2,778.54	5,558.16	0.15	58,739.89	119,341.46	139,692.73	85.9%
1,024	1,024	1x	33.29	40.22	6.19	146.36	146.36	24,210.11	100.0%
1,024	1,024	2x	60.88	72.14	3.39	123.98	179.14	26,851.59	100.0%
1,024	1,024	4x	120.04	145.41	2.39	171.56	264.86	26,727.35	100.0%
1,024	1,024	8x	203.74	245.68	2.05	277.35	402.53	31,239.74	100.0%
1,024	1,024	16x	378.02	492.29	1.09	458.91	659.60	31,467.14	100.0%
1,024	1,024	32x	692.66	940.76	0.58	722.39	1,153.06	29,436.83	100.0%
1,024	1,024	64x	1,266.40	1,810.66	0.33	1,221.16	1,937.84	33,018.07	100.0%
1,024	1,024	128x	2,391.21	3,300.04	0.20	2,119.49	3,183.60	36,360.26	100.0%
1,024	1,024	256x	2,107.71	2,925.52	0.21	13,060.96	38,618.61	69,371.86	100.0%
1,024	2,048	1x	32.77	39.01	6.36	148.27	148.27	24,962.72	100.0%
1,024	2,048	2x	63.32	78.76	4.41	143.78	144.57	24,669.58	100.0%
1,024	2,048	4x	117.39	133.51	3.20	142.34	187.14	28,972.86	100.0%
1,024	2,048	8x	218.18	286.74	1.58	266.15	393.42	26,890.91	100.0%
1,024	2,048	16x	346.41	421.53	1.17	406.71	566.17	32,859.39	100.0%
1,024	2,048	32x	529.59	683.95	0.71	843.90	1,119.40	31,280.26	100.0%
1,024	2,048	64x	991.44	1,339.75	0.40	1,273.87	1,803.55	32,436.35	100.0%
1,024	2,048	128x	1,925.68	2,696.44	0.22	2,078.98	3,493.84	36,707.03	100.0%
1,024	2,048	256x	1,996.04	2,701.90	0.23	13,147.31	38,455.19	69,071.08	100.0%
1,024	2,048	512x	2,643.33	3,612.81	0.19	43,171.09	96,704.53	124,074.76	100.0%
1,024	2,048	1024x	2,631.57	3,575.98	0.19	51,157.90	114,053.56	139,858.27	56.2%
2,048	128	1x	31.64	486.41	0.89	268.91	268.91	4,046.12	100.0%
2,048	128	2x	62.85	962.19	0.57	164.69	263.69	4,058.78	100.0%
2,048	128	4x	116.18	1,778.76	0.40	384.52	511.44	4,406.34	100.0%
2,048	128	8x	208.43	3,185.83	0.23	661.23	993.34	4,910.94	100.0%
2,048	128	16x	417.53	6,377.37	0.14	682.57	986.43	4,897.04	100.0%
2,048	128	32x	656.73	10,032.39	0.09	1,495.55	2,171.23	6,228.75	100.0%
2,048	128	64x	972.33	14,927.36	0.07	2,342.56	4,078.58	8,334.44	100.0%
2,048	128	128x	1,031.97	15,815.01	0.07	5,221.40	11,241.14	15,699.58	100.0%
2,048	128	256x	966.14	14,786.52	0.08	12,942.20	24,070.79	29,738.16	100.0%
2,048	512	1x	32.89	126.42	2.95	255.03	255.03	15,567.00	100.0%
2,048	512	2x	64.20	245.72	1.48	384.77	491.64	15,945.90	100.0%
2,048	512	4x	120.27	471.63	1.07	462.52	756.91	16,613.38	100.0%
2,048	512	8x	226.75	936.18	0.69	537.15	962.94	16,712.60	100.0%
2,048	512	16x	423.39	1,822.48	0.45	735.34	1,257.35	17,154.96	100.0%
2,048	512	32x	795.58	3,348.42	0.25	1,639.07	2,178.59	18,668.58	100.0%
2,048	512	64x	1,466.30	5,957.30	0.15	2,217.64	3,587.15	20,949.30	100.0%
2,048	512	128x	1,472.14	5,904.89	0.16	9,143.17	24,688.80	42,272.93	100.0%
2,048	512	256x	1,451.84	5,768.09	0.16	27,386.80	67,944.56	82,943.81	100.0%
2,048	1,024	1x	32.80	77.50	4.13	258.26	258.26	25,362.65	100.0%
2,048	1,024	2x	65.98	230.04	2.13	158.32	252.09	17,003.82	100.0%
2,048	1,024	4x	87.84	252.04	2.27	258.37	262.28	29,652.61	100.0%
2,048	1,024	8x	194.75	592.36	1.00	410.15	552.35	25,560.66	100.0%
2,048	1,024	16x	349.72	1,234.84	0.62	964.77	1,462.04	24,796.90	100.0%
2,048	1,024	32x	634.20	1,837.70	0.41	1,581.93	2,134.66	32,918.56	100.0%
2,048	1,024	64x	910.44	2,663.69	0.27	2,274.11	3,581.50	37,518.96	100.0%
2,048	1,024	128x	1,252.15	3,484.94	0.22	12,027.46	36,494.57	62,568.39	100.0%
2,048	1,024	256x	1,467.12	4,175.27	0.20	34,903.34	79,090.46	104,346.81	100.0%
2,048	1,024	512x	1,643.52	4,584.44	0.19	52,857.39	111,931.54	136,031.33	67.8%
2,048	2,048	1x	32.89	95.91	3.54	259.10	259.10	20,489.26	100.0%
2,048	2,048	2x	64.22	215.29	2.28	158.53	251.41	18,123.76	100.0%
2,048	2,048	4x	65.62	126.32	3.35	259.40	263.78	56,044.10	100.0%
2,048	2,048	8x	174.63	585.91	0.77	894.97	1,258.19	25,336.93	100.0%
2,048	2,048	16x	296.65	1,007.21	0.68	661.13	1,221.09	28,544.45	100.0%
2,048	2,048	32x	629.96	1,946.19	0.39	1,503.38	2,166.41	31,317.08	100.0%
2,048	2,048	64x	952.25	2,745.08	0.27	2,225.22	4,245.41	34,600.00	100.0%
2,048	2,048	128x	1,227.16	3,281.21	0.24	11,320.31	32,361.08	62,137.38	100.0%
2,048	2,048	256x	1,253.61	3,475.10	0.23	34,637.15	78,950.27	104,836.75	100.0%
2,048	2,048	512x	1,533.63	4,182.23	0.20	52,995.04	114,087.13	139,697.81	67.8%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H100 80GB HBM3

GPU Count8

GPU Memory (Total)632 GB

GPU Driver570.195.03

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)700 W

CPU ModelIntel(R) Xeon(R) Platinum 8480+

RAM1,772 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.11.0

OSUbuntu

OS Version22.04.5 LTS (Jammy Jellyfish)

Kernel Version5.15.0-88-generic

Python Version3.10.12

Model Configuration

Providermeta-llama

Model Namellama-3.3-70b-instruct

QuantizationFP16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length8192

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.95

Temperature0.70

Top-P1.00

Top-K-1