NVIDIA H20 (8x) - llama-3.3-70b-instruct

October 24, 2025 at 10:44 PM

Dataset: reference (v1.0)

NVIDIA NVIDIA H20 meta-llama llama-3.3-70b-instruct

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

3,370.98

Peak generation speed

Best Input TPS

6,350.24

Peak prefill speed

Best Energy Efficiency

0.11 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

40.07 ms

Lowest latency

Best E2E (P95)

3,620.39 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	512	1024x	3,370.98★	888.16	0.25	55,518.94	109,926.10	126,853.82	92.7%
128	128	1x	32.04	30.54	6.29	645.39	645.39	3,995.19	100.0%
128	128	2x	63.78	62.28	4.29	655.98	662.87	4,012.42	100.0%
128	128	4x	127.17	126.18	2.26	292.21	609.61	3,963.44	100.0%
128	128	8x	277.13	277.94	1.33	194.05	297.19	3,692.72	100.0%
128	128	16x	510.98	514.72	0.85	353.83	578.61	4,001.40	100.0%
128	128	32x	968.78	970.20	0.46	455.21	652.65	4,215.20	100.0%
128	128	64x	1,588.83	1,581.46	0.28	747.25	1,345.91	5,123.60	100.0%
128	128	128x	1,701.64	1,706.32	0.22	1,502.80	3,806.45	7,865.77	100.0%
128	128	256x	2,125.27	2,132.87	0.20	4,018.02	11,041.55	14,655.89	100.0%
128	128	512x	2,503.42	2,525.59	0.18	8,716.23	17,930.88	22,190.62	100.0%
128	128	1024x	2,457.56	2,470.81	0.18	19,636.55	39,741.03	44,767.40	100.0%
128	512	1x	37.75	9.00	10.12	77.51	77.51	13,563.35	100.0%
128	512	2x	75.41	18.41	7.07	90.70	95.41	13,577.77	100.0%
128	512	4x	150.09	37.23	4.59	47.39	64.30	13,638.21	100.0%
128	512	8x	277.02	74.15	2.99	105.77	185.36	13,848.25	100.0%
128	512	16x	562.64	146.35	1.50	201.83	309.69	14,083.53	100.0%
128	512	32x	1,002.92	265.81	0.81	394.96	808.46	15,413.59	100.0%
128	512	64x	1,904.88	500.09	0.43	546.62	877.37	16,271.36	100.0%
128	512	128x	2,020.72	523.61	0.36	1,510.16	3,492.45	19,498.75	100.0%
128	512	256x	2,937.99	778.13	0.26	9,230.70	19,247.97	35,569.16	100.0%
128	512	512x	3,177.72	840.98	0.26	26,149.58	55,326.21	71,462.93	100.0%
128	1,024	1x	37.60	6.27	10.98	85.33	85.33	19,417.22	100.0%
128	1,024	2x	73.20	16.14	7.38	81.87	90.31	15,424.91	100.0%
128	1,024	4x	115.98	21.45	5.55	115.00	164.12	22,942.75	100.0%
128	1,024	8x	234.32	42.60	3.27	116.86	188.03	23,883.09	100.0%
128	1,024	16x	428.89	76.47	1.94	247.01	390.46	25,269.06	100.0%
128	1,024	32x	830.87	140.72	1.06	429.32	531.03	27,791.71	100.0%
128	1,024	64x	1,524.04	264.17	0.58	670.69	1,051.03	29,793.40	100.0%
128	1,024	128x	1,846.67	313.18	0.41	3,061.64	20,348.42	46,110.33	100.0%
128	1,024	256x	2,529.34	439.68	0.33	13,551.11	34,474.37	61,043.95	100.0%
128	1,024	512x	2,953.22	513.99	0.29	38,562.13	87,480.11	111,869.22	100.0%
128	1,024	1024x	3,216.00	559.02	0.28	54,127.00	114,775.21	139,005.68	64.4%
128	2,048	1x	37.68	5.98	10.83	78.78	78.78	20,383.18	100.0%
128	2,048	2x	63.61	11.67	8.10	89.51	90.51	21,068.54	100.0%
128	2,048	4x	124.71	21.05	5.59	105.07	172.29	23,812.41	100.0%
128	2,048	8x	213.92	38.14	2.53	259.53	311.33	25,352.71	100.0%
128	2,048	16x	422.85	71.89	1.90	278.65	470.15	27,057.06	100.0%
128	2,048	32x	821.18	146.91	1.08	429.62	652.26	26,892.78	100.0%
128	2,048	64x	1,161.83	194.94	0.73	702.32	1,149.91	33,834.57	100.0%
128	2,048	128x	1,978.32	333.10	0.41	1,809.11	4,480.83	37,667.60	100.0%
128	2,048	256x	2,643.18	461.12	0.32	13,835.22	36,729.25	60,784.45	100.0%
128	2,048	512x	2,725.94	470.80	0.30	39,119.53	87,719.97	114,427.63	100.0%
128	2,048	1024x	3,024.26	522.37	0.30	53,516.26	112,798.49	137,881.32	64.1%
512	128	1x	35.36	137.02	2.67	288.79	288.79	3,620.39	100.0%
512	128	2x	70.56	274.26	1.70	288.62	289.07	3,627.23	100.0%
512	128	4x	138.94	536.77	1.08	174.92	318.18	3,679.12	100.0%
512	128	8x	243.52	939.36	0.74	462.98	822.79	4,202.03	100.0%
512	128	16x	452.10	1,751.88	0.40	783.38	1,084.53	4,523.46	100.0%
512	128	32x	709.02	2,762.85	0.27	1,357.63	2,072.85	5,760.35	100.0%
512	128	64x	991.60	3,878.54	0.17	2,828.17	4,292.61	8,192.03	100.0%
512	128	128x	1,006.60	3,925.99	0.15	5,195.98	11,902.95	15,380.16	100.0%
512	128	256x	1,270.39	4,937.29	0.13	9,025.75	16,817.62	22,159.26	100.0%
512	128	512x	1,315.02	5,131.22	0.13	18,755.74	36,738.78	44,592.51	100.0%
512	128	1024x	1,146.79	4,467.06	0.14	45,758.12	91,587.96	97,997.33	100.0%
512	512	1x	37.15	35.99	6.49	285.58	285.58	13,781.07	100.0%
512	512	2x	72.20	71.69	3.21	172.81	289.43	13,847.60	100.0%
512	512	4x	145.02	140.07	2.90	355.85	528.36	14,116.36	100.0%
512	512	8x	275.20	277.10	1.83	368.16	578.67	14,239.66	100.0%
512	512	16x	474.68	523.86	1.02	752.62	1,336.56	15,141.44	100.0%
512	512	32x	898.03	962.43	0.54	1,531.25	2,384.94	16,562.79	100.0%
512	512	64x	1,527.11	1,552.74	0.31	2,967.93	4,315.29	20,502.20	100.0%
512	512	128x	1,559.60	1,629.41	0.26	5,376.47	16,065.15	31,836.52	100.0%
512	512	256x	1,930.72	1,994.11	0.22	15,425.30	29,546.32	49,162.70	100.0%
512	512	512x	2,442.70	2,476.93	0.20	38,251.58	77,935.80	95,352.75	100.0%
512	512	1024x	2,424.45	2,468.27	0.20	56,625.59	108,305.63	127,284.62	66.1%
512	1,024	1x	37.08	35.30	6.52	286.49	286.49	14,024.59	100.0%
512	1,024	2x	59.05	52.88	5.20	164.54	277.26	18,388.59	100.0%
512	1,024	4x	128.95	87.47	3.70	358.72	524.81	22,202.23	100.0%
512	1,024	8x	216.65	157.14	1.78	990.19	1,352.48	24,914.45	100.0%
512	1,024	16x	397.52	324.46	1.35	876.06	1,571.08	22,085.27	100.0%
512	1,024	32x	745.73	561.95	0.77	1,496.25	2,093.06	27,128.41	100.0%
512	1,024	64x	1,363.96	942.65	0.45	2,846.65	4,082.57	31,783.78	100.0%
512	1,024	128x	2,231.74	1,539.18	0.27	4,771.45	6,796.26	39,179.48	100.0%
512	1,024	256x	2,346.57	1,602.71	0.25	18,835.69	43,445.02	72,241.25	100.0%
512	1,024	512x	2,480.89	1,690.25	0.24	50,370.47	109,282.33	134,873.42	98.2%
512	1,024	1024x	2,531.13	1,726.47	0.24	53,992.26	112,515.55	140,087.07	50.8%
512	2,048	1x	37.03	36.59	6.53	292.89	292.89	13,529.64	100.0%
512	2,048	2x	58.99	52.83	4.25	441.43	564.62	18,455.10	100.0%
512	2,048	4x	122.06	83.08	3.93	171.14	286.25	23,268.02	100.0%
512	2,048	8x	212.44	171.14	2.55	503.58	850.64	23,040.88	100.0%
512	2,048	16x	400.94	309.40	1.41	872.75	1,606.08	24,883.00	100.0%
512	2,048	32x	661.64	466.45	0.86	1,700.50	2,618.60	32,235.41	100.0%
512	2,048	64x	909.07	603.47	0.57	2,851.08	4,353.57	35,803.04	100.0%
512	2,048	128x	1,423.92	964.62	0.37	5,284.05	7,081.05	43,020.78	100.0%
512	2,048	256x	2,220.19	1,454.70	0.27	19,007.66	48,226.96	76,763.07	100.0%
512	2,048	512x	2,507.55	1,685.17	0.24	51,802.48	110,453.56	137,639.18	100.0%
512	2,048	1024x	2,515.23	1,695.44	0.24	54,393.26	114,036.42	141,199.01	50.5%
1,024	128	1x	32.77	249.62	1.54	541.84	541.84	3,906.02	100.0%
1,024	128	2x	65.34	498.72	1.04	547.14	550.50	3,916.87	100.0%
1,024	128	4x	114.80	876.46	0.74	677.74	999.27	4,455.22	100.0%
1,024	128	8x	204.60	1,565.03	0.41	945.52	1,587.34	4,999.89	100.0%
1,024	128	16x	336.29	2,579.80	0.27	1,420.30	2,611.14	6,082.75	100.0%
1,024	128	32x	516.00	3,991.24	0.18	2,520.95	4,079.25	7,854.14	100.0%
1,024	128	64x	709.45	5,447.65	0.13	4,268.29	7,505.46	11,500.44	100.0%
1,024	128	128x	685.26	5,288.06	0.12	7,698.25	16,823.76	20,309.04	100.0%
1,024	512	1x	37.67	71.73	4.33	40.07	40.07	13,591.73	100.0%
1,024	512	2x	75.27	143.63	3.03	40.21	41.15	13,601.13	100.0%
1,024	512	4x	149.21	284.79	1.98	50.78	68.27	13,719.78	100.0%
1,024	512	8x	287.07	551.54	1.02	255.31	575.68	14,197.58	100.0%
1,024	512	16x	541.61	1,045.10	0.64	496.37	1,065.29	15,026.26	100.0%
1,024	512	32x	950.57	1,823.33	0.36	1,156.79	2,071.11	17,213.68	100.0%
1,024	512	64x	1,561.08	3,104.99	0.21	2,456.02	4,138.93	20,233.15	100.0%
1,024	512	128x	1,391.03	2,785.50	0.19	7,093.19	24,973.30	42,755.56	100.0%
1,024	1,024	1x	36.93	39.52	6.15	547.07	547.07	24,642.86	100.0%
1,024	1,024	2x	72.90	77.37	4.43	475.92	537.25	25,188.60	100.0%
1,024	1,024	4x	124.18	143.79	3.47	268.22	521.23	26,514.78	100.0%
1,024	1,024	8x	248.82	317.06	1.79	374.12	576.87	24,451.64	100.0%
1,024	1,024	16x	413.06	524.33	1.07	893.03	2,110.64	29,953.88	100.0%
1,024	1,024	32x	736.70	992.77	0.58	1,759.96	2,542.07	29,453.49	100.0%
1,024	1,024	64x	1,253.59	1,708.28	0.34	3,951.41	5,575.11	33,857.86	100.0%
1,024	1,024	128x	1,504.16	2,075.41	0.24	7,608.60	23,808.11	47,487.74	100.0%
1,024	1,024	256x	1,739.98	2,381.37	0.22	25,477.41	57,432.75	89,605.42	100.0%
1,024	1,024	512x	1,982.52	2,716.27	0.21	53,847.21	113,239.73	141,004.24	81.6%
1,024	2,048	1x	36.81	45.55	5.73	542.35	542.35	21,380.27	100.0%
1,024	2,048	2x	67.73	86.79	4.12	543.94	546.90	22,308.79	100.0%
1,024	2,048	4x	137.17	164.68	2.13	441.14	582.16	23,536.73	100.0%
1,024	2,048	8x	212.44	251.86	1.77	1,191.36	1,586.00	30,252.11	100.0%
1,024	2,048	16x	363.52	465.62	1.11	1,391.61	2,089.17	29,055.18	100.0%
1,024	2,048	32x	441.89	536.15	0.82	2,799.92	4,051.69	33,131.56	100.0%
1,024	2,048	64x	988.63	1,331.81	0.39	4,807.29	8,086.91	36,174.92	100.0%
1,024	2,048	128x	1,608.33	2,211.24	0.24	7,620.72	13,204.27	47,811.40	100.0%
1,024	2,048	256x	1,761.72	2,380.54	0.22	26,324.93	59,716.14	91,265.53	100.0%
1,024	2,048	512x	1,984.75	2,643.29	0.21	52,953.44	112,229.76	140,618.72	78.9%
2,048	128	1x	28.96	445.25	0.90	1,053.24	1,053.24	4,420.01	100.0%
2,048	128	2x	57.81	885.05	0.63	1,057.48	1,060.70	4,426.99	100.0%
2,048	128	4x	115.52	1,768.73	0.48	1,051.24	1,059.38	4,431.03	100.0%
2,048	128	8x	155.88	2,382.71	0.30	1,584.27	3,119.11	6,564.51	100.0%
2,048	128	16x	236.95	3,619.23	0.20	2,682.75	4,478.12	8,616.31	100.0%
2,048	128	32x	339.33	5,183.66	0.14	4,515.22	8,075.24	12,046.74	100.0%
2,048	128	64x	375.12	5,778.78	0.11	7,226.10	16,586.76	21,590.49	100.0%
2,048	128	128x	413.43	6,350.24	0.11	13,914.48	26,554.80	36,025.11	100.0%
2,048	128	256x	357.21	5,461.05	0.12	32,192.70	74,805.70	91,424.87	100.0%
2,048	512	1x	35.01	134.57	2.70	1,060.26	1,060.26	14,624.52	100.0%
2,048	512	2x	70.00	267.89	1.92	1,060.71	1,061.25	14,626.30	100.0%
2,048	512	4x	105.15	464.64	1.10	2,072.49	3,084.30	16,865.73	100.0%
2,048	512	8x	220.39	915.00	0.79	1,687.26	3,061.77	17,103.29	100.0%
2,048	512	16x	374.78	1,612.59	0.47	3,316.78	5,124.15	19,367.17	100.0%
2,048	512	32x	518.80	2,136.95	0.30	5,949.58	13,095.16	29,256.39	100.0%
2,048	512	64x	825.93	3,298.83	0.19	6,999.60	18,636.21	37,866.38	100.0%
2,048	512	128x	918.24	3,622.52	0.16	14,947.53	38,036.37	59,474.00	100.0%
2,048	512	256x	1,086.20	4,342.90	0.15	39,813.97	78,340.69	107,484.92	100.0%
2,048	512	512x	1,113.80	4,420.97	0.16	55,641.09	114,750.46	137,237.39	61.9%
2,048	1,024	1x	35.21	118.04	3.06	1,050.64	1,050.64	16,645.82	100.0%
2,048	1,024	2x	64.61	175.35	2.61	1,065.56	1,070.87	22,108.15	100.0%
2,048	1,024	4x	107.64	275.57	1.90	553.69	1,072.22	27,538.80	100.0%
2,048	1,024	8x	191.86	574.30	1.20	1,824.01	3,076.97	26,096.41	100.0%
2,048	1,024	16x	283.79	958.04	0.70	3,466.94	5,531.31	29,224.01	100.0%
2,048	1,024	32x	527.58	1,540.39	0.44	5,154.62	10,145.51	38,741.41	100.0%
2,048	1,024	64x	895.01	2,723.87	0.27	7,900.85	13,733.76	44,357.33	100.0%
2,048	1,024	128x	1,059.25	2,953.35	0.20	13,763.94	44,792.62	76,494.56	100.0%
2,048	1,024	256x	1,354.32	3,787.48	0.18	42,272.01	91,134.05	122,340.13	100.0%
2,048	1,024	512x	1,322.18	3,726.25	0.18	54,536.92	112,584.20	143,982.56	56.8%
2,048	2,048	1x	35.76	93.21	3.70	1,053.66	1,053.66	21,086.37	100.0%
2,048	2,048	2x	65.67	219.40	2.20	1,065.52	1,068.92	17,712.98	100.0%
2,048	2,048	4x	70.41	141.63	3.10	1,049.65	1,066.88	49,831.98	100.0%
2,048	2,048	8x	181.62	658.95	1.02	1,703.60	3,096.60	23,285.41	100.0%
2,048	2,048	16x	299.32	877.74	0.70	3,820.95	6,093.39	33,485.99	100.0%
2,048	2,048	32x	335.91	949.90	0.59	5,316.71	9,092.17	35,963.94	100.0%
2,048	2,048	64x	921.79	2,677.89	0.27	7,496.30	14,154.75	44,086.72	100.0%
2,048	2,048	128x	1,099.00	3,109.87	0.22	14,283.69	39,067.24	70,100.14	100.0%
2,048	2,048	256x	1,227.23	3,400.06	0.19	44,171.46	98,211.26	133,375.05	100.0%
2,048	2,048	512x	1,323.67	3,659.88	0.18	53,419.33	111,541.69	142,419.14	55.9%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H20

GPU Count8

GPU Memory (Total)760 GB

GPU Driver570.195.03

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)500 W

CPU ModelIntel(R) Xeon(R) Platinum 8469C

RAM1,007 GB

Software Configuration

Inference FrameworkvLLM

Framework Versionv0.9.0

OSUbuntu

OS Version24.04.3 LTS (Noble Numbat)

Kernel Version6.8.0-79-generic

Python Version3.12.3

Model Configuration

Providermeta-llama

Model Namellama-3.3-70b-instruct

QuantizationBF16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model LengthUnknown

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory UtilizationUnknown

Temperature0.70

Top-P1.00

Top-K-1