NVIDIA H20 (8x) - gemma-3-27b-it

October 24, 2025 at 08:17 PM

Dataset: reference (v1.0)

NVIDIA NVIDIA H20 google gemma-3-27b-it

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

6,567.80

Peak generation speed

Best Input TPS

12,358.95

Peak prefill speed

Best Energy Efficiency

0.05 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

57.78 ms

Lowest latency

Best E2E (P95)

3,225.24 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	1,024	1024x	6,567.80★	839.93	0.12	33,343.71	52,848.89	99,852.68	100.0%
128	128	1x	22.87	21.80	6.51	2,648.81	2,648.81	5,595.31	100.0%
128	128	2x	44.26	43.22	3.94	2,727.59	2,809.68	5,771.60	100.0%
128	128	4x	87.51	86.82	2.34	1,543.84	2,874.55	5,816.02	100.0%
128	128	8x	168.39	168.89	1.42	1,698.38	3,092.40	6,066.69	100.0%
128	128	16x	514.70	518.47	0.60	870.24	986.74	3,970.64	100.0%
128	128	32x	824.48	825.68	0.38	1,438.64	1,766.58	4,950.66	100.0%
128	128	64x	1,402.74	1,396.23	0.23	2,204.80	2,612.27	5,783.81	100.0%
128	128	128x	1,880.39	1,880.05	0.17	4,144.27	5,176.60	8,493.23	100.0%
128	128	256x	1,690.99	1,694.03	0.17	12,862.70	14,493.81	18,760.80	100.0%
128	512	1x	42.65	10.16	6.44	89.49	89.49	12,004.81	100.0%
128	512	2x	85.22	20.81	3.93	82.16	82.44	12,012.27	100.0%
128	512	4x	168.70	41.85	2.36	153.16	167.96	12,132.94	100.0%
128	512	8x	328.08	82.50	1.57	295.53	359.77	12,434.13	100.0%
128	512	16x	628.42	160.37	1.10	549.11	594.92	12,857.50	100.0%
128	512	32x	1,195.01	300.16	0.59	934.65	1,056.28	13,640.72	100.0%
128	512	64x	2,078.71	518.01	0.32	1,708.86	2,098.44	15,662.52	100.0%
128	512	128x	3,277.04	823.07	0.21	4,572.59	5,459.99	19,526.46	100.0%
128	512	256x	4,185.95	1,049.52	0.15	11,763.77	13,888.30	30,396.39	100.0%
128	512	512x	4,044.33	1,016.71	0.14	25,749.26	30,569.31	52,883.80	100.0%
128	1,024	1x	42.74	5.09	7.40	58.05	58.05	23,959.06	100.0%
128	1,024	2x	83.88	10.24	3.70	142.91	153.29	24,412.53	100.0%
128	1,024	4x	168.32	20.88	3.09	191.87	195.48	24,324.08	100.0%
128	1,024	8x	311.96	41.91	1.88	293.95	324.46	24,502.14	100.0%
128	1,024	16x	615.00	80.53	1.22	501.53	577.80	25,606.21	100.0%
128	1,024	32x	1,174.96	150.93	0.66	975.49	1,161.92	27,162.73	100.0%
128	1,024	64x	2,135.83	272.54	0.37	1,775.62	2,169.48	29,830.49	100.0%
128	1,024	128x	3,623.01	463.05	0.22	4,372.01	5,377.03	34,902.70	100.0%
128	1,024	256x	4,723.77	603.67	0.16	13,792.27	15,386.17	53,457.94	100.0%
128	1,024	512x	4,742.70	608.10	0.14	29,754.08	32,810.70	82,127.94	100.0%
128	2,048	1x	42.60	4.31	7.59	93.79	93.79	28,283.46	100.0%
128	2,048	2x	75.21	6.57	5.08	134.99	136.72	37,616.21	100.0%
128	2,048	4x	154.13	12.95	3.03	206.35	248.73	39,032.55	100.0%
128	2,048	8x	269.35	24.83	1.79	319.93	391.95	41,109.30	100.0%
128	2,048	16x	471.11	42.51	1.52	504.88	589.91	44,439.88	100.0%
128	2,048	32x	992.31	86.62	0.81	955.00	1,050.79	46,643.20	100.0%
128	2,048	64x	1,789.52	146.96	0.45	1,728.65	2,156.05	53,583.57	100.0%
128	2,048	128x	3,004.04	253.23	0.27	4,342.28	5,493.84	58,612.63	100.0%
128	2,048	256x	4,518.29	381.83	0.18	14,575.17	16,468.38	79,820.33	100.0%
128	2,048	512x	4,576.16	383.61	0.16	31,121.42	33,142.80	134,301.05	100.0%
128	2,048	1024x	5,701.51	476.17	0.14	60,416.02	105,778.57	198,057.64	98.6%
512	128	1x	39.68	153.75	1.76	238.70	238.70	3,225.24	100.0%
512	128	2x	78.14	303.72	1.01	276.63	276.66	3,273.74	100.0%
512	128	4x	154.12	595.42	0.70	320.94	326.95	3,317.83	100.0%
512	128	8x	283.26	1,092.67	0.53	421.28	544.96	3,602.21	100.0%
512	128	16x	501.59	1,943.67	0.27	751.71	1,008.92	4,070.35	100.0%
512	128	32x	791.19	3,083.06	0.18	1,460.60	1,912.91	5,153.64	100.0%
512	128	64x	1,333.55	5,188.02	0.10	2,143.57	2,781.45	6,077.80	100.0%
512	128	128x	1,467.57	5,701.90	0.09	5,371.19	7,256.77	10,869.68	100.0%
512	128	256x	1,273.78	4,948.80	0.09	16,336.09	19,570.88	25,220.39	100.0%
512	512	1x	42.25	40.93	4.20	57.78	57.78	12,116.99	100.0%
512	512	2x	83.49	81.13	2.11	115.42	126.47	12,262.41	100.0%
512	512	4x	166.82	161.11	1.74	199.43	305.91	12,270.42	100.0%
512	512	8x	329.79	318.04	1.25	284.59	392.60	12,395.44	100.0%
512	512	16x	591.65	583.06	0.73	653.18	961.14	13,597.75	100.0%
512	512	32x	1,107.80	1,090.38	0.38	1,130.79	1,480.66	14,614.22	100.0%
512	512	64x	1,894.60	1,857.88	0.22	2,194.09	3,105.07	17,111.08	100.0%
512	512	128x	3,074.71	3,015.73	0.14	4,467.95	5,909.52	20,903.14	100.0%
512	512	256x	3,355.91	3,270.09	0.12	16,828.43	19,469.49	38,678.23	100.0%
512	512	512x	3,349.95	3,272.61	0.11	31,034.74	37,403.72	65,445.06	100.0%
512	1,024	1x	42.17	20.43	5.65	238.85	238.85	24,280.52	100.0%
512	1,024	2x	84.07	40.84	3.44	274.16	275.60	24,354.20	100.0%
512	1,024	4x	166.58	80.44	2.37	338.81	344.28	24,557.73	100.0%
512	1,024	8x	313.51	151.17	1.55	528.80	851.14	26,124.44	100.0%
512	1,024	16x	564.40	283.38	0.91	850.29	1,186.62	27,993.74	100.0%
512	1,024	32x	1,080.48	551.75	0.53	1,460.20	1,952.58	28,905.63	100.0%
512	1,024	64x	1,942.19	969.16	0.29	2,123.36	3,466.04	32,840.45	100.0%
512	1,024	128x	3,292.26	1,645.37	0.18	4,606.43	6,146.74	38,441.58	100.0%
512	1,024	256x	4,309.82	2,163.74	0.14	16,642.54	19,766.56	58,175.09	100.0%
512	1,024	512x	3,931.21	1,958.16	0.14	33,226.52	39,045.67	110,309.27	100.0%
512	2,048	1x	42.15	11.50	6.64	238.56	238.56	43,086.70	100.0%
512	2,048	2x	79.75	22.10	4.14	276.81	276.99	44,749.07	100.0%
512	2,048	4x	147.32	42.21	2.95	298.48	345.74	46,186.31	100.0%
512	2,048	8x	270.26	83.48	1.54	544.27	814.21	45,403.14	100.0%
512	2,048	16x	498.53	188.41	1.16	780.64	1,032.60	39,906.09	100.0%
512	2,048	32x	916.70	308.14	0.67	1,341.47	1,945.74	47,632.06	100.0%
512	2,048	64x	1,736.41	577.91	0.38	2,308.16	3,255.62	53,786.66	100.0%
512	2,048	128x	2,654.82	888.39	0.24	5,010.63	7,504.65	67,637.97	100.0%
512	2,048	256x	3,471.19	1,172.10	0.17	13,569.51	17,655.84	90,089.08	100.0%
512	2,048	512x	4,316.65	1,440.92	0.15	14,736.96	18,851.86	132,110.15	100.0%
512	2,048	1024x	4,298.00	1,442.70	0.15	92,247.54	162,998.75	250,955.63	83.4%
1,024	128	1x	37.44	285.17	1.04	435.06	435.06	3,417.08	100.0%
1,024	128	2x	74.14	565.88	0.61	460.72	466.63	3,449.63	100.0%
1,024	128	4x	127.74	975.30	0.40	540.87	873.08	4,002.81	100.0%
1,024	128	8x	249.51	1,908.63	0.29	571.81	878.60	4,090.86	100.0%
1,024	128	16x	354.57	2,720.05	0.18	1,466.08	2,383.67	5,765.75	100.0%
1,024	128	32x	632.68	4,854.34	0.12	2,080.47	3,051.79	6,461.02	100.0%
1,024	128	64x	861.68	6,616.60	0.09	3,486.36	5,794.65	9,460.51	100.0%
1,024	128	128x	1,099.52	8,444.00	0.06	6,969.19	9,853.25	14,846.10	100.0%
1,024	128	256x	1,059.90	8,139.70	0.07	17,903.82	23,073.08	30,396.23	100.0%
1,024	512	1x	42.41	80.75	2.86	58.46	58.46	12,072.56	100.0%
1,024	512	2x	80.93	154.43	1.78	285.30	454.10	12,627.19	100.0%
1,024	512	4x	161.26	307.80	1.19	337.83	524.01	12,691.77	100.0%
1,024	512	8x	304.76	582.81	0.79	531.67	871.83	13,426.24	100.0%
1,024	512	16x	571.55	1,096.14	0.46	979.04	1,563.12	14,320.97	100.0%
1,024	512	32x	1,003.30	1,956.00	0.28	1,621.79	2,222.90	16,034.20	100.0%
1,024	512	64x	1,623.58	3,140.80	0.16	2,802.68	5,186.00	19,975.43	100.0%
1,024	512	128x	2,383.55	4,622.99	0.11	6,660.24	10,008.31	27,054.97	100.0%
1,024	512	256x	2,715.60	5,241.88	0.09	17,223.41	22,821.04	47,320.97	100.0%
1,024	512	512x	2,580.91	5,000.55	0.10	31,681.69	52,429.10	91,006.32	100.0%
1,024	1,024	1x	41.78	39.78	4.35	412.68	412.68	24,505.20	100.0%
1,024	1,024	2x	82.63	78.84	2.65	452.75	453.97	24,771.54	100.0%
1,024	1,024	4x	166.00	158.42	1.82	436.99	531.29	24,671.98	100.0%
1,024	1,024	8x	303.79	290.48	1.08	787.24	1,399.16	26,958.83	100.0%
1,024	1,024	16x	564.52	556.93	0.75	1,089.52	1,599.71	28,201.20	100.0%
1,024	1,024	32x	1,055.02	1,045.41	0.41	1,790.85	2,629.61	30,026.79	100.0%
1,024	1,024	64x	1,817.77	1,796.38	0.24	3,340.55	5,196.15	34,954.60	100.0%
1,024	1,024	128x	2,909.76	2,895.77	0.15	6,651.46	10,074.43	43,388.95	100.0%
1,024	1,024	256x	2,791.06	2,747.54	0.13	13,834.30	20,144.40	75,059.32	100.0%
1,024	2,048	1x	41.87	25.87	5.26	430.41	430.41	37,661.30	100.0%
1,024	2,048	2x	74.91	44.67	3.48	459.98	468.32	43,275.88	100.0%
1,024	2,048	4x	142.35	87.07	2.13	532.54	540.72	44,152.84	100.0%
1,024	2,048	8x	261.61	158.05	1.34	639.67	1,010.43	49,345.97	100.0%
1,024	2,048	16x	474.16	294.88	0.89	939.09	1,584.01	51,871.79	100.0%
1,024	2,048	32x	894.39	591.00	0.56	2,263.55	3,784.22	52,008.95	100.0%
1,024	2,048	64x	1,569.62	1,054.48	0.33	2,971.52	4,169.52	58,514.26	100.0%
1,024	2,048	128x	2,652.53	1,817.66	0.20	6,881.84	9,403.66	65,816.72	100.0%
1,024	2,048	256x	3,365.94	2,246.89	0.15	16,835.62	21,948.63	99,797.55	100.0%
1,024	2,048	512x	3,544.39	2,401.40	0.14	37,818.74	45,696.57	186,457.84	100.0%
1,024	2,048	1024x	3,534.42	2,400.62	0.15	75,525.53	160,782.37	232,170.64	62.7%
2,048	128	1x	33.48	514.78	0.63	797.47	797.47	3,821.53	100.0%
2,048	128	2x	66.56	1,018.98	0.36	831.71	835.77	3,843.62	100.0%
2,048	128	4x	129.26	1,979.05	0.24	925.67	931.02	3,952.86	100.0%
2,048	128	8x	211.88	3,238.57	0.17	880.28	1,477.00	4,818.83	100.0%
2,048	128	16x	347.77	5,311.77	0.12	1,873.72	2,664.89	5,876.89	100.0%
2,048	128	32x	472.82	7,222.90	0.08	3,073.86	4,613.45	8,631.78	100.0%
2,048	128	64x	577.72	8,825.88	0.06	5,379.49	10,221.38	14,116.01	100.0%
2,048	128	128x	674.85	10,310.57	0.05	10,197.49	16,898.50	24,154.43	100.0%
2,048	128	256x	728.03	11,127.75	0.05	22,851.51	33,721.90	41,525.89	100.0%
2,048	128	512x	765.23	11,695.70	0.05	43,008.96	64,331.38	76,361.24	100.0%
2,048	128	1024x	806.99	12,358.95	0.05	79,075.22	125,243.56	138,040.48	98.5%
2,048	512	1x	39.83	153.10	1.82	803.76	803.76	12,853.38	100.0%
2,048	512	2x	79.09	302.70	1.10	844.18	849.09	12,944.28	100.0%
2,048	512	4x	156.50	599.04	0.75	705.60	892.15	13,058.63	100.0%
2,048	512	8x	251.69	1,024.28	0.51	1,388.78	2,526.02	15,276.20	100.0%
2,048	512	16x	478.31	1,913.56	0.32	1,943.74	3,396.32	16,337.89	100.0%
2,048	512	32x	838.11	3,282.55	0.16	3,676.29	4,558.24	19,044.63	100.0%
2,048	512	64x	1,390.23	5,407.73	0.12	5,144.70	7,739.11	23,107.46	100.0%
2,048	512	128x	2,016.44	7,810.54	0.08	9,573.85	14,924.83	31,891.25	100.0%
2,048	512	256x	1,739.11	6,781.52	0.08	22,932.77	55,794.81	71,496.03	100.0%
2,048	1,024	1x	40.96	78.73	2.96	801.50	801.50	24,996.36	100.0%
2,048	1,024	2x	81.14	155.28	1.81	833.06	839.30	25,225.00	100.0%
2,048	1,024	4x	141.82	312.55	1.20	717.09	904.21	25,006.37	100.0%
2,048	1,024	8x	255.49	588.60	0.79	1,013.79	1,521.83	26,252.13	100.0%
2,048	1,024	16x	420.95	1,011.74	0.46	2,555.48	4,100.70	30,908.40	100.0%
2,048	1,024	32x	863.63	1,890.34	0.29	3,278.36	4,859.40	33,077.15	100.0%
2,048	1,024	64x	1,546.72	3,222.30	0.17	5,015.83	8,041.09	38,779.76	100.0%
2,048	1,024	128x	2,353.14	4,762.83	0.11	10,088.50	16,531.08	52,445.33	100.0%
2,048	1,024	256x	2,142.53	4,360.37	0.11	24,260.11	75,321.79	106,380.18	100.0%
2,048	2,048	1x	41.05	67.16	3.28	802.55	802.55	29,277.27	100.0%
2,048	2,048	2x	81.78	129.60	2.03	836.11	839.78	30,206.30	100.0%
2,048	2,048	4x	122.53	224.57	1.30	902.86	921.12	34,151.92	100.0%
2,048	2,048	8x	204.46	440.26	0.85	1,186.78	1,759.37	34,895.57	100.0%
2,048	2,048	16x	342.69	623.09	0.69	2,464.84	4,437.35	47,483.55	100.0%
2,048	2,048	32x	690.95	1,100.61	0.42	3,250.73	5,549.96	52,235.86	100.0%
2,048	2,048	64x	1,371.40	2,084.53	0.24	5,203.41	7,694.88	56,055.43	100.0%
2,048	2,048	128x	2,356.93	3,345.18	0.15	9,401.44	14,961.40	69,836.73	100.0%
2,048	2,048	256x	2,238.10	3,255.86	0.14	24,786.64	87,727.47	134,611.01	100.0%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H20

GPU Count8

GPU Memory (Total)760 GB

GPU Driver570.195.03

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)500 W

CPU ModelIntel(R) Xeon(R) Platinum 8469C

RAM1,007 GB

Software Configuration

Inference FrameworkvLLM

Framework Versionv0.9.0

OSUbuntu

OS Version24.04.3 LTS (Noble Numbat)

Kernel Version6.8.0-79-generic

Python Version3.12.3

Model Configuration

Providergoogle

Model Namegemma-3-27b-it

QuantizationBF16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length8192

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization90.00

Temperature0.70

Top-P1.00

Top-K-1