NVIDIA H20 (8x) - mistral-nemo-instruct-2407

October 24, 2025 at 08:17 PM

Dataset: reference (v1.0)

NVIDIA NVIDIA H20 mistralai mistral-nemo-instruct-2407

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

12,605.43

Peak generation speed

Best Input TPS

24,969.58

Peak prefill speed

Best Energy Efficiency

0.02 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

44.44 ms

Lowest latency

Best E2E (P95)

1,330.68 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	1,024	1024x	12,605.43★	2,255.57	0.06	5,663.82	15,323.96	30,680.05	100.0%
128	128	1x	76.05	72.49	1.97	476.52	476.52	1,682.07	100.0%
128	128	2x	145.04	141.64	1.16	515.63	522.43	1,761.89	100.0%
128	128	4x	280.39	278.20	0.72	364.89	604.45	1,821.30	100.0%
128	128	8x	495.88	497.34	0.44	571.20	838.81	2,059.54	100.0%
128	128	16x	883.90	890.38	0.30	764.87	1,081.14	2,303.50	100.0%
128	128	32x	1,661.66	1,664.10	0.16	1,103.32	1,194.92	2,447.41	100.0%
128	128	64x	2,248.69	2,243.19	0.11	2,153.55	2,280.06	3,606.49	100.0%
128	128	128x	2,597.74	2,611.45	0.10	3,991.91	4,814.86	6,144.33	100.0%
128	128	256x	1,486.92	1,519.70	0.13	18,120.79	19,759.20	21,359.29	100.0%
128	512	1x	101.34	29.79	3.20	44.44	44.44	4,083.88	100.0%
128	512	2x	202.73	49.50	1.61	89.54	95.61	5,045.34	100.0%
128	512	4x	401.10	99.49	0.95	151.20	158.45	5,096.22	100.0%
128	512	8x	738.67	195.62	0.66	224.71	248.69	5,241.44	100.0%
128	512	16x	1,407.45	375.09	0.38	386.96	414.10	5,490.12	100.0%
128	512	32x	2,673.36	698.21	0.24	717.42	783.69	5,852.77	100.0%
128	512	64x	4,260.48	1,127.96	0.14	1,494.65	1,850.38	7,113.07	100.0%
128	512	128x	5,519.18	1,437.64	0.10	4,698.28	5,623.92	11,105.32	100.0%
128	512	256x	4,652.77	1,215.59	0.10	17,453.06	19,319.69	26,323.45	100.0%
128	1,024	1x	101.73	26.41	2.65	44.77	44.77	4,609.01	100.0%
128	1,024	2x	191.94	47.51	1.68	88.12	88.50	5,219.68	100.0%
128	1,024	4x	342.85	64.72	1.18	143.24	150.64	7,704.35	100.0%
128	1,024	8x	601.55	124.53	0.90	210.90	221.89	8,056.73	100.0%
128	1,024	16x	1,040.35	197.25	0.60	387.06	408.54	10,337.69	100.0%
128	1,024	32x	2,116.10	373.52	0.31	715.05	788.78	10,858.11	100.0%
128	1,024	64x	3,897.05	681.94	0.19	1,388.11	1,719.43	11,747.92	100.0%
128	1,024	128x	5,779.18	1,045.65	0.12	3,335.11	4,447.87	14,125.07	100.0%
128	1,024	256x	5,917.12	1,046.81	0.09	16,238.71	17,924.21	29,967.42	100.0%
128	1,024	512x	6,480.02	1,163.97	0.08	31,407.68	36,744.78	52,144.58	100.0%
128	2,048	1x	101.85	25.41	2.79	47.16	47.16	4,789.39	100.0%
128	2,048	2x	173.84	39.15	1.90	83.39	84.14	6,276.32	100.0%
128	2,048	4x	324.53	57.10	1.08	149.44	158.36	8,580.82	100.0%
128	2,048	8x	536.44	124.76	0.89	214.03	239.74	7,915.82	100.0%
128	2,048	16x	1,021.96	199.56	0.56	373.96	408.85	9,226.13	100.0%
128	2,048	32x	1,582.85	276.98	0.38	728.97	776.97	12,185.49	100.0%
128	2,048	64x	2,849.63	489.67	0.23	1,505.52	1,822.57	14,120.19	100.0%
128	2,048	128x	4,904.05	844.50	0.13	3,734.74	4,802.63	15,885.77	100.0%
128	2,048	256x	5,621.32	980.44	0.11	14,073.72	15,594.29	28,150.37	100.0%
128	2,048	512x	6,575.94	1,126.80	0.09	24,447.39	29,030.37	46,270.42	100.0%
128	2,048	1024x	7,250.91	1,248.40	0.07	52,370.13	60,272.14	84,894.29	96.3%
512	128	1x	96.17	372.65	0.69	127.48	127.48	1,330.68	100.0%
512	128	2x	188.24	731.62	0.40	150.54	153.54	1,358.62	100.0%
512	128	4x	321.00	1,240.13	0.23	297.94	353.90	1,591.35	100.0%
512	128	8x	637.61	2,459.53	0.18	308.83	379.84	1,602.60	100.0%
512	128	16x	970.62	3,761.14	0.11	573.72	814.49	2,099.30	100.0%
512	128	32x	1,757.94	6,850.21	0.07	868.18	1,020.72	2,313.10	100.0%
512	128	64x	2,253.88	8,833.15	0.05	1,609.96	2,107.40	3,564.95	100.0%
512	128	128x	2,506.45	9,775.18	0.05	3,538.07	4,757.71	6,342.01	100.0%
512	128	256x	1,606.54	6,286.50	0.06	15,622.16	17,698.49	19,716.35	100.0%
512	512	1x	102.77	99.56	1.69	122.87	122.87	4,981.21	100.0%
512	512	2x	207.50	201.62	1.02	70.83	70.94	4,932.66	100.0%
512	512	4x	405.79	391.92	0.61	130.77	138.30	5,042.57	100.0%
512	512	8x	752.60	746.55	0.41	262.25	341.98	5,282.13	100.0%
512	512	16x	1,305.64	1,393.99	0.28	441.38	567.69	5,686.95	100.0%
512	512	32x	2,397.51	2,482.27	0.17	826.38	1,127.31	6,414.35	100.0%
512	512	64x	4,103.62	4,138.42	0.10	1,499.68	2,003.27	7,661.27	100.0%
512	512	128x	5,871.79	5,953.05	0.07	3,518.15	4,686.84	10,409.66	100.0%
512	512	256x	4,921.80	4,982.70	0.06	16,034.65	17,865.09	25,209.23	100.0%
512	1,024	1x	103.72	91.87	1.89	46.44	46.44	5,388.42	100.0%
512	1,024	2x	159.92	100.96	1.53	86.38	87.05	9,625.56	100.0%
512	1,024	4x	300.47	198.38	0.94	146.12	187.98	9,584.33	100.0%
512	1,024	8x	647.39	484.96	0.58	229.59	290.89	8,099.89	100.0%
512	1,024	16x	1,022.96	781.87	0.42	403.89	526.89	8,655.77	100.0%
512	1,024	32x	1,950.93	1,466.73	0.23	798.82	948.51	10,699.79	100.0%
512	1,024	64x	3,487.21	2,440.28	0.13	1,426.06	1,844.17	12,058.17	100.0%
512	1,024	128x	5,662.23	4,013.62	0.08	3,363.97	4,350.17	14,773.99	100.0%
512	1,024	256x	6,393.94	4,375.90	0.07	10,944.61	14,090.35	26,969.68	100.0%
512	1,024	512x	6,548.69	4,659.79	0.06	29,400.26	33,898.64	51,069.54	100.0%
512	1,024	1024x	6,754.84	4,820.89	0.05	53,641.88	62,628.37	89,414.98	92.2%
512	2,048	1x	103.72	98.55	1.74	48.10	48.10	5,022.45	100.0%
512	2,048	2x	185.20	172.86	1.13	88.95	89.29	5,683.99	100.0%
512	2,048	4x	357.13	270.55	0.86	124.84	125.49	7,226.79	100.0%
512	2,048	8x	612.21	417.37	0.53	200.16	210.19	9,216.68	100.0%
512	2,048	16x	1,080.78	854.80	0.38	369.46	444.58	8,552.28	100.0%
512	2,048	32x	1,647.22	1,249.59	0.24	726.76	868.08	9,678.93	100.0%
512	2,048	64x	2,823.85	2,015.05	0.15	1,345.91	1,700.15	11,684.53	100.0%
512	2,048	128x	3,507.20	2,386.09	0.11	3,700.98	4,666.88	15,643.16	100.0%
512	2,048	256x	4,994.69	3,428.53	0.08	14,759.54	16,350.30	29,649.59	100.0%
512	2,048	512x	5,942.79	4,066.28	0.07	30,249.73	34,296.57	52,984.26	100.0%
512	2,048	1024x	6,958.75	4,792.16	0.06	42,249.62	56,817.70	85,157.60	100.0%
1,024	128	1x	89.82	684.21	0.42	209.22	209.22	1,423.82	100.0%
1,024	128	2x	175.58	1,340.19	0.24	242.41	242.63	1,455.86	100.0%
1,024	128	4x	340.20	2,597.34	0.15	286.06	288.24	1,501.57	100.0%
1,024	128	8x	531.95	4,069.09	0.11	470.53	679.34	1,921.25	100.0%
1,024	128	16x	974.77	7,477.87	0.07	611.56	845.00	2,089.52	100.0%
1,024	128	32x	1,379.25	10,585.05	0.06	1,166.16	1,598.51	2,943.27	100.0%
1,024	128	64x	1,879.42	14,447.40	0.04	1,994.64	2,761.60	4,308.12	100.0%
1,024	128	128x	2,047.80	15,745.71	0.03	3,989.41	5,628.36	7,907.41	100.0%
1,024	128	256x	1,415.60	10,920.28	0.04	16,699.56	19,653.77	22,437.13	100.0%
1,024	512	1x	100.47	191.33	1.19	205.35	205.35	5,094.69	100.0%
1,024	512	2x	199.14	380.01	0.72	159.25	230.99	5,133.42	100.0%
1,024	512	4x	383.59	732.16	0.38	221.49	312.52	5,335.89	100.0%
1,024	512	8x	762.61	1,458.39	0.31	255.60	385.95	5,363.47	100.0%
1,024	512	16x	1,345.42	2,612.84	0.19	548.38	835.16	6,002.20	100.0%
1,024	512	32x	2,244.71	4,432.58	0.11	1,032.98	1,581.08	7,061.29	100.0%
1,024	512	64x	3,840.88	7,531.61	0.07	1,570.87	2,161.66	8,306.20	100.0%
1,024	512	128x	5,005.09	10,002.70	0.05	3,929.81	5,569.49	12,287.02	100.0%
1,024	512	256x	4,531.02	9,051.72	0.05	16,019.51	18,396.27	27,140.95	100.0%
1,024	1,024	1x	103.82	145.65	1.44	49.41	49.41	6,683.30	100.0%
1,024	1,024	2x	167.41	197.17	1.12	68.84	69.18	9,719.38	100.0%
1,024	1,024	4x	303.18	400.51	0.65	247.94	353.43	9,368.98	100.0%
1,024	1,024	8x	593.00	818.07	0.41	275.34	475.01	9,330.51	100.0%
1,024	1,024	16x	1,122.12	1,554.77	0.26	375.23	515.99	10,074.12	100.0%
1,024	1,024	32x	1,905.42	2,661.05	0.17	805.42	1,027.57	10,402.99	100.0%
1,024	1,024	64x	3,409.72	4,889.55	0.09	1,621.24	2,297.79	12,297.09	100.0%
1,024	1,024	128x	5,135.23	7,286.96	0.07	3,687.85	5,393.79	16,214.06	100.0%
1,024	1,024	256x	5,308.24	7,240.60	0.06	16,348.43	18,508.48	32,461.80	100.0%
1,024	1,024	512x	5,861.32	8,069.45	0.05	29,646.65	34,511.73	57,006.53	100.0%
1,024	1,024	1024x	6,333.15	8,872.76	0.05	47,863.91	62,245.62	98,674.36	99.0%
1,024	2,048	1x	101.23	149.31	1.43	206.90	206.90	6,518.81	100.0%
1,024	2,048	2x	169.64	218.08	1.06	159.00	230.22	8,792.41	100.0%
1,024	2,048	4x	285.95	360.58	0.78	208.65	289.33	10,285.19	100.0%
1,024	2,048	8x	514.97	673.98	0.44	420.41	555.86	10,796.08	100.0%
1,024	2,048	16x	930.33	1,265.49	0.30	498.72	710.24	10,334.31	100.0%
1,024	2,048	32x	1,804.57	2,326.38	0.19	939.91	1,155.85	12,253.66	100.0%
1,024	2,048	64x	2,448.66	3,310.56	0.11	1,791.90	2,502.57	13,745.00	100.0%
1,024	2,048	128x	4,970.19	6,895.60	0.07	3,475.27	4,583.06	16,398.86	100.0%
1,024	2,048	256x	4,457.88	6,055.76	0.06	16,869.11	20,239.37	36,117.94	100.0%
2,048	128	1x	79.90	1,228.46	0.24	373.86	373.86	1,601.19	100.0%
2,048	128	2x	156.00	2,388.18	0.14	408.99	413.69	1,639.00	100.0%
2,048	128	4x	304.04	4,654.99	0.09	371.04	456.86	1,681.74	100.0%
2,048	128	8x	480.53	7,344.91	0.06	658.49	870.45	2,126.55	100.0%
2,048	128	16x	769.06	11,746.53	0.05	943.90	1,330.28	2,655.45	100.0%
2,048	128	32x	1,015.87	15,518.85	0.04	1,731.65	2,326.94	4,017.83	100.0%
2,048	128	64x	1,376.41	21,030.25	0.03	2,568.85	3,640.52	5,908.57	100.0%
2,048	128	128x	1,630.92	24,969.58	0.02	5,377.28	7,559.11	9,858.42	100.0%
2,048	128	256x	1,216.00	18,671.67	0.03	16,103.15	20,963.91	26,223.32	100.0%
2,048	512	1x	102.20	392.81	0.69	51.49	51.49	5,008.50	100.0%
2,048	512	2x	189.84	726.55	0.45	293.43	437.20	5,376.49	100.0%
2,048	512	4x	312.50	1,349.69	0.29	386.28	749.09	5,702.72	100.0%
2,048	512	8x	615.61	2,820.69	0.21	331.22	539.24	5,409.42	100.0%
2,048	512	16x	1,108.34	5,035.58	0.12	615.79	791.13	6,194.76	100.0%
2,048	512	32x	1,707.52	7,268.21	0.08	1,311.94	2,500.45	8,585.35	100.0%
2,048	512	64x	2,680.65	11,095.93	0.05	2,154.18	3,731.08	11,229.20	100.0%
2,048	512	128x	3,635.18	14,639.45	0.04	5,392.66	8,115.94	16,812.97	100.0%
2,048	512	256x	3,991.27	16,309.52	0.03	15,529.59	19,407.48	30,204.02	100.0%
2,048	512	512x	4,111.06	16,611.00	0.03	29,179.79	37,899.37	56,834.72	100.0%
2,048	512	1024x	2,262.89	9,066.70	0.05	56,603.88	67,766.60	88,148.72	43.9%
2,048	1,024	1x	96.70	338.03	0.80	376.84	376.84	5,809.59	100.0%
2,048	1,024	2x	197.95	557.71	0.56	244.50	390.76	7,008.72	100.0%
2,048	1,024	4x	262.10	918.88	0.37	287.55	453.58	8,200.36	100.0%
2,048	1,024	8x	513.82	1,857.36	0.22	536.46	546.66	8,343.90	100.0%
2,048	1,024	16x	858.67	2,885.70	0.17	1,013.28	1,668.83	10,590.21	100.0%
2,048	1,024	32x	1,481.93	4,642.87	0.12	1,476.31	2,439.41	12,284.39	100.0%
2,048	1,024	64x	2,572.91	7,687.41	0.07	2,191.99	3,230.74	15,540.07	100.0%
2,048	1,024	128x	3,963.10	11,236.20	0.05	4,484.30	7,289.92	20,019.48	100.0%
2,048	1,024	256x	4,423.92	12,709.32	0.04	15,293.51	20,321.31	36,552.62	100.0%
2,048	1,024	512x	4,788.74	13,724.90	0.04	33,951.30	41,283.00	68,820.17	100.0%
2,048	1,024	1024x	5,209.43	14,644.25	0.03	58,123.97	74,792.74	119,539.87	94.4%
2,048	2,048	1x	95.13	392.50	0.72	375.56	375.56	5,002.71	100.0%
2,048	2,048	2x	181.63	518.04	0.61	409.22	412.22	7,501.32	100.0%
2,048	2,048	4x	291.45	914.60	0.40	374.17	463.96	8,416.07	100.0%
2,048	2,048	8x	488.08	1,776.62	0.24	526.13	859.95	8,787.59	100.0%
2,048	2,048	16x	767.46	2,613.72	0.19	912.31	1,616.70	11,577.80	100.0%
2,048	2,048	32x	1,369.48	4,178.43	0.13	1,563.32	3,185.93	12,690.26	100.0%
2,048	2,048	64x	2,596.52	7,826.34	0.07	2,515.31	3,741.22	14,364.60	100.0%
2,048	2,048	128x	3,435.82	9,634.73	0.06	5,727.21	8,028.98	21,976.18	100.0%
2,048	2,048	256x	3,588.10	9,930.58	0.05	18,228.97	22,170.63	40,607.47	100.0%
2,048	2,048	512x	4,329.13	12,178.62	0.04	32,496.29	40,232.39	68,335.42	100.0%
2,048	2,048	1024x	6,245.09	17,179.12	0.03	16,907.07	25,915.49	89,642.90	100.0%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H20

GPU Count8

GPU Memory (Total)760 GB

GPU Driver570.195.03

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)500 W

CPU ModelIntel(R) Xeon(R) Platinum 8469C

RAM1,007 GB

Software Configuration

Inference FrameworkvLLM

Framework Versionv0.9.0

OSUbuntu

OS Version24.04.3 LTS (Noble Numbat)

Kernel Version6.8.0-79-generic

Python Version3.12.3

Model Configuration

Providermistralai

Model Namemistral-nemo-instruct-2407

QuantizationBF16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length32768

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization85.00

Temperature0.70

Top-P1.00

Top-K-1