NVIDIA H200 NVL (2x) - mistral-nemo-instruct-2407

November 7, 2025 at 04:42 AM

Dataset: reference (v1.0)

NVIDIA NVIDIA H200 NVL mistralai mistral-nemo-instruct-2407

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

12,204.48

Peak generation speed

Best Input TPS

47,690.47

Peak prefill speed

Best Energy Efficiency

0.01 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

68.02 ms

Lowest latency

Best E2E (P95)

1,203.89 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	2,048	1024x	12,204.48★	2,169.01	0.02	1,856.57	4,985.40	19,485.78	100.0%
128	128	1x	69.68	66.41	0.49	702.56	702.56	1,836.58	100.0%
128	128	2x	202.05	197.32	0.26	119.93	127.74	1,260.44	100.0%
128	128	4x	279.48	277.29	0.19	287.06	631.23	1,748.18	100.0%
128	128	8x	724.70	726.82	0.09	229.06	261.00	1,400.90	100.0%
128	128	16x	1,349.14	1,359.03	0.04	276.20	326.03	1,498.17	100.0%
128	128	32x	2,530.32	2,538.37	0.03	292.63	347.70	1,550.68	100.0%
128	128	64x	4,310.38	4,386.23	0.02	362.19	413.66	1,694.31	100.0%
128	128	128x	3,927.23	4,054.21	0.02	1,236.39	2,469.52	3,891.30	100.0%
128	512	1x	112.48	26.80	0.80	92.47	92.47	4,551.90	100.0%
128	512	2x	217.58	55.62	0.40	135.37	136.40	4,462.76	100.0%
128	512	4x	447.55	111.01	0.31	145.74	168.75	4,568.10	100.0%
128	512	8x	839.00	223.75	0.16	201.98	223.05	4,579.30	100.0%
128	512	16x	1,196.29	318.86	0.10	225.31	261.21	6,444.62	100.0%
128	512	32x	2,085.68	550.01	0.06	327.38	410.19	7,376.47	100.0%
128	512	64x	5,405.75	1,429.77	0.03	310.36	356.00	5,485.44	100.0%
128	512	128x	8,038.04	2,186.20	0.02	306.26	410.72	6,435.18	100.0%
128	512	256x	8,397.62	2,437.95	0.02	402.51	760.72	7,047.09	100.0%
128	512	512x	9,057.00	2,747.04	0.02	994.99	3,864.42	11,657.82	100.0%
128	512	1024x	10,837.68	3,411.94	0.02	1,245.94	3,747.99	10,674.00	100.0%
128	1,024	1x	108.60	29.98	0.81	78.91	78.91	4,059.39	100.0%
128	1,024	2x	219.95	48.23	0.64	79.07	81.53	5,158.56	100.0%
128	1,024	4x	372.63	71.30	0.41	114.82	122.63	7,038.65	100.0%
128	1,024	8x	663.11	148.17	0.20	181.92	210.34	6,885.42	100.0%
128	1,024	16x	1,125.52	209.68	0.12	226.93	261.61	9,728.69	100.0%
128	1,024	32x	2,312.61	408.77	0.07	265.74	313.70	9,909.10	100.0%
128	1,024	64x	4,202.86	742.42	0.04	307.05	387.27	10,746.09	100.0%
128	1,024	128x	6,766.04	1,232.15	0.03	300.19	454.46	11,700.08	100.0%
128	1,024	256x	8,352.47	1,597.27	0.02	391.28	879.52	13,019.80	100.0%
128	1,024	512x	10,274.08	2,013.10	0.02	666.49	1,496.55	14,521.35	100.0%
128	1,024	1024x	11,047.33	2,294.49	0.02	2,332.12	5,789.89	19,348.84	100.0%
128	2,048	1x	113.31	22.81	0.91	89.32	89.32	5,337.62	100.0%
128	2,048	2x	191.86	47.30	0.50	126.60	132.57	5,207.93	100.0%
128	2,048	4x	368.38	78.43	0.37	128.50	152.76	6,340.38	100.0%
128	2,048	8x	682.96	150.32	0.22	170.83	202.10	6,733.27	100.0%
128	2,048	16x	1,088.25	208.30	0.14	241.45	276.03	9,582.11	100.0%
128	2,048	32x	1,907.73	337.34	0.08	281.07	332.17	11,233.38	100.0%
128	2,048	64x	3,110.18	519.89	0.05	296.64	335.82	13,039.77	100.0%
128	2,048	128x	4,600.62	805.50	0.03	316.61	406.62	12,705.36	100.0%
128	2,048	256x	8,568.58	1,584.31	0.02	385.19	839.31	14,284.56	100.0%
128	2,048	512x	9,457.33	1,716.22	0.02	611.26	1,387.13	16,407.42	100.0%
512	128	1x	106.31	411.96	0.22	116.17	116.17	1,203.89	100.0%
512	128	2x	212.10	824.36	0.13	89.08	108.14	1,204.93	100.0%
512	128	4x	387.29	1,496.22	0.07	150.89	184.66	1,307.86	100.0%
512	128	8x	747.99	2,885.32	0.04	233.26	251.57	1,357.80	100.0%
512	128	16x	1,320.45	5,200.52	0.02	277.80	306.26	1,506.48	100.0%
512	128	32x	2,265.08	8,832.87	0.01	376.38	455.51	1,759.32	100.0%
512	128	64x	3,710.45	14,486.36	0.01	401.58	649.47	2,061.17	100.0%
512	128	128x	5,639.83	22,087.44	0.01	457.01	711.95	2,625.68	100.0%
512	128	256x	6,075.28	26,114.46	0.01	406.32	604.80	3,722.33	100.0%
512	128	512x	5,928.91	24,659.18	0.01	1,382.32	3,390.60	6,475.88	100.0%
512	128	1024x	4,393.17	20,312.10	0.01	4,094.47	8,631.40	11,388.98	100.0%
512	512	1x	110.08	106.64	0.58	68.02	68.02	4,650.20	100.0%
512	512	2x	223.73	218.25	0.39	71.74	71.94	4,547.90	100.0%
512	512	4x	450.01	434.63	0.19	114.18	118.89	4,545.26	100.0%
512	512	8x	829.15	806.29	0.10	192.79	222.86	4,883.39	100.0%
512	512	16x	1,520.55	1,598.71	0.06	203.40	250.20	4,946.15	100.0%
512	512	32x	2,840.07	2,947.55	0.03	269.81	312.58	5,347.57	100.0%
512	512	64x	5,017.33	5,113.11	0.02	298.92	356.84	6,036.25	100.0%
512	512	128x	8,285.62	8,612.64	0.01	344.02	475.00	7,025.60	100.0%
512	512	256x	8,454.57	9,306.82	0.01	365.05	615.42	8,147.38	100.0%
512	512	512x	10,293.75	11,221.56	0.01	548.93	1,520.98	9,820.31	100.0%
512	512	1024x	9,793.78	10,922.49	0.01	4,870.68	13,158.59	21,459.89	100.0%
512	1,024	1x	112.09	89.39	0.61	86.68	86.68	5,538.41	100.0%
512	1,024	2x	170.99	108.85	0.56	94.09	96.51	8,925.21	100.0%
512	1,024	4x	358.66	212.85	0.32	159.80	178.66	9,227.02	100.0%
512	1,024	8x	692.67	502.67	0.15	208.56	243.60	7,664.15	100.0%
512	1,024	16x	1,075.25	925.91	0.09	283.78	364.81	8,191.71	100.0%
512	1,024	32x	2,109.43	1,592.12	0.05	291.26	326.54	8,584.55	100.0%
512	1,024	64x	3,961.48	2,802.74	0.03	404.35	596.72	11,081.75	100.0%
512	1,024	128x	7,168.14	4,927.70	0.02	350.03	481.79	12,498.84	100.0%
512	1,024	256x	8,254.25	6,080.82	0.02	376.13	723.47	15,082.69	100.0%
512	1,024	512x	9,794.10	7,186.37	0.02	1,415.22	4,208.14	19,596.39	100.0%
512	1,024	1024x	11,035.23	8,000.49	0.01	6,939.05	14,278.89	28,329.94	100.0%
512	2,048	1x	113.95	95.31	0.56	111.52	111.52	5,194.05	100.0%
512	2,048	2x	224.70	178.86	0.30	130.36	149.07	5,549.27	100.0%
512	2,048	4x	366.67	247.28	0.28	127.38	145.98	7,851.58	100.0%
512	2,048	8x	630.89	465.36	0.15	211.89	249.25	8,133.60	100.0%
512	2,048	16x	1,084.54	862.33	0.09	301.86	383.21	8,383.82	100.0%
512	2,048	32x	1,865.00	1,422.30	0.05	315.76	359.17	9,439.40	100.0%
512	2,048	64x	3,792.58	2,627.37	0.03	371.18	526.55	10,996.24	100.0%
512	2,048	128x	4,302.45	2,877.89	0.03	458.69	715.25	13,757.68	100.0%
512	2,048	256x	7,144.44	4,825.20	0.02	508.18	779.17	17,324.46	100.0%
512	2,048	512x	8,207.83	5,955.42	0.02	1,004.02	3,364.23	20,255.05	100.0%
512	2,048	1024x	10,223.85	7,396.28	0.02	6,139.10	14,628.50	28,345.80	100.0%
1,024	128	1x	104.32	794.62	0.11	128.73	128.73	1,226.72	100.0%
1,024	128	2x	205.62	1,569.48	0.07	108.79	132.71	1,237.50	100.0%
1,024	128	4x	383.81	2,930.28	0.04	170.16	215.16	1,328.39	100.0%
1,024	128	8x	680.85	5,208.11	0.02	265.34	334.02	1,493.71	100.0%
1,024	128	16x	1,228.55	9,424.72	0.01	334.99	380.17	1,652.48	100.0%
1,024	128	32x	2,004.90	15,390.30	0.01	512.30	620.80	1,986.45	100.0%
1,024	128	64x	3,068.99	23,586.05	0.01	610.82	997.97	2,570.15	100.0%
1,024	128	128x	4,146.71	32,048.90	0.01	735.54	1,357.54	3,744.96	100.0%
1,024	128	256x	4,478.03	34,448.60	0.01	1,859.30	3,771.87	7,056.80	100.0%
1,024	128	512x	5,004.63	39,504.51	0.01	2,254.88	4,604.16	9,693.22	100.0%
1,024	128	1024x	5,207.31	41,923.76	0.01	3,861.25	11,262.96	14,070.71	100.0%
1,024	512	1x	107.36	204.45	0.37	110.68	110.68	4,768.86	100.0%
1,024	512	2x	215.17	410.59	0.27	95.35	109.67	4,741.91	100.0%
1,024	512	4x	425.07	811.33	0.14	141.66	161.07	4,814.59	100.0%
1,024	512	8x	822.49	1,572.89	0.07	217.77	273.87	4,962.44	100.0%
1,024	512	16x	1,454.07	2,824.20	0.04	310.79	397.17	5,537.62	100.0%
1,024	512	32x	2,780.48	5,505.78	0.02	309.03	388.29	5,643.09	100.0%
1,024	512	64x	3,020.53	6,033.96	0.02	1,147.69	4,011.65	10,279.75	100.0%
1,024	512	128x	5,306.32	10,655.83	0.01	731.05	1,579.77	11,617.70	100.0%
1,024	512	256x	7,868.92	15,677.96	0.01	1,163.11	2,218.17	12,812.71	100.0%
1,024	512	512x	8,187.55	16,642.48	0.01	2,992.34	10,205.97	23,187.44	100.0%
1,024	512	1024x	8,051.35	16,957.46	0.01	7,244.87	20,471.84	32,474.42	100.0%
1,024	1,024	1x	112.01	120.15	0.53	116.99	116.99	8,105.19	100.0%
1,024	1,024	2x	201.19	264.91	0.36	118.11	152.69	7,277.54	100.0%
1,024	1,024	4x	358.15	467.14	0.21	190.96	250.62	8,137.48	100.0%
1,024	1,024	8x	664.10	901.38	0.12	270.69	314.81	8,626.64	100.0%
1,024	1,024	16x	1,166.07	1,522.24	0.07	375.60	456.18	10,310.05	100.0%
1,024	1,024	32x	2,194.15	2,899.97	0.04	482.23	621.70	10,572.99	100.0%
1,024	1,024	64x	3,541.12	4,988.82	0.03	671.58	1,138.90	12,356.82	100.0%
1,024	1,024	128x	5,990.24	8,414.63	0.02	809.53	1,405.67	14,378.36	100.0%
1,024	1,024	256x	7,368.38	10,177.01	0.01	1,487.67	3,723.63	20,970.09	100.0%
1,024	1,024	512x	8,355.17	11,666.36	0.01	5,473.89	13,068.43	31,362.48	100.0%
1,024	1,024	1024x	9,158.93	13,390.29	0.01	8,288.49	20,222.25	36,298.65	100.0%
1,024	2,048	1x	111.46	160.76	0.43	133.49	133.49	6,054.36	100.0%
1,024	2,048	2x	185.63	204.93	0.31	152.44	175.74	9,344.93	100.0%
1,024	2,048	4x	389.57	491.08	0.20	171.10	191.53	7,916.79	100.0%
1,024	2,048	8x	646.95	786.76	0.11	307.39	365.24	9,883.49	100.0%
1,024	2,048	16x	1,074.91	1,421.30	0.08	325.09	394.41	10,838.65	100.0%
1,024	2,048	32x	2,140.73	2,900.24	0.04	512.78	637.40	10,469.82	100.0%
1,024	2,048	64x	3,308.78	4,416.49	0.03	549.42	893.10	12,424.01	100.0%
1,024	2,048	128x	4,022.91	5,609.13	0.02	824.47	1,670.41	14,477.22	100.0%
1,024	2,048	256x	6,932.46	9,553.80	0.02	1,195.15	2,341.08	19,901.69	100.0%
1,024	2,048	512x	7,861.06	10,863.85	0.01	5,729.68	14,761.58	33,087.58	100.0%
1,024	2,048	1024x	8,817.98	12,405.08	0.01	9,741.81	21,527.84	39,762.38	100.0%
2,048	128	1x	100.55	1,545.95	0.06	169.63	169.63	1,272.69	100.0%
2,048	128	2x	184.04	2,817.40	0.03	233.20	266.10	1,384.03	100.0%
2,048	128	4x	354.08	5,421.16	0.02	221.09	296.98	1,438.78	100.0%
2,048	128	8x	636.02	9,721.74	0.01	266.68	374.52	1,590.17	100.0%
2,048	128	16x	997.55	15,311.31	0.01	492.37	746.47	2,021.00	100.0%
2,048	128	32x	1,578.48	24,131.12	0.01	638.48	1,063.87	2,551.46	100.0%
2,048	128	64x	2,304.36	35,433.47	0.01	951.09	1,704.83	3,425.81	100.0%
2,048	128	128x	2,547.36	38,936.07	0.01	1,616.20	3,621.07	6,225.80	100.0%
2,048	128	256x	2,840.12	43,495.35	0.01	2,358.40	7,786.21	10,081.43	100.0%
2,048	128	512x	3,112.03	47,690.47	0.01	5,168.94	13,691.07	17,860.64	100.0%
2,048	128	1024x	2,932.90	45,241.38	0.01	14,626.17	31,166.66	37,831.28	100.0%
2,048	512	1x	110.30	423.96	0.23	146.76	146.76	4,641.44	100.0%
2,048	512	2x	222.75	852.51	0.16	148.17	152.08	4,593.56	100.0%
2,048	512	4x	360.24	1,543.11	0.06	280.91	324.15	5,071.59	100.0%
2,048	512	8x	686.18	3,116.69	0.05	307.18	446.57	5,015.09	100.0%
2,048	512	16x	1,238.17	5,502.37	0.03	396.66	559.33	5,655.40	100.0%
2,048	512	32x	2,253.63	9,357.26	0.02	586.43	930.80	6,611.41	100.0%
2,048	512	64x	3,271.86	13,458.54	0.01	1,043.28	2,066.72	9,187.38	100.0%
2,048	512	128x	5,004.05	20,257.34	0.01	1,400.26	2,948.78	12,119.59	100.0%
2,048	512	256x	6,600.23	26,646.57	0.01	2,343.34	5,607.22	18,348.60	100.0%
2,048	512	512x	6,297.64	25,322.79	0.01	9,668.00	22,137.93	34,725.82	100.0%
2,048	512	1024x	4,353.12	16,892.41	0.01	41,756.34	81,539.19	95,689.78	100.0%
2,048	1,024	1x	107.10	328.82	0.28	156.14	156.14	5,974.14	100.0%
2,048	1,024	2x	194.54	610.91	0.14	142.63	144.16	6,345.17	100.0%
2,048	1,024	4x	313.18	1,072.51	0.13	155.23	200.06	7,125.15	100.0%
2,048	1,024	8x	544.21	1,845.32	0.07	249.49	293.45	8,036.32	100.0%
2,048	1,024	16x	935.52	3,361.74	0.04	312.13	388.03	8,249.38	100.0%
2,048	1,024	32x	1,832.66	5,528.54	0.03	586.91	769.12	11,238.94	100.0%
2,048	1,024	64x	2,929.88	8,646.01	0.02	953.63	2,117.62	14,283.44	100.0%
2,048	1,024	128x	4,549.94	12,634.13	0.01	1,456.65	4,610.67	19,300.41	100.0%
2,048	1,024	256x	6,007.34	17,341.22	0.01	3,281.32	8,225.39	27,349.94	100.0%
2,048	1,024	512x	6,552.09	18,933.22	0.01	12,466.77	31,520.06	46,939.03	100.0%
2,048	1,024	1024x	6,444.40	18,223.99	0.01	37,450.01	80,752.71	98,674.39	100.0%
2,048	2,048	1x	111.34	337.62	0.27	159.68	159.68	5,818.77	100.0%
2,048	2,048	2x	204.79	551.97	0.23	154.41	165.40	7,020.07	100.0%
2,048	2,048	4x	321.06	998.73	0.12	303.08	375.65	7,820.41	100.0%
2,048	2,048	8x	537.97	1,967.81	0.08	317.68	467.60	7,495.69	100.0%
2,048	2,048	16x	762.90	2,542.55	0.06	459.87	626.31	9,881.09	100.0%
2,048	2,048	32x	1,489.72	4,835.55	0.03	688.87	1,211.08	11,135.92	100.0%
2,048	2,048	64x	2,345.05	7,069.88	0.02	990.78	1,870.56	13,594.13	100.0%
2,048	2,048	128x	3,280.73	8,850.55	0.02	1,464.33	3,030.25	19,351.30	100.0%
2,048	2,048	256x	5,485.44	15,397.95	0.01	2,468.71	5,759.08	27,811.93	100.0%
2,048	2,048	512x	6,022.90	17,082.97	0.01	11,761.15	31,396.88	47,875.87	100.0%
2,048	2,048	1024x	6,189.46	16,978.32	0.01	39,877.95	87,219.64	103,400.26	100.0%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H200 NVL

GPU Count2

GPU Memory (Total)280 GB

GPU Driver580.95.05

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)600 W

CPU ModelIntel(R) Xeon(R) 6960P

RAM2,267 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.11.0

OSUbuntu

OS Version22.04.5 LTS (Jammy Jellyfish)

Kernel Version5.15.0-88-generic

Python Version3.10.12

Model Configuration

Providermistralai

Model Namemistral-nemo-instruct-2407

QuantizationFP16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length8192

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.90

Temperature0.70

Top-P1.00

Top-K-1