NVIDIA A100-PCIE-40GB (1x) - Mistral-Nemo-Instruct

November 20, 2025 at 08:26 PM

Dataset: reference (v1.0)

NVIDIA NVIDIA A100-PCIE-40GB mistral Mistral-Nemo-Instruct

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

3,541.62

Peak generation speed

Best Input TPS

6,567.89

Peak prefill speed

Best Energy Efficiency

0.01 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

42.95 ms

Lowest latency

Best E2E (P95)

2,738.58 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	128	256x	3,541.62★	3,548.86	0.01	605.77	1,810.47	8,568.62	100.0%
128	128	1x	46.73	44.54	0.56	75.00	75.00	2,738.58	100.0%
128	128	2x	92.19	90.03	0.28	51.29	51.94	2,775.82	100.0%
128	128	4x	178.34	176.94	0.14	81.76	94.46	2,865.33	100.0%
128	128	8x	350.68	351.71	0.07	101.09	116.74	2,912.95	100.0%
128	128	16x	663.86	668.72	0.04	135.49	183.77	3,063.16	100.0%
128	128	32x	1,176.00	1,177.72	0.02	160.06	301.48	3,428.62	100.0%
128	128	64x	2,005.39	1,997.55	0.02	224.94	490.62	3,974.29	100.0%
128	128	128x	2,863.37	2,865.47	0.01	346.19	951.61	5,445.18	100.0%
128	128	512x	3,450.70	3,457.02	0.01	4,327.00	9,960.65	17,161.11	100.0%
128	512	1x	47.13	11.23	1.17	52.27	52.27	10,862.78	100.0%
128	512	2x	89.69	23.68	0.57	50.13	50.96	10,502.51	100.0%
128	512	4x	171.93	45.16	0.30	80.79	94.73	11,241.86	100.0%
128	512	8x	308.39	89.37	0.16	106.22	118.98	11,484.52	100.0%
128	512	16x	644.96	170.30	0.08	114.85	174.17	12,090.50	100.0%
128	512	32x	1,177.29	309.07	0.04	150.19	280.09	13,218.55	100.0%
128	512	64x	2,036.17	535.32	0.03	219.02	477.08	15,117.42	100.0%
128	512	128x	3,038.08	791.45	0.02	351.55	960.75	20,413.65	100.0%
128	512	256x	2,733.09	707.86	0.02	622.71	1,935.19	45,043.69	100.0%
128	1,024	1x	46.89	10.59	1.20	42.95	42.95	11,493.46	100.0%
128	1,024	2x	77.72	19.87	0.66	59.60	70.53	12,359.66	100.0%
128	1,024	4x	155.28	28.77	0.36	69.35	86.53	17,486.28	100.0%
128	1,024	8x	244.99	57.36	0.22	95.15	114.93	17,492.53	100.0%
128	1,024	16x	466.01	87.06	0.12	125.71	161.25	22,701.34	100.0%
128	1,024	32x	908.55	159.97	0.06	167.27	292.72	25,569.64	100.0%
128	1,024	64x	1,583.62	281.57	0.04	220.18	488.66	28,831.92	100.0%
128	1,024	128x	2,459.47	439.48	0.02	360.01	987.73	36,949.98	100.0%
128	1,024	256x	2,231.10	397.61	0.03	1,065.07	2,420.96	76,877.80	100.0%
128	2,048	1x	46.88	14.05	1.16	46.64	46.64	8,658.62	100.0%
128	2,048	2x	87.87	21.45	0.59	53.92	54.65	11,576.83	100.0%
128	2,048	4x	147.90	25.75	0.38	86.32	98.62	19,204.90	100.0%
128	2,048	8x	235.80	51.76	0.23	117.40	133.73	19,379.29	100.0%
128	2,048	16x	434.47	86.49	0.13	160.48	185.38	22,383.38	100.0%
128	2,048	32x	781.03	141.63	0.07	599.81	643.18	25,260.16	100.0%
128	2,048	64x	1,243.74	212.63	0.05	226.30	490.98	30,838.98	100.0%
128	2,048	128x	1,991.15	333.15	0.03	355.55	953.04	41,929.82	100.0%
128	2,048	256x	2,037.39	360.65	0.03	1,145.55	2,399.97	76,531.50	100.0%
128	2,048	512x	1,976.42	335.77	0.03	43,305.37	117,534.39	164,155.22	97.3%
512	128	1x	46.01	178.29	0.24	80.71	80.71	2,782.02	100.0%
512	128	2x	90.30	350.97	0.12	82.67	83.43	2,834.17	100.0%
512	128	4x	172.68	667.12	0.06	123.39	160.48	2,960.13	100.0%
512	128	8x	320.50	1,236.31	0.04	265.89	303.32	3,191.14	100.0%
512	128	16x	570.95	2,212.43	0.02	333.34	520.46	3,563.45	100.0%
512	128	32x	916.54	3,571.49	0.01	380.25	1,038.12	4,388.89	100.0%
512	128	64x	1,305.71	5,079.69	0.01	791.01	2,185.66	6,082.95	100.0%
512	128	128x	1,686.44	6,567.89	0.01	1,214.28	4,012.67	9,275.80	100.0%
512	128	256x	1,498.10	5,835.27	0.01	5,319.56	15,460.51	20,961.25	100.0%
512	512	1x	46.65	45.19	0.73	80.99	80.99	10,975.00	100.0%
512	512	2x	91.26	88.67	0.35	81.76	82.92	11,218.38	100.0%
512	512	4x	177.76	171.69	0.19	122.18	158.50	11,515.16	100.0%
512	512	8x	334.67	330.82	0.10	264.01	300.13	11,934.49	100.0%
512	512	16x	585.85	616.29	0.05	319.32	525.48	12,852.25	100.0%
512	512	32x	1,039.88	1,076.92	0.03	398.02	1,035.25	14,734.45	100.0%
512	512	64x	1,709.61	1,734.70	0.02	648.43	1,992.59	18,188.10	100.0%
512	512	128x	1,587.85	1,615.22	0.02	1,226.56	4,002.59	38,133.13	100.0%
512	1,024	1x	46.63	41.90	0.78	81.84	81.84	11,816.75	100.0%
512	1,024	2x	73.71	48.70	0.55	84.43	85.23	20,005.93	100.0%
512	1,024	4x	144.80	90.06	0.29	114.94	175.18	21,145.87	100.0%
512	1,024	8x	250.58	182.25	0.16	251.72	288.70	21,056.84	100.0%
512	1,024	16x	467.46	370.72	0.08	342.37	535.17	20,593.62	100.0%
512	1,024	32x	804.22	585.40	0.05	393.34	1,026.92	25,274.94	100.0%
512	1,024	64x	1,356.65	993.48	0.03	642.38	2,018.03	31,684.68	100.0%
512	1,024	128x	1,490.85	1,017.93	0.03	1,235.85	4,012.13	59,921.59	100.0%
512	1,024	256x	1,508.52	1,063.22	0.03	29,329.40	84,725.74	114,269.19	100.0%
512	1,024	512x	625.20	455.26	0.04	23,134.90	83,717.80	115,130.89	43.2%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA A100-PCIE-40GB

GPU Count1

GPU Memory (Total)40 GB

GPU Driver570.148.08

CUDA VersionUnknown

Compute Capability8.0

Power Limit (per GPU)250 W

CPU ModelAMD EPYC 7H12 64-Core Processor

RAM221 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.10.0

OSUbuntu

OS Version22.04.5 LTS (Jammy Jellyfish)

Kernel Version6.8.0-60-generic

Python Version3.10.12

Model Configuration

Providermistral

Model NameMistral-Nemo-Instruct

QuantizationFP16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length8192

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.90

Temperature0.70

Top-P1.00

Top-K-1