NVIDIA H200 NVL (2x) - allam-7b-instruct-preview

November 6, 2025 at 07:07 AM

Dataset: reference (v1.0)

NVIDIA NVIDIA H200 NVL humain-ai allam-7b-instruct-preview

Best Performance

Click a metric to highlight the best run in the table below

Best Output TPS

11,481.64

Peak generation speed

Best Input TPS

45,184.12

Peak prefill speed

Best Energy Efficiency

0.01 kWh/MT

Energy cost per 1M tokens

Best TTFT (P95)

45.78 ms

Lowest latency

Best E2E (P95)

821.66 ms

Lowest latency

Test Matrix Results

Performance across different input/output token combinations and concurrency levels

Input Tokens	Output Tokens	Concurrency	Output TPS	Input TPS	Energy Cost (kWh/MT)	TTFT Mean	TTFT P95	E2E P95	Success Rate
Best Run for Output TPS
128	512	1024x	11,481.64★	3,305.38	0.02	1,004.27	2,648.07	11,446.45	100.0%
128	128	1x	79.60	75.87	0.40	846.69	846.69	1,607.79	100.0%
128	128	2x	163.06	159.24	0.22	774.05	775.74	1,565.28	100.0%
128	128	4x	582.48	577.93	0.06	86.34	112.44	871.50	100.0%
128	128	8x	1,015.87	1,018.85	0.06	132.10	173.28	990.56	100.0%
128	128	16x	1,935.73	1,949.91	0.03	151.22	203.53	1,029.41	100.0%
128	128	32x	3,429.65	3,435.51	0.02	163.17	205.18	1,141.07	100.0%
128	128	64x	4,186.57	4,487.62	0.02	156.54	237.84	1,246.44	100.0%
128	128	128x	5,083.06	6,046.14	0.02	240.47	451.22	1,464.58	100.0%
128	128	256x	6,132.06	7,106.95	0.01	253.04	464.98	1,598.83	100.0%
128	128	512x	5,560.85	7,362.31	0.02	441.00	1,591.19	3,286.10	100.0%
128	512	1x	170.78	40.69	0.55	46.75	46.75	2,997.98	100.0%
128	512	2x	324.97	79.42	0.36	53.48	59.09	3,142.63	100.0%
128	512	4x	626.87	155.49	0.21	73.73	85.14	3,256.42	100.0%
128	512	8x	838.20	225.47	0.13	90.45	102.63	4,548.67	100.0%
128	512	16x	1,481.67	400.19	0.07	128.75	160.56	5,106.86	100.0%
128	512	32x	3,926.31	1,000.98	0.03	150.17	178.76	4,032.41	100.0%
128	512	64x	6,246.53	1,642.30	0.02	142.02	196.06	4,519.35	100.0%
128	512	128x	7,847.36	2,062.85	0.02	202.18	411.53	5,401.76	100.0%
128	512	256x	9,466.91	2,555.82	0.02	288.83	620.58	6,642.71	100.0%
128	512	512x	10,702.51	2,900.87	0.02	955.30	2,788.34	12,366.08	100.0%
128	1,024	1x	157.04	19.63	0.62	56.24	56.24	6,205.59	100.0%
128	1,024	2x	254.28	39.61	0.48	51.83	55.96	6,172.50	100.0%
128	1,024	4x	471.42	74.65	0.21	93.87	98.56	6,659.77	100.0%
128	1,024	8x	711.82	151.54	0.18	114.24	120.20	6,652.21	100.0%
128	1,024	16x	1,596.77	284.79	0.09	161.91	193.95	7,184.86	100.0%
128	1,024	32x	3,066.49	520.49	0.05	165.15	191.62	7,813.03	100.0%
128	1,024	64x	5,070.11	862.22	0.04	152.46	208.36	9,324.32	100.0%
128	1,024	128x	6,990.34	1,172.11	0.03	189.01	414.60	11,146.71	100.0%
128	1,024	256x	8,977.95	1,530.61	0.02	304.19	722.33	14,216.69	100.0%
128	1,024	512x	10,492.76	1,800.59	0.02	627.96	1,444.98	18,565.60	100.0%
128	1,024	1024x	11,322.90	1,922.74	0.02	1,058.05	2,398.37	27,024.78	100.0%
128	2,048	1x	172.00	23.82	0.59	57.38	57.38	5,115.29	100.0%
128	2,048	2x	323.62	59.23	0.28	63.70	65.08	4,198.09	100.0%
128	2,048	4x	508.78	79.62	0.28	62.14	65.95	6,294.49	100.0%
128	2,048	8x	719.65	145.49	0.17	110.68	138.70	6,272.11	100.0%
128	2,048	16x	1,309.22	250.85	0.10	142.71	188.30	7,097.18	100.0%
128	2,048	32x	2,289.52	369.28	0.07	149.28	173.28	9,360.21	100.0%
128	2,048	64x	4,110.64	614.98	0.04	153.94	195.62	12,638.20	100.0%
128	2,048	128x	4,651.90	740.46	0.04	161.18	272.48	14,255.68	100.0%
128	2,048	256x	7,274.34	1,153.57	0.03	561.30	2,724.37	20,094.99	100.0%
128	2,048	512x	9,036.25	1,429.96	0.02	601.07	1,791.17	24,934.97	100.0%
128	2,048	1024x	10,987.18	1,722.33	0.02	817.88	2,594.79	32,590.06	100.0%
512	128	1x	155.72	603.41	0.08	52.41	52.41	821.66	100.0%
512	128	2x	298.72	1,161.03	0.05	69.26	89.05	851.64	100.0%
512	128	4x	568.89	2,197.78	0.05	77.52	87.29	892.11	100.0%
512	128	8x	1,006.88	3,883.97	0.03	110.38	138.80	1,000.47	100.0%
512	128	16x	1,727.67	6,949.21	0.02	154.12	181.20	1,121.85	100.0%
512	128	32x	2,960.29	11,736.03	0.01	224.54	287.16	1,318.03	100.0%
512	128	64x	4,544.28	17,864.35	0.01	213.80	366.25	1,678.86	100.0%
512	128	128x	5,363.54	21,607.60	0.01	184.10	292.07	2,112.03	100.0%
512	128	256x	6,250.30	25,110.06	0.01	223.64	364.76	2,590.08	100.0%
512	128	512x	6,620.15	27,143.50	0.01	355.70	907.23	4,345.16	100.0%
512	128	1024x	7,549.57	30,444.45	0.01	380.31	728.15	4,002.21	100.0%
512	512	1x	168.53	163.27	0.32	46.97	46.97	3,036.84	100.0%
512	512	2x	330.00	320.66	0.24	45.49	45.78	3,097.91	100.0%
512	512	4x	547.47	601.95	0.14	85.55	105.74	3,271.22	100.0%
512	512	8x	1,032.63	1,111.11	0.08	106.11	129.13	3,544.04	100.0%
512	512	16x	1,569.47	1,937.97	0.05	171.14	219.95	4,070.94	100.0%
512	512	32x	3,103.38	3,306.61	0.03	279.22	382.96	4,783.49	100.0%
512	512	64x	5,451.40	5,736.14	0.02	216.57	257.44	5,435.62	100.0%
512	512	128x	7,884.26	8,034.33	0.02	212.82	315.42	7,677.98	100.0%
512	512	256x	7,171.22	7,376.75	0.02	603.01	2,233.35	11,673.92	100.0%
512	1,024	1x	164.05	235.85	0.28	49.87	49.87	2,096.81	100.0%
512	1,024	2x	200.94	161.37	0.36	50.22	53.76	5,925.69	100.0%
512	1,024	4x	438.73	313.17	0.16	97.99	102.34	6,043.87	100.0%
512	1,024	8x	932.04	766.99	0.10	102.81	125.03	4,819.54	100.0%
512	1,024	16x	1,141.95	933.32	0.08	125.09	146.26	7,673.13	100.0%
512	1,024	32x	2,441.57	1,911.04	0.04	153.50	179.55	7,647.95	100.0%
512	1,024	64x	3,992.89	3,105.33	0.03	145.64	222.94	9,657.40	100.0%
512	1,024	128x	6,277.75	4,495.48	0.02	171.41	323.49	12,502.55	100.0%
512	1,024	256x	7,622.53	5,710.67	0.02	261.27	624.85	17,369.64	100.0%
512	1,024	512x	8,428.06	6,048.08	0.02	498.52	1,507.71	25,677.13	100.0%
512	1,024	1024x	9,095.47	6,563.47	0.02	2,342.22	11,292.16	45,640.48	100.0%
512	2,048	1x	164.69	210.53	0.30	62.09	62.09	2,348.53	100.0%
512	2,048	2x	264.59	238.90	0.21	77.54	82.40	4,100.13	100.0%
512	2,048	4x	508.93	371.87	0.17	118.17	150.28	5,253.40	100.0%
512	2,048	8x	811.87	565.09	0.11	149.87	166.25	6,699.02	100.0%
512	2,048	16x	1,334.48	1,135.82	0.07	181.74	232.67	6,752.33	100.0%
512	2,048	32x	1,509.43	1,041.23	0.06	224.30	293.21	10,139.93	100.0%
512	2,048	64x	2,993.93	2,172.61	0.04	243.75	334.38	11,410.53	100.0%
512	2,048	128x	4,225.96	2,895.82	0.03	186.96	281.88	15,446.53	100.0%
512	2,048	256x	6,084.62	4,188.59	0.02	259.27	574.54	19,721.13	100.0%
512	2,048	512x	7,602.99	5,094.06	0.02	440.74	1,049.70	30,827.68	100.0%
512	2,048	1024x	8,551.50	5,744.01	0.02	3,175.27	14,443.09	53,117.24	100.0%
1,024	128	1x	145.12	1,105.44	0.05	91.99	91.99	881.63	100.0%
1,024	128	2x	286.35	2,185.68	0.03	65.58	80.90	888.68	100.0%
1,024	128	4x	527.84	4,029.90	0.03	109.51	136.10	968.02	100.0%
1,024	128	8x	948.15	7,252.78	0.02	123.77	142.19	1,068.95	100.0%
1,024	128	16x	1,586.37	12,169.64	0.01	218.64	256.46	1,275.52	100.0%
1,024	128	32x	2,419.87	18,584.86	0.01	274.05	400.64	1,636.36	100.0%
1,024	128	64x	3,429.05	26,408.06	0.01	349.74	549.53	2,285.79	100.0%
1,024	128	128x	4,592.20	35,806.49	0.01	390.02	691.84	3,308.66	100.0%
1,024	128	256x	4,934.48	39,160.75	0.01	341.97	800.28	4,482.06	100.0%
1,024	128	512x	5,030.18	39,368.60	0.01	672.07	1,936.57	9,145.26	100.0%
1,024	128	1024x	4,802.19	37,235.42	0.01	7,364.70	15,463.55	20,710.74	100.0%
1,024	512	1x	157.30	346.98	0.20	79.70	79.70	2,803.20	100.0%
1,024	512	2x	308.62	591.23	0.17	73.77	80.03	3,290.69	100.0%
1,024	512	4x	590.03	1,126.19	0.10	86.89	105.90	3,457.94	100.0%
1,024	512	8x	1,021.14	1,971.56	0.05	150.01	186.47	3,961.01	100.0%
1,024	512	16x	1,831.02	3,511.62	0.03	201.14	274.45	4,456.23	100.0%
1,024	512	32x	2,962.47	5,867.63	0.02	292.71	384.63	5,308.54	100.0%
1,024	512	64x	4,339.53	8,796.53	0.02	344.01	521.88	7,056.72	100.0%
1,024	512	128x	5,659.83	11,580.67	0.02	375.92	771.46	10,641.76	100.0%
1,024	512	256x	6,978.59	14,405.35	0.01	305.21	738.13	15,509.89	100.0%
1,024	512	512x	6,878.85	13,983.36	0.01	1,020.90	4,475.27	22,512.72	100.0%
1,024	1,024	1x	150.35	234.54	0.32	73.04	73.04	4,148.88	100.0%
1,024	1,024	2x	285.77	379.86	0.17	78.04	92.46	5,128.08	100.0%
1,024	1,024	4x	473.62	597.80	0.16	114.54	151.06	6,507.83	100.0%
1,024	1,024	8x	899.02	1,448.68	0.08	122.45	145.18	5,381.90	100.0%
1,024	1,024	16x	1,308.35	1,902.06	0.06	202.23	250.80	7,549.92	100.0%
1,024	1,024	32x	2,351.07	3,240.23	0.04	262.16	377.19	9,614.02	100.0%
1,024	1,024	64x	3,673.86	5,470.86	0.03	318.01	488.27	11,108.98	100.0%
1,024	1,024	128x	4,989.30	7,357.27	0.02	412.30	766.81	16,808.61	100.0%
1,024	1,024	256x	6,212.13	9,145.48	0.02	377.66	826.25	25,940.52	100.0%
1,024	1,024	512x	6,484.06	9,546.37	0.02	4,654.48	29,567.41	47,767.38	100.0%
1,024	1,024	1024x	6,294.79	9,255.38	0.02	28,559.17	68,209.11	93,701.66	100.0%
1,024	2,048	1x	167.69	259.93	0.27	83.33	83.33	3,743.82	100.0%
1,024	2,048	2x	294.70	386.47	0.23	86.08	94.13	5,008.42	100.0%
1,024	2,048	4x	545.17	874.11	0.11	94.09	107.51	4,397.39	100.0%
1,024	2,048	8x	907.13	1,332.37	0.08	141.04	161.99	5,653.59	100.0%
1,024	2,048	16x	1,063.12	1,437.29	0.07	212.43	287.74	9,531.88	100.0%
1,024	2,048	32x	2,178.97	2,906.68	0.04	275.74	434.44	9,793.31	100.0%
1,024	2,048	64x	2,752.69	3,598.63	0.04	366.61	565.01	12,822.68	100.0%
1,024	2,048	128x	3,642.88	5,340.35	0.03	363.41	790.26	17,237.66	100.0%
1,024	2,048	256x	5,107.52	7,182.94	0.02	526.40	1,249.08	27,332.66	100.0%
1,024	2,048	512x	5,634.46	7,851.74	0.02	4,843.06	31,115.90	53,216.70	100.0%
1,024	2,048	1024x	6,187.18	8,465.40	0.02	31,226.53	75,435.23	100,768.48	100.0%
2,048	128	1x	144.47	2,221.22	0.03	108.97	108.97	886.12	100.0%
2,048	128	2x	262.30	4,015.37	0.02	84.44	106.71	964.85	100.0%
2,048	128	4x	458.78	7,024.19	0.02	138.24	168.41	1,109.33	100.0%
2,048	128	8x	842.11	12,871.71	0.01	180.00	244.16	1,195.83	100.0%
2,048	128	16x	1,260.95	20,458.47	0.01	301.50	409.77	1,495.79	100.0%
2,048	128	32x	1,821.53	28,781.97	0.01	421.28	624.37	2,137.44	100.0%
2,048	128	64x	2,439.14	37,798.55	0.01	614.12	1,095.26	3,226.38	100.0%
2,048	128	128x	2,917.15	45,184.12	0.01	897.10	1,908.00	5,278.90	100.0%
2,048	128	256x	2,841.37	44,139.33	0.01	1,904.89	7,665.57	9,992.67	100.0%
2,048	512	1x	161.03	627.55	0.14	102.93	102.93	3,129.51	100.0%
2,048	512	2x	224.18	1,208.45	0.08	115.43	117.38	3,152.09	100.0%
2,048	512	4x	511.12	2,126.70	0.06	132.46	167.00	3,633.86	100.0%
2,048	512	8x	785.78	3,746.29	0.04	196.55	289.56	4,162.25	100.0%
2,048	512	16x	1,366.73	6,856.86	0.02	317.07	508.30	4,546.62	100.0%
2,048	512	32x	2,289.72	10,133.12	0.02	421.14	651.08	6,104.96	100.0%
2,048	512	64x	3,253.40	13,628.55	0.02	475.48	906.22	9,064.97	100.0%
2,048	512	128x	4,004.68	16,749.41	0.02	903.97	1,899.87	14,727.57	100.0%
2,048	512	256x	3,762.59	15,831.12	0.01	3,521.68	22,285.29	30,216.11	100.0%
2,048	1,024	1x	154.99	770.25	0.13	107.55	107.55	2,548.35	100.0%
2,048	1,024	2x	291.46	851.77	0.14	76.43	96.11	4,559.88	100.0%
2,048	1,024	4x	451.45	1,349.69	0.09	122.16	194.46	5,770.78	100.0%
2,048	1,024	8x	698.23	2,765.37	0.05	178.65	244.08	5,378.37	100.0%
2,048	1,024	16x	976.40	3,432.95	0.04	261.07	369.03	8,101.61	100.0%
2,048	1,024	32x	1,764.45	5,786.74	0.03	435.28	757.73	10,115.20	100.0%
2,048	1,024	64x	2,386.72	7,919.95	0.02	650.42	1,389.14	15,595.77	100.0%
2,048	1,024	128x	3,604.32	10,737.82	0.02	851.36	1,876.05	22,798.14	100.0%
2,048	1,024	256x	3,988.14	12,243.00	0.02	3,713.05	22,964.52	37,260.50	100.0%
2,048	1,024	512x	3,925.93	12,408.70	0.02	22,465.78	54,663.07	73,676.78	100.0%
2,048	2,048	1x	146.54	396.14	0.20	109.57	109.57	4,959.86	100.0%
2,048	2,048	2x	229.77	961.01	0.12	98.33	102.14	3,972.07	100.0%
2,048	2,048	4x	324.45	1,016.96	0.11	89.44	103.04	5,548.85	75.0%

Hardware Configuration

GPU ManufacturerNVIDIA

GPU ModelNVIDIA H200 NVL

GPU Count2

GPU Memory (Total)280 GB

GPU Driver580.95.05

CUDA VersionUnknown

Compute Capability9.0

Power Limit (per GPU)600 W

CPU ModelIntel(R) Xeon(R) 6960P

RAM2,267 GB

Software Configuration

Inference FrameworkvLLM

Framework Version0.11.0

OSUbuntu

OS Version22.04.5 LTS (Jammy Jellyfish)

Kernel Version5.15.0-88-generic

Python Version3.10.12

Model Configuration

Providerhumain-ai

Model Nameallam-7b-instruct-preview

QuantizationFP16

Inference Configuration

Runtime parameters used across all benchmark runs

Max Model Length4096

Tensor Parallel Size1

Pipeline Parallel Size1

GPU Memory Utilization0.90

Temperature0.70

Top-P1.00

Top-K-1