November 6, 2025 at 05:32 AM
Dataset: reference (v1.0)
Click a metric to highlight the best run in the table below
Performance across different input/output token combinations and concurrency levels
| Input Tokens | Output Tokens | Concurrency | Output TPS | Input TPS | Energy Cost (kWh/MT) | TTFT Mean | TTFT P95 | E2E P95 | Success Rate |
|---|---|---|---|---|---|---|---|---|---|
Best Run for Output TPS | |||||||||
| 128 | 512 | 1024x | 4,620.81★ | 1,285.70 | 0.05 | 13,890.51 | 45,498.05 | 75,536.60 | 99.7% |
| 128 | 128 | 1x | 40.70 | 38.79 | 2.06 | 249.17 | 249.17 | 3,144.66 | 100.0% |
| 128 | 128 | 2x | 84.58 | 85.27 | 1.21 | 68.94 | 79.71 | 2,921.35 | 100.0% |
| 128 | 128 | 4x | 132.08 | 167.33 | 0.65 | 98.40 | 112.04 | 3,026.15 | 100.0% |
| 128 | 128 | 8x | 290.96 | 330.54 | 0.32 | 158.48 | 181.50 | 3,098.11 | 100.0% |
| 128 | 128 | 16x | 429.53 | 432.68 | 0.21 | 1,713.49 | 1,756.24 | 4,744.26 | 100.0% |
| 128 | 128 | 32x | 1,062.73 | 1,102.41 | 0.10 | 335.31 | 509.94 | 3,505.37 | 96.9% |
| 128 | 128 | 64x | 1,873.37 | 1,929.94 | 0.06 | 365.14 | 622.47 | 4,079.11 | 100.0% |
| 128 | 128 | 128x | 2,806.30 | 3,015.84 | 0.05 | 461.99 | 1,103.72 | 5,017.62 | 100.0% |
| 128 | 128 | 256x | 2,730.77 | 3,254.06 | 0.05 | 2,215.28 | 3,906.33 | 9,279.74 | 100.0% |
| 128 | 512 | 1x | 42.79 | 53.82 | 1.56 | 57.54 | 57.54 | 2,243.45 | 100.0% |
| 128 | 512 | 2x | 53.62 | 81.25 | 1.67 | 63.57 | 75.82 | 2,938.26 | 100.0% |
| 128 | 512 | 4x | 130.33 | 42.88 | 1.31 | 91.88 | 101.89 | 11,841.40 | 100.0% |
| 128 | 512 | 8x | 262.65 | 79.45 | 0.63 | 112.31 | 129.96 | 12,908.75 | 100.0% |
| 128 | 512 | 16x | 460.65 | 136.19 | 0.34 | 159.64 | 177.10 | 15,109.66 | 100.0% |
| 128 | 512 | 32x | 1,232.45 | 321.47 | 0.16 | 161.22 | 192.47 | 12,688.76 | 100.0% |
| 128 | 512 | 64x | 2,094.19 | 570.37 | 0.10 | 159.15 | 224.91 | 13,812.23 | 98.4% |
| 128 | 512 | 128x | 3,606.33 | 956.10 | 0.07 | 153.98 | 237.20 | 15,618.61 | 100.0% |
| 128 | 512 | 256x | 4,385.66 | 1,197.76 | 0.06 | 545.95 | 2,438.67 | 22,224.11 | 99.6% |
| 128 | 512 | 512x | 4,586.66 | 1,263.56 | 0.05 | 2,147.84 | 4,978.44 | 44,935.51 | 99.6% |
| 128 | 1,024 | 1x | 42.73 | 5.18 | 4.70 | 56.32 | 56.32 | 23,542.34 | 100.0% |
| 128 | 1,024 | 2x | 63.24 | 10.70 | 3.11 | 71.62 | 73.65 | 22,697.16 | 100.0% |
| 128 | 1,024 | 4x | 138.51 | 21.43 | 1.44 | 108.94 | 124.79 | 23,698.66 | 100.0% |
| 128 | 1,024 | 8x | 171.09 | 43.36 | 1.10 | 176.07 | 199.06 | 23,661.75 | 100.0% |
| 128 | 1,024 | 16x | 572.52 | 82.73 | 0.37 | 205.08 | 248.19 | 24,895.21 | 100.0% |
| 128 | 1,024 | 32x | 1,059.84 | 160.32 | 0.22 | 283.33 | 425.72 | 25,503.85 | 100.0% |
| 128 | 1,024 | 64x | 2,076.91 | 288.09 | 0.12 | 339.40 | 566.79 | 28,165.67 | 100.0% |
| 128 | 1,024 | 128x | 3,290.92 | 499.37 | 0.08 | 464.35 | 1,073.86 | 32,070.36 | 99.2% |
| 128 | 1,024 | 256x | 4,418.04 | 645.46 | 0.06 | 729.18 | 1,796.83 | 48,636.28 | 99.6% |
| 128 | 1,024 | 512x | 4,462.24 | 657.55 | 0.06 | 1,524.86 | 3,810.41 | 92,176.35 | 99.6% |
| 128 | 1,024 | 1024x | 3,445.74 | 507.68 | 0.07 | 8,253.12 | 50,705.05 | 148,280.87 | 61.9% |
| 128 | 2,048 | 1x | 43.54 | 2.59 | 5.02 | 75.11 | 75.11 | 47,039.28 | 100.0% |
| 128 | 2,048 | 2x | 49.18 | 5.32 | 4.24 | 82.34 | 93.87 | 44,914.89 | 100.0% |
| 128 | 2,048 | 4x | 125.37 | 10.77 | 1.72 | 95.40 | 108.35 | 47,180.83 | 100.0% |
| 128 | 2,048 | 8x | 341.78 | 21.42 | 0.66 | 166.10 | 183.24 | 47,921.86 | 100.0% |
| 128 | 2,048 | 16x | 507.51 | 41.08 | 0.45 | 267.32 | 300.71 | 50,194.55 | 100.0% |
| 128 | 2,048 | 32x | 1,049.47 | 79.21 | 0.24 | 246.18 | 404.91 | 51,690.45 | 100.0% |
| 128 | 2,048 | 64x | 1,502.65 | 137.33 | 0.17 | 572.38 | 736.04 | 59,289.11 | 100.0% |
| 128 | 2,048 | 128x | 3,175.12 | 239.08 | 0.10 | 527.09 | 1,067.38 | 68,093.49 | 100.0% |
| 128 | 2,048 | 256x | 4,069.91 | 326.50 | 0.07 | 3,921.75 | 4,456.26 | 99,645.97 | 99.6% |
| 128 | 2,048 | 512x | 3,412.85 | 270.33 | 0.09 | 2,408.95 | 4,874.55 | 237,316.31 | 99.6% |
| 512 | 128 | 1x | 42.11 | 163.16 | 0.87 | 122.34 | 122.34 | 3,040.15 | 100.0% |
| 512 | 128 | 2x | 84.57 | 328.71 | 0.46 | 134.54 | 135.92 | 3,022.41 | 100.0% |
| 512 | 128 | 4x | 162.54 | 627.94 | 0.26 | 186.28 | 210.92 | 3,147.19 | 100.0% |
| 512 | 128 | 8x | 307.23 | 1,185.12 | 0.15 | 333.18 | 375.53 | 3,323.56 | 100.0% |
| 512 | 128 | 16x | 494.28 | 2,112.32 | 0.09 | 477.20 | 665.09 | 3,721.80 | 100.0% |
| 512 | 128 | 32x | 920.24 | 3,585.94 | 0.05 | 639.93 | 1,269.74 | 4,370.13 | 100.0% |
| 512 | 128 | 64x | 1,327.72 | 5,309.90 | 0.04 | 1,019.72 | 2,223.40 | 5,873.71 | 100.0% |
| 512 | 128 | 128x | 1,703.84 | 7,070.54 | 0.03 | 1,862.52 | 4,495.59 | 8,663.99 | 99.2% |
| 512 | 128 | 256x | 1,925.04 | 7,784.95 | 0.03 | 2,968.33 | 8,556.92 | 15,231.40 | 99.6% |
| 512 | 128 | 512x | 1,924.50 | 7,897.10 | 0.03 | 5,269.31 | 16,052.70 | 26,054.14 | 99.4% |
| 512 | 512 | 1x | 43.35 | 52.83 | 2.39 | 60.55 | 60.55 | 9,364.70 | 100.0% |
| 512 | 512 | 2x | 87.49 | 85.01 | 1.28 | 70.46 | 83.18 | 11,701.14 | 100.0% |
| 512 | 512 | 4x | 171.40 | 165.54 | 0.66 | 83.86 | 91.57 | 11,941.06 | 100.0% |
| 512 | 512 | 8x | 309.53 | 329.11 | 0.36 | 116.45 | 130.34 | 11,992.62 | 100.0% |
| 512 | 512 | 16x | 613.37 | 593.96 | 0.20 | 166.61 | 192.61 | 12,498.51 | 93.8% |
| 512 | 512 | 32x | 1,057.27 | 1,233.56 | 0.11 | 179.87 | 198.46 | 12,854.04 | 100.0% |
| 512 | 512 | 64x | 1,528.89 | 1,747.26 | 0.07 | 4,259.35 | 4,367.72 | 18,181.65 | 100.0% |
| 512 | 512 | 128x | 3,372.60 | 3,663.02 | 0.05 | 188.10 | 281.42 | 16,684.77 | 100.0% |
| 512 | 512 | 256x | 3,334.69 | 3,555.79 | 0.05 | 2,994.70 | 8,656.51 | 34,082.19 | 100.0% |
| 512 | 1,024 | 1x | 43.26 | 70.35 | 1.92 | 61.79 | 61.79 | 7,026.31 | 100.0% |
| 512 | 1,024 | 2x | 42.14 | 40.83 | 2.66 | 863.62 | 906.76 | 23,198.44 | 100.0% |
| 512 | 2,048 | 1x | 43.46 | 10.52 | 4.31 | 51.53 | 51.53 | 47,126.23 | 100.0% |
| 512 | 2,048 | 2x | 51.70 | 68.86 | 1.90 | 73.49 | 83.37 | 13,838.71 | 100.0% |
| 512 | 2,048 | 4x | 100.27 | 41.75 | 1.65 | 87.47 | 95.13 | 44,550.29 | 100.0% |
| 512 | 2,048 | 8x | 321.17 | 81.98 | 0.60 | 113.17 | 127.38 | 48,166.78 | 100.0% |
| 512 | 2,048 | 16x | 488.09 | 158.81 | 0.39 | 142.52 | 160.70 | 49,933.75 | 100.0% |
| 512 | 2,048 | 32x | 991.49 | 299.97 | 0.21 | 805.26 | 861.23 | 53,165.25 | 100.0% |
| 512 | 2,048 | 64x | 1,835.62 | 540.92 | 0.13 | 182.63 | 247.64 | 58,736.83 | 100.0% |
| 512 | 2,048 | 128x | 2,857.34 | 851.80 | 0.09 | 1,797.36 | 3,975.00 | 74,388.41 | 100.0% |
| 512 | 2,048 | 256x | 2,982.35 | 862.35 | 0.08 | 2,967.73 | 8,685.29 | 141,902.96 | 100.0% |
| 512 | 2,048 | 512x | 2,618.11 | 803.74 | 0.09 | 11,511.43 | 21,940.83 | 254,397.10 | 81.8% |
| 1,024 | 128 | 1x | 9.69 | 497.45 | 0.12 | 1,519.79 | 1,519.79 | 1,936.12 | 100.0% |
| 1,024 | 128 | 2x | 82.74 | 631.54 | 0.28 | 133.74 | 194.72 | 3,085.61 | 100.0% |
| 1,024 | 128 | 4x | 152.93 | 1,167.56 | 0.16 | 298.57 | 369.67 | 3,339.51 | 100.0% |
| 1,024 | 128 | 8x | 202.10 | 1,680.54 | 0.11 | 1,602.00 | 1,684.22 | 4,651.79 | 100.0% |
| 1,024 | 128 | 16x | 271.02 | 2,198.27 | 0.07 | 2,309.51 | 4,002.79 | 7,111.53 | 100.0% |
| 1,024 | 128 | 32x | 445.74 | 3,745.32 | 0.05 | 1,904.33 | 4,833.67 | 8,042.69 | 96.9% |
| 1,024 | 128 | 64x | 830.59 | 6,837.55 | 0.04 | 2,187.68 | 5,153.33 | 8,906.10 | 98.4% |
| 1,024 | 128 | 128x | 857.53 | 6,828.21 | 0.03 | 4,397.35 | 10,507.61 | 17,954.48 | 99.2% |
| 1,024 | 128 | 256x | 1,032.84 | 8,288.91 | 0.03 | 10,138.35 | 21,984.10 | 29,981.81 | 99.6% |
| 1,024 | 128 | 512x | 1,092.93 | 8,844.22 | 0.03 | 14,013.45 | 43,277.18 | 52,334.98 | 99.8% |
| 1,024 | 128 | 1024x | 685.89 | 5,606.02 | 0.05 | 60,078.96 | 113,340.03 | 128,184.25 | 73.5% |
| 1,024 | 512 | 1x | 31.02 | 1,440.18 | 0.05 | 183.14 | 183.14 | 652.55 | 100.0% |
| 1,024 | 512 | 2x | 85.93 | 163.97 | 0.90 | 128.13 | 188.75 | 11,909.97 | 100.0% |
| 1,024 | 512 | 4x | 156.54 | 320.38 | 0.48 | 291.51 | 367.23 | 12,188.99 | 100.0% |
| 1,024 | 512 | 8x | 321.52 | 622.46 | 0.26 | 477.74 | 601.26 | 12,560.76 | 100.0% |
| 1,024 | 512 | 16x | 590.29 | 1,149.05 | 0.14 | 798.79 | 1,208.26 | 13,638.44 | 100.0% |
| 1,024 | 512 | 32x | 948.33 | 2,060.79 | 0.09 | 1,206.60 | 2,346.03 | 15,166.26 | 100.0% |
| 1,024 | 512 | 64x | 1,505.77 | 3,301.70 | 0.06 | 1,829.48 | 4,394.43 | 18,881.47 | 100.0% |
| 1,024 | 512 | 128x | 2,097.30 | 4,384.79 | 0.05 | 3,303.03 | 9,726.76 | 28,008.01 | 99.2% |
| 1,024 | 512 | 256x | 2,446.92 | 5,275.40 | 0.04 | 5,439.27 | 18,069.73 | 46,143.28 | 100.0% |
| 1,024 | 512 | 512x | 1,825.26 | 4,024.35 | 0.05 | 24,749.71 | 93,250.52 | 119,667.35 | 99.8% |
| 1,024 | 1,024 | 1x | 43.08 | 41.02 | 2.75 | 188.65 | 188.65 | 23,769.04 | 100.0% |
| 1,024 | 1,024 | 2x | 62.24 | 82.34 | 1.60 | 145.88 | 207.36 | 23,067.85 | 100.0% |
| 1,024 | 1,024 | 4x | 168.14 | 160.47 | 0.71 | 296.28 | 370.58 | 24,355.57 | 100.0% |
| 1,024 | 1,024 | 8x | 314.69 | 300.90 | 0.38 | 1,786.46 | 1,863.89 | 26,015.40 | 100.0% |
| 1,024 | 1,024 | 16x | 458.83 | 526.07 | 0.23 | 1,345.14 | 4,549.04 | 29,825.89 | 100.0% |
| 1,024 | 1,024 | 32x | 933.03 | 998.82 | 0.13 | 3,475.68 | 5,066.08 | 31,399.46 | 100.0% |
| 1,024 | 1,024 | 64x | 1,419.52 | 1,772.64 | 0.09 | 2,193.38 | 6,274.14 | 35,307.02 | 100.0% |
| 1,024 | 1,024 | 128x | 2,373.51 | 2,529.94 | 0.06 | 6,542.23 | 12,501.92 | 49,579.72 | 100.0% |
| 1,024 | 1,024 | 256x | 2,608.18 | 3,006.57 | 0.05 | 10,830.80 | 24,957.29 | 81,464.75 | 100.0% |
| 1,024 | 1,024 | 512x | 1,978.49 | 2,256.15 | 0.07 | 14,339.67 | 35,081.08 | 161,952.43 | 75.4% |
| 1,024 | 2,048 | 1x | 42.70 | 20.33 | 3.61 | 978.17 | 978.17 | 47,956.82 | 100.0% |
| 1,024 | 2,048 | 2x | 87.49 | 41.74 | 1.81 | 132.12 | 194.24 | 46,812.05 | 100.0% |
| 1,024 | 2,048 | 4x | 170.36 | 81.29 | 0.93 | 327.71 | 371.53 | 48,079.55 | 100.0% |
| 1,024 | 2,048 | 8x | 267.68 | 161.02 | 0.56 | 478.01 | 604.04 | 48,617.19 | 100.0% |
| 1,024 | 2,048 | 16x | 595.18 | 302.06 | 0.29 | 687.61 | 1,214.41 | 51,970.53 | 100.0% |
| 1,024 | 2,048 | 32x | 936.98 | 559.04 | 0.18 | 1,341.59 | 2,388.55 | 56,127.24 | 100.0% |
| 1,024 | 2,048 | 64x | 1,507.27 | 913.95 | 0.12 | 5,965.82 | 6,993.19 | 67,506.85 | 98.4% |
| 1,024 | 2,048 | 128x | 2,487.84 | 1,493.64 | 0.08 | 3,249.22 | 9,338.93 | 83,896.30 | 100.0% |
| 1,024 | 2,048 | 256x | 2,239.68 | 1,301.36 | 0.09 | 5,993.57 | 19,325.88 | 183,930.16 | 99.6% |
| 2,048 | 128 | 1x | 39.35 | 604.98 | 0.29 | 325.10 | 325.10 | 3,252.94 | 100.0% |
| 2,048 | 128 | 2x | 29.06 | 882.86 | 0.19 | 1,397.48 | 1,529.51 | 4,292.67 | 100.0% |
| 2,048 | 512 | 1x | 43.46 | 167.06 | 1.10 | 66.43 | 66.43 | 11,779.43 | 100.0% |
| 2,048 | 512 | 2x | 59.76 | 332.65 | 0.57 | 82.63 | 92.13 | 11,406.95 | 100.0% |
| 2,048 | 512 | 4x | 163.03 | 624.02 | 0.29 | 367.43 | 633.56 | 12,552.53 | 100.0% |
| 2,048 | 512 | 8x | 307.39 | 1,174.63 | 0.16 | 1,063.43 | 1,213.83 | 13,315.82 | 100.0% |
| 2,048 | 512 | 16x | 510.75 | 2,068.44 | 0.10 | 1,135.89 | 2,330.28 | 15,072.88 | 100.0% |
| 2,048 | 512 | 32x | 790.78 | 3,332.60 | 0.07 | 2,092.53 | 4,577.24 | 18,089.91 | 96.9% |
| 2,048 | 512 | 64x | 1,168.50 | 4,812.39 | 0.05 | 3,425.18 | 9,691.27 | 25,821.04 | 100.0% |
| 2,048 | 512 | 128x | 1,330.64 | 5,654.14 | 0.04 | 11,200.16 | 23,041.55 | 44,109.18 | 100.0% |
| 2,048 | 512 | 256x | 1,151.44 | 4,949.83 | 0.05 | 20,999.01 | 79,174.78 | 99,405.53 | 100.0% |
| 2,048 | 1,024 | 1x | 9.07 | 4,462.59 | 0.05 | 347.24 | 347.24 | 417.20 | 100.0% |
| 2,048 | 1,024 | 2x | 85.97 | 164.50 | 0.92 | 203.37 | 325.81 | 23,816.30 | 100.0% |
| 2,048 | 1,024 | 4x | 146.63 | 280.62 | 0.49 | 3,667.37 | 3,813.33 | 27,924.54 | 100.0% |
| 2,048 | 1,024 | 8x | 262.00 | 612.24 | 0.28 | 837.25 | 1,093.74 | 25,539.01 | 100.0% |
| 2,048 | 1,024 | 16x | 453.08 | 1,088.72 | 0.16 | 2,541.60 | 2,700.34 | 28,715.24 | 100.0% |
| 2,048 | 1,024 | 32x | 766.32 | 1,872.46 | 0.10 | 4,596.24 | 5,711.80 | 33,371.68 | 100.0% |
| 2,048 | 1,024 | 64x | 1,211.88 | 2,868.20 | 0.07 | 4,116.15 | 11,827.31 | 43,470.76 | 100.0% |
| 2,048 | 1,024 | 128x | 1,749.03 | 4,195.99 | 0.05 | 6,169.51 | 18,528.04 | 59,352.02 | 100.0% |
| 2,048 | 1,024 | 256x | 1,329.91 | 2,981.27 | 0.07 | 22,969.04 | 113,885.97 | 152,654.48 | 91.8% |
Runtime parameters used across all benchmark runs
