[TensorRT-LLM] TensorRT LLM version: 1.2.0rc8
max_position_embeddings: 8192
[01/15/2026-00:32:20] [TRT-LLM] [I] Using LLM with PyTorch backend
[01/15/2026-00:32:20] [TRT-LLM] [W] Using default gpus_per_node: 8
[01/15/2026-00:32:20] [TRT-LLM] [I] neither checkpoint_format nor checkpoint_loader were provided, checkpoint_format will be set to HF.
[TensorRT-LLM] TensorRT LLM version: 1.2.0rc8
[TensorRT-LLM] TensorRT LLM version: 1.2.0rc8
[TensorRT-LLM] TensorRT LLM version: 1.2.0rc8
[TensorRT-LLM] TensorRT LLM version: 1.2.0rc8
[TensorRT-LLM] TensorRT LLM version: 1.2.0rc8
[TensorRT-LLM] TensorRT LLM version: 1.2.0rc8
[TensorRT-LLM] TensorRT LLM version: 1.2.0rc8
[TensorRT-LLM] TensorRT LLM version: 1.2.0rc8
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
Model init total -- 153.34s
Model init total -- 153.28s
Model init total -- 153.49s
Model init total -- 153.71s
Model init total -- 153.61s
Model init total -- 153.41s
Model init total -- 153.51s
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=6399, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 35.93 GiB for max tokens in paged KV cache (204768).
Model init total -- 154.18s
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=6399, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 35.93 GiB for max tokens in paged KV cache (204768).
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=6399, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=6399, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 35.93 GiB for max tokens in paged KV cache (204768).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 35.93 GiB for max tokens in paged KV cache (204768).
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=6399, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 35.93 GiB for max tokens in paged KV cache (204768).
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=6399, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 35.93 GiB for max tokens in paged KV cache (204768).
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=6399, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 35.93 GiB for max tokens in paged KV cache (204768).
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=6399, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 35.93 GiB for max tokens in paged KV cache (204768).
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 2050
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 2048
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 2049
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 2051
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 2052
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 2054
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 2053
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 2055
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=15907, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=15907, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=15907, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=15907, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=15907, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=15907, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=15907, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 6337 [window size=202753], tokens per block=32, primary blocks=15907, secondary blocks=0, max sequence length=202753
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 89.32 GiB for max tokens in paged KV cache (509024).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 89.32 GiB for max tokens in paged KV cache (509024).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 89.32 GiB for max tokens in paged KV cache (509024).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 89.32 GiB for max tokens in paged KV cache (509024).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 89.32 GiB for max tokens in paged KV cache (509024).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 89.32 GiB for max tokens in paged KV cache (509024).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 89.32 GiB for max tokens in paged KV cache (509024).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 89.32 GiB for max tokens in paged KV cache (509024).
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 134309376 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 0 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 134309376 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 134309376 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 134309376 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 134309376 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 134309376 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 134309376 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 134309376 bytes to 311427072 bytes
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 134309376 bytes to 311427072 bytes
trt-llm (tokenizer=/root/work/huggingface/hub/models--zai-org--GLM-4.7/snapshots/475d85cda16beac79cde7f4cf4cae8d1260566f5,checkpoint_dir=/root/work/quantized_models//saved_models_475d85cda16beac79cde7f4cf4cae8d1260566f5_nvfp4_kv_fp8,max_gen_toks=4096), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 64
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9348|±  |0.0068|
|     |       |strict-match    |     5|exact_match|↑  |0.9325|±  |0.0069|