Phase-Technologies commited on
Commit
0acbc79
·
verified ·
1 Parent(s): 1d4ff47

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +17 -1
README.md CHANGED
@@ -227,4 +227,20 @@ CUDA | 21,212,115 | 17.3300 | 5.50 MiB/s | 1.22 M
227
 
228
  *Note on the CUDA/ROCm Truncation Bug:*
229
  - **Root Cause**: The GPU backends ([gpu_engine_cuda.cu](src/crayon/c_ext/gpu_engine_cuda.cu) and [rocm_engine.hip](src/crayon/c_ext/rocm_engine.hip)) calculated the output buffer token capacity per sequence based on the batch average sentence length, but enforced a strict hardcoded limit of `4096` tokens. Any sequence generating more than 4,096 tokens was silently truncated, leading to data loss on large inputs.
230
- - **What Changed (Fix)**: The static `4096` cap was replaced with dynamic capacity planning. The engine now determines the maximum sentence length in the batch (`max_len`) and sets sequence capacity to `max_len + 64`. To prevent GPU Out-Of-Memory (OOM) failures under massive batch workloads, it enforces a dynamic safety ceiling based on a total allocation budget of 512M elements (~2 GB). This enables processing massive individual documents safely.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
227
 
228
  *Note on the CUDA/ROCm Truncation Bug:*
229
  - **Root Cause**: The GPU backends ([gpu_engine_cuda.cu](src/crayon/c_ext/gpu_engine_cuda.cu) and [rocm_engine.hip](src/crayon/c_ext/rocm_engine.hip)) calculated the output buffer token capacity per sequence based on the batch average sentence length, but enforced a strict hardcoded limit of `4096` tokens. Any sequence generating more than 4,096 tokens was silently truncated, leading to data loss on large inputs.
230
+ - **What Changed (Fix)**: The static `4096` cap was replaced with dynamic capacity planning. The engine now determines the maximum sentence length in the batch (`max_len`) and sets sequence capacity to `max_len + 64`. To prevent GPU Out-Of-Memory (OOM) failures under massive batch workloads, it enforces a dynamic safety ceiling based on a total allocation budget of 512M elements (~2 GB). This enables processing massive individual documents safely.
231
+
232
+ ## Citation
233
+
234
+ If you use CRAYON in your research or applications, please cite it using the following BibTeX entry:
235
+
236
+ ```bibtex
237
+ @misc{pal2026crayon,
238
+ author = {Pal, Soham},
239
+ title = {{CRAYON: Hardware-Accelerated BPE Tokenization via Zero-Copy Double-Array Tries and SIMD Vectorization}},
240
+ month = jun,
241
+ year = 2026,
242
+ publisher = {Xerv Research \& Engineering Division},
243
+ doi = {10.5281/zenodo.20761376},
244
+ url = {https://doi.org/10.5281/zenodo.20761376}
245
+ }
246
+ ```