naver

/

provence-reranker-debertav3-v1

like 71

Follow

NAVER 225

Hello,

I’ve started experimenting with a Rust implementation of this model using Candle, mostly for fun, and you can find the early code here: https://github.com/matthewhaynesonline/provence-rs. It’s still a rough proof of concept and nowhere near feature parity yet, but I wanted to share it early before going too far in the wrong direction.

A few rough edges at the moment:

1. Splitting: right now I’m splitting contexts before tokenization (based on char count as a proxy for token limit).
   1. Not ideal but it was just a quick way to get something working before proper token level splitting.
2. Batching performance: on Metal / Candle, batching inputs together seems unexpectedly slower than sequential processing.
   1. Larger batches seem to create notable latency spikes (~2x - 3x slower).
   2. Probably a code bug, but I was going in circles and planning to revisit later and benchmark on CUDA.
   3. The sequential approach seems to be roughly on par with the pytorch implementation (in my limited testing)
3. Sentence splitting: the punkt rs crate seems abandoned, so I’m using a fork as well as having a basic punctuation whitespace splitter (feature flagged).

I’d be interested to hear what the you think, whether this sort of port aligns with how you imagine the model being used and if there are any thoughts / concerns. Thanks!