learned-sparse retriever (SPLADE doc-v3-gte S1)

97c1846 verified 5 days ago

54.5 kB

tags:
  - sentence-transformers
  - sparse-encoder
  - sparse
  - asymmetric
  - inference-free
  - splade
  - generated_from_trainer
  - dataset_size:46954
  - loss:CachedSpladeLoss
  - loss:SparseMultipleNegativesRankingLoss
  - loss:FlopsLoss
base_model: opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte
widget:
  - text: >
      title: Partial Matrix Completion


      summary: This paper considers a twist on the standard matrix completion
      problem where one is required to only

      complete a subset of entries (not the whole matrix) that includes the
      entries shown. This allows them to consider

      substantially more observation patterns unlike the standard
      missing-at-random response. In this context

      the paper makes the following contributions:


      1. Propose a computationally-inefficient algorithm that with high
      probability recovers a subset of entries

      that is at least as large as the revealed set, with low target accuracy.

      2. Develop a computationally-efficient relaxation of this algorithm that
      has worse statistical dependence

      on the target accuracy than the inefficient algorithm.

      3. Provide an online variant of the algorithm that is iterative/gradient
      based that applies to adversarial

      online matrix completion.



      weaknesses and questions: Some weaknesses, though these are better
      understood as interesting avenues for future work:


      1. The proposed algorithms provide noisy completion even when the
      observations are noiseless (i.e. no obvious notion of completion to the
      inherent noise level). 

      This does seem inherent to the algorithm/proof techniques used here, it is
      unclear to me that this can be overcome here. 

      2. There is an obvious sample complexity gap between the efficient and
      inefficient algorithms. Is this inherent, or a result of the proof
      technique here?




      1. The algorithmic viewpoint here is to separate the coverage computation
      from the completion method. This is advantageous in some ways (notably the
      generality and e.g. obtaining an essentially free proof of the risk), but
      potentially disallows using structure in the completion. E.g. in the
      causal inference setting of (say) row $1$ being partially revealed from
      columns $1, 2\ldots, K \leq \text{dim}(M)$.

      2. What is $\mu_\max$? It is defined in the appendix (and easy to guess)
      but is not in the text (unless I missed)
  - text: >-
      title: IncVGGT: Incremental VGGT for Memory-Bounded Long-Range 3D
      Reconstruction


      summary: This work presents IncVGGT, a training-free incremental variant
      of VGGT designed to address the quadratic memory growth and excessive
      computation issues in large-scale visual geometry transformers. The method
      tackles redundancy from both the input and history sides: on the input
      side, it performs registration and composition to merge short temporal
      windows into compact composite views; on the history side, it retains only
      the top-k historically important slots together with the most recent one
      for the next step. Experimental results demonstrate notable improvements
      in both inference time and memory efficiency.


      weaknesses and questions: - The registration-based redundancy reduction
      component has been widely applied in the community (e.g., [1]). Moreover,
      introducing a separate registration-and-composition preprocessing module
      somewhat compromises the clean and unified design philosophy of VGGT,
      which originally aimed to eliminate all explicit priors.

      A more elegant approach would be to integrate the registration and
      composition mechanism directly into VGGT’s internal architecture,
      preserving its end-to-end structure.


      - Limited Novelty: While the contributions toward improving memory and
      speed are meaningful for industrial and practical applications, the
      registration-based redundancy reduction and global-local cache pruning
      techniques are incremental relative to VGGT. The authors themselves
      describe IncVGGT as an “incremental variant” of VGGT, suggesting that it
      contributes less conceptual novelty to academic research, even though it
      provides clear engineering value.


      [1] Xiaoshui Huang, Guofeng Mei, Jian Zhang, and Rana Abbas. A
      Comprehensive Survey on Point Cloud Registration.

      N/A
  - text: >-
      title: Beyond Safe Answers: A Benchmark for Evaluating True Risk Awareness
      in Large Reasoning Models


      summary: This paper investigates an important phenomenon: models produce
      superficially safe outputs while internal reasoning processes harmful
      content, and summary this as SSA (Superficial Safety Alignment (SSA)). The
      authors introduce a novel benchmark: Beyond Safe Answers (BSA) with over
      2000 instances and report results on 23 reasoning models. The evaluation
      results show significant emergency to improve reasoning models' internal
      safety as their think process is also open to the public (except some
      close-source reasoning models like o3).


      weaknesses and questions: 1. The presentation could be improved. Line
      127-132 has some space that making this page appear somewhat empty.


      2. How many GPU hours, total tokens, and dollar cost does one evaluation
      pipeline consume?

      Please see the weakness.
  - text: >-
      title: Physics of Language Models: Part 2.2, How to Learn From Mistakes on
      Grade-School Math Problems


      summary: This paper studies the impact of training language models with
      "error-then-retry" format data on the reasoning performance. With
      experiments on a synthetic GSM8k-style math reasoning datasets, the
      authors conclude that pretraining with such data improves reasoning
      accuracy than on same amount of "error-free" normal data, and LoRA
      fine-tuning with such data does not help.


      weaknesses and questions: The paper mainly experiments with one type of
      error: inserting a wrong parameter that cannot be computed next. While
      this is easy to implement, it would be hard to simulate all kinds of
      errors that language models can make in real-world reasonings scenarios.
      This cast doubts on how the suggested approach can be deployed to train
      LMs on non-synthetic data. If the authors could provide some discussion on
      how the method can be generalized to different errors (e.g., math
      calculation error, context misunderstanding, ...) in a scalable and
      controllable way, that would be very promising.

      I would like to hear the authors' thoughts on the inference-time scaling
      properties of model trained with "retry" data: given a fixed
      inference-time computation budget, is it better for model to produce
      reasoning chains with retry (more tokens to reach final answer, but
      average accuracy is higher) or without retry (less tokens to reach final
      answer, but average accuracy is lower)?
  - text: >-
      title: Generalizable Multi-Camera 3D Object Detection from a Single Source
      via Fourier Cross-View Learning


      summary: This paper proposed a Fourier Cross-View Learning (FCVL)
      framework, which augments the data in the frequency domain and includes a
      contrastive-style semantic consistency loss to improve the model 
      generalization ability from a single source.


      weaknesses and questions: Pros:

      1. This paper augments the data in frequency domain by jitterring both
      amplitude and phase, which diversify the dataset. The phase term typically
      captures high-frequency features that may be more transferable.

      2. The designed contrastive loss, which utilizes the adjacent image
      regions, is also a good idea.

      3. The proposed method is adaptable to different approaches and achieved
      SOTA results with various baseline models.

      4. The t-SNE visualization and other visualization results validate that
      the learned features are domain-invariant and the images become more
      diverse.


      Cons:

      1. There may be some false negative samples for the contrastive loss
      design since some large vehicles may distribute across several adjacent
      frames.

      1. Regarding the semantic consistency loss, it appears to work well for
      small vehicles. But what happens when the vehicle is large? Will the rear
      part of a large vehicle be treated as a negative sample of the front part?


      2. Besides large vehicles, what if there are other vehicles in the
      background image, will they also be regarded as negative samples?


      3. Could you also show some examples where the proposed model still cannot
      detect correctly? How to further improve the model the future?


      4. The idea of augmenting data in the frequency domain is not new and have
      been tried by previous researchers. What are advantages of the proposed
      frequency domain augmentation?
pipeline_tag: feature-extraction
library_name: sentence-transformers
metrics:
  - dot_accuracy@1
  - dot_accuracy@10
  - dot_precision@10
  - dot_precision@100
  - dot_recall@10
  - dot_recall@100
  - dot_ndcg@10
  - dot_mrr@10
  - dot_map@100
  - query_active_dims
  - query_sparsity_ratio
  - corpus_active_dims
  - corpus_sparsity_ratio
  - avg_flops
model-index:
  - name: Asymmetric Inference-free SPLADE Sparse Encoder
    results:
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: reviewsearch
          type: reviewsearch
        metrics:
          - type: dot_accuracy@1
            value: 0.40155296727676093
            name: Dot Accuracy@1
          - type: dot_accuracy@10
            value: 0.7879796299097463
            name: Dot Accuracy@10
          - type: dot_precision@10
            value: 0.21599858821156656
            name: Dot Precision@10
          - type: dot_precision@100
            value: 0.06021781878687037
            name: Dot Precision@100
          - type: dot_recall@10
            value: 0.19603705746446795
            name: Dot Recall@10
          - type: dot_recall@100
            value: 0.47140266480788
            name: Dot Recall@100
          - type: dot_ndcg@10
            value: 0.2849832143549761
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.5243879048787551
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.18859069597551725
            name: Dot Map@100
          - type: query_active_dims
            value: 7.151616096496582
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9997656897943615
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 902.127325119251
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9704433744473083
            name: Corpus Sparsity Ratio
          - type: avg_flops
            value: 1.6038551330566406
            name: Avg Flops

Asymmetric Inference-free SPLADE Sparse Encoder

This is a Asymmetric Inference-free SPLADE Sparse Encoder model finetuned from opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte using the sentence-transformers library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

Model Type: Asymmetric Inference-free SPLADE Sparse Encoder
Base model: opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte
Maximum Sequence Length: 512 tokens
Output Dimensionality: 30522 dimensions
Similarity Function: Dot Product
Supported Modality: Text

Model Sources

Documentation: Sentence Transformers Documentation
Documentation: Sparse Encoder Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): Router(
    default_route='document'
    (sub_modules): ModuleDict(
      (query): Sequential(
        (0): SparseStaticEmbedding({'frozen': False}, dim=30522, tokenizer=DistilBertTokenizer)
      )
      (document): Sequential(
        (0): Transformer({'transformer_task': 'fill-mask', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'token_embeddings', 'architecture': 'NewForMaskedLM'})
        (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'log1p_relu', 'embedding_dimension': 30522})
      )
    )
  )
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("sparse_encoder_model_id")
# Run inference
sentences = [
    'video detector frequency bias',
    'title: Seeing What Matters: Generalizable AI-generated Video Detection with Forensic-Oriented Augmentation\n\nsummary: Recently, with the rapid development of video generative models, forensic detectors have also emerged; however, many of them lack generalizability. To address this issue, the authors propose a method that first identifies discriminative features that are less biased toward forensic-irrelevant patterns and more robust for detecting generated videos. Based on frequency domain analysis and previous literature, the authors identify that mid-high frequency components are less susceptible to compression artifacts and are thus suitable for forensic detection. To utilize these cues, two novel augmentation strategies are introduced during training: (1) Injection of forensic cues, by reconstructing real videos using the same autoencoder that generated the fake videos, aligning both real and fake samples in the representation space, and (2) Wavelet-based augmentation, by mixing real and fake videos using wavelet decomposition to enhance the mid-high frequency learning and push the detector to focus on those features. In the experimental evaluation, the model was trained using videos generated by the Pyramid Flow model, and testing was performed on both the GenVideo dataset (publicly available) and a newly constructed dataset comprising 2,400 videos from recent generative models. Accuracy was used as the evaluation metric, and the proposed method outperformed recent forensic detection methods.\n\nweaknesses and questions: 1. Quality: \nThe submission is technically sound, presenting a relevant formulation and effective use of forensic feature extraction from videos. The claims are well-supported by experimental results. The methods employed are appropriate, and the work is presented as a complete and cohesive study. The authors are transparent about both the strengths and limitations of their approach.\n\n2. Clarity:\nThe submission is clearly written and easy to follow.\n\n3. Significance:\nThe results are impactful for the community. While the code and new dataset have not been released, if shared, this work has strong potential to be used by other researchers for further development and benchmarking.\n\n4. Originality:\nThe work integrates several ideas from previous methods, with appropriate citations. Although the application of these ideas to video forensics is a novel context, the core techniques themselves are adapted rather than newly proposed. Therefore, the method is not entirely novel in its formulation.\na. In lines 186–193, the authors state that replacing the low-frequency bands with those from the real counterpart forces the detector to focus on mid-high frequency features. However, how can we confidently claim that the detector will focus on mid-high frequencies? It is possible that the detector still learns from the low-frequency content provided by the real counterpart, potentially leading to misleading or incorrect detection. Do the authors have empirical evidence or observations supporting this assumption?\nb. The proposed method appears to heavily rely on the reconstruction quality of the Pyramid Flow model. While the paper mentions that this model generally reconstructs videos without visible artifacts, what happens if it fails in certain cases? Would such failure compromise the robustness or generalizability of the detector trained using this reconstruction-based augmentation?\nc. The paper makes valuable contributions, including a new dataset and a promising method. However, neither the code nor the dataset has been made available. The authors are strongly encouraged to release both the code and dataset to enable transparent evaluation, reproducibility, and faster progress in the rapidly evolving field of forensic video detection.',
    'title: FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies\n\nsummary: This paper proposes a fusion-based object detection pipeline that attempts to promote event+RGB fused object detection at varying frequencies. More specifically, a FlexFuse module is deigned to align the rich semantic information from RGB frames with the high-frequency event data, a FlexTune strategy is proposed to generate frequency-adjusted labels for unlabelled event streams. Experiments on the DSEC datasets demonstrate a superior performance of the proposed method over previous fusion-based methods. Also, the model maintains a robust performance across events of varying frequencies (up to 180 Hz).\n\nweaknesses and questions: Pros:\n- Fusing the complementary advantages of events and RGB images for object detection is a promising direction that can power applications that are sensitive to operational latency, for example, autonomous driving, industrial anomaly detection. In this context, exploring the robust detection across varying event frequencies and pushing its speed upper bound is important.\n- The proposed FlexTune strategy extends the classic pseudo label strategy to the event-RGB fused object detection, where high-frequency events are treated as unlabeled images, and several strategies grounded on the characteristics of events are deaigned to generate pseudo labels to boost the performance across different frequencies. \n- Experiments on the DSEC datasets (Car and Pedestrian classes) demonstrate the significant performance improvements of the proposed method. Meanwhile, the proposed methods maintain robustness across varying frequencies.\n- This paper is overall well-structured and easy to follow.\n\nCons:\nAlthough the task and method sound compelling, I have some major concerns about the data configuration and experimental comparison:\n- The quality of “high-frequency events” and the resulting performance degradation seems to be dependent on the proposed way of generating event frames. Section 3.2 mentions that the event is divided into several sub-intervals and aggregated within each interval to simulate high-frequency scenarios, while this setting may make the definition of high-frequency events problematic. For example, we can simply aggregate information for a longer period for each event aggregation, while still maintaining a low temporal gap between two sampling points. It will be helpful to provide the results on this setting with both high-frequency events and richer semantic information.\n- In the experimental section, this paper didn’t compare with models with solely RGB input. Therefore, it is hard to see the advantages of Event+RGB fusion compared to only using events or RGB frames. Also, it is not clear why only one fusion-based method is compared on the DSEC-Det dataset while more are compared on other datasets. \n- Although three datasets have been used for experiments, results on only 3 classes (Car, Pedestrian L-Veh.) are provided across all these datasets. This makes it hard to verify the methods’ generalization ability across more classes and scenarios, especially considering that pretrained weights are used during experiments.\n- It is not explained why using two significantly different backbones for events (RVT) and RGB frames (ResNet-50). Moreover, both pretrained weights are used for RVT and ResNet50 while it is not clear what are the pretrained datasets and how much they would contribute to the final performance.\nMy major concern is about the configuration of high-frequency events and the fairness of experimental comparison, I could reassess my score if these weaknesses are properly addressed.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 30522]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[16.0392,  9.8499,  6.5218],
#         [ 9.8499, 55.4061, 15.9871],
#         [ 6.5218, 15.9871, 58.2849]])

Evaluation

Metrics

Sparse Information Retrieval

Dataset: reviewsearch
Evaluated with SparseInformationRetrievalEvaluator

Metric	Value
dot_accuracy@1	0.4016
dot_accuracy@10	0.788
dot_precision@10	0.216
dot_precision@100	0.0602
dot_recall@10	0.196
dot_recall@100	0.4714
dot_ndcg@10	0.285
dot_mrr@10	0.5244
dot_map@100	0.1886
query_active_dims	7.1516
query_sparsity_ratio	0.9998
corpus_active_dims	902.1273
corpus_sparsity_ratio	0.9704
avg_flops	1.6039

Training Details

Training Dataset

Unnamed Dataset

Size: 46,954 training samples
Columns: anchor, positive, negative_1, negative_2, and negative_3

Approximate statistics based on the first 100 samples:

	anchor	positive	negative_1	negative_2	negative_3
type	string	string	string	string	string
modality	text	text	text	text	text
details	min: 5 tokens mean: 8.65 tokens max: 14 tokens	min: 197 tokens mean: 421.56 tokens max: 512 tokens	min: 161 tokens mean: 435.95 tokens max: 512 tokens	min: 171 tokens mean: 451.66 tokens max: 512 tokens	min: 104 tokens mean: 431.57 tokens max: 512 tokens

Samples:

anchor	positive	negative_1	negative_2	negative_3
`meta-learning unclear contribution`	title: Extending Contextual Self-Modulation: Meta-Learning Across Modalities, Task Dimensionalities, and Data Regimes summary: This paper proposes to extend Contextual Self-Modulation (CSM), which is an uncertainty-handling mechanism of Neural Context Flow (NCF). NCF is a robust meta-learning framework of neural ODE that includes self-modulation with high-order Taylor expansion around contexts. The authors focus on generalizing CSM into high data regimes and designed StochasticNCF and FlashCAVIA, which take CSM in various directions. From comparative experiments, the author verified that the NCF can be successfully extended to various optimization and meta-learning techniques. weaknesses and questions: 1. I believe this paper is not clearly written. * iCSM: It is briefly mentioned that the space of iCSM is "an infinite-dimensional variation" that utilizes "a space of multi-layer perceptrons, whose weights are flattened into a 1-dimensional tensor." I do not find a good enough ex...	title: Generalizing to Unseen Domains for Regression summary: The authors tackle domain generalization for regression tasks, or DGR. DGR is often different from classification tasks in DG as well as imbalanced regression tasks due to shift in labels. Popular feature alignment type of algorithms using IPM may not be applicable to DGR because the labels of different domains may not align, which may cause a model to fit to only one domain. The authors propose margin-aware meta learning framework to solve DGR by weighting query based on the distance to support, which mitigates sampling bias. The authors provide experiment results on multiple datasets compared to several feature alignment methods in domain adaptation as well as self-supervised learning and meta-learning methods. weaknesses and questions: 1. The relationship between the margin and hardness that the authors demonstrated is not something new. It is somewhat obvious. For example, in Gaussian Processes, the uncertainty incre...	title: The Meta-Representation Hypothesis summary: The paper proposes to combine Deep Mutual Learning with RL. In Deep Mutual Learning, several learners learn independently but at the same try to minimize the KL between their predictive distributions. The paper hypothesizes that two RL policies can learn from different MDPs — where each MDP has its own randomly sampled observation function while the policies try to minimize the KL between them. This would lead to the learning of robust representation functions. The randomly perturbed observation function is a key aspect of the paper — in their paper they apply a CNN with random weights to the observation to map the true observation to a perturbed one. The paper tests this hypothesis via PPO and shows that Deep Mutual Learning is helpful for generalization on the Procgen Benchmark. weaknesses and questions: ## Pros 1. Tackles an important problem about having a robust perception function for RL. 2. A positive thing is that the whole ...	title: Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliance summary: Paper proposes a graph metanetwork that edits trained MLPs in one forward pass to satisfy requirements (data minimization, equalized-odds fairness, pruning) while preserving behavior. It outputs masks and residuals, trained over model families with JSD + requirement losses, yielding faster, better Pareto trade-offs on Adult/Bank. weaknesses and questions: 1. For each dataset, each requirement, and each weighting coefficient, a separate model must be designed and trained. 2. Given the data-hungry training (needing many trainings to construct weight samples) and poor cross-dataset generalization, the claim of “no costly retraining” seems hard to stand. 3. The datasets are extremely limited and very low-dimensional, which makes me worry about applicability to higher-dimensional real-world settings. I’d like to know how it performs on higher-dimensional datasets, and how much the final res...
`meta-learning scalability benchmarks`	title: Extending Contextual Self-Modulation: Meta-Learning Across Modalities, Task Dimensionalities, and Data Regimes summary: This paper proposes to extend Contextual Self-Modulation (CSM), which is an uncertainty-handling mechanism of Neural Context Flow (NCF). NCF is a robust meta-learning framework of neural ODE that includes self-modulation with high-order Taylor expansion around contexts. The authors focus on generalizing CSM into high data regimes and designed StochasticNCF and FlashCAVIA, which take CSM in various directions. From comparative experiments, the author verified that the NCF can be successfully extended to various optimization and meta-learning techniques. weaknesses and questions: 1. I believe this paper is not clearly written. * iCSM: It is briefly mentioned that the space of iCSM is "an infinite-dimensional variation" that utilizes "a space of multi-layer perceptrons, whose weights are flattened into a 1-dimensional tensor." I do not find a good enough ex...	title: OPTFM: A Scalable Multi-View Graph Transformer for Hierarchical Pre-Training in Combinatorial Optimization summary: The paper proposes a foundation model for combinatorial optimization with a focus on mixed integer programming. Specifically, the model is trained on the factor (bipartite) graph representation of MIPs. The paper uses a novel efficient transformer architecture. The pretraining has two phases. In the first, the graph is partitioned into subgraphs and the model learns to predict which constraints each variable is connected to. In the second, the model is trained in contrastive fashion to distinguish between pairs of subgraphs that belong to the same instance vs pairs that belong to different instances. Finally, once the variables have been embedded those embeddings are used for several different downstream tasks. The method shows strong empirical results on solving combinatorial optimization instances and on solver configuration tasks. weaknesses and questions: ## ...	title: Large-Scale Pretraining Offers Modest Benefits for Tabular Transfer Learning summary: The paper evaluates seven open-source tabular foundation models (TFMs) on 88 classification and 82 regression datasets, in both full-data and few-shot regimes, to check whether the performance gains reported in prior work actually hold under stricter evaluation. It shows that common practices—min–max normalizing metrics and only doing rank-based tests—can overstate the benefits of tabular pretraining; when performance is kept on the original scale and per-dataset bootstrap significance is applied, TFMs are usually tied with strong tree/NN baselines like CatBoost, only clearly winning on small classification tables. The authors also compare three TFMs (TabuLa-8B, TP-BERTa, XTab) with their non-pretrained counterparts and find that pretraining mainly helps LLM-style tabular ICL, but brings little or no benefit to other architectures, so the overall gains from large-scale tabular pretraining are...	title: Scaling Laws for Pre-training Agents and World Models summary: This paper investigates the existence and characteristics of scaling laws in embodied AI tasks, specifically world modeling (WM) and behavior cloning (BC), drawing parallels to scaling laws observed in large language models (LLMs). Through extensive experiments on large-scale video game datasets (e.g., Bleeding Edge) and robotics data (RT-1), the authors demonstrate that power-law relationships between model size, dataset size, compute, and pre-training loss also apply to WM and BC. Key findings include: - Scaling laws for WM are influenced by tokenizer compression rates. Higher compression rates lead to more reliance on dataset size. - BC with tokenized observations requires prioritizing dataset scaling under modest compute budgets, while BC with CNN-based architectures favors model scaling. - Small-scale language modeling experiments validate mechanisms behind observed scaling phenomena (e.g., sparse supervisio...
`anchor quality upper bounds training`	title: Tournament Style RL: Stabilizing Policy Optimization on Non Verifiable Problems summary: The paper proposes a novel tournament reward calculated against a given set of anchor answers to generate reward supervision for LLM training in tasks without verifiable rewards. For each input prompt, a set of anchors is generated before training using a stronger LLM and ranked. Then the generated answers are compared against this ranked set of anchors to generate a reward for each answer which is then used for GRPO fine-tuning. weaknesses and questions: Strength 1. the proposed method is well motivated and clearly presented 2. can be easily implemented upon GRPO style fine-tuning pipelines 3. is robust against noise in evaluator LLMs Weakness 1. The proposed method relies heavily on anchor model quality. And the score itself will saturate if the model being fine-tuned surpasses the anchor model's quality. On the other hand, the performance of anchor model upper limits the model ...	title: On Proper Learnability between Average- and Worst-case Robustness summary: This paper studies the setting of robust PAC learning to test time attacks, using a relaxed notion of robustness on average instead of robustness to the worst-case attack. The contributions are as follows. -Negative result: even when using the relaxed notion of robustness, improper learning is impossible. This is a stronger result from the example in the worst-case setting [Montasser et al. 2019]. Moreover, this is achieved by a natural example of $\ell_p$ balls and the uniform measure. The intuition is that the non-Lipschitzness of the 0-1 loss enforces to use improper learning. -Positive results: 1. When considering Lipschitz loss functions, uniform convergence hold, and as a result, ERM is sufficient for learnability. 2. Instead of relaxing the robust loss, it is possible to relax the benchmark we compare to, i.e. the best function in the class but with a smaller parameter in the probabilist...	title: Spectral Perturbation Bounds for Low-Rank Approximation with Applications to Privacy summary: The paper presents the error bound (in terms of the spectral norm) for the rank-$p$ approximation of a symmetric matrix perturbed with a symmetric noise. The paper is written nicely with a clear motivation and description. The mathematical tools used to solve this hard problem namely contour bootstrapping are interesting in their own right and the authors seem to have found a novel application for this tool in this work. The simulation results also seem to highlight the utility of the derived bound when compared to classical baselines. weaknesses and questions: Strengths: 1. The derived bound is tighter than the EYM bound and leverages the symmetry of data and noise matrices. Also, the presented bounds are high probability asymptotic bounds, which are preferable over the bounds on expectation. 2. The paper adopts a new analytical approach, contour bootstrapping. Weaknesses: 1. The ...	title: Utility Boundary of Dataset Distillation: Scaling and Configuration-Coverage Laws summary: This paper proposes a unified configuration–dynamics–error framework that integrates gradient, distribution, and trajectory matching within a generalization-error analysis. It establishes the scaling law and coverage law linking distilled sample size to performance and configuration diversity, theoretically and empirically unifying major dataset distillation methods. weaknesses and questions: 1. The framework relies on PL conditions and Lipschitz continuity. While these assumptions are standard in convergence analysis, they may not strictly hold for modern deep networks with non-smooth activations, normalization layers, and stochastic training components. The practical relevance of the theoretical results could be further clarified by discussing their validity under relaxed or empirically realistic assumptions. 2. The validation of the proposed laws relies mainly on curve-fitting without...

Loss: CachedSpladeLoss with these parameters:

{
    "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=True, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)",
    "document_regularizer_weight": 0.003,
    "mini_batch_size": 128
}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 1024
num_train_epochs: 1.0
learning_rate: 2e-05
warmup_steps: 0.1
bf16: True
eval_on_start: True
router_mapping: {'anchor': 'query', 'positive': 'document', 'negative_1': 'document', 'negative_2': 'document', 'negative_3': 'document'}
learning_rate_mapping: {'0\.sub_modules\.query\.0\.weight': 0.001}

All Hyperparameters

Click to expand

per_device_train_batch_size: 1024
num_train_epochs: 1.0
max_steps: -1
learning_rate: 2e-05
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_steps: 0.1
optim: adamw_torch_fused
optim_args: None
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: True
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: None
trackio_bucket_id: None
trackio_static_space_id: None
per_device_eval_batch_size: 8
prediction_loss_only: True
eval_on_start: True
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: False
hub_private_repo: None
hub_model_id: None
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: False
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: True
dataloader_num_workers: 0
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
remove_unused_columns: True
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_static_graph: None
ddp_backend: None
ddp_timeout: 1800
fsdp: None
fsdp_config: None
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
warmup_ratio: None
local_rank: -1
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {'anchor': 'query', 'positive': 'document', 'negative_1': 'document', 'negative_2': 'document', 'negative_3': 'document'}
learning_rate_mapping: {'0\.sub_modules\.query\.0\.weight': 0.001}

Training Logs

Epoch	Step	Training Loss	reviewsearch_dot_ndcg@10
0	0	-	0.2377
0.0909	2	1.7752	-
0.1818	4	1.6872	-
0.2727	6	1.4462	0.2689
0.3636	8	1.3025	-
0.4545	10	1.2209	-
0.5455	12	1.1711	0.2882
0.6364	14	1.1102	-
0.7273	16	1.1213	-
0.8182	18	1.0570	0.2865
0.9091	20	1.0478	-
1.0	22	1.0844	0.2850

Training Time

Training: 8.1 minutes
Evaluation: 16.8 minutes
Total: 25.0 minutes

Framework Versions

Python: 3.12.9
Sentence Transformers: 5.6.0
Transformers: 5.12.1
PyTorch: 2.8.0+cu128
Accelerate: 1.14.0
Datasets: 5.0.0
Tokenizers: 0.22.2

Additional Resources

Training and Finetuning Sparse Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning SPLADE and other sparse encoder models.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedSpladeLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
@misc{formal2022distillationhardnegativesampling,
    title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
    author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and St\'ephane Clinchant},
    year={2022},
    eprint={2205.04733},
    archivePrefix={arXiv},
    primaryClass={cs.IR},
}

SparseMultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}