yjoonjang/reviewsearch-sparse

Asymmetric Inference-free SPLADE Sparse Encoder

This is a Asymmetric Inference-free SPLADE Sparse Encoder model finetuned from opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte using the sentence-transformers library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

• Model Type: Asymmetric Inference-free SPLADE Sparse Encoder
• Base model: opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 30522 dimensions
• Similarity Function: Dot Product
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): Router(
    default_route='document'
    (sub_modules): ModuleDict(
      (query): Sequential(
        (0): SparseStaticEmbedding({'frozen': False}, dim=30522, tokenizer=DistilBertTokenizer)
      )
      (document): Sequential(
        (0): Transformer({'transformer_task': 'fill-mask', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'token_embeddings', 'architecture': 'NewForMaskedLM'})
        (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'log1p_relu', 'embedding_dimension': 30522})
      )
    )
  )
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("sparse_encoder_model_id")
# Run inference
sentences = [
    'video detector frequency bias',
    'title: Seeing What Matters: Generalizable AI-generated Video Detection with Forensic-Oriented Augmentation\n\nsummary: Recently, with the rapid development of video generative models, forensic detectors have also emerged; however, many of them lack generalizability. To address this issue, the authors propose a method that first identifies discriminative features that are less biased toward forensic-irrelevant patterns and more robust for detecting generated videos. Based on frequency domain analysis and previous literature, the authors identify that mid-high frequency components are less susceptible to compression artifacts and are thus suitable for forensic detection. To utilize these cues, two novel augmentation strategies are introduced during training: (1) Injection of forensic cues, by reconstructing real videos using the same autoencoder that generated the fake videos, aligning both real and fake samples in the representation space, and (2) Wavelet-based augmentation, by mixing real and fake videos using wavelet decomposition to enhance the mid-high frequency learning and push the detector to focus on those features. In the experimental evaluation, the model was trained using videos generated by the Pyramid Flow model, and testing was performed on both the GenVideo dataset (publicly available) and a newly constructed dataset comprising 2,400 videos from recent generative models. Accuracy was used as the evaluation metric, and the proposed method outperformed recent forensic detection methods.\n\nweaknesses and questions: 1. Quality: \nThe submission is technically sound, presenting a relevant formulation and effective use of forensic feature extraction from videos. The claims are well-supported by experimental results. The methods employed are appropriate, and the work is presented as a complete and cohesive study. The authors are transparent about both the strengths and limitations of their approach.\n\n2. Clarity:\nThe submission is clearly written and easy to follow.\n\n3. Significance:\nThe results are impactful for the community. While the code and new dataset have not been released, if shared, this work has strong potential to be used by other researchers for further development and benchmarking.\n\n4. Originality:\nThe work integrates several ideas from previous methods, with appropriate citations. Although the application of these ideas to video forensics is a novel context, the core techniques themselves are adapted rather than newly proposed. Therefore, the method is not entirely novel in its formulation.\na. In lines 186–193, the authors state that replacing the low-frequency bands with those from the real counterpart forces the detector to focus on mid-high frequency features. However, how can we confidently claim that the detector will focus on mid-high frequencies? It is possible that the detector still learns from the low-frequency content provided by the real counterpart, potentially leading to misleading or incorrect detection. Do the authors have empirical evidence or observations supporting this assumption?\nb. The proposed method appears to heavily rely on the reconstruction quality of the Pyramid Flow model. While the paper mentions that this model generally reconstructs videos without visible artifacts, what happens if it fails in certain cases? Would such failure compromise the robustness or generalizability of the detector trained using this reconstruction-based augmentation?\nc. The paper makes valuable contributions, including a new dataset and a promising method. However, neither the code nor the dataset has been made available. The authors are strongly encouraged to release both the code and dataset to enable transparent evaluation, reproducibility, and faster progress in the rapidly evolving field of forensic video detection.',
    'title: FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies\n\nsummary: This paper proposes a fusion-based object detection pipeline that attempts to promote event+RGB fused object detection at varying frequencies. More specifically, a FlexFuse module is deigned to align the rich semantic information from RGB frames with the high-frequency event data, a FlexTune strategy is proposed to generate frequency-adjusted labels for unlabelled event streams. Experiments on the DSEC datasets demonstrate a superior performance of the proposed method over previous fusion-based methods. Also, the model maintains a robust performance across events of varying frequencies (up to 180 Hz).\n\nweaknesses and questions: Pros:\n- Fusing the complementary advantages of events and RGB images for object detection is a promising direction that can power applications that are sensitive to operational latency, for example, autonomous driving, industrial anomaly detection. In this context, exploring the robust detection across varying event frequencies and pushing its speed upper bound is important.\n- The proposed FlexTune strategy extends the classic pseudo label strategy to the event-RGB fused object detection, where high-frequency events are treated as unlabeled images, and several strategies grounded on the characteristics of events are deaigned to generate pseudo labels to boost the performance across different frequencies. \n- Experiments on the DSEC datasets (Car and Pedestrian classes) demonstrate the significant performance improvements of the proposed method. Meanwhile, the proposed methods maintain robustness across varying frequencies.\n- This paper is overall well-structured and easy to follow.\n\nCons:\nAlthough the task and method sound compelling, I have some major concerns about the data configuration and experimental comparison:\n- The quality of “high-frequency events” and the resulting performance degradation seems to be dependent on the proposed way of generating event frames. Section 3.2 mentions that the event is divided into several sub-intervals and aggregated within each interval to simulate high-frequency scenarios, while this setting may make the definition of high-frequency events problematic. For example, we can simply aggregate information for a longer period for each event aggregation, while still maintaining a low temporal gap between two sampling points. It will be helpful to provide the results on this setting with both high-frequency events and richer semantic information.\n- In the experimental section, this paper didn’t compare with models with solely RGB input. Therefore, it is hard to see the advantages of Event+RGB fusion compared to only using events or RGB frames. Also, it is not clear why only one fusion-based method is compared on the DSEC-Det dataset while more are compared on other datasets. \n- Although three datasets have been used for experiments, results on only 3 classes (Car, Pedestrian L-Veh.) are provided across all these datasets. This makes it hard to verify the methods’ generalization ability across more classes and scenarios, especially considering that pretrained weights are used during experiments.\n- It is not explained why using two significantly different backbones for events (RVT) and RGB frames (ResNet-50). Moreover, both pretrained weights are used for RVT and ResNet50 while it is not clear what are the pretrained datasets and how much they would contribute to the final performance.\nMy major concern is about the configuration of high-frequency events and the fairness of experimental comparison, I could reassess my score if these weaknesses are properly addressed.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 30522]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[16.0392,  9.8499,  6.5218],
#         [ 9.8499, 55.4061, 15.9871],
#         [ 6.5218, 15.9871, 58.2849]])

Evaluation

Metrics

Sparse Information Retrieval

• Dataset: reviewsearch
• Evaluated with SparseInformationRetrievalEvaluator

Metric	Value
dot_accuracy@1	0.4016
dot_accuracy@10	0.788
dot_precision@10	0.216
dot_precision@100	0.0602
dot_recall@10	0.196
dot_recall@100	0.4714
dot_ndcg@10	0.285
dot_mrr@10	0.5244
dot_map@100	0.1886
query_active_dims	7.1516
query_sparsity_ratio	0.9998
corpus_active_dims	902.1273
corpus_sparsity_ratio	0.9704
avg_flops	1.6039

Training Details

Training Dataset

Unnamed Dataset

• Size: 46,954 training samples
• Columns: anchor, positive, negative_1, negative_2, and negative_3
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative_1	negative_2	negative_3
  type	string	string	string	string	string
  modality	text	text	text	text	text
  details	min: 5 tokens mean: 8.65 tokens max: 14 tokens	min: 197 tokens mean: 421.56 tokens max: 512 tokens	min: 161 tokens mean: 435.95 tokens max: 512 tokens	min: 171 tokens mean: 451.66 tokens max: 512 tokens	min: 104 tokens mean: 431.57 tokens max: 512 tokens

• Samples:

  anchor	positive	negative_1	negative_2	negative_3
  meta-learning unclear contribution	title: Extending Contextual Self-Modulation: Meta-Learning Across Modalities, Task Dimensionalities, and Data Regimes summary: This paper proposes to extend Contextual Self-Modulation (CSM), which is an uncertainty-handling mechanism of Neural Context Flow (NCF). NCF is a robust meta-learning framework of neural ODE that includes self-modulation with high-order Taylor expansion around contexts. The authors focus on generalizing CSM into high data regimes and designed StochasticNCF and FlashCAVIA, which take CSM in various directions. From comparative experiments, the author verified that the NCF can be successfully extended to various optimization and meta-learning techniques. weaknesses and questions: 1. I believe this paper is not clearly written. * iCSM: It is briefly mentioned that the space of iCSM is "an infinite-dimensional variation" that utilizes "a space of multi-layer perceptrons, whose weights are flattened into a 1-dimensional tensor." I do not find a good enough ex...	title: Generalizing to Unseen Domains for Regression summary: The authors tackle domain generalization for regression tasks, or DGR. DGR is often different from classification tasks in DG as well as imbalanced regression tasks due to shift in labels. Popular feature alignment type of algorithms using IPM may not be applicable to DGR because the labels of different domains may not align, which may cause a model to fit to only one domain. The authors propose margin-aware meta learning framework to solve DGR by weighting query based on the distance to support, which mitigates sampling bias. The authors provide experiment results on multiple datasets compared to several feature alignment methods in domain adaptation as well as self-supervised learning and meta-learning methods. weaknesses and questions: 1. The relationship between the margin and hardness that the authors demonstrated is not something new. It is somewhat obvious. For example, in Gaussian Processes, the uncertainty incre...	title: The Meta-Representation Hypothesis summary: The paper proposes to combine Deep Mutual Learning with RL. In Deep Mutual Learning, several learners learn independently but at the same try to minimize the KL between their predictive distributions. The paper hypothesizes that two RL policies can learn from different MDPs — where each MDP has its own randomly sampled observation function while the policies try to minimize the KL between them. This would lead to the learning of robust representation functions. The randomly perturbed observation function is a key aspect of the paper — in their paper they apply a CNN with random weights to the observation to map the true observation to a perturbed one. The paper tests this hypothesis via PPO and shows that Deep Mutual Learning is helpful for generalization on the Procgen Benchmark. weaknesses and questions: ## Pros 1. Tackles an important problem about having a robust perception function for RL. 2. A positive thing is that the whole ...	title: Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliance summary: Paper proposes a graph metanetwork that edits trained MLPs in one forward pass to satisfy requirements (data minimization, equalized-odds fairness, pruning) while preserving behavior. It outputs masks and residuals, trained over model families with JSD + requirement losses, yielding faster, better Pareto trade-offs on Adult/Bank. weaknesses and questions: 1. For each dataset, each requirement, and each weighting coefficient, a separate model must be designed and trained. 2. Given the data-hungry training (needing many trainings to construct weight samples) and poor cross-dataset generalization, the claim of “no costly retraining” seems hard to stand. 3. The datasets are extremely limited and very low-dimensional, which makes me worry about applicability to higher-dimensional real-world settings. I’d like to know how it performs on higher-dimensional datasets, and how much the final res...
  meta-learning scalability benchmarks	title: Extending Contextual Self-Modulation: Meta-Learning Across Modalities, Task Dimensionalities, and Data Regimes summary: This paper proposes to extend Contextual Self-Modulation (CSM), which is an uncertainty-handling mechanism of Neural Context Flow (NCF). NCF is a robust meta-learning framework of neural ODE that includes self-modulation with high-order Taylor expansion around contexts. The authors focus on generalizing CSM into high data regimes and designed StochasticNCF and FlashCAVIA, which take CSM in various directions. From comparative experiments, the author verified that the NCF can be successfully extended to various optimization and meta-learning techniques. weaknesses and questions: 1. I believe this paper is not clearly written. * iCSM: It is briefly mentioned that the space of iCSM is "an infinite-dimensional variation" that utilizes "a space of multi-layer perceptrons, whose weights are flattened into a 1-dimensional tensor." I do not find a good enough ex...	title: OPTFM: A Scalable Multi-View Graph Transformer for Hierarchical Pre-Training in Combinatorial Optimization summary: The paper proposes a foundation model for combinatorial optimization with a focus on mixed integer programming. Specifically, the model is trained on the factor (bipartite) graph representation of MIPs. The paper uses a novel efficient transformer architecture. The pretraining has two phases. In the first, the graph is partitioned into subgraphs and the model learns to predict which constraints each variable is connected to. In the second, the model is trained in contrastive fashion to distinguish between pairs of subgraphs that belong to the same instance vs pairs that belong to different instances. Finally, once the variables have been embedded those embeddings are used for several different downstream tasks. The method shows strong empirical results on solving combinatorial optimization instances and on solver configuration tasks. weaknesses and questions: ## ...	title: Large-Scale Pretraining Offers Modest Benefits for Tabular Transfer Learning summary: The paper evaluates seven open-source tabular foundation models (TFMs) on 88 classification and 82 regression datasets, in both full-data and few-shot regimes, to check whether the performance gains reported in prior work actually hold under stricter evaluation. It shows that common practices—min–max normalizing metrics and only doing rank-based tests—can overstate the benefits of tabular pretraining; when performance is kept on the original scale and per-dataset bootstrap significance is applied, TFMs are usually tied with strong tree/NN baselines like CatBoost, only clearly winning on small classification tables. The authors also compare three TFMs (TabuLa-8B, TP-BERTa, XTab) with their non-pretrained counterparts and find that pretraining mainly helps LLM-style tabular ICL, but brings little or no benefit to other architectures, so the overall gains from large-scale tabular pretraining are...	title: Scaling Laws for Pre-training Agents and World Models summary: This paper investigates the existence and characteristics of scaling laws in embodied AI tasks, specifically world modeling (WM) and behavior cloning (BC), drawing parallels to scaling laws observed in large language models (LLMs). Through extensive experiments on large-scale video game datasets (e.g., Bleeding Edge) and robotics data (RT-1), the authors demonstrate that power-law relationships between model size, dataset size, compute, and pre-training loss also apply to WM and BC. Key findings include: - Scaling laws for WM are influenced by tokenizer compression rates. Higher compression rates lead to more reliance on dataset size. - BC with tokenized observations requires prioritizing dataset scaling under modest compute budgets, while BC with CNN-based architectures favors model scaling. - Small-scale language modeling experiments validate mechanisms behind observed scaling phenomena (e.g., sparse supervisio...
  anchor quality upper bounds training	title: Tournament Style RL: Stabilizing Policy Optimization on Non Verifiable Problems summary: The paper proposes a novel tournament reward calculated against a given set of anchor answers to generate reward supervision for LLM training in tasks without verifiable rewards. For each input prompt, a set of anchors is generated before training using a stronger LLM and ranked. Then the generated answers are compared against this ranked set of anchors to generate a reward for each answer which is then used for GRPO fine-tuning. weaknesses and questions: Strength 1. the proposed method is well motivated and clearly presented 2. can be easily implemented upon GRPO style fine-tuning pipelines 3. is robust against noise in evaluator LLMs Weakness 1. The proposed method relies heavily on anchor model quality. And the score itself will saturate if the model being fine-tuned surpasses the anchor model's quality. On the other hand, the performance of anchor model upper limits the model ...	title: On Proper Learnability between Average- and Worst-case Robustness summary: This paper studies the setting of robust PAC learning to test time attacks, using a relaxed notion of robustness on average instead of robustness to the worst-case attack. The contributions are as follows. -Negative result: even when using the relaxed notion of robustness, improper learning is impossible. This is a stronger result from the example in the worst-case setting [Montasser et al. 2019]. Moreover, this is achieved by a natural example of $\ell_p$ balls and the uniform measure. The intuition is that the non-Lipschitzness of the 0-1 loss enforces to use improper learning. -Positive results: 1. When considering Lipschitz loss functions, uniform convergence hold, and as a result, ERM is sufficient for learnability. 2. Instead of relaxing the robust loss, it is possible to relax the benchmark we compare to, i.e. the best function in the class but with a smaller parameter in the probabilist...	title: Spectral Perturbation Bounds for Low-Rank Approximation with Applications to Privacy summary: The paper presents the error bound (in terms of the spectral norm) for the rank-$p$ approximation of a symmetric matrix perturbed with a symmetric noise. The paper is written nicely with a clear motivation and description. The mathematical tools used to solve this hard problem namely contour bootstrapping are interesting in their own right and the authors seem to have found a novel application for this tool in this work. The simulation results also seem to highlight the utility of the derived bound when compared to classical baselines. weaknesses and questions: Strengths: 1. The derived bound is tighter than the EYM bound and leverages the symmetry of data and noise matrices. Also, the presented bounds are high probability asymptotic bounds, which are preferable over the bounds on expectation. 2. The paper adopts a new analytical approach, contour bootstrapping. Weaknesses: 1. The ...	title: Utility Boundary of Dataset Distillation: Scaling and Configuration-Coverage Laws summary: This paper proposes a unified configuration–dynamics–error framework that integrates gradient, distribution, and trajectory matching within a generalization-error analysis. It establishes the scaling law and coverage law linking distilled sample size to performance and configuration diversity, theoretically and empirically unifying major dataset distillation methods. weaknesses and questions: 1. The framework relies on PL conditions and Lipschitz continuity. While these assumptions are standard in convergence analysis, they may not strictly hold for modern deep networks with non-smooth activations, normalization layers, and stochastic training components. The practical relevance of the theoretical results could be further clarified by discussing their validity under relaxed or empirically realistic assumptions. 2. The validation of the proposed laws relies mainly on curve-fitting without...

• Loss: CachedSpladeLoss with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=True, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)",
      "document_regularizer_weight": 0.003,
      "mini_batch_size": 128
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 1024
• num_train_epochs: 1.0
• learning_rate: 2e-05
• warmup_steps: 0.1
• bf16: True
• eval_on_start: True
• router_mapping: {'anchor': 'query', 'positive': 'document', 'negative_1': 'document', 'negative_2': 'document', 'negative_3': 'document'}
• learning_rate_mapping: {'0\.sub_modules\.query\.0\.weight': 0.001}

All Hyperparameters

Click to expand

• per_device_train_batch_size: 1024
• num_train_epochs: 1.0
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: True
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {'anchor': 'query', 'positive': 'document', 'negative_1': 'document', 'negative_2': 'document', 'negative_3': 'document'}
• learning_rate_mapping: {'0\.sub_modules\.query\.0\.weight': 0.001}

Training Logs

Epoch	Step	Training Loss	reviewsearch_dot_ndcg@10
0	0	-	0.2377
0.0909	2	1.7752	-
0.1818	4	1.6872	-
0.2727	6	1.4462	0.2689
0.3636	8	1.3025	-
0.4545	10	1.2209	-
0.5455	12	1.1711	0.2882
0.6364	14	1.1102	-
0.7273	16	1.1213	-
0.8182	18	1.0570	0.2865
0.9091	20	1.0478	-
1.0	22	1.0844	0.2850

Training Time

• Training: 8.1 minutes
• Evaluation: 16.8 minutes
• Total: 25.0 minutes

Framework Versions

• Python: 3.12.9
• Sentence Transformers: 5.6.0
• Transformers: 5.12.1
• PyTorch: 2.8.0+cu128
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Sparse Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning SPLADE and other sparse encoder models.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedSpladeLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
@misc{formal2022distillationhardnegativesampling,
    title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
    author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and St\'ephane Clinchant},
    year={2022},
    eprint={2205.04733},
    archivePrefix={arXiv},
    primaryClass={cs.IR},
}

SparseMultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}