yjoonjang/reviewsearch-dense

SentenceTransformer

This is a sentence-transformers model trained. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Maximum Sequence Length: 2048 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'unpad_inputs': False, 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (4): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'theory on finite-trial estimates',
]
documents = [
    'title: The Number of Trials Matters in Infinite-Horizon General-Utility Markov Decision Processes\n\nsummary: The paper analyzes the impact of the number of trails in estimating the objectives for GUMDPs. For both the discounted and average settings, it is shown by examples that there are mismatches between the finite-trial estimates and the actual infinite-trail objectives. Bounds on the mismatches are provided, with numerical results supporting the theoretical claims.\n\nweaknesses and questions: None\nIn Fig 3b, it looks like there are some discontinuities in the performance of $M_{f, 3}$ around $\\gamma=0.9$ where the finite-trail performance seems to diverge away from the infinite-trail one, but then converges back to it. Is that expected from theoretical analysis?',
    "title: The Minimax Rate of HSIC Estimation for Translation-Invariant Kernels\n\nsummary: The rate at which HSIC can be estimated is an important and open problem, in this paper, the authors prove that\nthe minimax optimal rate of HSIC estimation for Borel measures is $\\mathcal{O}(n^{-0.5})$ with M>=2 components, which is very important as existing conclusion only holds for M=2. Other byproducts can be naturally introduced, implying the minimax lower bound for the estimation of\ncross-covariance operator, which can be further specialized to get back the minimax result on the estimation of the covariance operator.\n\nweaknesses and questions: 1. Overall, the paper is not easy to follow as the paper's main contribution seems to be the proof part. \n2. I wouldn't say it is the weakness or the author's problem, as this is a theoretical paper, experiments are not necessary. Still is it possible to design toy experiments to validate the conclusions in the paper?\nNA",
    'title: Stronger Neyman Regret Guarantees for Adaptive Experimental Design\n\nsummary: This paper explores efficient ATE estimation in adaptive experimental designs. The authors focus on Neyman regret, which quantifies the variance difference between the inverse-propensity-weighted (IPW) estimator under the proposed adaptive design and the best fixed design in hindsight. Prior work (e.g., Dai et al., 2023) established a sublinear $O(\\sqrt{T})$ bound on Neyman regret. This paper strengthens that result, achieving an $O(\\log T)$ bound under slightly stronger assumptions. The analysis is further extended to contextual (multigroup) settings, introducing a method that ensures $O(\\sqrt{T})$ regret across multiple overlapping subpopulations. The approach is validated both theoretically and empirically.\n\nweaknesses and questions: None.\nNone.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.5815, 0.5067, 0.4693]])

Evaluation

Metrics

Information Retrieval

• Dataset: reviewsearch
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "query_prompt": "task: search result | query: ",
      "corpus_prompt": "title: none | text: "
  }
  

Metric	Value
cosine_accuracy@1	0.3588
cosine_accuracy@10	0.775
cosine_precision@10	0.1914
cosine_precision@100	0.056
cosine_recall@10	0.1702
cosine_recall@100	0.4395
cosine_ndcg@10	0.2512
cosine_mrr@10	0.4889
cosine_map@100	0.1569

Training Details

Training Dataset

Unnamed Dataset

• Size: 46,935 training samples
• Columns: anchor, positive, negative_1, negative_2, negative_3, negative_4, negative_5, negative_6, and negative_7
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7
  type	string	string	string	string	string	string	string	string	string
  modality	text	text	text	text	text	text	text	text	text
  details	min: 5 tokens mean: 7.66 tokens max: 12 tokens	min: 200 tokens mean: 504.56 tokens max: 1380 tokens	min: 159 tokens mean: 558.75 tokens max: 1575 tokens	min: 146 tokens mean: 516.22 tokens max: 1170 tokens	min: 160 tokens mean: 573.96 tokens max: 1957 tokens	min: 90 tokens mean: 487.22 tokens max: 1527 tokens	min: 138 tokens mean: 594.55 tokens max: 1959 tokens	min: 127 tokens mean: 616.91 tokens max: 1398 tokens	min: 64 tokens mean: 603.93 tokens max: 2048 tokens

• Samples:

  anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7
  meta-learning unclear contribution	title: Extending Contextual Self-Modulation: Meta-Learning Across Modalities, Task Dimensionalities, and Data Regimes summary: This paper proposes to extend Contextual Self-Modulation (CSM), which is an uncertainty-handling mechanism of Neural Context Flow (NCF). NCF is a robust meta-learning framework of neural ODE that includes self-modulation with high-order Taylor expansion around contexts. The authors focus on generalizing CSM into high data regimes and designed StochasticNCF and FlashCAVIA, which take CSM in various directions. From comparative experiments, the author verified that the NCF can be successfully extended to various optimization and meta-learning techniques. weaknesses and questions: 1. I believe this paper is not clearly written. * iCSM: It is briefly mentioned that the space of iCSM is "an infinite-dimensional variation" that utilizes "a space of multi-layer perceptrons, whose weights are flattened into a 1-dimensional tensor." I do not find a good enough ex...	title: Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning summary: This paper introduces FAME (Fast and Meta knowledge learners), a novel framework for continual reinforcement learning that draws inspiration from the human brain’s memory systems. It proposes a dual-learner architecture comprising a fast learner, which rapidly adapts to new tasks, and a meta learner, which incrementally integrates knowledge to prevent catastrophic forgetting. To support efficient adaptation and stability, the authors define principled metrics such as MDP distance (for task similarity) and catastrophic forgetting (for performance degradation). A key contribution is the adaptive meta warm-up mechanism, which selects the best initialization strategy for new tasks using statistical hypothesis testing, thereby mitigating negative transfer. Theoretical foundations are coupled with practical algorithms for both value-based and policy-based RL, using divergence-based incremental u...	title: Meta-Reinforcement Learning for Compiler Optimization: A Kernel-Embedded CompilerLLM with Verified Assumptions and Practical Guarantees summary: This paper proposes methods to improve optimization, specifically by trying to find surprising or interesting optimizations that nonetheless pass validation. weaknesses and questions: 1. This paper is not complete. There are obvious issues with the writing such as "Meta-optimization" and "Meta-reinforcement learning" being repeated many times in the intro, the "Meta-Learning Theory." section in the related work being empty, etc. 2. Clarity could use improvement. In many places (e.g. the abstract) the writing was either to jargon-filled or incomplete for me to understand well. 3. The description in section 4.2 and beyond is not clear enough for me to fully understand the method. There are many places that are unclear, but for instance it is not stated how k_cfg, k_data, k_inst etc. are calculated. 4. There is no comparison with o...	title: Meta-Router: Bridging Gold-standard and Preference-based Evaluations in LLM Routing summary: This paper addresses the challenge of training LLM routers to balance response quality and inference cost. It proposes a novel causal inference framework, viewing gold-standard and preference-based evaluation data as a treatment assignment problem. The key insight is that bias in preference-based data corresponds to the conditional average treatment effect (CATE). The proposed Meta-Router framework corrects this bias via meta-learners (S-learner, T-learner, X-learner), incorporates propensity score weighting to address data source imbalances, and applies distributionally robust optimization (DRO) for routing robustness. Experiments on benchmarks (AlpacaEval, MT-Bench, MMLU, GSM8K) show improved cost-quality trade-offs compared to standard baselines. weaknesses and questions: Limited Baseline Scope: Compares mostly to classical methods (IPW, DR); recent LLM routing methods and stron...	title: The Meta-Representation Hypothesis summary: The paper proposes to combine Deep Mutual Learning with RL. In Deep Mutual Learning, several learners learn independently but at the same try to minimize the KL between their predictive distributions. The paper hypothesizes that two RL policies can learn from different MDPs — where each MDP has its own randomly sampled observation function while the policies try to minimize the KL between them. This would lead to the learning of robust representation functions. The randomly perturbed observation function is a key aspect of the paper — in their paper they apply a CNN with random weights to the observation to map the true observation to a perturbed one. The paper tests this hypothesis via PPO and shows that Deep Mutual Learning is helpful for generalization on the Procgen Benchmark. weaknesses and questions: ## Pros 1. Tackles an important problem about having a robust perception function for RL. 2. A positive thing is that the whole ...	title: MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation summary: This paper proposes to achieve generalizable tool learning by additionally training models on meta-reasoning QA tasks. The meta-reasoning data are constructed by asking questions about the tool-using process in multiple directions, including action effect, decision-making, reversion, action input boundary, etc. Experiment results show improved tool learning performance on tasks including SAW, BW, LOG, Toolbench and BFCL. weaknesses and questions: 1. In lines 224-226, "In order to maintain the general ability of the model in the first stage, only the parameters of the query and value projection layers of the Transformer are updated instead of full-parameter training." This constraint might also affect learning ability and make comparisons unfair. Results ensuring similar settings will make results more convincing. 2. The "LLaMA3-solution" baselines are updated fewer times (10k*3) ...	title: Meta-learning Representations for Learning from Multiple Annotators summary: This paper proposes a meta-learning approach that utilizes noisy labels from multiple annotators to build a classifier without relying on true labels. The authors employ a probabilistic framework where latent class representations in a Mixture of Gaussians model are optimized via EM. This approach maximizes the likelihood of observed noisy labels given the latent variables, assuming these noisy annotations can guide the learning of true underlying classes. weaknesses and questions: 1. The paper assumes isotropic variance in the latent space, simplifying computation but potentially limiting flexibility. Real-world data often exhibit complex, class-specific structures that may not align with uniform variance assumptions, particularly in nuanced classification tasks. 2. Modeling A as a KKr matrix may lead to over-parameterization, especially with limited data. Without visualization of learned matrices,...	title: Meta ControlNet: Enhancing Task Adaptation via Meta Learning summary: This paper introduces Meta ControlNet, leveraging meta-learning and a novel layer-freezing approach to significantly reduce the training steps needed for ControlNet from 5000 to 1000. Additionally, it enables zero-shot control in edge tasks and rapid adaptation in complex tasks like Human Pose with only 100 finetuning steps. weaknesses and questions: There are three critical issues for this paper: 1. No quantitative results are provided. The paper only shows qualitative results, which makes it hard to evaluate the performance of the proposed method. 2. Zero-shot capability. This paper claims that the proposed method can achieve zero-shot control in edge tasks. The model is trained on HED, Segmentation, and Depth map which belong to the edge tasks. It raises a question about whether the model has achieved zero-shot control in edge tasks. If we train ControlNet on three tasks, and then test the model on Canny ...
  meta-learning scalability benchmarks	title: Extending Contextual Self-Modulation: Meta-Learning Across Modalities, Task Dimensionalities, and Data Regimes summary: This paper proposes to extend Contextual Self-Modulation (CSM), which is an uncertainty-handling mechanism of Neural Context Flow (NCF). NCF is a robust meta-learning framework of neural ODE that includes self-modulation with high-order Taylor expansion around contexts. The authors focus on generalizing CSM into high data regimes and designed StochasticNCF and FlashCAVIA, which take CSM in various directions. From comparative experiments, the author verified that the NCF can be successfully extended to various optimization and meta-learning techniques. weaknesses and questions: 1. I believe this paper is not clearly written. * iCSM: It is briefly mentioned that the space of iCSM is "an infinite-dimensional variation" that utilizes "a space of multi-layer perceptrons, whose weights are flattened into a 1-dimensional tensor." I do not find a good enough ex...	title: A solvable model of inference-time scaling summary: This paper introduces an analytically tractable model of inference-time scaling using Bayesian linear regression with reward-weighted sampling, deriving closed-form expressions for generalization error in the high-dimensional limit. The authors prove that when the reward model is well-aligned with the teacher, error decreases monotonically with inference samples $k$ (scaling as $\Theta(1/k^2)$ in the best-of-k limit), but substantial reward misspecification induces a finite optimal $k$ and optimal temperature. The theory delineates parameter regimes where scaling inference-time compute is provably more effective than collecting additional training data, though this advantage degrades as task difficulty increases. weaknesses and questions: 1. Oversimplified model: The paper only studies linear regression with quadratic rewards and Gaussian assumptions, while real LLMs involve highly nonlinear neural networks, complex reward mo...	title: MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? summary: This paper presents a new evaluation benchmark for Multimodal Large Language Models (MLLMs), dubbed MME-RealWorld, which focuses on challenges that models face in the real world. Specifically, MME-RealWorld covers 29,429 question-answer pairs across 5 real-world scenarios. Experimental results on MME-RealWorld show that even the most advanced models still struggled in real-life scenarios. Besides, the authors have also conducted detailed analyses to explain the unsatisfying performance of MLLMs. weaknesses and questions: - The evaluation on MME-RealWorld seems to require lots of computation resources, which may limit the accessibility for researchers with fewer resources. Do the authors have plans to expand or adapt MME-RealWorld to include new tasks or modalities as MLLMs capabilities evolve?	title: On the Embedding Collapse When Scaling up Recommendation Models summary: This paper studies recommendation model performance when scaling up the embedding layers of the model. The paper identifies a phenomenon of embedding collapse, wherein the embedding matrix tends to reside in a low-dimensional subspace. Through empirical experiments on FFM and DCNv2 and theoretical analysis on FM, the paper shows that the feature interaction process of recommendation models leads to embedding collapse and thus limits the model scalability. The paper also performed empirical experiments on regularized DCNv2 and DNN which led to less collapsed embeddings, but the model performance got worse. The paper proposes multi-embedding, which leads to better performance when scaling up the embedding layers. Experiments demonstrate that this proposed design provides consistent scalability for various recommendation models. weaknesses and questions: - In section 3, the paper proposes Information Abundan...	title: Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encoder summary: This paper presents Meta-learned Masked Auto-Encoder (MetaMAE), a novel modality-agnostic self-supervised learning (SSL) framework that leverages meta-learning to improve the transfer abilities of Masked Auto-Encoder (MAE). The authors reinterpret the mask reconstruction task of MAE as a meta-learning task and propose the integration of two advanced meta-learning techniques: gradient-based meta-learning and task contrastive learning. MetaMAE is evaluated on various data modalities from modality-agnostic SSL benchmarks, demonstrating significant improvements over previous modality-agnostic SSL methods in linear evaluation. The proposed approach also shows improved transferability on cross-domain datasets. weaknesses and questions: 1. There are significant differences in pretraining and fine-tuning hyperparameters for various downstream tasks, such as masking ratio, batch size, and decoder ...	title: GraphBench: Next-generation graph learning benchmarking summary: This paper introduces GraphBench, a contribution of around 20 unique datasets from 7 broad and diverse categories for graph learning benchmarking. It complements the existing popular graph learning benchmarks which may be significant for molecular and citation networks, as examples, but often missing for other areas such as chip design, circuit design and weather forecasting, among others (though there are individual areas in the literature that tackle these problems). The paper also highlights the current limitations with graph benchmarks in terms of data diversity reflecting multiple real world scenarios, in/out distribution splits, evaluation consistencies and framework for usage. It finally presents a framework based on Pytorch and Pytorch Geometric which acts as the interface for loaders, optimizers and evaluators. weaknesses and questions: - The manuscript includes reasonable discussion points on limitation...	title: (Out-of-context) Meta-learning in Language Models summary: The paper shows the existence of a phenomenon that the authors refer to as out-of-contect meta learning in large language models. The authors design experiments that show that this phenomenon causes the internalization of text that is broadly useful, meaning that the LLM is more likely to treat this content as true. The paper shows two forms of internalization, namely weak and strong internalization, the later being a form of meta learning. Two reasons are suggested for this phenomenon, one based on the parameters of the model, and another one relying on the implicit gradient alignment bias of gradient-based optimization methods. weaknesses and questions: * There is no conclusive explanation of the reasons why internalization happens * The phenomenon is hard to formalize and study, which limits the advantage of the insights in the paper None	title: Training-Free Generalization on Heterogeneous Tabular Data via Meta-Representation summary: This paper introduces a novel approach to enable training-free generalization for tabular datasets. The core idea is something like: For any given dataset, the input label data (x, y) is restructured into a new format: (distance to prototypes of class c, likelihood of the label of class c). This uniform data representation allows different datasets to be organized in a consistent manner. Thus, a model trained on this standardized format can effectively generalize across various tabular datasets. On unseen datasets, the proposed model achieves superior performances and saves training time. weaknesses and questions: The dataset used in this study is somewhat limited. Although I have confidence in the model's ability to generalize effectively to new datasets by representing data points in terms of their similarity to prototypes, there are concerns about its adaptability to other dataset...
  anchor quality upper bounds training	title: Tournament Style RL: Stabilizing Policy Optimization on Non Verifiable Problems summary: The paper proposes a novel tournament reward calculated against a given set of anchor answers to generate reward supervision for LLM training in tasks without verifiable rewards. For each input prompt, a set of anchors is generated before training using a stronger LLM and ranked. Then the generated answers are compared against this ranked set of anchors to generate a reward for each answer which is then used for GRPO fine-tuning. weaknesses and questions: Strength 1. the proposed method is well motivated and clearly presented 2. can be easily implemented upon GRPO style fine-tuning pipelines 3. is robust against noise in evaluator LLMs Weakness 1. The proposed method relies heavily on anchor model quality. And the score itself will saturate if the model being fine-tuned surpasses the anchor model's quality. On the other hand, the performance of anchor model upper limits the model ...	title: Unified Stability Bounds for Structured World Models: Geometry, Equivariance, and Identifiability as Sufficient Conditions summary: Overview： This paper addresses a key challenge in model-based reinforcement learning: the lack of a principled and low-overhead framework for diagnosing the quality of learned world-model representations. Motivated by the need to move beyond expensive, end-to-end evaluations and the limitations of existing theories, the authors aim to explain which properties of a representation govern downstream control performance and how to test them on existing model checkpoints. To solve this, the paper introduces a unified stability bound that decomposes the policy's suboptimality gap into three verifiable channels: geometric distortion (κ), an identifiability defect proxied by Total Correlation (TC), and an equivariance defect proxied by Local Equivariance Error (LEE). The authors then propose a practical diagnostic protocol where these proxies are measured ...	title: Quality Control at Your Fingertips: Quality-Aware Translation Models summary: The paper proposed two methods to make the NMT model quality aware. One is to prompt the NMT model with a quality score during training, but using the best score during inference time. The other is similar to multi-task learning but in a more unified way by appending the quality score in the target side. Both approaches show promising improvements in translation quality and one of them can work well with the MBR decoding to boost the translation quality further. weaknesses and questions: My concerns are in the questions. If they can be addressed properly, they won't be weakness to me. In conclusion, which one between QA prompting and prediction approaches is your recommendation in the situations including latency sensitive inference and large scale distillation. Please also describe how do you scale your methods in large scale multilingual machine translation system. The experiments highly relies on ...	title: Utility Boundary of Dataset Distillation: Scaling and Configuration-Coverage Laws summary: This paper proposes a unified configuration–dynamics–error framework that integrates gradient, distribution, and trajectory matching within a generalization-error analysis. It establishes the scaling law and coverage law linking distilled sample size to performance and configuration diversity, theoretically and empirically unifying major dataset distillation methods. weaknesses and questions: 1. The framework relies on PL conditions and Lipschitz continuity. While these assumptions are standard in convergence analysis, they may not strictly hold for modern deep networks with non-smooth activations, normalization layers, and stochastic training components. The practical relevance of the theoretical results could be further clarified by discussing their validity under relaxed or empirically realistic assumptions. 2. The validation of the proposed laws relies mainly on curve-fitting without...	title: Task-Robust Pre-Training for Worst-Case Downstream Adaptation summary: In order to improve the robustness of the pre-trained model on downstream tasks, the authors propose a simple optimization algorithm, softmax weighted gradient descent, to minimize the worst-case expected risk of upstream tasks. weaknesses and questions: In the experimental part, the method does not seem to show a consistent improvement. For example, as shown in Table 1, although the author mentioned that the model has significantly improved performance on many more challenging tasks, it has worse performance than the previous model on some downstream tasks that have performed well. See Table 2 for the same reason. Could this be improved with some tweaks for consistency? According to the weekness I mentioned above, my question is whether such a strategy is a trade-off in the performance between the best case and the worst case on the downstream task, and cannot achieve consistency improvement?	title: Steering Beyond the Support: Adversarial Training on Unsupervised Jailbroken Activation Simulation summary: This paper studies activation steering for jailbreak defense under unseen or out-of-distribution attacks. Instead of learning steering only from a fixed supervised jailbreak set, the paper proposes to simulate diverse jailbroken activations through unsupervised latent direction discovery, and then train a potential-induced steering field with a bi-level adversarial objective. The method is evaluated on three aligned LLMs and six jailbreak families, and the paper reports improved robustness together with increasing subspace coverage during training. weaknesses and questions: Strengths - The paper targets an important problem in LLM safety, namely whether activation steering can generalize beyond a fixed supervised jailbreak set. - The overall method is reasonably well motivated. In particular, combining unsupervised jailbreak activation simulation with a learned steer...	title: Learning to Quantize for Training Vector-Quantized Networks summary: This paper proposes a novel vector quantization training framework Meta-Quantization inspired by meta-learning, which decouples the optimization of codebook and autoencoder into two stages, enabling dynamic codebook generation and task-specific training. The proposed method outperforms existing vector quantization approaches on image construction and generation tasks. weaknesses and questions: The description of convergence is inconsistent. While Figure 2 states that $\phi$ and $\theta$ are trained to convergence before training $\psi$, Algorithm 1 shows that they are updated together. This discrepancy creates ambiguity regarding the actual optimization procedure implemented in the paper. In the introduction, it is mentioned that the codebook utilization in previous methods is low. However, in the experiments (Table 3, 4, 5), the codebook utilization of VQGAN-LC is also quite high. Please provide a justificat...	title: Estimating Fréchet bounds for validating programmatic weak supervision summary: This paper proposes solutions via convex programs to estimate Frechet bounds for Programmatic Weak Supervision (PWS). This approach uses estimates of the true labels via labelmodels (i.e., different aggregation schemes that exist in the literature). With these estimates of the labels, they provide an approach to estimate bounds on the accuracy (and other quantities) of the weak labelers. They provide experiments to check the validity of their bounds and also provide experiments with weak labelers generated via prompting to examine how their bounds perform under instances of weak labelers with different qualities/accuracies. weaknesses and questions: 1. One weakness is that this approach is fundamentally reliant on the quality of the label model. This is manifested in assumption 2.3, which states that the estimate of the conditional distribution of $Y | Z$ should approach the true conditional distri...

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "GuideGISTEmbedLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 1024
• num_train_epochs: 1.0
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• warmup_steps: 0.1
• bf16: True
• eval_on_start: True
• dataloader_num_workers: 4
• ddp_find_unused_parameters: False
• prompts: {'anchor': 'task: search result | query: ', 'positive': 'title: none | text: ', 'negative_1': 'title: none | text: ', 'negative_2': 'title: none | text: ', 'negative_3': 'title: none | text: ', 'negative_4': 'title: none | text: ', 'negative_5': 'title: none | text: ', 'negative_6': 'title: none | text: ', 'negative_7': 'title: none | text: '}
• batch_sampler: no_duplicates_hashed

All Hyperparameters

Click to expand

• per_device_train_batch_size: 1024
• num_train_epochs: 1.0
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: True
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 4
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: False
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: {'anchor': 'task: search result | query: ', 'positive': 'title: none | text: ', 'negative_1': 'title: none | text: ', 'negative_2': 'title: none | text: ', 'negative_3': 'title: none | text: ', 'negative_4': 'title: none | text: ', 'negative_5': 'title: none | text: ', 'negative_6': 'title: none | text: ', 'negative_7': 'title: none | text: '}
• batch_sampler: no_duplicates_hashed
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	reviewsearch_cosine_ndcg@10
0	0	-	0.1704
0.0909	2	34.5974	-
0.1364	3	-	0.2356
0.1818	4	25.3299	-
0.2727	6	16.6895	0.2372
0.3636	8	13.9725	-
0.4091	9	-	0.2423
0.4545	10	12.2704	-
0.5455	12	11.2867	0.2473
0.6364	14	10.7076	-
0.6818	15	-	0.2492
0.7273	16	10.2320	-
0.8182	18	10.0333	0.2509
0.9091	20	9.9865	-
0.9545	21	-	0.2507
1.0	22	9.9844	0.2512

Training Time

• Training: 4.2 hours
• Evaluation: 1.6 hours
• Total: 5.7 hours

Framework Versions

• Python: 3.12.9
• Sentence Transformers: 5.6.0
• Transformers: 5.12.1
• PyTorch: 2.8.0+cu128
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}