---
tags:
- sentence-transformers
- sparse-encoder
- sparse
- asymmetric
- inference-free
- splade
- generated_from_trainer
- dataset_size:46954
- loss:CachedSpladeLoss
- loss:SparseMultipleNegativesRankingLoss
- loss:FlopsLoss
base_model: opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte
widget:
- text: "title: Partial Matrix Completion\n\nsummary: This paper considers a twist\
\ on the standard matrix completion problem where one is required to only\ncomplete\
\ a subset of entries (not the whole matrix) that includes the entries shown.\
\ This allows them to consider\nsubstantially more observation patterns unlike\
\ the standard missing-at-random response. In this context\nthe paper makes the\
\ following contributions:\n\n1. Propose a computationally-inefficient algorithm\
\ that with high probability recovers a subset of entries\nthat is at least as\
\ large as the revealed set, with low target accuracy.\n2. Develop a computationally-efficient\
\ relaxation of this algorithm that has worse statistical dependence\non the target\
\ accuracy than the inefficient algorithm.\n3. Provide an online variant of the\
\ algorithm that is iterative/gradient based that applies to adversarial\nonline\
\ matrix completion.\n\n\nweaknesses and questions: Some weaknesses, though these\
\ are better understood as interesting avenues for future work:\n\n1. The proposed\
\ algorithms provide noisy completion even when the observations are noiseless\
\ (i.e. no obvious notion of completion to the inherent noise level). \nThis does\
\ seem inherent to the algorithm/proof techniques used here, it is unclear to\
\ me that this can be overcome here. \n2. There is an obvious sample complexity\
\ gap between the efficient and inefficient algorithms. Is this inherent, or a\
\ result of the proof technique here?\n\n\n\n1. The algorithmic viewpoint here\
\ is to separate the coverage computation from the completion method. This is\
\ advantageous in some ways (notably the generality and e.g. obtaining an essentially\
\ free proof of the risk), but potentially disallows using structure in the completion.\
\ E.g. in the causal inference setting of (say) row $1$ being partially revealed\
\ from columns $1, 2\\ldots, K \\leq \\text{dim}(M)$.\n2. What is $\\mu_\\max$?\
\ It is defined in the appendix (and easy to guess) but is not in the text (unless\
\ I missed)\n"
- text: 'title: IncVGGT: Incremental VGGT for Memory-Bounded Long-Range 3D Reconstruction
summary: This work presents IncVGGT, a training-free incremental variant of VGGT
designed to address the quadratic memory growth and excessive computation issues
in large-scale visual geometry transformers. The method tackles redundancy from
both the input and history sides: on the input side, it performs registration
and composition to merge short temporal windows into compact composite views;
on the history side, it retains only the top-k historically important slots together
with the most recent one for the next step. Experimental results demonstrate notable
improvements in both inference time and memory efficiency.
weaknesses and questions: - The registration-based redundancy reduction component
has been widely applied in the community (e.g., [1]). Moreover, introducing a
separate registration-and-composition preprocessing module somewhat compromises
the clean and unified design philosophy of VGGT, which originally aimed to eliminate
all explicit priors.
A more elegant approach would be to integrate the registration and composition
mechanism directly into VGGT’s internal architecture, preserving its end-to-end
structure.
- Limited Novelty: While the contributions toward improving memory and speed are
meaningful for industrial and practical applications, the registration-based redundancy
reduction and global-local cache pruning techniques are incremental relative to
VGGT. The authors themselves describe IncVGGT as an “incremental variant” of VGGT,
suggesting that it contributes less conceptual novelty to academic research, even
though it provides clear engineering value.
[1] Xiaoshui Huang, Guofeng Mei, Jian Zhang, and Rana Abbas. A Comprehensive Survey
on Point Cloud Registration.
N/A'
- text: 'title: Beyond Safe Answers: A Benchmark for Evaluating True Risk Awareness
in Large Reasoning Models
summary: This paper investigates an important phenomenon: models produce superficially
safe outputs while internal reasoning processes harmful content, and summary this
as SSA (Superficial Safety Alignment (SSA)). The authors introduce a novel benchmark:
Beyond Safe Answers (BSA) with over 2000 instances and report results on 23 reasoning
models. The evaluation results show significant emergency to improve reasoning
models'' internal safety as their think process is also open to the public (except
some close-source reasoning models like o3).
weaknesses and questions: 1. The presentation could be improved. Line 127-132
has some space that making this page appear somewhat empty.
2. How many GPU hours, total tokens, and dollar cost does one evaluation pipeline
consume?
Please see the weakness.'
- text: 'title: Physics of Language Models: Part 2.2, How to Learn From Mistakes on
Grade-School Math Problems
summary: This paper studies the impact of training language models with "error-then-retry"
format data on the reasoning performance. With experiments on a synthetic GSM8k-style
math reasoning datasets, the authors conclude that pretraining with such data
improves reasoning accuracy than on same amount of "error-free" normal data, and
LoRA fine-tuning with such data does not help.
weaknesses and questions: The paper mainly experiments with one type of error:
inserting a wrong parameter that cannot be computed next. While this is easy to
implement, it would be hard to simulate all kinds of errors that language models
can make in real-world reasonings scenarios. This cast doubts on how the suggested
approach can be deployed to train LMs on non-synthetic data. If the authors could
provide some discussion on how the method can be generalized to different errors
(e.g., math calculation error, context misunderstanding, ...) in a scalable and
controllable way, that would be very promising.
I would like to hear the authors'' thoughts on the inference-time scaling properties
of model trained with "retry" data: given a fixed inference-time computation budget,
is it better for model to produce reasoning chains with retry (more tokens to
reach final answer, but average accuracy is higher) or without retry (less tokens
to reach final answer, but average accuracy is lower)?'
- text: 'title: Generalizable Multi-Camera 3D Object Detection from a Single Source
via Fourier Cross-View Learning
summary: This paper proposed a Fourier Cross-View Learning (FCVL) framework, which
augments the data in the frequency domain and includes a contrastive-style semantic
consistency loss to improve the model generalization ability from a single source.
weaknesses and questions: Pros:
1. This paper augments the data in frequency domain by jitterring both amplitude
and phase, which diversify the dataset. The phase term typically captures high-frequency
features that may be more transferable.
2. The designed contrastive loss, which utilizes the adjacent image regions, is
also a good idea.
3. The proposed method is adaptable to different approaches and achieved SOTA
results with various baseline models.
4. The t-SNE visualization and other visualization results validate that the learned
features are domain-invariant and the images become more diverse.
Cons:
1. There may be some false negative samples for the contrastive loss design since
some large vehicles may distribute across several adjacent frames.
1. Regarding the semantic consistency loss, it appears to work well for small
vehicles. But what happens when the vehicle is large? Will the rear part of a
large vehicle be treated as a negative sample of the front part?
2. Besides large vehicles, what if there are other vehicles in the background
image, will they also be regarded as negative samples?
3. Could you also show some examples where the proposed model still cannot detect
correctly? How to further improve the model the future?
4. The idea of augmenting data in the frequency domain is not new and have been
tried by previous researchers. What are advantages of the proposed frequency domain
augmentation?'
pipeline_tag: feature-extraction
library_name: sentence-transformers
metrics:
- dot_accuracy@1
- dot_accuracy@10
- dot_precision@10
- dot_precision@100
- dot_recall@10
- dot_recall@100
- dot_ndcg@10
- dot_mrr@10
- dot_map@100
- query_active_dims
- query_sparsity_ratio
- corpus_active_dims
- corpus_sparsity_ratio
- avg_flops
model-index:
- name: Asymmetric Inference-free SPLADE Sparse Encoder
results:
- task:
type: sparse-information-retrieval
name: Sparse Information Retrieval
dataset:
name: reviewsearch
type: reviewsearch
metrics:
- type: dot_accuracy@1
value: 0.40155296727676093
name: Dot Accuracy@1
- type: dot_accuracy@10
value: 0.7879796299097463
name: Dot Accuracy@10
- type: dot_precision@10
value: 0.21599858821156656
name: Dot Precision@10
- type: dot_precision@100
value: 0.06021781878687037
name: Dot Precision@100
- type: dot_recall@10
value: 0.19603705746446795
name: Dot Recall@10
- type: dot_recall@100
value: 0.47140266480788
name: Dot Recall@100
- type: dot_ndcg@10
value: 0.2849832143549761
name: Dot Ndcg@10
- type: dot_mrr@10
value: 0.5243879048787551
name: Dot Mrr@10
- type: dot_map@100
value: 0.18859069597551725
name: Dot Map@100
- type: query_active_dims
value: 7.151616096496582
name: Query Active Dims
- type: query_sparsity_ratio
value: 0.9997656897943615
name: Query Sparsity Ratio
- type: corpus_active_dims
value: 902.127325119251
name: Corpus Active Dims
- type: corpus_sparsity_ratio
value: 0.9704433744473083
name: Corpus Sparsity Ratio
- type: avg_flops
value: 1.6038551330566406
name: Avg Flops
---
# Asymmetric Inference-free SPLADE Sparse Encoder
This is a [Asymmetric Inference-free SPLADE Sparse Encoder](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) model finetuned from [opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte) using the [sentence-transformers](https://www.SBERT.net) library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.
## Model Details
### Model Description
- **Model Type:** Asymmetric Inference-free SPLADE Sparse Encoder
- **Base model:** [opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte)
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 30522 dimensions
- **Similarity Function:** Dot Product
- **Supported Modality:** Text
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Documentation:** [Sparse Encoder Documentation](https://www.sbert.net/docs/sparse_encoder/usage/usage.html)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers)
- **Hugging Face:** [Sparse Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=sparse-encoder)
### Full Model Architecture
```
SparseEncoder(
(0): Router(
default_route='document'
(sub_modules): ModuleDict(
(query): Sequential(
(0): SparseStaticEmbedding({'frozen': False}, dim=30522, tokenizer=DistilBertTokenizer)
)
(document): Sequential(
(0): Transformer({'transformer_task': 'fill-mask', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'token_embeddings', 'architecture': 'NewForMaskedLM'})
(1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'log1p_relu', 'embedding_dimension': 30522})
)
)
)
)
```
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SparseEncoder
# Download from the 🤗 Hub
model = SparseEncoder("sparse_encoder_model_id")
# Run inference
sentences = [
'video detector frequency bias',
'title: Seeing What Matters: Generalizable AI-generated Video Detection with Forensic-Oriented Augmentation\n\nsummary: Recently, with the rapid development of video generative models, forensic detectors have also emerged; however, many of them lack generalizability. To address this issue, the authors propose a method that first identifies discriminative features that are less biased toward forensic-irrelevant patterns and more robust for detecting generated videos. Based on frequency domain analysis and previous literature, the authors identify that mid-high frequency components are less susceptible to compression artifacts and are thus suitable for forensic detection. To utilize these cues, two novel augmentation strategies are introduced during training: (1) Injection of forensic cues, by reconstructing real videos using the same autoencoder that generated the fake videos, aligning both real and fake samples in the representation space, and (2) Wavelet-based augmentation, by mixing real and fake videos using wavelet decomposition to enhance the mid-high frequency learning and push the detector to focus on those features. In the experimental evaluation, the model was trained using videos generated by the Pyramid Flow model, and testing was performed on both the GenVideo dataset (publicly available) and a newly constructed dataset comprising 2,400 videos from recent generative models. Accuracy was used as the evaluation metric, and the proposed method outperformed recent forensic detection methods.\n\nweaknesses and questions: 1. Quality: \nThe submission is technically sound, presenting a relevant formulation and effective use of forensic feature extraction from videos. The claims are well-supported by experimental results. The methods employed are appropriate, and the work is presented as a complete and cohesive study. The authors are transparent about both the strengths and limitations of their approach.\n\n2. Clarity:\nThe submission is clearly written and easy to follow.\n\n3. Significance:\nThe results are impactful for the community. While the code and new dataset have not been released, if shared, this work has strong potential to be used by other researchers for further development and benchmarking.\n\n4. Originality:\nThe work integrates several ideas from previous methods, with appropriate citations. Although the application of these ideas to video forensics is a novel context, the core techniques themselves are adapted rather than newly proposed. Therefore, the method is not entirely novel in its formulation.\na. In lines 186–193, the authors state that replacing the low-frequency bands with those from the real counterpart forces the detector to focus on mid-high frequency features. However, how can we confidently claim that the detector will focus on mid-high frequencies? It is possible that the detector still learns from the low-frequency content provided by the real counterpart, potentially leading to misleading or incorrect detection. Do the authors have empirical evidence or observations supporting this assumption?\nb. The proposed method appears to heavily rely on the reconstruction quality of the Pyramid Flow model. While the paper mentions that this model generally reconstructs videos without visible artifacts, what happens if it fails in certain cases? Would such failure compromise the robustness or generalizability of the detector trained using this reconstruction-based augmentation?\nc. The paper makes valuable contributions, including a new dataset and a promising method. However, neither the code nor the dataset has been made available. The authors are strongly encouraged to release both the code and dataset to enable transparent evaluation, reproducibility, and faster progress in the rapidly evolving field of forensic video detection.',
'title: FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies\n\nsummary: This paper proposes a fusion-based object detection pipeline that attempts to promote event+RGB fused object detection at varying frequencies. More specifically, a FlexFuse module is deigned to align the rich semantic information from RGB frames with the high-frequency event data, a FlexTune strategy is proposed to generate frequency-adjusted labels for unlabelled event streams. Experiments on the DSEC datasets demonstrate a superior performance of the proposed method over previous fusion-based methods. Also, the model maintains a robust performance across events of varying frequencies (up to 180 Hz).\n\nweaknesses and questions: Pros:\n- Fusing the complementary advantages of events and RGB images for object detection is a promising direction that can power applications that are sensitive to operational latency, for example, autonomous driving, industrial anomaly detection. In this context, exploring the robust detection across varying event frequencies and pushing its speed upper bound is important.\n- The proposed FlexTune strategy extends the classic pseudo label strategy to the event-RGB fused object detection, where high-frequency events are treated as unlabeled images, and several strategies grounded on the characteristics of events are deaigned to generate pseudo labels to boost the performance across different frequencies. \n- Experiments on the DSEC datasets (Car and Pedestrian classes) demonstrate the significant performance improvements of the proposed method. Meanwhile, the proposed methods maintain robustness across varying frequencies.\n- This paper is overall well-structured and easy to follow.\n\nCons:\nAlthough the task and method sound compelling, I have some major concerns about the data configuration and experimental comparison:\n- The quality of “high-frequency events” and the resulting performance degradation seems to be dependent on the proposed way of generating event frames. Section 3.2 mentions that the event is divided into several sub-intervals and aggregated within each interval to simulate high-frequency scenarios, while this setting may make the definition of high-frequency events problematic. For example, we can simply aggregate information for a longer period for each event aggregation, while still maintaining a low temporal gap between two sampling points. It will be helpful to provide the results on this setting with both high-frequency events and richer semantic information.\n- In the experimental section, this paper didn’t compare with models with solely RGB input. Therefore, it is hard to see the advantages of Event+RGB fusion compared to only using events or RGB frames. Also, it is not clear why only one fusion-based method is compared on the DSEC-Det dataset while more are compared on other datasets. \n- Although three datasets have been used for experiments, results on only 3 classes (Car, Pedestrian L-Veh.) are provided across all these datasets. This makes it hard to verify the methods’ generalization ability across more classes and scenarios, especially considering that pretrained weights are used during experiments.\n- It is not explained why using two significantly different backbones for events (RVT) and RGB frames (ResNet-50). Moreover, both pretrained weights are used for RVT and ResNet50 while it is not clear what are the pretrained datasets and how much they would contribute to the final performance.\nMy major concern is about the configuration of high-frequency events and the fairness of experimental comparison, I could reassess my score if these weaknesses are properly addressed.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 30522]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[16.0392, 9.8499, 6.5218],
# [ 9.8499, 55.4061, 15.9871],
# [ 6.5218, 15.9871, 58.2849]])
```
## Evaluation
### Metrics
#### Sparse Information Retrieval
* Dataset: `reviewsearch`
* Evaluated with [SparseInformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseInformationRetrievalEvaluator)
| Metric | Value |
|:----------------------|:----------|
| dot_accuracy@1 | 0.4016 |
| dot_accuracy@10 | 0.788 |
| dot_precision@10 | 0.216 |
| dot_precision@100 | 0.0602 |
| dot_recall@10 | 0.196 |
| dot_recall@100 | 0.4714 |
| **dot_ndcg@10** | **0.285** |
| dot_mrr@10 | 0.5244 |
| dot_map@100 | 0.1886 |
| query_active_dims | 7.1516 |
| query_sparsity_ratio | 0.9998 |
| corpus_active_dims | 902.1273 |
| corpus_sparsity_ratio | 0.9704 |
| avg_flops | 1.6039 |
## Training Details
### Training Dataset
#### Unnamed Dataset
* Size: 46,954 training samples
* Columns: anchor, positive, negative_1, negative_2, and negative_3
* Approximate statistics based on the first 100 samples:
| | anchor | positive | negative_1 | negative_2 | negative_3 |
|:---------|:---------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------|
| type | string | string | string | string | string |
| modality | text | text | text | text | text |
| details | - min: 5 tokens
- mean: 8.65 tokens
- max: 14 tokens
| - min: 197 tokens
- mean: 421.56 tokens
- max: 512 tokens
| - min: 161 tokens
- mean: 435.95 tokens
- max: 512 tokens
| - min: 171 tokens
- mean: 451.66 tokens
- max: 512 tokens
| - min: 104 tokens
- mean: 431.57 tokens
- max: 512 tokens
|
* Samples:
| anchor | positive | negative_1 | negative_2 | negative_3 |
|:--------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| meta-learning unclear contribution | title: Extending Contextual Self-Modulation: Meta-Learning Across Modalities, Task Dimensionalities, and Data Regimes
summary: This paper proposes to extend Contextual Self-Modulation (CSM), which is an uncertainty-handling mechanism of Neural Context Flow (NCF). NCF is a robust meta-learning framework of neural ODE that includes self-modulation with high-order Taylor expansion around contexts. The authors focus on generalizing CSM into high data regimes and designed StochasticNCF and FlashCAVIA, which take CSM in various directions. From comparative experiments, the author verified that the NCF can be successfully extended to various optimization and meta-learning techniques.
weaknesses and questions: 1. I believe this paper is not clearly written.
* iCSM: It is briefly mentioned that the space of iCSM is "an infinite-dimensional variation" that utilizes "a space of multi-layer perceptrons, whose weights are flattened into a 1-dimensional tensor." I do not find a good enough ex... | title: Generalizing to Unseen Domains for Regression
summary: The authors tackle domain generalization for regression tasks, or DGR. DGR is often different from classification tasks in DG as well as imbalanced regression tasks due to shift in labels.
Popular feature alignment type of algorithms using IPM may not be applicable to DGR because the labels of different domains may not align, which may cause a model to fit to only one domain.
The authors propose margin-aware meta learning framework to solve DGR by weighting query based on the distance to support, which mitigates sampling bias.
The authors provide experiment results on multiple datasets compared to several feature alignment methods in domain adaptation as well as self-supervised learning and meta-learning methods.
weaknesses and questions: 1. The relationship between the margin and hardness that the authors demonstrated is not something new. It is somewhat obvious. For example, in Gaussian Processes, the uncertainty incre... | title: The Meta-Representation Hypothesis
summary: The paper proposes to combine Deep Mutual Learning with RL. In Deep Mutual Learning, several learners learn independently but at the same try to minimize the KL between their predictive distributions. The paper hypothesizes that two RL policies can learn from different MDPs — where each MDP has its own randomly sampled observation function while the policies try to minimize the KL between them. This would lead to the learning of robust representation functions. The randomly perturbed observation function is a key aspect of the paper — in their paper they apply a CNN with random weights to the observation to map the true observation to a perturbed one. The paper tests this hypothesis via PPO and shows that Deep Mutual Learning is helpful for generalization on the Procgen Benchmark.
weaknesses and questions: ## Pros
1. Tackles an important problem about having a robust perception function for RL.
2. A positive thing is that the whole ... | title: Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliance
summary: Paper proposes a graph metanetwork that edits trained MLPs in one forward pass to satisfy requirements (data minimization, equalized-odds fairness, pruning) while preserving behavior. It outputs masks and residuals, trained over model families with JSD + requirement losses, yielding faster, better Pareto trade-offs on Adult/Bank.
weaknesses and questions: 1. For each dataset, each requirement, and each weighting coefficient, a separate model must be designed and trained.
2. Given the data-hungry training (needing many trainings to construct weight samples) and poor cross-dataset generalization, the claim of “no costly retraining” seems hard to stand.
3. The datasets are extremely limited and very low-dimensional, which makes me worry about applicability to higher-dimensional real-world settings.
I’d like to know how it performs on higher-dimensional datasets, and how much the final res... |
| meta-learning scalability benchmarks | title: Extending Contextual Self-Modulation: Meta-Learning Across Modalities, Task Dimensionalities, and Data Regimes
summary: This paper proposes to extend Contextual Self-Modulation (CSM), which is an uncertainty-handling mechanism of Neural Context Flow (NCF). NCF is a robust meta-learning framework of neural ODE that includes self-modulation with high-order Taylor expansion around contexts. The authors focus on generalizing CSM into high data regimes and designed StochasticNCF and FlashCAVIA, which take CSM in various directions. From comparative experiments, the author verified that the NCF can be successfully extended to various optimization and meta-learning techniques.
weaknesses and questions: 1. I believe this paper is not clearly written.
* iCSM: It is briefly mentioned that the space of iCSM is "an infinite-dimensional variation" that utilizes "a space of multi-layer perceptrons, whose weights are flattened into a 1-dimensional tensor." I do not find a good enough ex... | title: OPTFM: A Scalable Multi-View Graph Transformer for Hierarchical Pre-Training in Combinatorial Optimization
summary: The paper proposes a foundation model for combinatorial optimization with a focus on mixed integer programming. Specifically, the model is trained on the factor (bipartite) graph representation of MIPs. The paper uses a novel efficient transformer architecture. The pretraining has two phases. In the first, the graph is partitioned into subgraphs and the model learns to predict which constraints each variable is connected to. In the second, the model is trained in contrastive fashion to distinguish between pairs of subgraphs that belong to the same instance vs pairs that belong to different instances. Finally, once the variables have been embedded those embeddings are used for several different downstream tasks. The method shows strong empirical results on solving combinatorial optimization instances and on solver configuration tasks.
weaknesses and questions: ## ... | title: Large-Scale Pretraining Offers Modest Benefits for Tabular Transfer Learning
summary: The paper evaluates seven open-source tabular foundation models (TFMs) on 88 classification and 82 regression datasets, in both full-data and few-shot regimes, to check whether the performance gains reported in prior work actually hold under stricter evaluation. It shows that common practices—min–max normalizing metrics and only doing rank-based tests—can overstate the benefits of tabular pretraining; when performance is kept on the original scale and per-dataset bootstrap significance is applied, TFMs are usually tied with strong tree/NN baselines like CatBoost, only clearly winning on small classification tables.
The authors also compare three TFMs (TabuLa-8B, TP-BERTa, XTab) with their non-pretrained counterparts and find that pretraining mainly helps LLM-style tabular ICL, but brings little or no benefit to other architectures, so the overall gains from large-scale tabular pretraining are... | title: Scaling Laws for Pre-training Agents and World Models
summary: This paper investigates the existence and characteristics of scaling laws in embodied AI tasks, specifically world modeling (WM) and behavior cloning (BC), drawing parallels to scaling laws observed in large language models (LLMs). Through extensive experiments on large-scale video game datasets (e.g., Bleeding Edge) and robotics data (RT-1), the authors demonstrate that power-law relationships between model size, dataset size, compute, and pre-training loss also apply to WM and BC. Key findings include:
- Scaling laws for WM are influenced by tokenizer compression rates. Higher compression rates lead to more reliance on dataset size.
- BC with tokenized observations requires prioritizing dataset scaling under modest compute budgets, while BC with CNN-based architectures favors model scaling.
- Small-scale language modeling experiments validate mechanisms behind observed scaling phenomena (e.g., sparse supervisio... |
| anchor quality upper bounds training | title: Tournament Style RL: Stabilizing Policy Optimization on Non Verifiable Problems
summary: The paper proposes a novel tournament reward calculated against a given set of anchor answers to generate reward supervision for LLM training in tasks without verifiable rewards. For each input prompt, a set of anchors is generated before training using a stronger LLM and ranked. Then the generated answers are compared against this ranked set of anchors to generate a reward for each answer which is then used for GRPO fine-tuning.
weaknesses and questions: **Strength**
1. the proposed method is well motivated and clearly presented
2. can be easily implemented upon GRPO style fine-tuning pipelines
3. is robust against noise in evaluator LLMs
**Weakness**
1. The proposed method relies heavily on anchor model quality. And the score itself will saturate if the model being fine-tuned surpasses the anchor model's quality. On the other hand, the performance of anchor model upper limits the model ... | title: On Proper Learnability between Average- and Worst-case Robustness
summary: This paper studies the setting of robust PAC learning to test time attacks, using a relaxed notion of robustness on average instead of robustness to the worst-case attack.
The contributions are as follows.
-Negative result: even when using the relaxed notion of robustness, improper learning is impossible. This is a stronger result from the example in the worst-case setting [Montasser et al. 2019]. Moreover, this is achieved by a natural example of $\ell_p$ balls and the uniform measure.
The intuition is that the non-Lipschitzness of the 0-1 loss enforces to use improper learning.
-Positive results:
1. When considering Lipschitz loss functions, uniform convergence hold, and as a result, ERM is sufficient for learnability.
2. Instead of relaxing the robust loss, it is possible to relax the benchmark we compare to, i.e. the best function in the class but with a smaller parameter in the probabilist... | title: Spectral Perturbation Bounds for Low-Rank Approximation with Applications to Privacy
summary: The paper presents the error bound (in terms of the spectral norm) for the rank-$p$ approximation of a symmetric matrix perturbed with a symmetric noise. The paper is written nicely with a clear motivation and description. The mathematical tools used to solve this hard problem namely contour bootstrapping are interesting in their own right and the authors seem to have found a novel application for this tool in this work. The simulation results also seem to highlight the utility of the derived bound when compared to classical baselines.
weaknesses and questions: Strengths:
1. The derived bound is tighter than the EYM bound and leverages the symmetry of data and noise matrices. Also, the presented bounds are high probability asymptotic bounds, which are preferable over the bounds on expectation.
2. The paper adopts a new analytical approach, contour bootstrapping.
Weaknesses:
1. The ... | title: Utility Boundary of Dataset Distillation: Scaling and Configuration-Coverage Laws
summary: This paper proposes a unified configuration–dynamics–error framework that integrates gradient, distribution, and trajectory matching within a generalization-error analysis. It establishes the scaling law and coverage law linking distilled sample size to performance and configuration diversity, theoretically and empirically unifying major dataset distillation methods.
weaknesses and questions: 1. The framework relies on PL conditions and Lipschitz continuity. While these assumptions are standard in convergence analysis, they may not strictly hold for modern deep networks with non-smooth activations, normalization layers, and stochastic training components. The practical relevance of the theoretical results could be further clarified by discussing their validity under relaxed or empirically realistic assumptions.
2. The validation of the proposed laws relies mainly on curve-fitting without... |
* Loss: [CachedSpladeLoss](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#cachedspladeloss) with these parameters:
```json
{
"loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=True, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)",
"document_regularizer_weight": 0.003,
"mini_batch_size": 128
}
```
### Training Hyperparameters
#### Non-Default Hyperparameters
- `per_device_train_batch_size`: 1024
- `num_train_epochs`: 1.0
- `learning_rate`: 2e-05
- `warmup_steps`: 0.1
- `bf16`: True
- `eval_on_start`: True
- `router_mapping`: {'anchor': 'query', 'positive': 'document', 'negative_1': 'document', 'negative_2': 'document', 'negative_3': 'document'}
- `learning_rate_mapping`: {'0\\.sub_modules\\.query\\.0\\.weight': 0.001}
#### All Hyperparameters
Click to expand
- `per_device_train_batch_size`: 1024
- `num_train_epochs`: 1.0
- `max_steps`: -1
- `learning_rate`: 2e-05
- `lr_scheduler_type`: linear
- `lr_scheduler_kwargs`: None
- `warmup_steps`: 0.1
- `optim`: adamw_torch_fused
- `optim_args`: None
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `optim_target_modules`: None
- `gradient_accumulation_steps`: 1
- `average_tokens_across_devices`: True
- `max_grad_norm`: 1.0
- `label_smoothing_factor`: 0.0
- `bf16`: True
- `fp16`: False
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `use_liger_kernel`: False
- `liger_kernel_config`: None
- `use_cache`: False
- `neftune_noise_alpha`: None
- `torch_empty_cache_steps`: None
- `auto_find_batch_size`: False
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `include_num_input_tokens_seen`: no
- `log_level`: passive
- `log_level_replica`: warning
- `disable_tqdm`: False
- `project`: huggingface
- `trackio_space_id`: None
- `trackio_bucket_id`: None
- `trackio_static_space_id`: None
- `per_device_eval_batch_size`: 8
- `prediction_loss_only`: True
- `eval_on_start`: True
- `eval_do_concat_batches`: True
- `eval_use_gather_object`: False
- `eval_accumulation_steps`: None
- `include_for_metrics`: []
- `batch_eval_metrics`: False
- `save_only_model`: False
- `save_on_each_node`: False
- `enable_jit_checkpoint`: False
- `push_to_hub`: False
- `hub_private_repo`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_always_push`: False
- `hub_revision`: None
- `load_best_model_at_end`: False
- `ignore_data_skip`: False
- `restore_callback_states_from_checkpoint`: False
- `full_determinism`: False
- `seed`: 42
- `data_seed`: None
- `use_cpu`: False
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `parallelism_config`: None
- `dataloader_drop_last`: True
- `dataloader_num_workers`: 0
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `dataloader_prefetch_factor`: None
- `remove_unused_columns`: True
- `label_names`: None
- `train_sampling_strategy`: random
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `ddp_static_graph`: None
- `ddp_backend`: None
- `ddp_timeout`: 1800
- `fsdp`: None
- `fsdp_config`: None
- `deepspeed`: None
- `debug`: []
- `skip_memory_metrics`: True
- `do_predict`: False
- `resume_from_checkpoint`: None
- `warmup_ratio`: None
- `local_rank`: -1
- `prompts`: None
- `batch_sampler`: batch_sampler
- `multi_dataset_batch_sampler`: proportional
- `router_mapping`: {'anchor': 'query', 'positive': 'document', 'negative_1': 'document', 'negative_2': 'document', 'negative_3': 'document'}
- `learning_rate_mapping`: {'0\\.sub_modules\\.query\\.0\\.weight': 0.001}