metadata
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:46935
- loss:MatryoshkaLoss
- loss:GuideGISTEmbedLoss
widget:
- source_sentence: multi-modal imputation computational cost
sentences:
- >-
title: Learning Flexible Forward Trajectories for Masked Molecular
Diffusion
summary: This paper focuses on the adaptation problem of Masked
Diffusion Models (MDMs) for discrete data in molecular graph generation.
It points out that "fixed, element-independent" forward masking
scheduling leads to different molecules collapsing to the same
intermediate state in the forward trajectory, making reverse denoising,
typically unimodal and predicting independently by node or edge,
difficult to learn the correct reconstruction target. To address this,
the paper proposes MELD: which learns the forward masking rate at the
element level (node/edge) and assigns an independent erosion trajectory
to each graph element through a parameterized noise scheduling network;
it is jointly optimized with the reverse denoising network during
training. The authors claim that MELD achieves high efficiency in
unconditional generation of QM9 and ZINC250K graphs and outperforms
standard MDM and several diffusion baselines in distribution alignment
and property alignment.
weaknesses and questions: 1.The element-level kernel renders the forward
process non-equivariant, meaning the intermediate state distribution is
affected by vertex permutations. For molecular graphs, this contradicts
the fundamental principle that isomorphism should not alter the
generative distribution. Current methods merely introduce a learnable
embedding H for each graph element and "randomly permutate columns" to
"distinguish graph states with the same number of nodes/edges," but this
does not restore the guarantee of permutation equivariance. It needs to
be proven that this forward process, which breaks equivariance, does not
induce dependencies on node labels and generalization issues, especially
whether relabeling input nodes during testing maintains a consistent
sampling distribution.
2.The abstract and main text claim that MELD is "the first diffusion
model to achieve 100% chemigenicity in unconditional generation on QM9
and ZINC250K," but several MDM baselines in Table 1 also show 100%. The
wording needs to be corrected.
3.The paper does not provide an explicit collision risk function or
upper and lower bound analysis; the loss in Equation (3) does not
directly minimize the "collision probability". It is suggested to
provide a computable proxy metric and its relationship with the gradient
direction, or to supplement the appendix with a simplified derivation of
the "collision probability as a function of {𝑤_{𝑖}}".
4.The manuscript states that "unless otherwise specified, standard MDM
and MELD use the same DiT backbone," but were the other
discrete/continuous diffusion baselines in Table 1 also retrained and
had their backbones and training budgets aligned? If comparisons are
only made within the MDM family without aligning the
backbones/hyperparameters of external distributed models, the
conclusions may overestimate the advantages of MELD. Please provide the
number of training epochs, GPU configuration, total duration, and FLOPs
in the appendix, as well as the retraining/reproduction practices for
each baseline.
5.The use of V.U.N.↑ in Tables 3 and 6 lacks a clear explanation of its
meaning and calculation in the text (it seems to be a composite score
for Validity/Uniqueness/Novelty?). Please define it explicitly at its
first appearance in the text.
1.The statement "first 100% validity" conflicts with Table 1. It is
recommended to change it to "significantly reduced FCD/NSPDK while
maintaining 100% validity." Could you please report the confidence
intervals for inefficiency (multiple sampling)?
2.Can a more systematic comparison be made between the key differences
and complexity of existing "adaptive/category-level" scheduling (such as
DiffusionBERT, GenMD4, TabDiff) and the "element-level" scheduling in
this paper? Currently, only a rough comparison is made in Table 3,
lacking a theoretical analysis of the differences in expressive power.
3.Please list the number of training epochs, learning rate, scheduler,
backbone, number of GPUs, and training time for all baselines; and
specify which baselines were retrained by the authors and which were
reproduced from the original paper.
4.Please add "Node relabeling robustness test" (variance of distribution
index/property MAE under multiple labels of the same molecule).
5.Table 5 only performs isomorphism counting on 12 nodes/131 samples,
which is costly but has a small sample size; it is recommended to
provide estimation methods for larger scales (such as approximate GI or
fingerprint hash upper/lower bounds) and statistical confidence
intervals.
- >-
title: ICFI: A Feature Importance Measure For Multi-Class Classification
summary: The paper proposes a novel feature importance method for
multi-class classification. The method tackles the problem of providing
not only one set of feature importances, but one for each pair of
classes, thus offering more insight into the classifier. The method is
based on the idea of observing decrease in empirical risk when two
classes are combined, in combination with permutation-based feature
importance. Some experiments are provided showing that the method gives
sensible results and outperforms GSHAP adapted to the same problem.
weaknesses and questions: The proposed method is relatively simple and
does not bring any extremely innovative methodology or theoretical
results, which is nothing wrong by itself, but the I would expect a very
strong empirical evaluation or (even better) a practical use-case that
demonstrates not only that the method works but that the problem of
requiring additional insights into (pairwise) relationships between
classes is really a problem in need of a solution.
The current experiments do not convince me (see Questions). As the
authors also say, evaluation of XAI is a big challenge and there doesn't
seem to be any shortcut to a sound empirical evaluation
(https://icml.cc/virtual/2025/poster/40169). The first two experiments
establish that there is nothing clearly wrong with the method, which is
OK. The retraining experiment and comparison with GSHAP I do not
understand. If the goal of the method is to provide insights into how
the model classifies, then this is far from a realistic assessment (yes,
it is common to do this in XAI/ML papers, but it doesn't make it any
less unrealistic). Also, it seems to me that GSHAP was forced into this
comparison, not being a method developed for the same purpose. I might
be wrong, but the paper doesn't do a good job of describing exactly what
GSHAP is or how it was adapted.
And I might have other issues with the paper on things that I currently
don't quite understand and/or were not explained clearly enough (also
see Questions).
Minor comments:
- Some extra effort seems to have gone into squeezing this to fit the
page limit (Figure 6 caption has no space to breathe, etc.).
- ).One
- The proposed method operates on model risk not on model predictions
directly. So, technically, it is not explaining what the model does, but
what features contribute to the models predictive performance. Often the
same, but not always.
Q1: Finally, why not include some global feature importance into the
comparison? The problem of masking the feature importance of globally
less important features that are important for certain pairs of classes
might be exaggerated. I'd imagine that for a low number of classes the
global ordering would be decent (definitely better than random).
Q2. Computational complexity: First, it would really help if the
computational complexity is stated more explicitly, instead of "in line
with existing permutation methods but cheaper than SHAP". Second, I'm
not convinced that the latter is correct. The proposed method requires
for each feature a constant number of permutations and each permutation
requires a model prediction? Any decent implementation of SHAP should
also be linear in the number of features and will contain the model
prediction (you don't go through all subsets of coalitions).
Q3. Permutation importance has certain failure cases, compared to SHAP,
for example. Why not combine the idea of combining two classes but then
use Shapley values instead of permutation importance?
Q4. I'd remove the explicit "Definition 1" from definition of the
pairwise feature importance problem. It is not necessary and it is not
precise. Informally we would probably agree on what "as it pertains to
separating the target classes $\sigma$ and $\rho$" means, but what does
it really mean? A model never trully 100% focuses on separating only two
classes (unless there are only two classes).
Q5. The interval computation in A.3 seems like overkill. The (Bayesian
posterior) mean and standard deviation of a process where 100
independent samples are given is estimated using Markov Chain Monte
Carlo? Unless I'm missing something, the only possible justification
would be that we use uniform priors on the two parameters and therefore
can't use the analytical solution. But if we are going to be so precise
as to not allow values outside of [0,1] then why use a Gaussian
likelihood, which is clearly not appropriate. Burn-in also doesn't make
sense (why not just pick a sensible starting value, like the empirical
mean and standard deviation). To summarize, average +/- 1.96 * standard
deviation of the sample / sqrt(100) should give essentially the same
results.
Q6. I'm unsure about the upper/lower bound requirement. First, the
requirements, as stated, would allow for a method that assigns
arbitrarily low negative feature importances (we require irrelevant
features to have 0 and to have an upper bound; there is nothing saying
that a relevant feature can't have a negative importance, for example,
if it decreases predictive performance). I'll assume that the intention
was for them to be bounded between 0 and an upper bound (which might as
well be 1). I'm not convinced by the argument that people prefer bounded
things therefore bounding is better. That is, it is mathematically easy
to bound things, but with it we change the scale of the feature
importance. Are these importances even comparable across class pairs for
same risk? Are they comparable across different risks?
- >-
title: Learning Optimal Multimodal Information Bottleneck
Representations
summary: The author introduces a theoretically guaranteed multimodal
information bottleneck approach. This method dynamically adjusts the
regularization weights of each modality by considering the varying
degrees of task-relevant information across different modalities.
Theoretically, the optimization objective proposed by the author is of a
remarkably straightforward form, and the practical loss function serves
as an upper bound to this theoretical objective, thereby ensuring the
feasibility.
#### update after rebuttal: I don't change my assessment.
weaknesses and questions: The article is logically structured, with
clearly defined and reader-friendly symbols. The theoretical section is
particularly detailed and rigorous.
In the experiments, the non-MIB-based methods only include some basic
approaches. I am curious about how they compare with newer methods:
Peng, Xiaokang, et al. "Balanced multimodal learning via on-the-fly
gradient modulation." Proceedings of the IEEE/CVF conference on computer
vision and pattern recognition. 2022.
Zhang, Xiaohui, et al. "Multimodal representation learning by
alternating unimodal adaptation." Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition. 2024.
I harbor a degree of skepticism towards the reasoning in Equation (3),
which posits that concatenating $e_i$ enhances the model's learning by
improving the signal-to-noise ratio. Typically, the introduction of
signal-to-noise ratio considerations involves additive noise, such as (
z_i^{noise} = z_i + e_i ), rather than direct concatenation. Moreover,
the ablation studies do not include relevant content to substantiate
this.
- >-
title: A Structured Pruning Algorithm for Model-based Deep Learning
summary: This paper proposes a structured pruning method for model-based
deep learning in inverse problems. The proposed method, SPADE, reduces
the computational complexity of model-based networks at test-time by
pruning its non-essential weights. In addition, three different
fine-tuning methods are introduced for the pruned networks to reduce
performance loss. SPADE is evaluated on compressed sensing MRI and image
super-resolution, and is shown to speed up inference with minimal
performance degradation.
weaknesses and questions: Weaknesses:
- The contributions of the paper are mostly comprised of a combination
of existing techniques such as the pruning algorithm and the fine-tuning
techniques.
- The method is not compared with other methods for improving inference
speed, such as [1] or [2] mentioned in the paper. The lack of this
comparison makes it difficult to quantify the significance of the
results. As an example, there is a 0.77 dB PSNR drop with a 51% speed up
at test-time (Table 1) for compressed sensing MRI which seems to be a
large performance reduction, and it is unclear how this compares to
existing methods.
[1] J. Liu, Y. Sun, W. Gan, X. Xu, B. Wohlberg, and U. S. Kamilov.
SGD-Net: Efficient Model-Based
Deep Learning With Theoretical Guarantees. IEEE Trans. Computational
Imag., 7:598–610,
2021.
[2] J. Tang and M. Davies. A fast stochastic plug-and-play ADMM for
imaging inverse problems. arXiv
preprint arXiv:2006.11630, 2020.
- How much does the training time increase for SPADE, compared with the
baseline unpruned model-based network?
- Is it possible to combine fine-tuning losses, rather than view them as
independent techniques, and could that help preserve performance?
- How does the memory complexity change at test-time? Memory complexity
is also a quite important consideration for which discussion has not
been included.
Suggestions:
- The introduction, and the "DL and MBDL." subsection in the background
are repetitive. For instance, the equation for PnP/RED does not seem to
contribute to the story of the paper. The background can be shortened to
include more experiments in the main paper, such as the visual results
(Figure 6-8) in the supplemental, which are crucial for compressed
sensing MRI.
- Typographical errors should be fixed via proofreading.
- >-
title: FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion
summary: This paper proposes an MOE-based model to handle multimodal
data fusion. It addresses two challenges: missing modalities and
irregularly sampled data trajectories. A Laplace gating function is
applied to the MoE Backbone. An entropy regularization loss is proposed
to ensure balanced and stable expert utilization. The author validates
the method in diverse datasets.
weaknesses and questions: Addressing the following weaknesses may
improve the paper:
1. The author should clearly distinguish the proposed method and others’
modules. From the paper, the Laplace gating function is proposed as a
new one. I am not sure the author made some contributions to the encoder
design, router design, and loss design. The author should make more
illustrations about the contributions, not just combine other people’s
work together.
2. The experimental results are not extensive. The author should clearly
demonstrate the data modality of the chosen benchmarks. It seems that
other modalities, such as text, are not included. The author should
explain this.
3. Please explain more concisely how the gating functions can stabilize
the imbalance and sparse multi-modal data. The paper should point this
out more concisely and better with experimental results. Too many
mathematical proofs and theorems in Sec.3 seem not helpful in
illustrating the advantages of the proposed method.
Please see the weakness part to answer the questions.
- >-
title: Compute Where It Counts: Adaptive Compute Allocation for Large
Language Models via Learned Granular Sparsity
summary: CWIC (Compute Where it Counts) introduces a novel method for
training sparse LLMs by making sparsity thresholds learnable parameters.
The key contributions are: (1) learned contextual thresholds that are
optimized using STEs, allowing models to dynamically allocate different
amounts of compute to different tokens and weight matrices, and (2)
granular sparsity that partitions matrix columns into smaller "stripes"
for more expressive sparsity patterns.
weaknesses and questions: Major Concerns
1) I am not sure about the claim - *STE improves performance by
removing the variance imparted on the grads when the values of G
fluctuate...*: Using STE seems overly aggressive for the use-case? I
think there needs to be more justification around this choice.
Supporting experiments to compare that this is as a winning choice might
also be helpful. Right now, this seems to more like a empirical tuning
based selection?
- If this choice is derived from JumpReLU - their gradient estimator
worked for SAE reconstruction doesn't automatically validate these
additional modifications for multi-layer distillation. Current
explanation is not sufficient for a convincing argument.
In general, I think section 3.3 is poorly written. Authors should
re-word their ideas more clearly here.
2) No analysis of why related kernel choice is better than alternatives?
3) The psuedo-derivative bandwidth (lines 220-224): The interaction
between adaptive bandwidth $(\epsilon_i = \alpha\epsilon · std(x_i))$
and input whitening (Section 3.4) is unclear - is std computed on
whitened or raw values? How does this affect gradient scales?
- No comparison of adaptive vs. fixed bandwidth for pseudo-derivatives.
4) Experimental concern: The distillation data includes benchmark
training sets (MMLU, ARC, WinoGrande) repeated 5×, which may inflate
evaluation scores (refer: appendix C I believe). I think reporting
scores on benchmarks not in training data would be a stronger claim.
5) Loss function: Authors need to explain how to interpret this. Right
now, it looks asymmetrical: the loss only penalizes using too many
parameters, not too few? In other words, it will keep on minimizing, i.e
nothing stops the model from becoming 10× sparse if that happens to
minimize distillation loss? It seems one would need careful tuning/warm
starting to counter this or is there a implicit effect from distillation
loss (or am I understanding this wrong)?
I strongly think the presentation wrt to the key ideas (section 3.2-3.6)
need to be improved to make the author's work more accessible and make a
convincing argument towards a lot of different empirical choices.
Additionally, I found the concatenation operator to be a bit confusing
(line 180-181), maybe there is a better way to mathematically denote it?
NA
- "title: Unified Insights: Harnessing Multi-modal Data for Phenotype Imputation via View Decoupling\n\nsummary: This paper focuses on the task of phenotype imputation and proposes utilizing multi-modal data to gain insights that facilitate the evaluation of patients' overall health status. Specifically, the authors design a framework based on view decoupling, which involves segregating the modeling of biological data and phenotype data to avoid the impact of data heterogeneity and view conflict. To alleviate the influence of noise and irrelevant information in the biological data, a novel contrastive knowledge distillation method is proposed. Furthermore, the authors conduct extensive experiments to demonstrate the superiority of the proposed model.\n\nweaknesses and questions: 1.\tThe model includes multiple components. It would be beneficial to discuss the time complexity of the proposed method. Specifically, an analysis of the computational efficiency for each component, as well as the overall model, would provide valuable insights.\n2.\tThe patients in the experiments are selected from those with Alzheimer's disease and related dementias. It would be helpful to explain the rationale behind selecting this particular patient set. Additionally, it is important to discuss whether the model is applicable to other cohorts.\n3.\tWhy can't recent models, such as M3Care, Graph, and MUSE, directly address the need for integrating biological data and EHR data?\n4.\tThe proposed method involves multiple loss functions. Adding these losses to Figure 1 would aid understanding. Including pseudocode for the algorithm would also be helpful.\nPlease refer to weaknesses."
- "title: Distributed Parallel Gradient Stacking(DPGS): Solving Whole Slide Image Stacking Challenge in Multi-Instance Learning\n\nsummary: This paper introduces Distributed Parallel Gradient Stacking (DPGS), a framework designed to address the challenge of non-stackable data in Multiple Instance Learning (MIL) for Whole Slide Image (WSI) analysis. The authors propose two key components: (1) DPGS, which enables parallel processing of variable-length MIL bags by distributing them across multiple GPUs and aggregating their gradients, and (2) Deep Model-Gradient Compression (DMGC), which reduces communication overhead during distributed training through joint compression of gradients and model parameters. Experiments on Camelyon16 and TCGA-Lung datasets demonstrate significant improvements in both training speed (up to 31× faster) and classification accuracy (up to 9.3% increase) compared to baseline methods.\n\nweaknesses and questions: Strengths:\n1.\tThe mathematical derivations showing equivalence to mini-batch training are sound.\n2.\tThe gradient compression approach (DMGC) offers an interesting extension to existing gradient compression techniques.\n3.\tThe ablation studies provide useful insights into the factors affecting performance within their framework.\nWeaknesses:\n1.\tHigh resource requirements: Although the speed-up performance is impressive, the method requires multiple GPUs and high-bandwidth connections, significantly limiting its practical applicability in many research and clinical settings.\n2.\tMissing comparisons to simpler alternatives: The paper doesn't evaluate simple alternatives such as uniform sampling that could potentially achieve similar results with much less complexity.\n3.\tOutdated baselines: The paper relies on comparisons with older MIL methods rather than current state-of-the-art approaches.\n4.\tUnclear feature extraction: The \"multi-scale\" features referenced throughout the paper are not adequately defined.\n5.\tLimited relevance given foundation models: The paper doesn't acknowledge or compare against foundation models that have demonstrated near-perfect performance on the same datasets.\n1.\tCould you provide a clear definition of the \"multi-scale\" feature extraction process used in your experiments, including architectures and implementation details?\n2.\tHave you compared your approach with simpler methods like uniform sampling from each WSI to create fixed-length bags that could be trained with standard batch processing?\n3.\tCould you explain your experimental setting and the reason why you did not consider current foundation model-based approaches (UNI [1], CONCH [2], PLIP [3], etc.) that have demonstrated state-of-the-art performance on the same datasets?\n4.\tWhat is the minimum hardware configuration required to achieve meaningful benefits from your approach compared to single-GPU training?\n5.\tWhy did you choose to compare against older MIL methods rather than more recent approaches that might represent stronger baselines?\nI will consider raising the overall recommendation score if these questions are solved in the rebuttal phase.\n[1] Chen, Richard J., et al. \"Towards a general-purpose foundation model for computational pathology.\" Nature Medicine 30.3 (2024): 850-862.\n[2] Lu, Ming Y., et al. \"A visual-language foundation model for computational pathology.\" Nature Medicine 30.3 (2024): 863-874.\n[3] Huang, Zhi, et al. \"A visual–language foundation model for pathology image analysis using medical twitter.\" Nature medicine 29.9 (2023): 2307-2316."
- source_sentence: quantization prediction flips
sentences:
- >-
title: Block Rotation is All You Need for MXFP4 Quantization
summary: This paper presents an investigation into the applicability of
existing post-training quantization (PTQ) methods for LLMs under the
emerging MXFP4 format. The core finding that global rotation-based
methods are fundamentally incompatible with MXFP4's block-wise scaling,
thus they posit that a simple block-wise rotation (BRQ) is able to
mitigate this issue. They provide extensive experiments and analysis.
While the topic is timely, the paper suffers from a fundamentally
incremental contribution and a lack of technical novelty that is enough
to question its value to the quantization community. The core insight is
deemed obvious for the target audience, and the solution does not
constitute a significant algorithmic advance.
weaknesses and questions: 1. The central problem and its solution are a
straightforward, expected outcome for anyone with deep expertise in
quantization. Applying a block-level transformation to align with a
block-level quantization scheme is a natural and almost trivial
engineering adjustment, not a novel research contribution. The MXFP4
format, by design, uses local block scaling (PoT) to contain outliers.
Applying a global operation that deliberately spreads out outlier energy
directly counteracts the format's core design principle. Therefore,
observing a performance collapse is not a discovery; it is a
confirmation of a predictable hardware/algorithm mismatch.
2. The proposed BRQ method is a direct and obvious application of
existing concepts. It simply restricts the well-known rotation transform
to the block granularity defined by the hardware. This does not
represent a new algorithm or a conceptual breakthrough.
3. They selected 'datasets and benchmarks' as the primary area, but
there is no new datasets or benchmarks provided.
NA
- >-
title: Robust Training of Neural Networks at Arbitrary Precision and
Sparsity
summary: Quantization aware training (QAT) is a widely used method to
prepare models for quantization. To overcome the rounding operation
during backward pass, QAT uses straight through estimator (STE) which
bypasses the gradients in the backward pass outside of the rounding
operation. This work, characterizes this feature as the backward pass
being quantization oblivious. To overcome this, the authors present a
workaround that exposes the backward pass to the quantization error
using a combination of ridge regression and affine quantization.
Experiments on a wide array of models, and bit regimes shows superior
performance of the proposed method when compared to plain STE-based QAT.
weaknesses and questions: * **QAT and error minimization:** The
fundamental hypothesis of this work is that during the backward pass,
QAT is oblivious to the quantization error. While this is possible, it
has been shown by now that STE creates a different type of dynamic that
results in weight oscillations. And these oscillations have factors that
are trying to compensate the errors due to QAT [1,2]. Given this, what
do the authors make of these explanations? And how does it alter their
hypothesis, or not? If not, what is their argument?
* **STE beyond 2 bits:** Even if one were to attribute STE for poor
performance in extremely low-bit width, how is this not manifested to
the same degree in higher bit regimes. What is the explanation? Is STE
more problematic only in extremely low-bit regimes?
* **Ternary quantization with STE:** There are several works that use
STE for ternary quantization [3,4]. The critique that STE cannot be used
for extremely low-bit regimes does not hold up.
* **Presentation clarity:** The ideas, experiments, and results are
quite compelling in this work. However, the presentation is unclear in
many places. There are vague statements, unsubstantiated by evidence
(discussions around biological neurons, intelligence), and presentation
of results makes it difficult to parse them. Many of the interesting
results are in the Appendix (Fig. 3, Table 1) whereas the main results
in Fig. 1 and Fig. 2 are illegible, with no clear captions, legends,
axes labels. This is unfortunate as it dilutes the impact of otherwise
nice contribution. I would suggest improving these aspects.
* L-67: Very vague statement with exaggerated claims. It is by now
common knowledge that large, quantized models outperform smaller ones.
And also the claim of biological intelligence is extremely misplaced.
### Other comments
* Reference to Figure 1-a in L-37 is not useful as none of the concepts
are fully introduced; consider dropping this reference or elaborating
the caption so that it can independently explain the concepts in the
figure.
* L-38: What are the heuristic-based modifications authors are pointing
to? No references to back this up.
* L-54: Strange sentence; perhaps missing a preposition somewhere. Did
the authors mean "full potential of the theoretically..."
### References
1. Wenshøj, Jonathan, Bob Pepin, and Raghavendra Selvan. "Oscillations
Make Neural Networks Robust to Quantization." arXiv preprint
arXiv:2502.00490 (2025).
2. Xie, Weiying, et al. "Allowing Oscillation Quantization: Overcoming
Solution Space Limitation in Low Bit-Width Quantization." Proceedings of
the IEEE/CVF International Conference on Computer Vision. 2025.
3. Choi, Jungwook, et al. "PACT: Parameterized Clipping Activation for
Quantized Neural Networks." (2018).
4. Wang, Jinheng, et al. "1-bit ai infra: Part 1.1, fast and lossless
bitnet b1. 58 inference on cpus." arXiv preprint arXiv:2410.16144
(2024).
See weaknesses above.
- >-
title: Nearly Lossless Adaptive Bit Switching
summary: The paper addresses challenges in model quantization for deep
neural networks (DNNs), focusing on optimizing quantization-aware
training (QAT) across multiple bit-widths with weight-sharing. To this
end, this paper introduces a novel quantization method that exploits the
highest integer precision to achieve nearly lossless bit-switching,
reducing storage without relying on full precision. Key contributions
include: (1) Adaptive Learning Rate Scaling: A technique that
dynamically adjusts learning rates for different precisions to address
competitive interference and inconsistent gradient issues during
one-shot joint training. (2) Double Rounding: An extension for one-step
rounding quantizer in fixed-precision quantization to improve accuracy.
Experimental results on the ImageNet-1K dataset show that the proposed
methods surpass state-of-the-art approaches in both multi-precision and
mixed-precision scenarios, achieving higher efficiency and accuracy.
weaknesses and questions: - Some analysis is missing. For example, I'm
wondering whether the second rounding leads to more quantization errors,
as the first rounding is used to produce INT8 weights and second
rounding is then performed to quantize lower bit-width, the twice
quantization is possible to cause more clipping errors and rounding
errors, some analysis could enhance the strength of proposed methods.
- Some designs should be further clarified, e.g., why ALRS is applied
only for the scaling factors? Intuitively, weights of small bit-width is
induced large gradient variance by STE, and thus the weights of small
bit-width should also benefit from using smaller LR.
- Fig. 1 is a bit confusing, some colored arrows are not well
explained.
- This works essentially lies in the research of mixed-precision
quantization, so I think it is better to compare more MPQ (e.g., HAQ,
DNAS, LIMPQ, etc) research in the Sec.4. Moreover, some recent papers on
multi bit-width quantization are missed on the , e.g., [1] (PTQ-based)
and [2][3] (QAT-based), which could be included into the Related Work.
[1] Xu, Ke, et al. "PTMQ: Post-training Multi-Bit Quantization of Neural
Networks." Proceedings of the AAAI Conference on Artificial
Intelligence. Vol. 38. No. 14. 2024.
[2] Tang, Chen, et al. "Retraining-free model quantization via one-shot
weight-coupling learning." Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition. 2024.
[3] Zhong, Yunshan, et al. "MultiQuant: A Novel Multi-Branch Topology
Method for Arbitrary Bit-width Network Quantization." arXiv preprint
arXiv:2305.08117 (2023).
Please refer to the weaknesses.
- >-
title: FPTQuant: Function-Preserving Transforms for LLM Quantization
summary: This paper introduces FPTQuant, a novel PTQ framework designed
to mitigate the severe activation outliers in LLMs. The authors cleverly
design four lightweight FPTs—including a mathematically elegant pre-RoPE
transform and MLP scaling techniques—that smooth activation
distributions without altering the exact mathematical outputs of the
unquantized model. By co-optimizing these transforms locally and
end-to-end, FPTQuant achieves highly accurate static INT4 quantization
with virtually zero inference overhead, completely bypassing the need
for slow, bespoke CUDA kernels.
weaknesses and questions: Strength:
1. The block-diagonal pre-RoPE transform is good. Instead of fighting
the architecture, exploiting the equivariances of modern Transformers to
commute with the RoPE matrix is exactly the kind of principled,
math-first solution the quantization community needs right now.
2. This method folds transforms directly into the weights. Avoiding
writing CUDA kernels for non-standard hardware is crucial for real-world
deployment and cross-hardware compatibility.
Weakness:
1. Claims of "virtually no overhead" are great, but theoretical FLOP
reduction isn't enough. Hardware metrics are needed—like e2e latency,
TTFT, and decode throughput (Tokens/sec) across varying batch sizes on
standard GPUs (A100/H100)—to prove that memory bandwidth isn't secretly
being bottlenecked.
2. Because FPTQuant requires both local and end-to-end training of the
transforms, what is the actual wall-clock time and peak GPU memory
required for the calibration phase? Especially compared to other
training-based pts methods like spinquant.
3. Real-world INT4 usually relies on group-wise quantization (e.g.,
group size 64/128) to maintain accuracy. Since the FPTs use channel-wise
scaling to tame activation outliers, they inevitably alter the weight
variance within those specific groups. It's a noticeable gap that the
paper doesn't analyze—either theoretically or empirically—whether these
transforms inadvertently blow up intra-group weight variance. If so,
this would severely degrade group-wise quantization fidelity.
See weakness
- >-
title: Large Language Models do Not Make Complete Use of Math Reasoning
Data
summary: The paper studies per-item generalization when fine-tuning LLMs
on math-reasoning tasks. As training data is incrementally increased,
many test items that were previously answered correctly flip to
incorrect, so net accuracy improves only marginally because “newly
correct” and “newly incorrect” items roughly cancel. The finding is
demonstrated for SFT (e.g., Llama-3-8B, Gemma-3-4B on GSM8K/MAWPS) and
RL (Qwen2.5-0.5B on GSM8K/MATH8K). The authors quantify a “Union vs
Final” gap (items solved by any intermediate model vs the final model),
show cross-seed “predictive multiplicity” (same data, different seeds →
different test items solved), and offer a high-level explanation via a
“strategy set” view of reasoning traces. Ablations suggest sample order
and LoRA dropout contribute to divergence across seeds. No new training
method is proposed; the work is diagnostic/observational.
weaknesses and questions: – Limited causal explanation: the paper
establishes that flips occur, but the proposed “strategy set/Rashomon”
lens is descriptive and not predictive; it does not isolate
necessary/sufficient causes of flipping or quantify their contributions
(e.g., data conflicts vs optimization noise vs under/over-fitting vs
decoding effects).
– Scope is narrow (math-reasoning, modest model sizes); it is unclear
how broadly the phenomenon holds (code, multilingual,
instruction-following, safety, etc.), or how it scales with much larger
base models and longer training.
– No actionable solution: beyond noting order/dropout effects, the work
stops short of proposing methods to reduce flips or close the
Union-vs-Final gap (e.g., curriculum, data reweighting, conflict
detection, checkpoint ensembling, order-invariant updates,
strategy-diversity regularizers).
– Experimental controls are thin in places: small number of seeds;
limited statistical testing; compute/training-length/early-stopping
effects not deeply probed; decoding settings (e.g.,
temperature/self-consistency) only partially explored.
– The “union” signal naturally suggests simple mitigations (checkpoint
ensembling across data-subset steps, EMA over training,
mixture-of-checkpoints) that are not tried; without testing such
baselines, the practical impact remains unclear.
– Theoretical component does not yield falsifiable predictions (e.g.,
when flips should increase/decrease given measurable dataset/model
properties).
• What fraction of flips can be attributed to measurable data conflicts
(near-duplicates with differing rationales/solutions, annotation noise)
vs optimization stochasticity? Can you quantify this via conflict
detection or per-sample gradient similarity analyses?
• Can you predict which items will flip when adding data? For instance,
are low-margin items (by log-prob gap), longer reasoning chains, or
particular operation types more flip-prone?
• Does the phenomenon persist with substantially larger base models and
longer training to convergence? How does it scale with training
steps/epochs and gradient noise scale?
• What is the effect of decoding schemes (temperature,
self-consistency, verifier-guided selection, tool-use) on flips beyond
majority voting?
• Do simple mitigations narrow the Union-vs-Final gap: (a) checkpoint
ensembling across subset steps, (b) EMA of weights, (c) curriculum or
order-invariant batching, (d) removing LoRA dropout and fixing order for
all conditions, (e) data deduplication/cluster-balanced sampling?
• Does full-parameter fine-tuning (no adapters) or different PEFT
choices alter the flip rate?
• Can the strategy-set view be made predictive (e.g., estimating
effective strategy entropy per item) and tested against flip rates?
- >-
title: Accuracy is Not All You Need
summary: The authors analyze changes in model predictions as an
additional performance metric to evaluate quantization schemes applied
to large language models (LLaMa2 and Yi). The authors show that during
model quantization that the predictions for a substantial number of
examples change, beyond the limited number of examples which labeled
incorrect post quantization. Additionally, the authors show that the
number of examples that change in prediction is correlated with the
KL-Divergence between the base and quantized model.
weaknesses and questions: The overall results are unsurprising and do
not provide additional characterizations of the effects of quantization
on model performance or example characteristics.
- It is expected that model predictions with lower confidence are more
likely to change upon compression. However, the current analysis does
not provide insight into how these examples change during compression.
For example it is not clear if the reason that low confidence examples
change in prediction is because they are initially close to the decision
boundary or if they observe greater change in model confidence during
compression.
- Change in model prediction has been studied both in the context of
continual learning and interpretability analysis of model training
dynamics [1,2]
Related Work
1. Toneva, Mariya, et al. "An Empirical Study of Example Forgetting
during Deep Neural Network Learning." International Conference on
Learning Representations. 2018.
2. Swayamdipta, Swabha, et al. "Dataset Cartography: Mapping and
Diagnosing Datasets with Training Dynamics." Proceedings of the 2020
Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
1. Are the flipped examples consistent across different compression,
quantization schemes, or models?
2. For a single method scheme with varying degrees of compression (GPTQ
W8A16 -> GPTQ W4A16; or pruning at different intensities), do the flips
increase monotonically, consistenlty with increasing degrees of
quantization? For example, are the examples that are forgotten at 8-bit
quantization relearned at 4-bit quantization?
3. The top margin (difference in probability) metric is used to
characterize the model performance pre-quantization; A. What does the
metric indicate post quantization? B. Does quantization result in a more
uniform probability distribution? C. Are the shifts in predicted
likelihood and confidence larger or smaller for examples that were
initially higher or lower in confidence?
- >-
title: Quantization Hurts Reasoning? An Empirical Study on Quantized
Reasoning Models
summary: This paper provides a comprehensive empirical study on the
quantization of reasoning models. They evaluate SoTA quantization
methods across weight-only, weight-activation, and KV cache
quantization. Their conclusion is:
Lossless settings: W8A8 or W4A16 are consistently < 1 % accuracy drop;
W4 weight‑only or KV‑cache is “near‑lossless” on most models.
Best algorithms: AWQ for weight‑only, QuaRot for KV‑cache (except small
Qwen), FlatQuant for weight‑activation.
Failure modes: 3‑bit quantization, small models, harder tasks and
RL‑trained models (QwQ) are much more fragile.
No extra thinking: mild quantization does not lengthen CoT while
aggressive low‑bit settings do.
Scaling trade‑offs: 4‑bit large models beat smaller BF16 models in both
accuracy‑per‑GB and accuracy‑per‑second.
weaknesses and questions: 1. No novel ideas are proposed. It is more
about empirical study paper. The main contribution of this paper is
doing experiments for the current quantization algorithms, and
conclusion might be change as new approaches evolve. This paper would be
benefit from this by proposing a leaderboard that support evaluations of
any quantization algorithms.
2. It does not really explain why certain quantization algorithms fail
or do well. For example, it states "AWQ is preferred over GPTQ", "QuaRot
is generally preferred over KVQuant*". This paper could be benefit if
author can connect those empirical conclusions (accuracy drop) with some
theoretically justifications and provide some insights in designing new
algorithms.
- >-
title: Unveiling the Potential of Quantization with MXFP4: Strategies
for Quantization Error Reduction
summary: MXFP4 and NVFP4 exhibit a significant gap in fidelity. The
authors trace MXFP4's lower accuracy to its rigid power-of-two scaling
factors and coarser block granularity, which struggle to preserve
outliers. To mitigate this without requiring hardware modifications, the
authors introduce two software-only techniques, Overflow-Aware Scaling
(OAS) and Macro Block Scaling (MBS), that improve MXFP4 quantization
fidelity without requiring hardware changes. These methods reduce the
gap between MXFP4 and NVFP4 to roughly one-tenth of its original size.
weaknesses and questions: Strengths:
1. The problem addressed in this paper is important and impactful, and
has significant value for the community.
2. The experimental results are substantial: the average accuracy gap is
reduced from 10% to 1%.
Weaknesses:
I am not very familiar with this field. The methods appear to be sound.
Other weaknesses and limitations can be referenced in the questions
below.
1. The evaluation appears to omit calibration-based post-training
quantization baselines. Modern deployments typically rely on algorithms
such as GPTQ, AWQ, or SmoothQuant to handle outliers before format
conversion. If weights and activations have already been smoothed, how
much additional benefit would the proposed methods provide?
2. Could the authors provide token generation latency benchmark results?
3. Could the authors include a histogram showing the frequency with
which the OAS threshold is triggered?
- source_sentence: semantic labels data efficient learning
sentences:
- >-
title: Language Semantic Graph Guided Data-Efficient Learning
summary: The paper introduces the Language Semantic Graph (LSG), a novel
approach to data-efficient learning that leverages semantic information
from labels. The LSG is used to train an auxiliary graph neural network,
which then guides the primary model's training, enhancing the
utilization of label knowledge. This method is applicable across various
modalities, including image, video, and audio, and has shown significant
performance enhancement in both Transfer Learning and Semi-Supervised
Learning scenarios.
Experiments were conducted on seven standard datasets covering images,
videos, and audios, using several deep neural networks with different
architectures and pretraining datasets. The results show that LSG
significantly outperforms other methods, especially when labeled data is
scarce. It also demonstrates promising potential in semi-supervised
settings, achieving the best performance across all labeling rates and
datasets. When applied to self-supervised pretrained models, LSG shows
consistent gains. It also improves model performance on both
in-distribution and out-of-distribution samples, indicating that label
semantic relations help the model learn more robust features.
In video and audio experiments, LSG consistently improves the
fine-tuning accuracy across all tasks with limited labeled samples. It
outperforms other methods, boosting accuracy significantly. For audio
experiments, LSG achieves an average of 5.56% accuracy enhancement from
the baseline, demonstrating its wide applicability across various
modalities.
"LSG consists of two parts: an auxiliary graph neural network that
extracts knowledge from the semantic graph and two novel optimization
objectives that transfer the knowledge to primary models." The authors
demonstrate that LSG is applicable on image, video and audio models and
brings significant performance gains to the model under Transfer
Learning and Semi-Supervised Learning scenarios.
weaknesses and questions: The effectiveness of the LSG method relies
heavily on the quality and semantic richness of the labels. In scenarios
where labels are sparse, ambiguous, or not well-defined, the performance
of the LSG method could be compromised.
*The LSG method relies heavily on the quality and semantic richness of
the labels. How does the quality of the labels impact the performance of
the LSG method? Could the LSG method be adapted to work effectively with
less informative or ambiguous labels, and if so, how?
* The paper primarily focuses on classification tasks. Could the LSG
method be adapted or extended to other tasks, and if so, what
modifications would be necessary?
- >-
title: Concept Bottleneck Models under Label Noise
summary: The paper is concerned with inherently interpretable models
called Concept Bottleneck Models (CBMs). These models require extensive
concept labeling; however, these labels are usually assumed to be
perfect. The paper explores how noise in these concept labels affects
the final target prediction. The authors perform extensive experiments
across all three variants of CBMs to show the detrimental effects of
concept labels. The authors then proposed SAM training to improve
concept and target accuracy.
weaknesses and questions: - While the experiments are comprehensive for
Section 3 and 4. Some of the results are pushed to the appendix (which
is fine), however it would have been nice to summarise them in brief in
the text.
- I enjoyed reading up to Section 4. Thank you. However, I would have
appreciated some theoretical intuition on why SAM works better (unless I
missed it).
- The paper performs experiments with CUB and AwA2 datasets, popular
benchmark datasets for CBMs. These datasets however have a strong
correlation between concept labels and the target label. I can imagine
label noise to be very detrimental (as observed from Figure 2). The
potential observed effect due to label noise might be weaker in the case
of diverse concepts for each target.
- The paper claims to be first the paper to looking at label noise in
CBMs, while I would not refute this, I would like to point out the
authors to some very relevant papers - [1] (noise added to concept
labels, similar to some of the exps in Sec3) [2]-(concept robustness and
adv attacks)
[1] - Sheth, Ivaxi, and Samira Ebrahimi Kahou. "Auxiliary losses for
learning generalizable concept-based models." Advances in Neural
Information Processing Systems 36 (2024).
[2] - Sinha, Sanchit, et al. "Understanding and enhancing robustness of
concept-based models." Proceedings of the AAAI Conference on Artificial
Intelligence. Vol. 37. No. 12. 2023.
- In Sec 6.1 authors show that other CBM variants are also susceptible
to label noise. This alings with CBMs, however the obvious question for
me is, does SAM training improve the robustness? Why or Why not?
- Concept labeling CBMs is very difficult, there is an increasing
interest in using LLMs for concept annotation. Can the authors kindly
comment (maybe in Limitations section of the paper), on whether label
noise will impact such concept labels.
- Interventions are useful aspect of CBMs. What is the impact of
interventions to reduce label noise? I assume interventions may be less
effective. Does SAM improve it?
Minor:
- Line 101, now caps for "We".
- Figure 3, which model is used not specified? Joint/Ind/Seq?
- >-
title: DREAM: Dual-Standard Semantic Homogeneity with Dynamic
Optimization for Graph Learning with Label Noise
summary: This paper addresses the important and timely problem of robust
graph learning under label noise, proposing a dynamic reweighting
scheme, DREAM, that integrates relational information. During training,
it dynamically reweights labeled nodes using semantic homogeneity
scores. This method is well-motivated and potentially applicable to many
graph learning settings. Its dual-criteria anchor selection mechanism
combines proximity-aware and topology-aware anchors. Experimental
results show that, on standard benchmark datasets and noise models, this
method achieves significant and stable improvements compared to several
strong baselines.
weaknesses and questions: Strengths:
* Semantic homogeneity scores based solely on graph structure
relationships, rather than solely on loss values or simple auxiliary
regularization terms, provide a direct, relation-informed reliability
signal.
* This method addresses a well-known problem in node classification
under label noise: instead of relying solely on small-loss heuristics or
global consistency terms, it explicitly utilizes graph structure
relationships to determine which labels are trustworthy. This method can
be integrated with standard GNN backbones, showing potential value for
practical applications in handling noisy graph labels.
* Experiments were conducted on six benchmark datasets and three noise
types. Compared to a series of LLN and GLN baseline methods, the
proposed method achieves significant and stable improvements, and
ablation experiments validate the effectiveness of each module.
Weaknesses:
* The theoretical guarantee of this method requires that the semantic
homogeneity score $H(n_i;A(n_i);\tau)$ must be a $(\beta,\epsilon)$
approximation of the ideal importance weight
$\alpha(x_i,\hat{y}_i,y_i)$. However, this paper neither provides the
conditions for this approximation to hold nor offers strong quantitative
evidence that $H$ can reliably reflect the cleanliness of the labels.
* Proximity-aware anchors relies on noisy labels and is prone to
confirmation bias. The nearest neighbor candidate set
$C_P(n_t)=\{n_i\mid i\in S \wedge y_i=y_t\}$ is constructed based on
observed labels (which may have been contaminated). In cases of high
noise rates, small sample classes, or weak representations in the early
stages of training, anchor point selection may be biased towards nodes
that have been mislabeled as "same kind," reinforcing erroneous
supervision signals before the embedding becomes discriminative.
* Equation (2) implicitly assumes that the candidate pool is large
enough to select exactly $k_P$ nearest neighbor anchors, but the paper
does not explain how to handle the situation where $|C_P(n_t)|<k_P$ in
scenarios with small categories and few labels.
* The potential sensitivity and stability issues caused by sharpening
temperature scaling are not fully explained. The homogeneity weight uses
$H=s^{1/\tau}$, where $\tau$ defaults to a small value (0.04), and the
similarity is normalized to $[0,1]$, which makes the weight distribution
extremely sharp. Although ablation experiments show that temperature
scaling has an effect, the paper does not discuss stability strategies,
nor does it perform sensitivity analysis on $\tau$ and $d_{\max}$ on
multiple datasets.
* Q1: When the class size is small, i.e., $|C_P(n_t)|<k_P$, how do you
handle this? Do you directly use all available candidate samples, or
dynamically adjust the value of $k_P$?
* Q2: How sensitive is the model performance to the temperature
parameter $\tau$ and the geodesic threshold dmax on different datasets?
Have you tried scheduling $\tau$ during training to mitigate potential
training instability issues?
* Q3: Can you provide empirical evidence that $H$ is related to label
cleanliness(e.g., how H changes with training epochs when distinguishing
clean labels from noisy labels), and that this conclusion still holds
true when the noise rate is higher ($\ge$ 50%)?
* Q4: In heterophilous graphs like Flickr (with low homophility), what
is the class purity of the topology-aware anchor $A_T(n_t)$? If the
class mixing of $A_T$ is high, have you considered incorporating a weak
class consistency prior (e.g., based on prediction consistency) when
selecting topology anchors? Would this approach improve the model's
robustness?
- >-
title: Learning to Label: A Reinforced Self-Evolving Framework for
Semi-supervised Referring Expression Segmentation
summary: This paper proposes Learning to Label (L2L), a framework for
semi-supervised referring expression segmentation (SS-RES). The method
uses a frozen MLLM (Qwen2.5-VL) to generate bounding box predictions for
unlabeled image–text pairs, which are then fed into a frozen SAM2 to
produce soft segmentation priors. These external priors are fused with
the segmentation model's own predictions via an uncertainty-aware module
(SPM), injected into the encoder via stage-adaptive gating (SESM), and
used to generate pseudo-labels whose pixel-level selection is governed
by a DDPG-based reinforcement learning agent (RPLE). The actual
segmentation network (Swin Transformer + BERT) is the only component
trained end-to-end. Experiments on RefCOCO, RefCOCO+, and RefCOCOg under
0.1%–10% label budgets show improvements over existing SS-RES methods.
weaknesses and questions: ## Strengths
- **Well-motivated observation.** The analysis of confidence mismatch
between MLLM priors and model predictions (Figure 1) is insightful and
clearly illustrates why fixed-threshold pseudo-labeling is suboptimal
for this task. The scatter plot effectively demonstrates the
sample-dependent nature of disagreement.
- **Practically relevant problem.** Reducing annotation cost for RES is
genuinely important, and the semi-supervised formulation is a natural
direction. The paper clearly articulates why standard SSL methods
struggle with referring expression segmentation due to referential
ambiguity and boundary uncertainty.
- **Clean framework design.** The overall pipeline—external prior
generation, uncertainty-aware fusion, conditional guidance injection,
and adaptive pseudo-label selection—is logically coherent and
well-structured.
## Weakness
- The dominant source of improvement is the external foundation model,
not the proposed method. This is the most critical concern. Table 1
reveals that the overwhelming majority of performance gain comes from
simply using MLLM+SAM2 priors, not from the proposed modules (SPM, SESM,
RPLE). For further rigorous demonstration, more advanced heuristic
should be added to the baseline. In my opinion, the common pseudo-label
method without MLLM little provide the information to demonstrate the
effectiveness of the proposed methods Specifically, the Baseline (w/
MLLM) uses a fixed mixing coefficient of 0.5, which is an intentionally
naive design. I suggest to include (a) Baseline (w/ MLLM) with a tuned
mixing coefficient, (b) agreement-based filtering (use pseudo-labels
only where both MLLM and model are confident), (c) any adaptive
thresholding method (e.g., FlexMatch, CATM from ST-SGG [1, 2]). These
comparison will provide how good the proposed modules are when doing
semi-supervised learning.
- I have concern on the RL formulation (RPLE) in terms of the problem's
complexity. The state space is 6-dimensional and the action space is
2-dimensional. DDPG is designed for high-dimensional continuous control
problems—using it for a 6 -> 2 mapping is excessive. The authors should
claim what is the advantage of using RL approaches compared to the
simple adaptive thresholding like [1, 2].
- While MLLM and SAM2 are used offline, the overhead of generating
priors for all unlabeled data, plus the RL agent's training cost, is
never reported.
[1] FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo
Labeling
[2] Adaptive Self-training Framework for Fine-grained Scene Graph
Generation
- Can you add natural heuristic baselines—using pseudo-labels only where
both the MLLM prior and model prediction are confident
(intersection-based filtering).
- Given that the state space is 6D and action space is 2D, have you
compared RPLE against simpler adaptive thresholding baselines?
- The Baseline (w/ MLLM) uses a fixed mixing coefficient of 0.5. What
happens when this coefficient is optimized via grid search on the
validation set?
- >-
title: Learning with Selectively Labeled Data from Multiple
Decision-makers
summary: This paper focuses on the problem setting of classification
with selective labeled data, that is, the labeled data at hand can be
biased because of decision-making in the past. This paper defines the
problem mathematically and solves this problem from the perspective of
the instrumental variable (IV) framework. There are two assumption
settings: (1) No unmeasured common effect modifiers (NUCEM), which is a
strong assumption that leads to a clean solution, and (2) Partial
identification, where a reasonable solution can be obtained. Theoretical
analyses of two assumption settings are provided. Furthermore, a
practical algorithm for both cases is also provided based on weighted
empirical risk minimization with calibration guarantee. Synthetic
experiments show that the proposed method outperforms baselines.
## update after rebuttal
After the rebuttal, I still think the idea of this paper is novel. It
studies the problem setting extensively theoretically and also provides
experimental results., Thus, I keep my score (4: accept). The authors
clarified in the rebuttal that their work has novelty and also admits
some current drawbacks of their methods (e.g., computation time).
weaknesses and questions: Strengths
1. Strong theoretical results that improve an understanding of a
complicated yet relevant problem setting. It is praiseworthy that this
paper not only focuses on a restrictive NUCEM assumption but also
considers the partial information assumption.
2. Practical algorithms with theoretical guarantee are provided, which
can be relatively easy to implement.
3. Experimental results (although synthetic) show that the proposed
method is effective compared with reasonable baselines.
Weaknesses
1. Proposed method's weakness is not much discussed in my understanding.
One might be that it could be computationally expensive (I'm not sure).
Moreover, the estimation of weight could be incorrect, and we don't see
much effect in the experiments, whether this can make the proposed
method not work well. I find the comment in the paper, why NUCEM lost to
partial under NUCEM assumption quite interesting that NUCEM requires a
ratio estimation. I think such discussions could be useful. Or ablation
study of the effect of imprecise weight estimation could also be useful.
(but I'm also aware that the paper is already dense unfortunately).
1. Could you please comment on the comparison of the computational cost
of the proposed unified cost-sensitive learning (point), (partial), and
vanilla training?
2. Since many weights have to be estimated, how important is the
accuracy of weight estimation? Is the solution highly sensitive to this?
3. Is this the first work to use instrumental variable (IV) framework
for selective labeled classification?
- >-
title: Hard Labels In! Rethinking the Role of Hard Labels in Mitigating
Local Semantic Drift
summary: This paper proposes to utilize hard labels to both address the
massive storage bottleneck and improve performance for dataset
distillation methods. The paper identifies that reducing cropped views
introduce a problem called local-view semantic drift, when crops are
assigned soft labels that deviate from the ground-truth image label. To
solve this, the paper proposes a 3-stage training scheme that consists
of 2 soft-label stages and a key hard-label stage in the middle that
theoretically improve the gradient alignment of the two label types.
Massive theoretical analysis and a semantic drift framework are provided
to support the method. Experiments demonstrate state-of-the-art
performance and a 100x reduction of storage cost.
weaknesses and questions: Strengths:
The theoretical framework of the soft-hard alignment is thorough in this
paper, with in-depth analysis of the Local-View Semantic Drift problem
studied in the paper. Meanwhile, the empirical validation shows strong
effectiveness and efficiency.
Weaknesses:
- The main method, i.e., training stage B, seems not using hard labels
for correcting signals, which is the primary motivation of the paper.
The method basically uses label smoothing + Cutmix, both still creating
soft labels.
- The theoretical part before Sec.3.2 is quite disconnected to the main
method.
- The number of compared methods are limited. The authors are suggested
to compare 1~2 more (e.g., GIFT).
- The evaluated datasets are limited to only Tiny-ImageNet/ImageNet-1K,
limiting it only in general domains.
- Minor issues:
- Notation of $s$ defined for the second time at page 4 stage A.
- Table 6 caption: right/left rather than top/bottom.
- The best-performing $\alpha$ is annotated wrong in Table 13.
1. Why is the one-hot labels not used for stage B directly?
2. How is the LS + Cutmix method able to perform gradient correction?
3. Is there intuitive explanation or empirical evidence on how the three
stages work to bound the gap in Theorem 3.5?
- >-
title: Semantic F1 Scores: Fair Evaluation Under Fuzzy Class Boundaries
summary: This paper introduces Semantic F1 Scores, a family of
evaluation metrics for multi-label classification that account for
semantic similarity between labels. The core idea is to give partial
credit when a predicted label is semantically related to the true label,
instead of counting it as a complete miss as in standard F1. The metric
uses a two-step matching (predict-to-true and true-to-predict) to
compute “semantic precision” and “semantic recall,” which are combined
into an F1 score. Semantic F1 is designed to be backward-compatible (it
reduces to standard F1 when no partial credit is allowed). Additionally,
to theoretical arguments, the authors conduct an empirical study on
eight datasets (synthetic and real) and demonstrate that Semantic F1
provides potentially fairer and more informative evaluation in tasks
with subjective or overlapping labels, correlating more closely with
real-world outcomes than classical F1.
weaknesses and questions: - I think the biggest weakness of the approach
is dependency on the similarity matrix. If the matrix S is poorly
specified or biased, scores could be misleading or unfair, which makes
it tricky to use as an evaluation metric. The defining similarity may be
non-trivial. Similarity derived from label co-occurrence or embeddings
might not truly reflect conceptual closeness. Results with different S
matrices are not comparable.
- Baseline comparison on real tasks is limited, as on real datasets,
evaluation focuses on hard F1 vs Semantic F1, with no direct comparison
to alternative semantic metrics, which also feels limited in a synthetic
study.
- I'm generally a bit confused by experiments; it is not clear to me
what is optimized and what is evaluated, and why should I care that
Semantic F1 better correlates with the task than hard F1? And what about
other soft/semantic metrics?
- Many labels on Figures 2 and 3 are so small that they are not readable
when printed.
Please, see weaknesses session.
- >-
title: Beyond One-Hot Labels: Semantic Mixing for Model Calibration
summary: This paper propose Calibration-aware Semantic Mixing, a model
calibration approach using diffusion-based data augmentation, like
“semantic mixup”.Unlike traditional one-hot labeling, CSM generates
mixed samples with soft labels with the CLIP. The authors introduce a
reannotation technique using CLIP features and investigate the influence
of loss functions prove L2 loss is good for enhances calibration.
weaknesses and questions: Weaknesses:
1. Limited discussion on computational efficiency
2. Hyperparameter sensitivity analysis is not well explored.
3. No Transformer model is included for comparison
1. How does the choice of diffusion model affect the performance of CSM?
How about we use other generative models. There are some generative
model doing interpolation , like GANs. Does it helps?
2. How does CSM compare to Mixup in terms of training efficiency and
memory usage?
- source_sentence: llm uncertainty curvature control
sentences:
- >-
title: VARIATIONAL QUANTUM ALGORITHMS ARE LIPSCHITZ SMOOTH
summary: This paper presents a quite rigorous theoretical analysis of
the L-smoothness property of VQA objective functions which scales
linearly on the # of layers P, providing a worst-case upper limit on
curvature that holds for general circuits. The authors provide a formal
proof of global L-smoothness and derive an explicit upper bound on the
smoothness constant L. Furthermore, they show that for certainc classes
of VQAs this bound may take a very simple form, all the way down to be
proportional to the depth of the underlying circuit. This result is then
connected to circuits often assumed to be relevant for practical
applications, including a diagnostic for ansatz overparameterization and
a heuristic for setting near-optimal learning rates. While the
contributions are welcome and well-supported, the analysis is confined
to an idealized, noiseless setting, which limits the direct
applicability of its conclusions to contemporary NISQ hardware.
This paper maybe useful since it can help better establish, for example,
learning rates. $L$, provides an upper bound on the curvature and
guarantees that the landscape is not infinitely "spiky and being able to
guarantee this it is crucial for gradient-based methods because it
ensures stability since if I know the maximum curvature, I can choose a
learning rate small enough ($\eta \approx 1/L$) to guarantee that the
optimization steps will not wildly overshoot a minimum.
However, to my view, this paper does not solve any of the ever present
issues of VQAs. While this paper provides a valuable formalization of
L-smoothness with the potential L-informed learning rate similar to many
classical ML problems, the more fundamental and unresolved problem for
VQAs is the lack of a meaningful lower bound on curvature, not the upper
one, a condition that manifests as the barren plateau problem where
vanishing gradients render optimization intractable regardless of the
landscape's theoretical smoothness.
weaknesses and questions: (1) The entire analysis is done in an
idealized noiseless setting. The authors do acknowledges this by
establishing the result as a theoretical baseline. However, this is a
significant limitation. The primary challenge in practical VQA
optimization stems from the stochastic nature of the objective function
landscape induced by shot noise and hardware errors of all shorts. An
analysis of L-smoothness in a setting where these dominant,
non-smoothness-inducing effects are absent provides limited guidance for
optimization on actual NISQ devices. The conclusions about stable,
predictable curvature scaling may not hold when the optimizer interacts
with a stochastic estimator of the objective function.
(2) The bound is potentially loose since the proof of Theorem 2 relies
on the inequality $\|H\|_2 \leq\|B\|_2$, where $B_{k
l}=4\|M\|_2\left\|G_k\right\|_2\left\|G_l\right\|_2$ is an element-wise
upper bound on the Hessian matrix $H$. This step can introduce a
substantial gap. The paper's own empirical results as shwon in Figure 1a
show that the measured maximum curvature, $\tilde{L}_{\text {max }}$, is
often only a small fraction of the theoretical upper bound $L_{\text
{upper }}$. So while the bound correctly captures scaling, its
significant looseness warrants a more detailed theoretical investigation
maybe. The analysis could be strengthened by discussing the conditions
under which the inequalities in the proof become equalities and what
circuit physical properties (entanglement structure, parameter
correlations) might govern the magnitude of this gap.
(3) The empirical ground truth for maximum curvature, $\tilde{L}_{\text
{max }}$, is estimated by taking the maximum Hessian norm over 1000
random parameter samples. While Appendix D. 2 provides a reasonable
justification for the stability of this estimate, this methodology
cannot guarantee that the true global maximum of $\left\|\nabla^2
f(\theta)\right\|_2$ has been found in general. For that problems where
the global optimizer is known are useful testbeds since
hiigh-dimensional landscapes may contain rare and isolated regions of
extreme curvature that are unlikely to be captured by uniform random
sampling.
(4) The proposed heuristic is designed to set a single global learning
rate. However, modern optimization heavily relies on adaptive methods
like adam. So, while the existence of such a constant is proven, this
framework is somewhat misaligned with the reality of modern, large-scale
optimization unless we want to restrict ourselves to only talk about
quantum optimization in isolation. As noted in the literature, e.g.
https://arxiv.org/abs/2210.02418 for many typical problems, objective
functions rarely satisfy uniform smoothness assumptions in a way that is
practically useful their gradients may only be locally Lipschitz
continuous, or the local curvature can vary dramatically across the
parameter space. Of course, the VQA objective is usually globally
L-bounded, as shown in this paper. But a global constant $L$, determined
by the region of maximum curvature is excessively conservative for the
majority of the landscape as far as using it for thelearning rate.
Standard gradient descent with a step size derived from this global $L$
(e.g., $\eta \approx 1/L$) would take impractically small steps thus
leading to slow convergence. This is precisely why SOTA optimizers
really care to account for local geometry. The paper's proposed learning
rate heuristic, while nice in principle, still provides a global rate,
which does not align with modern optimization paradigms. The analysis
would be significantly strengthened by contextualizing its findings
within more modern frameworks, such as local or relative smoothness of
the VQA objective in this sense.
(1) How do you expect the main results and particularly the predictable
linear scaling of curvature with depth, to change in the presence of
realistic shot noise and hardware noise? This is super crucial. Does the
concept of L-smoothness remain a useful descriptor for the stochastic
objective function that an optimizer actually interacts with?
(2) Could you provide more theoretical insight into the large gap
between the derived upper bound $L_{\text {upper }}$ and the empirically
observed $\tilde{L}_{\text {max }}$ ? Does this gap depend on properties
not captured by the bound, such as the circuit's entanglement capacity
or the locality of the observable?
(3) The trigonometric polynomial proof route in Appendix A. 6 bounds the
Fourier coefficients as $\left|d_\omega\right| \leq\|M\|_2$. Given that
these coefficients have a specific structure ( $d_\omega=\left\langle
u_\omega\right| M\left|v_\omega\right\rangle$ ), could a more refined
analysis that does not resort to this uniform worst-case bound yield a
tighter overall smoothness constant? These trigonometric polynomials,
note, are actually Hermitian trigonometric polynomials in $d$ complex
variables and the optimization takes place over the torus
$\mathbb{T}^d$. does this not induce some "structure" to be exploited so
as to further bound $L$?
(4) Regarding the learning rate heuristic, would it be more effective to
use the calibrated effective smoothness constant, $L_{\rm effective}$,
to rescale the global learning rate of an adaptive optimizer like adam,
rather than using it directly in a vanilla SGD context?
- >-
title: Text Has Curvature
summary: The paper is dedicated to the question of intrinsic curvature
of discrete textual data and asks, whether we could define a practical
measure for its estimation? The authors claim that curvature should be a
property of string data itself rather than an artifact of specific
embedding space (hyperbolic, spherical etc.) and propose "Texture," a
measure of curvature primitive that measures how left and right contexts
interact in-context. The authors claim three contributions: (1)
empirical and theoretical justification, proving that semantic inference
in natural corpora is non-flat; (2) a formal definition of Texture as a
signed curvature field via optimal transport; and (3) practical
applications for long-context inference via pruning (CURVPRUNE) and
retrieval (CURVFLAG).
weaknesses and questions: Strengths:
- The paper addresses an important question, that hasn't attracted much
attention: what does curvature mean for text itself, independent of
embedding spaces.
- The theory is rather sound and interesting. The use of holonomy and
contextual evidence interaction (CEI) as falsifiable null hypotheses is
methodologically strong. By testing against coherence-destroying
controls (suffix-swap, local-shuffle), the authors provide empirical
evidence that an approach is theoretically motivated.
- The paper shows interesting applications of their curvature estimates
with some performance improvements.
Weaknesses:
- The paper claims are somewhat misleading. Initially they state that
their curvature measure must be independent from the target model's
choice of embedding space. Although it's independent from the target
model the approach is evaluated at, it appears to be dependent on the
choice of bidirectional encoding/infilling model, isn't it?
($\textit{distilroberta-base}$ in this case)
- While the paper claims to establish discrete curvature at the word
level in the abstract, the notation is somewhat ambiguous with later
claims being made for token-level representations. Moreover, although
authors claim to take a union of Top-K tokens with an intention to
preserve the support -- however, I didn't find any discussion related to
the choice of K to be used. The use of character-level / byte-level
models for their approach looks rather more sound [1, 2].
- The computational cost of computing Texture (solving Sinkhorn scaling
per slot) may be prohibitive for very long contexts. The paper does not
provide a thorough cost-benefit analysis.
- The paper lacks extensive ablation on the choice of key
hyperparameters (support size K; context radii L, R; infilling model)
and their effects on curvature sign and magnitude. All the experiments
are limited to Llama-3-8B.
The appendices provides open-sourced code samples for reproduction,
algorithmic details, and additional experiments.
Although the topic is interesting, I think the paper would benefit from
additional revision and discussion of tokenization biases and their
impact on curvature estimates and experimental evaluation with ablations
on the choice of hyperparameters and the sensitivity of their approach
to these choises. Overall, I believe the work is still quite immature
and is not yet ready for publication.
[1] Juan Luis Gastaldi, John Terilla, Luca Malagutti, Brian DuSell, Tim
Vieira, & Ryan Cotterell (2025). The Foundations of Tokenization:
Statistical and Computational Concerns. In The Thirteenth International
Conference on Learning Representations, ICLR 2025, Singapore, April
24-28, 2025.
[2] Jonathan Hayase, Alisa Liu, Noah A. Smith, & Sewoong Oh (2025).
Sampling from Your Language Model One Byte at a Time. arXiv preprint
arXiv: 2506.14123.
- Is curvature dominated by local-interactions only, i.e. we should
primarilly consider small L and R?
- How is $s_{ref}$ chosen in Eq.(2)? Is the condition in Certificate I
necessary or sufficient?
- >-
title: Debiasing Mini-Batch Quadratics for Applications in Deep Learning
summary: This paper demonstrates a systematic bias in stochastic 2nd
order approximation of empirical risk for neural networks. The study
start from an observation showing that a particular curvature structure
on minibatch Hessian is biased towards underestimation. Then, authors
proposes a debiasing method.
weaknesses and questions: I believe the notion of unbiased estimate is
not well defined and studied. Define the Hessian associate with
full-batch training loss as $$H(x) = \frac{1}{n} \sum_{i=1}^n \nabla^2
f_i(x)$$. Similarly, we define the unbiased estimate of this matrix as
$$H'(x) = \frac{1}{m} \sum_{k=1}^m \nabla^2 f_{i_k}(x)$$ where $i_k$ are
uniformly drawn from $\{1,\dots, n\}$.
For each fixed vector $d$, we have $$E d^\top H' d^\top = d^\top H d$$.
Thus, the **directional curvature**, defined in the paper, is unbiased.
But, why this paper observe a bias in the estimate? The main issue is
that they consider random directions $d$ depending on minibatches. I do
not know why we need to choose random $d$s?
I believe the quantity of interest is not well defined and motivated,
here. It is easy to provide an unbiased estimation of $q$, directional
curvature and directional slop. But, the authors want to estimating
another quality that they do not exactly defined and it is not clear how
its estimation connect with Newton's method or Laplace approximation.
If you want to estimate the maximum/minimum eigenvalue of the Hessian or
even its condition number, the empirical Hessian provides an
asymptotically unbiased estimate (see for example
https://arxiv.org/pdf/1912.10754).
- I recommend to replace $u_1$ and $u_2$ in Figure 1 by leading
eigenvectors of full-batch Hessian to better grasp my comment in
weaknesses. I expect to see that increasing batch size will lead to a
better approximation of the curvature.
- What is exactly the quantity that you want to estimate and why it is
important to estimate? Directional curvature on which random directions?
- What do you exactly mean by bias? To prove an estimate has bias, we
need to take average. But, experiments in Figure 1 are not computing an
average. How can I conclude from these experiments that the estimate is
biased?
- >-
title: What's the Magic Word? A Control Theory of LLM Prompting
summary: This work presents a control perspective of the LLM
steerability by introducing the concept of $k-\epsilon$ controllability.
They also proved a bound on the controllability of self-attention in
terms of the singular values of its weight matrices. In addition,
several experimental studies have performed to compute the $k-\epsilon$
controllability of LLMs (Falcon-7b, Llama-7b, Falcon-40b). The authors
conclude that LLMs are very controllable and the control prompts of 10
tokens or less are sufficiently enough to ensure the LLM output the
target token.
weaknesses and questions: Section 4 only considers a self-attention
head, which is quite simple and limited (compared to the current model
used in LLM). What are the difficulties in generalizing such results to
a more complex model?
The presentation of Section 4 can be further improved. The relationship
between state controllability (Definition 7) and $k-\epsilon$
controllability (Definition 6) should be discussed, i.e., implications
of your theory result in Section 4. In addition, I am confused about
some notations: are $u_i$, $x_i$ the embeddings of the tokens?
Previously the u and x are presented as tokens, it does not make sense
to make $\|u_i\|\le 1$ and $\|x_i\| \le 1$ if they are tokens. In
addition, is this assumption valid in real LLMs?
Although introducing the controllability of LLM from a control
perspective is interesting, the experimental results of checking the
controllability of the LLMs are not very exciting given the existing
results from previous work [Zou 2023]. The experiment setup is almost
identical to GCG work and the obtained results are also within
expectation. Instead, proposing a new method to study the
controllability of black-box LLMs will be more interesting.
1. What are the connections between state controllability (Definition 7)
and $k-\epsilon$ controllability (Definition 6)? Does the former imply
the later?
2. Are the assumptions $\| u_i\| \le 1$ and $\| x_i \| \le 1$ realistic?
If not, is your results in Section 4 still hold?
- >-
title: Uncertainty-Aware LLM Probing
summary: The authors examine how effective uncertainty quantification
methods are at quantifying the uncertainty of LM probes, and design
their own gradient-based quantification method. They find that no method
is robust to domain shifts, but that their method can detect OOD
effectively.
weaknesses and questions: 1. The writing overall tends to be unclear and
not well-organized. Many sentences lack specificity or need to be better
defined. (ex. Line 58 “turns out to be sub-optimal”—how? Lines 75-77,
“Based on evaluations on uncertainty quantification in traditional ML,
we would expect the uncertainty methods to work reliably here”—which
evaluations, and why?)
2. Some of the results would benefit from further analysis, as
inconclusive results are reported without investigation into the cause.
For instance, lines 313-314, “More precisely, we sometimes observed very
high performance, but very low one at other times”, does not present any
hypothesis for why this was the case.
3. The performance benefit on ID data does not appear to be consistent
and significant. It only outperforms all other methods in “average rank”
with 2/5 models. Figure 2 also suggests to me that these metrics all
achieve similar scores, which does not present a compelling case for
using ADGrad over other established methods.
4. While OOD data detection is useful, it is not immediately obvious why
this should be compared to uncertainty quantification methods as
baselines and not baselines for OOD data detection. This seems like a
separate application.
Why report the average rank of the scores and not simply average over
all categories? This seems somewhat arbitrary.
- "title: Representational Curvature Modulates Behavioral Uncertainty in Large Language Models\n\nsummary: The paper argues that straighter hidden-state trajectories in LLMs are associated with lower next-token uncertainty. It shows that this curvature–entropy link is strongest in middle layers, emerges during training, and can be partly controlled with perturbations or regularization, suggesting that trajectory straightening may play a functional role in prediction rather than being just a descriptive pattern.\n\nweaknesses and questions: **Strengths:\L\L **\n\n1. This paper checks where the studied effect is strongest, how it emerges over training, and whether trajectory-aligned perturbations selectively change entropy while misaligned perturbations do not. That makes the mechanistic claim meaningfully stronger than a simple observational result.\n\n**Weaknesses:**\n\n1. The main variables in the study are scalars, ie, contextual curvature, computed over a local trajectory window, and next-token entropy. That makes the story elegant, but also lossy: very different geometric structures can share similar curvature, and very different output distributions can share similar entropy. So the paper may be capturing a real effect while still missing the richer mechanism underneath it.\n\n2. The regularization experiments are done on small-scale LLMs, and the paper says outright that it remains untested whether the same relationship holds for larger foundation models, different architectures, multimodal systems, or more heterogeneous datasets. That matters a lot here, because the paper is close to proposing a general principle of autoregressive prediction. Right now, I think the evidence is good for “this happens in the studied setting,” but weaker for “this is a broad law of LLM computation.”\n\n3. Even if curvature regularization reliably lowers entropy, the paper does not show that this improves calibration, robustness, task accuracy, or downstream usefulness. And because validation loss stays similar, one interpretation is that it mostly reshapes probability mass among wrong answers rather than improving decision quality.\n* The regularization result makes curvature look less like a correlate of entropy and more like a direct control variable. If entropy changes while validation loss stays the same, why shouldn’t we read this as evidence that curvature is effectively determining uncertainty?\n\n\n* The observation that middle layers are both straightest and most predictive of entropy sounds reminiscent of an information-bottleneck story, where intermediate representations compress context into a task-relevant predictive code. Do you see your result that way, or do you think “trajectory straightening” is a different phenomenon from compression in the information-bottleneck sense?"
- >-
title: Conf-Gen: Conformal Uncertainty Quantification for Generative
Models
summary: The authors present an extension of conformal risk control
(CRC) which is itself an extension of conformal prediction (CP), suited
to obtaining performance guarantees for the answers provided by LLMs, in
the case where there is a set of exchangeable (input, ground_truth)
pairs. One has to also interpret the LLM output to form a set or
sequence of candidate answers parametrized by a conservativenss
parameter lambda (e.g., to select the most promising candidates) and
provide both a way to obtain a score for each candidate answer and a way
to evaluate how good this output is given a ground truth answer. A
calibration method then allows to choose lambda to obtain a required
degree of expected goodness on a new example from the same distribution.
weaknesses and questions: Soundness and originality
This paper allows doing a form of calibration with conformal guarantees
on LLM outputs in settings not considered before, which makes
comparisons difficult in several of these settings.
The experiments suggest that the theoretical guarantees hold.
The paper appears to be very sound and original.
Presentation
The paper does a good job of presenting relevant prior work.
The formal aspect of the paper also appears much above the ICML average
and I did not detect any issue on that front, although I did not read
the appendix.
More generally, everything is very clear.
Significance
This extension of conformal prediction seems relevant for many
LLM-related tasks and could thus be impactful.
See the key questions for authors.
One limitation I would note is that the kind of data stream on which
LLMs are actually applied is not necessarily exchangeable, invalidating
the core assumption behind CP. But the guarantees may be useful in
practice nonetheless, e.g., to obtain conservative safety guarantees.
Another CP limitation of course is that these are only guarantees in
average, not for a particular query. This is of course not a specific
issue with this paper, but it may limit the usefulness of this family of
approach, e.g., in high-stakes domains like medical decision-making.
Please share any thoughts on these two issues (non-exchangeability and
average-guarantees).
- >-
title: Local Curvature Descent: Squeezing More Curvature out of Standard
and Polyak Gradient Descent
summary: Convexity and L-smoothness are standard assumptions in
optimization literature which are useful for easier analysis of
optimization algorithms and determining the right algorithm parameters.
These global conditions may not always take local differences in the
curvature into account. This paper proposes new analogues of the
assumptions to incorporate certain kinds of local curvature information.
The paper also proposes modifications of gradient descent using matrix
valued step sizes to take advantage of the modified assumptions.
weaknesses and questions: The first-order convexity condition and
L-smoothness imply that $$ f(y) + \langle \nabla f(y), x-y\rangle \leq
f(x) \leq f(y) + \langle \nabla f(y), x-y\rangle + \frac L2 ||
x-y||^2.$$ The authors' assumption 2.1 modifies these inequalities by
adding the term $\frac12 || x-y||^2_{\mathbf C}$ to the lower bound and
upper bound for $f(x)$ provided by these inequalities (since $\frac12 ||
x-y||^2_{\mathbf C+L\mathbf I} = \frac12 || x-y||^2_{\mathbf C} + \frac
L2 || x-y||^2$). Thus, assumption 2.1 seems to be more general than
L-smoothness but it is less general than convexity.
In the standard analyses of GD, L-smoothness guarantees a sufficient
decrease with each step (with the right step size) and convexity ensures
that that decrease pushes us towards the minimizer. The two inequalities
balance each other in a crucial way. This work exploits that tradeoff.
However, it is not surprising that if the same term is added to both the
lower bound and upper bound of $f(x)$ then they will cancel each other
out and the standard convergence proofs will still go through.
Furthermore, if $f$ is twice differentiable, assumption 2.1 is actually
equivalent to
$$\mathbf C(x) \preceq \nabla^2 f(x) \preceq \mathbf C(x) + L_C \mathbf
I.$$ This follows from the same kind of standard arguments used to show
that convex functions have positive semidefinite Hessians. Having
observed this second order condition, many of the remarks that the
authors make follow directly. The case when $\mathbf C = 0$ is the
standard case with convexity and L-smoothness, and the case when $L_C=0$
is the realm of second order methods like Newton's method. My impression
is that to give any useful advantage over standard GD, the map $\mathbf
C(x)$ will have to approximate the Hessian $\nabla^2 f(x)$. But then the
algorithms provided do not seem to be very useful unless there is a good
way to approximate the Hessian.
LCD1 actually just seems to be a version of Newton's method where the
Hessian is overestimated by its upper bound $\mathbf C(x) + L_C \mathbf
I$ to make it more stable. The convergence rate provided for LCD1 is the
same as GD (unless $L_C = 0$, in which case it is pure Newton's method
anyway), which does not provide any new insights either. The authors
present LCD2 as a generalization of Polyak's step size, but the step
size $\beta_k$ in that case does not even have a closed form. Computing
$\beta_k$ itself requires an optimization problem to be solved at each
step, and the benefits of doing that are not clear. LCD3 has a closed
form step-size but there are no convergence results provided for it, so
it's not clear how well it performs.
The assumption could still have been justified with examples of
interesting functions that satisfy assumption 2.1 in non-trivial ways.
Unfortunately, that does not seem to be the case. One of the curvature
matrices specified for each of the examples 6.1-6.4 is just the Hessian.
For examples 6.2 and 6.4, $\nabla f(x) \nabla f(x)^\top$ is proposed as
another candidate for the curvature matrix, but these kinds of
approximations of the Hessian are already covered by quasi-Newton
methods like Berndt–Hall–Hall–Hausman algorithm.
The experiments are also only performed on these trivial examples and
authors compare their proposed step sizes only against Polyak step size.
The first experiment is on a strongly convex and L-smooth function,
which is covered by the classical assumptions, and the optimal method
for which would have been a momentum-based algorithm like Nesterov's
accelerated gradient descent. The second experiment chooses the Hessian
as the curvature matrix, reducing it to the case where second order
methods would perform better.
Overall, the assumption proposed in the paper does to offer many new
theoretical insights nor do the algorithms proposed offer practical
advantages over existing algorithms.
- Can the authors comment on the second order characterization of
assumption 2.1?
- What is the time complexity of LCD2 compared to Polyak step size?
Specifically, how does the computation of $\beta_k$ affect the
complexity?
- source_sentence: theory on finite-trial estimates
sentences:
- >-
title: Complexity Analysis of Normalizing Constant Estimation: from
Jarzynski Equality to Annealed Importance Sampling and beyond
summary: This paper provides a full theoretical analysis of the error in
Annealed Importance Sampling (AIS), accounting for both the sampling
process that generates particles and the estimator of the normalizing
constant computed from these samples. The authors derive a general upper
bound on the estimation error as a function of the prescribed
probability path that guides the sampling process. Two important path
choices are examined: the standard geometric interpolation path, which
is shown to lead to exponential complexity in the difficulty of the
problem, and the reverse diffusion path, which achieves only polynomial
complexity—provided oracle access to score functions (otherwise, an
additional approximation error appears). The difficulty of the problem
is quantified in terms of the between-mode distance of the target
distribution.
weaknesses and questions: The writing could be clearer in some parts,
but overall the paper is clear.
Q1. Theorem 4 is particular to the geometric interpolation path? Is it
normal that the number of samples $N$, the number of SMC iterations $M$,
and the discretization of the Langevin process, do not appear in the
error bound (Eq 11)?
- >-
title: Stronger Neyman Regret Guarantees for Adaptive Experimental
Design
summary: This paper explores efficient ATE estimation in adaptive
experimental designs. The authors focus on Neyman regret, which
quantifies the variance difference between the
inverse-propensity-weighted (IPW) estimator under the proposed adaptive
design and the best fixed design in hindsight. Prior work (e.g., Dai et
al., 2023) established a sublinear $O(\sqrt{T})$ bound on Neyman regret.
This paper strengthens that result, achieving an $O(\log T)$ bound under
slightly stronger assumptions. The analysis is further extended to
contextual (multigroup) settings, introducing a method that ensures
$O(\sqrt{T})$ regret across multiple overlapping subpopulations. The
approach is validated both theoretically and empirically.
weaknesses and questions: None.
None.
- >-
title: Theory on Score-Mismatched Diffusion Models and Zero-Shot
Conditional Samplers
summary: This paper provides a non-asymptotic convergence theory for
diffusion models with a mismatched score function. The results can be
applied to zero-shot conditional samplers, especially linear conditional
models. The authors also propose the Bias-Optimal (BO) DDNM sampler and
provide a comparison with the previous algorithms.
weaknesses and questions: - Assumption 2 requires an upper bound on the
score estimation error $\epsilon^2 = \tilde{\mathcal{O}}(T^{-2})$, which
could be restrictive compared to the previous literature (e.g., Li et
al. 2024b) which applies to any $\epsilon$.
- The in-line equations in Section 2.1 and Section 5 are hard to follow.
I suggest the authors re-organize the equations for better readability,
especially by highlighting the definitions and differences of $q$, $p$,
and $\hat{p}$.
- The big-O notation in this work is confusing. To name a few, in
Assumptions 3 and 4, should $(1-\alpha_t)^m \mathbb{E}[\cdot] =
O((1-\alpha_t)^m)$ be equivalent to $\mathbb{E}[\cdot] = O(1)$? Or do
the hidden constants in these assumptions implicitly depend on
$(1-\alpha_t)^m$? In Definition 1, the noise schedule needs to satisfy
$\bar{\alpha}\_T = o (1/T)$ which is defined as an asymptotic bound
$\lim\sup_{T\to\infty} |\bar{\alpha}_T / (1/T)| \to 0$, while Theorem 1
presents a non-asymptotic analysis. How does the asymptotic assumption
apply to the non-asymptotic analysis? I suggest the authors clarify the
notations or even explicitly write the constants' dependency on the
hyperparameters if possible.
- Does $1-\alpha_1=\delta$ in equation (8) contradict with $1-\alpha_1
\lesssim \frac{\log T}{T}$ in Definition 1? What can we obtain from
Theorem 1 if $\alpha_t$ is chosen as (8)?
- >-
title: The Number of Trials Matters in Infinite-Horizon General-Utility
Markov Decision Processes
summary: The paper analyzes the impact of the number of trails in
estimating the objectives for GUMDPs. For both the discounted and
average settings, it is shown by examples that there are mismatches
between the finite-trial estimates and the actual infinite-trail
objectives. Bounds on the mismatches are provided, with numerical
results supporting the theoretical claims.
weaknesses and questions: None
In Fig 3b, it looks like there are some discontinuities in the
performance of $M_{f, 3}$ around $\gamma=0.9$ where the finite-trail
performance seems to diverge away from the infinite-trail one, but then
converges back to it. Is that expected from theoretical analysis?
- >-
title: The Minimax Rate of HSIC Estimation for Translation-Invariant
Kernels
summary: The rate at which HSIC can be estimated is an important and
open problem, in this paper, the authors prove that
the minimax optimal rate of HSIC estimation for Borel measures is
$\mathcal{O}(n^{-0.5})$ with M>=2 components, which is very important as
existing conclusion only holds for M=2. Other byproducts can be
naturally introduced, implying the minimax lower bound for the
estimation of
cross-covariance operator, which can be further specialized to get back
the minimax result on the estimation of the covariance operator.
weaknesses and questions: 1. Overall, the paper is not easy to follow as
the paper's main contribution seems to be the proof part.
2. I wouldn't say it is the weakness or the author's problem, as this is
a theoretical paper, experiments are not necessary. Still is it possible
to design toy experiments to validate the conclusions in the paper?
NA
- >-
title: Active Treatment Effect Estimation via Limited Samples
summary: Experimental design for estimating treatment effects does not
generally have strong finite-sample guarantees, especially as the
dimensionality of the covariates grows. Recent works implement
experimental design based on leverage scores. This work proposes an
alternative approach called IRD, which helps achieve a sample complexity
for the estimation error that is linear in the covariate dimensionality.
The method is validated with a variety of standard semi-synthetic
experiments.
**Update after rebuttal**: after considering the additional results
provided, I have decided to increase my score.
weaknesses and questions: The problem of active sampling for
treatment-effect estimation with high-dimensional covariates is clearly
significant. The solution appears to have clear benefits over other
recent works. It would be very helpful to better describe the method so
that readers can understand the key contributions.
1. Specifically what role do partitioning and subsampling play in the
proposed method?
2. Does this method easily extend to multiple treatments?
- >-
title: Beyond the Average: Distributional Causal Inference under
Imperfect Compliance
summary: This paper introduces an estimator for the local distributional
treatment effect, which is an instrumental-variable-type estimator for
the causal effect among compliers. They prove asymptotic properties of
the estimator and empirically argue using synthetic and real data that
the estimator is more effective than baselines.
weaknesses and questions: Strengths:
- while I do have some confusions outlined below, I think the
contribution here is probably theoretically solid and well-scoped
- experiments are mostly clear and compelling
Weaknesses:
- could use some more clarity on exactly what the contribution here is -
is it just adding a distributional component onto existing IV estimators
for complier effects? or something further. I think it's more but have
trouble totally figuring it out, the distributional component by itself
seems like probably not quite enough novelty but I could be wrong. Lemma
3.2 would be a nice place to do some of this clarification but anywhere
is fine
- confused about a couple fundamental pieces here: the authors define
\eta_z = E[D | Z, S, X] on L193. However according to the causal graph
in Fig 1 it looks like D should be independent of X, S given Z - why is
this not just E[D | Z] in that case? Similarly I wonder about the
equation below on L194 - I would assume from Fig 1 that Y should be
independent of X, S given Z
- Algo 1: should clarify what exactly M is trained on, inputs + outputs.
I don't see this notation elsewhere
- Assumption 5.1: (I) I find it confusing why we need a condition on the
difference but not on the individual components, (ii) the < \infty part
of this is confusing to me as well, doesn't this hold trivially for any
F bounded above by 1? which \mu_z should be since it is an Expectation
of an indicator function
- Theorem 5.2 - I may just not have the theory background here so
apologies but I don't really understand what it means for this to
converge to a GP. Don't we want it to be decreasing in some function of
n?
- Fig 4: your left and right plots look identical to me, I'm assuming
this is an error
- clarify the contribution beyond previous work
- correct Fig 4 so that the L and R parts are different
- clarify how I should interpret causal graph in Fig 1 and how it
interacts with the definitions of \eta and \mu
- >-
title: Distributed Least Squares in Small Space via Sketching and Bias
Reduction
summary: Sketched least squares involve estimating the term
$(X^TX)^{-1}$ which has a high bias when the sketch matrix $S$ is not
sub-Gaussian. This paper gives a sparse sketching method using a LESS
embedding which runs in optimal space and current matrix multiplication
time, where $S$ is sparse, and constructed based on the leverage scores
of the data matrix $A$ (Definition 2 $(s,\beta_1,\beta_2)$-LESS
embedding). The paper also improves the sharpness of the probability
bounds which is applicable to similar problems in RMT using LESS
embeddings. For $s=1$, nothing is different, but when $s > 1$, the bias
bound is reduced.
weaknesses and questions: 1. It would be nice to have experiments with
other sketching methods to (empirically) justify some statements on the
bias, variance (although not stated) and computational time, e.g.
estimators mentioned in Table 1. For example, are there sketching
estimators with higher bias, but less variance?
There are some minor typos, e.g. line 212 reference missing, lines 282
to Equation (2) at bottom of the page is missing a bracket for the
numerous expectations (in contrast to lines 542 onwards in the
appendix).
In Appendix A, notation for concentration inequalities should be looked
at and made consistent, e.g. Lemma 6 / H{\"o}lder's inequality should
have a $\frac{1}{q}$, Lemma 7 ($\lambda$ max isn't defined), Lemma 8 /
Azuma's inequality ($\lambda$, $m$ should be consistent), Lemma 10 is
missing a bracket for $\mathbb E[x_i^2]$.
The presentation of the proof for Theorem 5 was slightly jarring (due to
Lemma 11, Lemma 12 appearing in the proof), but there also doesn't seem
to be a good way to include them (since referring to the two lemmas
requires the upper bounds, and flipping back a page is also
inconvenient). Maybe a solution is to indent the Lemmas, or box them up?
The dot before line 626 (after 72) should be removed.
1. It would be nice to have experiments with other sketching methods to
(empirically) justify some statements on the bias, variance (although
not stated) and computational time, e.g. estimators mentioned in Table
1. For example, are there sketching estimators with higher bias, but
less variance?
2. Despite being clear to read, I had to go back and forth a bit to find
out what the novelty is. I appreciate the clarity and thoroughness of
explaining the bounds on the bias and variance, runtime, but I would
like it if Definition 1 & 2 came much earlier (or at least maybe an
informal Definition 2 after line 85?)
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
- cosine_accuracy@1
- cosine_accuracy@10
- cosine_precision@10
- cosine_precision@100
- cosine_recall@10
- cosine_recall@100
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
model-index:
- name: SentenceTransformer
results:
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: reviewsearch
type: reviewsearch
metrics:
- type: cosine_accuracy@1
value: 0.35884636716583473
name: Cosine Accuracy@1
- type: cosine_accuracy@10
value: 0.7750214289315787
name: Cosine Accuracy@10
- type: cosine_precision@10
value: 0.19142338526697927
name: Cosine Precision@10
- type: cosine_precision@100
value: 0.05602279029899663
name: Cosine Precision@100
- type: cosine_recall@10
value: 0.17017869143702086
name: Cosine Recall@10
- type: cosine_recall@100
value: 0.4395403885775364
name: Cosine Recall@100
- type: cosine_ndcg@10
value: 0.25119901516426524
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.4888785045607057
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.15688999637818096
name: Cosine Map@100
SentenceTransformer
This is a sentence-transformers model trained. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.
Model Details
Model Description
- Model Type: Sentence Transformer
- Maximum Sequence Length: 2048 tokens
- Output Dimensionality: 768 dimensions
- Similarity Function: Cosine Similarity
- Supported Modality: Text
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Sentence Transformers on Hugging Face
Full Model Architecture
SentenceTransformer(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'unpad_inputs': False, 'architecture': 'Gemma3TextModel'})
(1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
(2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
(3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
(4): Normalize({})
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
'theory on finite-trial estimates',
]
documents = [
'title: The Number of Trials Matters in Infinite-Horizon General-Utility Markov Decision Processes\n\nsummary: The paper analyzes the impact of the number of trails in estimating the objectives for GUMDPs. For both the discounted and average settings, it is shown by examples that there are mismatches between the finite-trial estimates and the actual infinite-trail objectives. Bounds on the mismatches are provided, with numerical results supporting the theoretical claims.\n\nweaknesses and questions: None\nIn Fig 3b, it looks like there are some discontinuities in the performance of $M_{f, 3}$ around $\\gamma=0.9$ where the finite-trail performance seems to diverge away from the infinite-trail one, but then converges back to it. Is that expected from theoretical analysis?',
"title: The Minimax Rate of HSIC Estimation for Translation-Invariant Kernels\n\nsummary: The rate at which HSIC can be estimated is an important and open problem, in this paper, the authors prove that\nthe minimax optimal rate of HSIC estimation for Borel measures is $\\mathcal{O}(n^{-0.5})$ with M>=2 components, which is very important as existing conclusion only holds for M=2. Other byproducts can be naturally introduced, implying the minimax lower bound for the estimation of\ncross-covariance operator, which can be further specialized to get back the minimax result on the estimation of the covariance operator.\n\nweaknesses and questions: 1. Overall, the paper is not easy to follow as the paper's main contribution seems to be the proof part. \n2. I wouldn't say it is the weakness or the author's problem, as this is a theoretical paper, experiments are not necessary. Still is it possible to design toy experiments to validate the conclusions in the paper?\nNA",
'title: Stronger Neyman Regret Guarantees for Adaptive Experimental Design\n\nsummary: This paper explores efficient ATE estimation in adaptive experimental designs. The authors focus on Neyman regret, which quantifies the variance difference between the inverse-propensity-weighted (IPW) estimator under the proposed adaptive design and the best fixed design in hindsight. Prior work (e.g., Dai et al., 2023) established a sublinear $O(\\sqrt{T})$ bound on Neyman regret. This paper strengthens that result, achieving an $O(\\log T)$ bound under slightly stronger assumptions. The analysis is further extended to contextual (multigroup) settings, introducing a method that ensures $O(\\sqrt{T})$ regret across multiple overlapping subpopulations. The approach is validated both theoretically and empirically.\n\nweaknesses and questions: None.\nNone.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.5815, 0.5067, 0.4693]])
Evaluation
Metrics
Information Retrieval
- Dataset:
reviewsearch - Evaluated with
InformationRetrievalEvaluatorwith these parameters:{ "query_prompt": "task: search result | query: ", "corpus_prompt": "title: none | text: " }
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.3588 |
| cosine_accuracy@10 | 0.775 |
| cosine_precision@10 | 0.1914 |
| cosine_precision@100 | 0.056 |
| cosine_recall@10 | 0.1702 |
| cosine_recall@100 | 0.4395 |
| cosine_ndcg@10 | 0.2512 |
| cosine_mrr@10 | 0.4889 |
| cosine_map@100 | 0.1569 |
Training Details
Training Dataset
Unnamed Dataset
- Size: 46,935 training samples
- Columns:
anchor,positive,negative_1,negative_2,negative_3,negative_4,negative_5,negative_6, andnegative_7 - Approximate statistics based on the first 100 samples:
anchor positive negative_1 negative_2 negative_3 negative_4 negative_5 negative_6 negative_7 type string string string string string string string string string modality text text text text text text text text text details - min: 5 tokens
- mean: 7.66 tokens
- max: 12 tokens
- min: 200 tokens
- mean: 504.56 tokens
- max: 1380 tokens
- min: 159 tokens
- mean: 558.75 tokens
- max: 1575 tokens
- min: 146 tokens
- mean: 516.22 tokens
- max: 1170 tokens
- min: 160 tokens
- mean: 573.96 tokens
- max: 1957 tokens
- min: 90 tokens
- mean: 487.22 tokens
- max: 1527 tokens
- min: 138 tokens
- mean: 594.55 tokens
- max: 1959 tokens
- min: 127 tokens
- mean: 616.91 tokens
- max: 1398 tokens
- min: 64 tokens
- mean: 603.93 tokens
- max: 2048 tokens
- Samples:
anchor positive negative_1 negative_2 negative_3 negative_4 negative_5 negative_6 negative_7 meta-learning unclear contributiontitle: Extending Contextual Self-Modulation: Meta-Learning Across Modalities, Task Dimensionalities, and Data Regimes
summary: This paper proposes to extend Contextual Self-Modulation (CSM), which is an uncertainty-handling mechanism of Neural Context Flow (NCF). NCF is a robust meta-learning framework of neural ODE that includes self-modulation with high-order Taylor expansion around contexts. The authors focus on generalizing CSM into high data regimes and designed StochasticNCF and FlashCAVIA, which take CSM in various directions. From comparative experiments, the author verified that the NCF can be successfully extended to various optimization and meta-learning techniques.
weaknesses and questions: 1. I believe this paper is not clearly written.
* iCSM: It is briefly mentioned that the space of iCSM is "an infinite-dimensional variation" that utilizes "a space of multi-layer perceptrons, whose weights are flattened into a 1-dimensional tensor." I do not find a good enough ex...title: Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning
summary: This paper introduces FAME (Fast and Meta knowledge learners), a novel framework for continual reinforcement learning that draws inspiration from the human brain’s memory systems. It proposes a dual-learner architecture comprising a fast learner, which rapidly adapts to new tasks, and a meta learner, which incrementally integrates knowledge to prevent catastrophic forgetting. To support efficient adaptation and stability, the authors define principled metrics such as MDP distance (for task similarity) and catastrophic forgetting (for performance degradation). A key contribution is the adaptive meta warm-up mechanism, which selects the best initialization strategy for new tasks using statistical hypothesis testing, thereby mitigating negative transfer. Theoretical foundations are coupled with practical algorithms for both value-based and policy-based RL, using divergence-based incremental u...title: Meta-Reinforcement Learning for Compiler Optimization: A Kernel-Embedded CompilerLLM with Verified Assumptions and Practical Guarantees
summary: This paper proposes methods to improve optimization, specifically by trying to find surprising or interesting optimizations that nonetheless pass validation.
weaknesses and questions: 1. This paper is not complete. There are obvious issues with the writing such as "Meta-optimization" and "Meta-reinforcement learning" being repeated many times in the intro, the "Meta-Learning Theory." section in the related work being empty, etc.
2. Clarity could use improvement. In many places (e.g. the abstract) the writing was either to jargon-filled or incomplete for me to understand well.
3. The description in section 4.2 and beyond is not clear enough for me to fully understand the method. There are many places that are unclear, but for instance it is not stated howk_cfg,k_data,k_instetc. are calculated.
4. There is no comparison with o...title: Meta-Router: Bridging Gold-standard and Preference-based Evaluations in LLM Routing
summary: This paper addresses the challenge of training LLM routers to balance response quality and inference cost. It proposes a novel causal inference framework, viewing gold-standard and preference-based evaluation data as a treatment assignment problem. The key insight is that bias in preference-based data corresponds to the conditional average treatment effect (CATE). The proposed Meta-Router framework corrects this bias via meta-learners (S-learner, T-learner, X-learner), incorporates propensity score weighting to address data source imbalances, and applies distributionally robust optimization (DRO) for routing robustness. Experiments on benchmarks (AlpacaEval, MT-Bench, MMLU, GSM8K) show improved cost-quality trade-offs compared to standard baselines.
weaknesses and questions: Limited Baseline Scope: Compares mostly to classical methods (IPW, DR); recent LLM routing methods and stron...title: The Meta-Representation Hypothesis
summary: The paper proposes to combine Deep Mutual Learning with RL. In Deep Mutual Learning, several learners learn independently but at the same try to minimize the KL between their predictive distributions. The paper hypothesizes that two RL policies can learn from different MDPs — where each MDP has its own randomly sampled observation function while the policies try to minimize the KL between them. This would lead to the learning of robust representation functions. The randomly perturbed observation function is a key aspect of the paper — in their paper they apply a CNN with random weights to the observation to map the true observation to a perturbed one. The paper tests this hypothesis via PPO and shows that Deep Mutual Learning is helpful for generalization on the Procgen Benchmark.
weaknesses and questions: ## Pros
1. Tackles an important problem about having a robust perception function for RL.
2. A positive thing is that the whole ...title: MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation
summary: This paper proposes to achieve generalizable tool learning by additionally training models on meta-reasoning QA tasks. The meta-reasoning data are constructed by asking questions about the tool-using process in multiple directions, including action effect, decision-making, reversion, action input boundary, etc. Experiment results show improved tool learning performance on tasks including SAW, BW, LOG, Toolbench and BFCL.
weaknesses and questions: 1. In lines 224-226, "In order to maintain the general ability of the model in the first stage, only the parameters of the query and value projection layers of the Transformer are updated instead of full-parameter training." This constraint might also affect learning ability and make comparisons unfair. Results ensuring similar settings will make results more convincing.
2. The "LLaMA3-solution" baselines are updated fewer times (10k*3) ...title: Meta-learning Representations for Learning from Multiple Annotators
summary: This paper proposes a meta-learning approach that utilizes noisy labels from multiple annotators to build a classifier without relying on true labels. The authors employ a probabilistic framework where latent class representations in a Mixture of Gaussians model are optimized via EM. This approach maximizes the likelihood of observed noisy labels given the latent variables, assuming these noisy annotations can guide the learning of true underlying classes.
weaknesses and questions: 1. The paper assumes isotropic variance in the latent space, simplifying computation but potentially limiting flexibility. Real-world data often exhibit complex, class-specific structures that may not align with uniform variance assumptions, particularly in nuanced classification tasks.
2. Modeling A as a KKr matrix may lead to over-parameterization, especially with limited data. Without visualization of learned matrices,...title: Meta ControlNet: Enhancing Task Adaptation via Meta Learning
summary: This paper introduces Meta ControlNet, leveraging meta-learning and a novel layer-freezing approach to significantly reduce the training steps needed for ControlNet from 5000 to 1000. Additionally, it enables zero-shot control in edge tasks and rapid adaptation in complex tasks like Human Pose with only 100 finetuning steps.
weaknesses and questions: There are three critical issues for this paper:
1. No quantitative results are provided. The paper only shows qualitative results, which makes it hard to evaluate the performance of the proposed method.
2. Zero-shot capability. This paper claims that the proposed method can achieve zero-shot control in edge tasks. The model is trained on HED, Segmentation, and Depth map which belong to the edge tasks. It raises a question about whether the model has achieved zero-shot control in edge tasks. If we train ControlNet on three tasks, and then test the model on Canny ...meta-learning scalability benchmarkstitle: Extending Contextual Self-Modulation: Meta-Learning Across Modalities, Task Dimensionalities, and Data Regimes
summary: This paper proposes to extend Contextual Self-Modulation (CSM), which is an uncertainty-handling mechanism of Neural Context Flow (NCF). NCF is a robust meta-learning framework of neural ODE that includes self-modulation with high-order Taylor expansion around contexts. The authors focus on generalizing CSM into high data regimes and designed StochasticNCF and FlashCAVIA, which take CSM in various directions. From comparative experiments, the author verified that the NCF can be successfully extended to various optimization and meta-learning techniques.
weaknesses and questions: 1. I believe this paper is not clearly written.
* iCSM: It is briefly mentioned that the space of iCSM is "an infinite-dimensional variation" that utilizes "a space of multi-layer perceptrons, whose weights are flattened into a 1-dimensional tensor." I do not find a good enough ex...title: A solvable model of inference-time scaling
summary: This paper introduces an analytically tractable model of inference-time scaling using Bayesian linear regression with reward-weighted sampling, deriving closed-form expressions for generalization error in the high-dimensional limit. The authors prove that when the reward model is well-aligned with the teacher, error decreases monotonically with inference samples $k$ (scaling as $\Theta(1/k^2)$ in the best-of-k limit), but substantial reward misspecification induces a finite optimal $k$ and optimal temperature. The theory delineates parameter regimes where scaling inference-time compute is provably more effective than collecting additional training data, though this advantage degrades as task difficulty increases.
weaknesses and questions: 1. Oversimplified model: The paper only studies linear regression with quadratic rewards and Gaussian assumptions, while real LLMs involve highly nonlinear neural networks, complex reward mo...title: MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?
summary: This paper presents a new evaluation benchmark for Multimodal Large Language Models (MLLMs), dubbed MME-RealWorld, which focuses on challenges that models face in the real world. Specifically, MME-RealWorld covers 29,429 question-answer pairs across 5 real-world scenarios. Experimental results on MME-RealWorld show that even the most advanced models still struggled in real-life scenarios. Besides, the authors have also conducted detailed analyses to explain the unsatisfying performance of MLLMs.
weaknesses and questions: - The evaluation on MME-RealWorld seems to require lots of computation resources, which may limit the accessibility for researchers with fewer resources.
Do the authors have plans to expand or adapt MME-RealWorld to include new tasks or modalities as MLLMs capabilities evolve?title: On the Embedding Collapse When Scaling up Recommendation Models
summary: This paper studies recommendation model performance when scaling up the embedding layers of the model. The paper identifies a phenomenon of embedding collapse, wherein the embedding matrix tends to reside in a low-dimensional subspace. Through empirical experiments on FFM and DCNv2 and theoretical analysis on FM, the paper shows that the feature interaction process of recommendation models leads to embedding collapse and thus limits the model scalability. The paper also performed empirical experiments on regularized DCNv2 and DNN which led to less collapsed embeddings, but the model performance got worse. The paper proposes multi-embedding, which leads to better performance when scaling up the embedding layers. Experiments demonstrate that this proposed design provides consistent scalability for various recommendation models.
weaknesses and questions: - In section 3, the paper proposes Information Abundan...title: Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encoder
summary: This paper presents Meta-learned Masked Auto-Encoder (MetaMAE), a novel modality-agnostic self-supervised learning (SSL) framework that leverages meta-learning to improve the transfer abilities of Masked Auto-Encoder (MAE). The authors reinterpret the mask reconstruction task of MAE as a meta-learning task and propose the integration of two advanced meta-learning techniques: gradient-based meta-learning and task contrastive learning. MetaMAE is evaluated on various data modalities from modality-agnostic SSL benchmarks, demonstrating significant improvements over previous modality-agnostic SSL methods in linear evaluation. The proposed approach also shows improved transferability on cross-domain datasets.
weaknesses and questions: 1. There are significant differences in pretraining and fine-tuning hyperparameters for various downstream tasks, such as masking ratio, batch size, and decoder ...title: GraphBench: Next-generation graph learning benchmarking
summary: This paper introduces GraphBench, a contribution of around 20 unique datasets from 7 broad and diverse categories for graph learning benchmarking. It complements the existing popular graph learning benchmarks which may be significant for molecular and citation networks, as examples, but often missing for other areas such as chip design, circuit design and weather forecasting, among others (though there are individual areas in the literature that tackle these problems). The paper also highlights the current limitations with graph benchmarks in terms of data diversity reflecting multiple real world scenarios, in/out distribution splits, evaluation consistencies and framework for usage. It finally presents a framework based on Pytorch and Pytorch Geometric which acts as the interface for loaders, optimizers and evaluators.
weaknesses and questions: - The manuscript includes reasonable discussion points on limitation...title: (Out-of-context) Meta-learning in Language Models
summary: The paper shows the existence of a phenomenon that the authors refer to as out-of-contect meta learning in large language models. The authors design experiments that show that this phenomenon causes the internalization of text that is broadly useful, meaning that the LLM is more likely to treat this content as true. The paper shows two forms of internalization, namely weak and strong internalization, the later being a form of meta learning. Two reasons are suggested for this phenomenon, one based on the parameters of the model, and another one relying on the implicit gradient alignment bias of gradient-based optimization methods.
weaknesses and questions: * There is no conclusive explanation of the reasons why internalization happens
* The phenomenon is hard to formalize and study, which limits the advantage of the insights in the paper
Nonetitle: Training-Free Generalization on Heterogeneous Tabular Data via Meta-Representation
summary: This paper introduces a novel approach to enable training-free generalization for tabular datasets.
The core idea is something like:
For any given dataset, the input label data (x, y) is restructured into a new format: (distance to prototypes of class c, likelihood of the label of class c). This uniform data representation allows different datasets to be organized in a consistent manner. Thus, a model trained on this standardized format can effectively generalize across various tabular datasets.
On unseen datasets, the proposed model achieves superior performances and saves training time.
weaknesses and questions: The dataset used in this study is somewhat limited. Although I have confidence in the model's ability to generalize effectively to new datasets by representing data points in terms of their similarity to prototypes, there are concerns about its adaptability to other dataset...anchor quality upper bounds trainingtitle: Tournament Style RL: Stabilizing Policy Optimization on Non Verifiable Problems
summary: The paper proposes a novel tournament reward calculated against a given set of anchor answers to generate reward supervision for LLM training in tasks without verifiable rewards. For each input prompt, a set of anchors is generated before training using a stronger LLM and ranked. Then the generated answers are compared against this ranked set of anchors to generate a reward for each answer which is then used for GRPO fine-tuning.
weaknesses and questions: Strength
1. the proposed method is well motivated and clearly presented
2. can be easily implemented upon GRPO style fine-tuning pipelines
3. is robust against noise in evaluator LLMs
Weakness
1. The proposed method relies heavily on anchor model quality. And the score itself will saturate if the model being fine-tuned surpasses the anchor model's quality. On the other hand, the performance of anchor model upper limits the model ...title: Unified Stability Bounds for Structured World Models: Geometry, Equivariance, and Identifiability as Sufficient Conditions
summary: Overview：
This paper addresses a key challenge in model-based reinforcement learning: the lack of a principled and low-overhead framework for diagnosing the quality of learned world-model representations. Motivated by the need to move beyond expensive, end-to-end evaluations and the limitations of existing theories, the authors aim to explain which properties of a representation govern downstream control performance and how to test them on existing model checkpoints. To solve this, the paper introduces a unified stability bound that decomposes the policy's suboptimality gap into three verifiable channels: geometric distortion (κ), an identifiability defect proxied by Total Correlation (TC), and an equivariance defect proxied by Local Equivariance Error (LEE). The authors then propose a practical diagnostic protocol where these proxies are measured ...title: Quality Control at Your Fingertips: Quality-Aware Translation Models
summary: The paper proposed two methods to make the NMT model quality aware. One is to prompt the NMT model with a quality score during training, but using the best score during inference time. The other is similar to multi-task learning but in a more unified way by appending the quality score in the target side. Both approaches show promising improvements in translation quality and one of them can work well with the MBR decoding to boost the translation quality further.
weaknesses and questions: My concerns are in the questions. If they can be addressed properly, they won't be weakness to me.
In conclusion, which one between QA prompting and prediction approaches is your recommendation in the situations including latency sensitive inference and large scale distillation. Please also describe how do you scale your methods in large scale multilingual machine translation system. The experiments highly relies on ...title: Utility Boundary of Dataset Distillation: Scaling and Configuration-Coverage Laws
summary: This paper proposes a unified configuration–dynamics–error framework that integrates gradient, distribution, and trajectory matching within a generalization-error analysis. It establishes the scaling law and coverage law linking distilled sample size to performance and configuration diversity, theoretically and empirically unifying major dataset distillation methods.
weaknesses and questions: 1. The framework relies on PL conditions and Lipschitz continuity. While these assumptions are standard in convergence analysis, they may not strictly hold for modern deep networks with non-smooth activations, normalization layers, and stochastic training components. The practical relevance of the theoretical results could be further clarified by discussing their validity under relaxed or empirically realistic assumptions.
2. The validation of the proposed laws relies mainly on curve-fitting without...title: Task-Robust Pre-Training for Worst-Case Downstream Adaptation
summary: In order to improve the robustness of the pre-trained model on downstream tasks, the authors propose a simple optimization algorithm, softmax weighted gradient descent, to minimize the worst-case expected risk of upstream tasks.
weaknesses and questions: In the experimental part, the method does not seem to show a consistent improvement. For example, as shown in Table 1, although the author mentioned that the model has significantly improved performance on many more challenging tasks, it has worse performance than the previous model on some downstream tasks that have performed well. See Table 2 for the same reason. Could this be improved with some tweaks for consistency?
According to the weekness I mentioned above, my question is whether such a strategy is a trade-off in the performance between the best case and the worst case on the downstream task, and cannot achieve consistency improvement?title: Steering Beyond the Support: Adversarial Training on Unsupervised Jailbroken Activation Simulation
summary: This paper studies activation steering for jailbreak defense under unseen or out-of-distribution attacks. Instead of learning steering only from a fixed supervised jailbreak set, the paper proposes to simulate diverse jailbroken activations through unsupervised latent direction discovery, and then train a potential-induced steering field with a bi-level adversarial objective. The method is evaluated on three aligned LLMs and six jailbreak families, and the paper reports improved robustness together with increasing subspace coverage during training.
weaknesses and questions: Strengths
- The paper targets an important problem in LLM safety, namely whether activation steering can generalize beyond a fixed supervised jailbreak set.
- The overall method is reasonably well motivated. In particular, combining unsupervised jailbreak activation simulation with a learned steer...title: Learning to Quantize for Training Vector-Quantized Networks
summary: This paper proposes a novel vector quantization training framework Meta-Quantization inspired by meta-learning, which decouples the optimization of codebook and autoencoder into two stages, enabling dynamic codebook generation and task-specific training. The proposed method outperforms existing vector quantization approaches on image construction and generation tasks.
weaknesses and questions: The description of convergence is inconsistent. While Figure 2 states that $\phi$ and $\theta$ are trained to convergence before training $\psi$, Algorithm 1 shows that they are updated together. This discrepancy creates ambiguity regarding the actual optimization procedure implemented in the paper.
In the introduction, it is mentioned that the codebook utilization in previous methods is low. However, in the experiments (Table 3, 4, 5), the codebook utilization of VQGAN-LC is also quite high. Please provide a justificat...title: Estimating Fréchet bounds for validating programmatic weak supervision
summary: This paper proposes solutions via convex programs to estimate Frechet bounds for Programmatic Weak Supervision (PWS). This approach uses estimates of the true labels via labelmodels (i.e., different aggregation schemes that exist in the literature). With these estimates of the labels, they provide an approach to estimate bounds on the accuracy (and other quantities) of the weak labelers. They provide experiments to check the validity of their bounds and also provide experiments with weak labelers generated via prompting to examine how their bounds perform under instances of weak labelers with different qualities/accuracies.
weaknesses and questions: 1. One weakness is that this approach is fundamentally reliant on the quality of the label model. This is manifested in assumption 2.3, which states that the estimate of the conditional distribution of $Y | Z$ should approach the true conditional distri... - Loss:
MatryoshkaLosswith these parameters:{ "loss": "GuideGISTEmbedLoss", "matryoshka_dims": [ 768, 512, 256, 128 ], "matryoshka_weights": [ 1, 1, 1, 1 ], "n_dims_per_step": -1 }
Training Hyperparameters
Non-Default Hyperparameters
per_device_train_batch_size: 1024num_train_epochs: 1.0learning_rate: 2e-05lr_scheduler_type: cosinewarmup_steps: 0.1bf16: Trueeval_on_start: Truedataloader_num_workers: 4ddp_find_unused_parameters: Falseprompts: {'anchor': 'task: search result | query: ', 'positive': 'title: none | text: ', 'negative_1': 'title: none | text: ', 'negative_2': 'title: none | text: ', 'negative_3': 'title: none | text: ', 'negative_4': 'title: none | text: ', 'negative_5': 'title: none | text: ', 'negative_6': 'title: none | text: ', 'negative_7': 'title: none | text: '}batch_sampler: no_duplicates_hashed
All Hyperparameters
Click to expand
per_device_train_batch_size: 1024num_train_epochs: 1.0max_steps: -1learning_rate: 2e-05lr_scheduler_type: cosinelr_scheduler_kwargs: Nonewarmup_steps: 0.1optim: adamw_torch_fusedoptim_args: Noneweight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08optim_target_modules: Nonegradient_accumulation_steps: 1average_tokens_across_devices: Truemax_grad_norm: 1.0label_smoothing_factor: 0.0bf16: Truefp16: Falsebf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonegradient_checkpointing: Falsegradient_checkpointing_kwargs: Nonetorch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneuse_liger_kernel: Falseliger_kernel_config: Noneuse_cache: Falseneftune_noise_alpha: Nonetorch_empty_cache_steps: Noneauto_find_batch_size: Falselog_on_each_node: Truelogging_nan_inf_filter: Trueinclude_num_input_tokens_seen: nolog_level: passivelog_level_replica: warningdisable_tqdm: Falseproject: huggingfacetrackio_space_id: Nonetrackio_bucket_id: Nonetrackio_static_space_id: Noneper_device_eval_batch_size: 8prediction_loss_only: Trueeval_on_start: Trueeval_do_concat_batches: Trueeval_use_gather_object: Falseeval_accumulation_steps: Noneinclude_for_metrics: []batch_eval_metrics: Falsesave_only_model: Falsesave_on_each_node: Falseenable_jit_checkpoint: Falsepush_to_hub: Falsehub_private_repo: Nonehub_model_id: Nonehub_strategy: every_savehub_always_push: Falsehub_revision: Noneload_best_model_at_end: Falseignore_data_skip: Falserestore_callback_states_from_checkpoint: Falsefull_determinism: Falseseed: 42data_seed: Noneuse_cpu: Falseaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}parallelism_config: Nonedataloader_drop_last: Truedataloader_num_workers: 4dataloader_pin_memory: Truedataloader_persistent_workers: Falsedataloader_prefetch_factor: Noneremove_unused_columns: Truelabel_names: Nonetrain_sampling_strategy: randomlength_column_name: lengthddp_find_unused_parameters: Falseddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falseddp_static_graph: Noneddp_backend: Noneddp_timeout: 1800fsdp: Nonefsdp_config: Nonedeepspeed: Nonedebug: []skip_memory_metrics: Truedo_predict: Falseresume_from_checkpoint: Nonewarmup_ratio: Nonelocal_rank: -1prompts: {'anchor': 'task: search result | query: ', 'positive': 'title: none | text: ', 'negative_1': 'title: none | text: ', 'negative_2': 'title: none | text: ', 'negative_3': 'title: none | text: ', 'negative_4': 'title: none | text: ', 'negative_5': 'title: none | text: ', 'negative_6': 'title: none | text: ', 'negative_7': 'title: none | text: '}batch_sampler: no_duplicates_hashedmulti_dataset_batch_sampler: proportionalrouter_mapping: {}learning_rate_mapping: {}
Training Logs
| Epoch | Step | Training Loss | reviewsearch_cosine_ndcg@10 |
|---|---|---|---|
| 0 | 0 | - | 0.1704 |
| 0.0909 | 2 | 34.5974 | - |
| 0.1364 | 3 | - | 0.2356 |
| 0.1818 | 4 | 25.3299 | - |
| 0.2727 | 6 | 16.6895 | 0.2372 |
| 0.3636 | 8 | 13.9725 | - |
| 0.4091 | 9 | - | 0.2423 |
| 0.4545 | 10 | 12.2704 | - |
| 0.5455 | 12 | 11.2867 | 0.2473 |
| 0.6364 | 14 | 10.7076 | - |
| 0.6818 | 15 | - | 0.2492 |
| 0.7273 | 16 | 10.2320 | - |
| 0.8182 | 18 | 10.0333 | 0.2509 |
| 0.9091 | 20 | 9.9865 | - |
| 0.9545 | 21 | - | 0.2507 |
| 1.0 | 22 | 9.9844 | 0.2512 |
Training Time
- Training: 4.2 hours
- Evaluation: 1.6 hours
- Total: 5.7 hours
Framework Versions
- Python: 3.12.9
- Sentence Transformers: 5.6.0
- Transformers: 5.12.1
- PyTorch: 2.8.0+cu128
- Accelerate: 1.14.0
- Datasets: 5.0.0
- Tokenizers: 0.22.2
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
MatryoshkaLoss
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}