Finding Widespread Cheating on Popular Agent Benchmarks
Agentic cheating is a widespread issue, affecting thousands of submitted agent runs on 28+ submissions across 9 different benchmarks.
All Posts
2026
2025
CTSketch: Compositional Tensor Sketching for Scalable Neurosymbolic Learning
Scaling neurosymbolic learning with program decomposition and tensor sketching.
Probabilistic Soundness Guarantees in LLM Reasoning Chains
We certify the soundness of LLM reasoning chains with probabilistic guarantees, especially under error propagation.
Instruction Following by Boosting Attention of Large Language Models
We improve instruction-following in large language models by boosting attention, a simple technique that outperforms existing steering methods.
Probabilistic Stability Guarantees for Feature Attributions
We scale certified explanations to provide practical guarantees in high-dimensional settings.
2024
The FIX Benchmark: Extracting Features Interpretable to eXperts
We present the FIX benchmark for evaluating how interpretable features are to real-world experts, ranging from gallbladder surgeons to supernova cosmologists.
Logicbreaks: A Framework for Understanding Subversion of Rule-based Inference
We study jailbreak attacks through propositional Horn inference.
Towards Compositionality in Concept Learning
A method for learning compositional concepts from pre-trained foundation models.
Data-Efficient Learning with Neural Programs
Combining neural perception with symbolic or GPT-based reasoning
2023
Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Overcoming fundamental barriers in feature attribution methods with grouped attributions
SmoothLLM: Defending LLMs Against Jailbreaking Attacks
LLMs, jailbreaking, and generative AI’s ‘biggest security flaw’
Stable Explanations with Multiplicative Smoothing
Achieving stability guarantees for feature attribution methods
Do Machine Learning Models Learn Statistical Rules Inferred from Data?
Understanding and improving model predictions using rules learned from data.
Faithful Chain-of-Thought Reasoning
A novel prompting method that provides faithful explanations and improves performance on complex reasoning tasks
In-context Learning Influences
An influence framework for studying in-context learning examples
Adversarial Prompting
Finding phrases that bias image and text generation towards unexpected outputs