From the arXiv
Thursday, 30 July 2026 · 20 papers
OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
This paper introduces OmegaUse-OfficeVal, a benchmark designed to evaluate LLM agents on complex, multi-step office tasks. Its core contribution is the inclusion of economic grounding, pairing each task with human labor time and a price proxy. This allows for cost-effective evaluation of LLM agents against human perfor…
On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment
This paper introduces Routing-based On-Policy Distillation (ROPD), a novel method to improve LLM safety by distilling knowledge from an aligned model to a potentially compromised one. Instead of relying on specific prompt templates, ROPD models the divergence between aligned and compromised output distributions. This a…
Progressive Multimodal Alignment for Continual Instruction Tuning
This paper addresses projector drift in multimodal continual learning by introducing Progressive Multimodal Alignment (PMA). PMA uses a lightweight descriptor to detect visual distribution shifts and progressively adds new "expert" projector components only when necessary, while retaining the original projector as a st…
Think Short, Defer Smart, Act, and Repeat: Calibrated Reasoning and Uncertainty-Aware Deferral for Edge LLM Agents
This paper introduces Think Short, Defer Smart (TSDS), a framework for edge LLM agents. TSDS optimizes reasoning by stopping early when an action is stable and defers to a cloud model only when local uncertainty is high. Its key contribution is a jointly calibrated approach that guarantees both performance and efficien…
DIRECT: Direct Decoding for Efficient and Aligned Sequence Labeling with Large Language Models
DIRECT improves sequence labeling with LLMs by optimizing for human preferences via DPO and enforcing structured outputs. Its core method uses controlled decoding and a template-filling mechanism to generate only label tokens, significantly boosting inference efficiency and domain alignment.
From Found to Designed: Concepts as a Design Axis for Large Language Models
This paper proposes treating "concepts" as a deliberate design axis for Large Language Models (LLMs), rather than an emergent property. The authors introduce a taxonomy for designing LLMs with explicit conceptual structure, considering when and how this structure is introduced (e.g., during training or inference) and i…
OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment
This paper introduces OptimismBench, a novel method for detecting directional bias in language model probability judgments. By presenting inverted scenarios (e.g., P(success) vs. P(failure)), OptimismBench quantifies a signed bias score without needing ground truth. The study reveals widespread optimism in most tested …
Budget-Aware LLM Discovery via Cost-Calibrated Frontier Utility
This paper addresses the problem of efficiently discovering useful large language models (LLMs) within a fixed token budget. The core method, **CostAda**, introduces "cost-calibrated frontier utility" to guide the search. This utility balances the quality improvement of a candidate LLM against the cost incurred to find…
Can AI agents conduct open-ended AI research? Early evidence from two case studies
This paper introduces "shadow evaluations" to assess AI agents' ability to conduct open-ended AI research. In this method, frontier AI agents tackle the core research question of unpublished papers, with the original authors grading their output. The study found that while agents could handle the engineering tasks, the…
Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agents
This paper addresses the challenge of LLM agents selecting the optimal number of tools from a ranked list, considering their diverse costs. The core method, CAM-DF, formulates this as a cost-aware stopping problem, training directly on the value of continuing versus stopping. The key contribution is a principled approa…
Forecasting Trajectory-Level Safety Risks in Black-Box Multi-Turn Interactions
This paper introduces Recast, a framework for predicting safety risks in multi-turn LLM interactions. Recast addresses the limitation of current reactive safety measures by analyzing both immediate dialogue and historical context to forecast how malicious intent can build up over time. Its core contribution is enabling…
Thinking Under Uncertainty: Evidence Use and Information-Seeking in Language Models
This paper investigates how large language models (LLMs) "think" under uncertainty by distinguishing between improved evidence use and information seeking. Using bandit trials, they found that LLM inference-time thinking primarily enhances value-guided decision-making and reduces random noise, rather than actively seek…
ToxScreen: Detecting Whether an LLM Has Been Poisoned
This paper introduces ToxScreen, a benchmark for evaluating LLM backdoor detection. Its core method involves assessing a defender's ability to recover hidden triggers within a poisoned LLM, given white-box access to model weights and behavior. The main contribution is ToxScreen itself, a diverse set of poisoned models …
Two Calls Beat Five Agents: Evaluating Multi-Agent Pipelines Against Self-Refinement for Local Language Models
This paper evaluates multi-agent LLM pipelines against self-refinement for local models. It finds that a five-agent pipeline struggles with error accumulation, while a two-call self-refinement strategy achieves competitive accuracy with significantly lower token usage. The contribution lies in demonstrating that simple…
Evaluating Regional Bias in LLMs From Abstract Stereotype to Concrete Social Decision-Making
This paper introduces Stereotypes-to-Decisions (S2D), a framework to systematically evaluate regional bias in LLMs. S2D assesses how LLMs perceive regions in terms of warmth and competence, and how these perceptions influence decisions related to education, occupation, and social interaction. The study finds significan…
Mental World Modeling
This paper introduces Mental World Modeling (MWM), a framework that extends traditional world models by incorporating agents' hidden mental states (beliefs, desires, intentions) alongside physical states. MWM's core method is to maintain a coupled physical-mental world state, predict how actions jointly update both, an…
TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning
TREK is a benchmark designed to rigorously evaluate LLM agents in complex travel planning. Its core method involves creating multi-constraint tasks that require agents to synthesize feasible itineraries, ensuring all aspects like bookings, travel times, and budgets are correct and executable. TREK's key contribution is…
A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communities
This paper investigates whether AI coding agents adhere to open-source community rules designed to regulate their contributions. The researchers created a benchmark dataset and tested frontier AI models, finding that current agents **rarely proactively consult or comply with these rules**. Their main contribution is hi…
Human diversity fuels collective creativity that large language models cannot simulate or sustain
This paper demonstrates that human diversity, particularly between native and non-native speakers, significantly enhances collective creativity. The core method involved a metaphor generation experiment comparing human groups (with and without AI assistance) to AI-simulated groups. The key contribution is showing that …
Linguistic Monoculture in LLM-Assisted Language Use
This paper introduces the concept of "linguistic monoculture" in LLM-assisted writing, where widespread reliance on shared models can reduce linguistic variation. The authors develop a mathematical framework to model the coevolution of authors and LLMs, demonstrating how shared models can lead to a homogenization of li…