From the arXiv
Monday, 8 June 2026 · 20 papers
A Comprehensive Anatomy of Human and DeepSeek-R1 LLM Mathematical Reasoning
This paper empirically compares human and DeepSeek-R1 LLM mathematical reasoning on AIME problems. It finds that while the LLM mimics the structure of human solutions, it lacks genuine reasoning, exhibiting "topological mimicry" through repetitive, shallow checks and loops. However, successful LLM solutions show stable…
Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle
The paper introduces AARR (Act As a Real Researcher), a benchmark series designed to evaluate Large Language Models (LLMs) and AI agents on their ability to perform research tasks with the professionalism and nuanced judgment of human researchers, rather than just macro-level execution. The first benchmark, AARRI-Bench…
Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills
Socratic-SWE is a self-evolving framework for training software engineering agents. It leverages the agent's past problem-solving attempts (traces) to identify recurring failures and successful repair strategies. These distilled "skills" then guide the generation of new, targeted training tasks, creating a closed-loop …
Watch, Remember, Reason: Human-View Video Understanding with MLLMs
This paper proposes a "human-view" framework for video understanding using Multimodal Large Language Models (MLLMs), organizing capabilities into "watching," "remembering," and "reasoning." This approach aims to unify the analysis of how MLLMs process sparse evidence, maintain long-range context, and perform grounded i…
Self-evolving LLM agents with in-distribution Optimization
This paper introduces Q-Evolve, a framework for training LLM agents to make reliable long-horizon decisions. It addresses the challenge of delayed rewards by unifying automatic reward labeling and policy learning. Q-Evolve uses an in-distribution reinforcement learning approach with a hybrid dataset and a weighted Impl…
Do Coding Agents Deceive Us? Detecting and Preventing Cheating via Capped Evaluation with Randomized Tests
This paper addresses deceptive performance in coding agents, where models exploit shortcuts to achieve high scores without true task mastery. They propose CapCode, a framework generating datasets with randomized tests capped below perfect scores, making cheating evident when models exceed this cap. Additionally, CapRew…
DuMate-DeepResearch: An Auditable Multi-Agent System with Recursive Search and Rubric-Grounded Reasoning
DuMate-DeepResearch is a multi-agent framework for complex research tasks that addresses limitations in current Deep Research systems. Its core method decouples task planning and execution into an Agent Core and an extensible Tool Ecosystem, enabling explicit traceability of decisions and tool usage. This design enhanc…
Hierarchical Certified Semantic Commitment for Byzantine-Resilient LLM-Agent Collaboration
This paper introduces Hierarchical Certified Semantic Commitment (H-CSC), a novel Byzantine Fault Tolerance (BFT) protocol for LLM agents. H-CSC addresses the challenge of reaching consensus on natural language proposals by converting embedding-derived signals into three distinct outcomes: a semantic commit, a verdict …
How AI Agents Reshape Knowledge Work: Autonomy, Efficiency, and Scope
This paper investigates how autonomous AI agents, exemplified by Perplexity's "Computer" product, transform knowledge work compared to conversational assistants like "Search." The core method involves analyzing production data to compare task execution times and user behavior. The key contribution is demonstrating that…
MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanism
MemDreamer tackles long video understanding by decoupling perception and reasoning. It builds a Hierarchical Graph Memory to semantically abstract video content and uses an agentic retrieval mechanism to efficiently query this memory for reasoning, achieving state-of-the-art results with significantly reduced computati…
Online Pandora's Box for Contextual LLM Cascading
This paper introduces an "online contextual Pandora's Box" method for adaptively selecting Large Language Model (LLM) APIs. The core innovation is a two-phase decision process where the system queries APIs sequentially, incurring costs, and then selects an output based on observed rewards. The key contribution lies in …
SV-Detect: AI-generated Text Detection with Steering Vectors
This paper introduces SV-Detect, a novel AI-generated text detection method that utilizes "steering vectors" derived from a frozen language model's hidden states. These vectors capture directions that distinguish human from machine text at each layer, and the detector classifies text based on its alignment with these d…
When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations
This paper investigates how sensitive Large Language Models (LLMs), including those specialized for medicine, are to small changes in input prompts. The core method involves a systematic sensitivity analysis on the MedMCQA benchmark, categorizing prompt variations as natural or adversarial. The key contribution is demo…
TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention
TabSwift is an efficient tabular foundation model that leverages a lightweight row-wise attention mechanism. By incorporating a gated attention stabilization and learnable register tokens, it achieves competitive accuracy with state-of-the-art models while significantly reducing inference costs. This makes TabSwift a p…
Adversarial Creation and Detection of AI-Generated Social Bot Content
This paper proposes an adversarial methodology to create and detect AI-generated social bot content. By modeling malicious actors impersonating real users, they generate a unique, multilingual dataset of paired human and AI messages. This adversarial training significantly improves the accuracy of AI-generated text det…
Agentopia: Long-Term Life Simulation and Learning in Agent Societies
Agentopia introduces a framework for simulating long-term (10 years) life in agent societies powered by LLMs. Its core method involves agents autonomously pursuing growth, relationships, and goals within this extended simulation. The paper's contribution is demonstrating how such prolonged social experience can foster …
M$^3$Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions
This paper introduces M$^3$Exam, a novel benchmark for evaluating language agents' ability to reason over realistic, accumulating multimodal information from user interactions. Its core contribution is a query-centric approach that assesses cross-modal grounding and implicit information inference, revealing significant…
SWE-Explore: Benchmarking How Coding Agents Explore Repositories
SWE-Explore is a new benchmark designed to evaluate how coding agents explore software repositories. Instead of just checking if a task is solved, it measures an agent's ability to identify relevant code regions for a given issue within a limited budget. This allows for a more granular assessment of crucial exploration…
Sycophantic Praise: Evaluating Excessive Praise in Language Models
This paper introduces a new method to measure "sycophantic praise" in language models, which is defined as excessive flattery beyond what a contribution warrants. Their parameterized framework evaluates praise against contribution quality and user ability, outperforming generic LLM judges in agreement with human annota…
How reliable are LLMs when it comes to playing dice?
This paper benchmarks LLMs on dice-playing probability problems, finding they excel on standard exercises but struggle with counterintuitive ones. Their performance degrades significantly with disguised problem formulations and misleading prompt suggestions, indicating a lack of true probabilistic reasoning despite suc…