CATArena Advances AI Research While Denario Simplifies Financial Analysis

Large language models (LLMs) have made significant progress in various tasks, including coding, reasoning, and social deduction. However, existing benchmarks often focus on narrow, verifiable tasks, excluding open-ended research. A recent study introduced a new way to measure progress towards AI R&D automation by having an agent take on the central research question of a high-quality unpublished paper and grading its output. The results showed that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle. Another study proposed a framework for evaluating objective misalignment in mixed-motive LLM multi-agent systems, which found that objective misalignment undermines outcomes in inherently adversarial environments. A novel Belief-Guided architecture was introduced for the game of Go, which disentangles the Policy head from a distinct Belief head and models epistemic uncertainty and strategic stability. A graph-guided LLM multi-agent system, UrbanDS, was proposed for data-intensive urban tasks, which constructs a unified dataset graph to organize reusable dataset skills and relationships among datasets. The system was evaluated on both general and urban benchmarks, demonstrating its effectiveness in handling data-intensive urban scenarios.

Despite the progress made by LLMs, there are still challenges to be addressed. For instance, existing benchmarks often rely on shared models, which can drive authors toward a common norm, reducing population-level variation in linguistic form. A study proposed a mathematical framework to analyze the effects of shared models on linguistic form, which showed that shared models can drive authors toward a common norm, but recursive feedback can relocate the shared norm without altering pairwise spread under common conformity. Another study introduced a benchmark for evaluating memory-augmented personalized agents with hierarchical user understanding from heterogeneous data, which found that existing systems perform well on semantic memory retrieval, but decline on episodic memory. A framework for evaluating evidence-ledger adjudication was proposed, which paired each claim with an evidence packet and assigned a support relation, and found that the agent evidence-ledger condition achieved 0.676 relation accuracy and 0.601 macro-F1.

The development of LLMs has also led to the creation of new benchmarks and evaluation methods. For instance, a study proposed a routing-based on-policy distillation framework for LLM safety, which models the divergence between aligned and compromised output probability distributions rather than fitting specific prompt templates. Another study introduced a benchmark for evaluating LLM agents on long-horizon office-suite tasks with economic grounding, which found that existing LLMs are substantially cheaper and faster than human workers, but have not yet approached human-level deliverable quality. A framework for evaluating the projectibility of benchmark inferences was proposed, which identified a non-composition principle: support for adjacent projections warrants their composition only when endpoints and assumptions align and dependence and uncertainty are carried through.

In addition to the progress made by LLMs, there are also new challenges and opportunities in the field. For instance, a study proposed a framework for evaluating the effectiveness of LLMs in handling data-intensive urban scenarios, which found that UrbanDS consistently outperformed existing data science agents on data-intensive tasks. Another study introduced a benchmark for evaluating the ability of LLMs to conduct open-ended AI research, which found that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle. A framework for evaluating the projectibility of benchmark inferences was proposed, which identified a non-composition principle: support for adjacent projections warrants their composition only when endpoints and assumptions align and dependence and uncertainty are carried through.