System-1 Models Show Mixed Accuracy in LLM Agent Decision Tasks

VERA Brief

AI-generated. Grounded in the article and its cited sources.

A study evaluated System-1 decision models for LLM agents, revealing significant accuracy differences between tested models. The research also identified self-audited errors in its own evaluation pipeline, impacting reported figures.

Key facts

• System-1 models aim to reduce cost and latency in LLM agents by making rapid decisions.
• Jev, a hosted model, showed significantly higher accuracy than Laya, an open-weight model, on most agent decision points.
• Neither model performed well on zero-shot model routing, and both had similar performance on RAG relevance gating.
• Laya's accuracy was negatively impacted by changes in option order and a larger number of candidate tools.
• The study identified self-audited errors in its evaluation pipeline, affecting cost reporting, accuracy metrics, and injection detection.