New Dataset Tracks AI Scientist Reasoning Processes

VERA Brief

AI-generated. Grounded in the article and its cited sources.

A new public dataset called OpenDiscoveryTrace has been released, containing 558 trajectories of AI scientific agents. This dataset details the step-by-step reasoning processes of AI models, aiming to enable auditing of AI scientific methodology and diagnosis of failure modes.

Key facts

• OpenDiscoveryTrace is a new public dataset containing 558 complete trajectories for AI scientific agents.
• The dataset captures step-by-step reasoning processes of AI models, including thoughts, tool calls, observations, errors, and confidence.
• It covers 124 scientific tasks across various domains like drug discovery and genomics.
• The data includes trajectories from seven models: three frontier models and four open-weight models.
• Process traces reveal behavioral differences not apparent in output-only evaluations, such as variations in error rates between frontier models.