Offline evaluation informs adaptive experiment design

VERA Brief

AI-generated. Grounded in the article and its cited sources.

Researchers are using historical A/B test data to inform the design of adaptive experiments with contextual bandits. The aim is to determine if adaptive strategies would have been more effective than static designs by evaluating their performance under simulated conditions.

Key facts

• Historical A/B test data can be used to inform the creation of adaptive experiments based on contextual bandits.
• The research assesses which adaptive policies could have outperformed original designs and under what conditions.
• The methodology combines off-policy evaluation with a controlled warm-start simulation using logged A/B test data.
• Adaptive, context-aware policies show improvements over fixed allocations when significant heterogeneity exists.
• This work offers a practical approach to assessing adaptive experimentation suitability using offline data.