Benchmarking Simulation-Based Inference

Download

• Paper
• Code
• Website

Abstract

Simulation-based inference (SBI) has emerged as a powerful framework for performing Bayesian inference when the likelihood function is intractable but simulations from the model are feasible. The rapid development of SBI methods has created a need for systematic evaluation and comparison. We present a comprehensive benchmark for SBI methods, including tasks with different properties: dimensionality of parameters and data, simulation budget, and requirements for amortization. Our benchmark includes reference posteriors obtained with expensive Monte Carlo methods, standardized implementations of competing algorithms, and metrics for accuracy and computational efficiency. We evaluate six SBI algorithms across ten benchmark tasks, revealing strengths and weaknesses of different approaches. Sequential methods generally achieve better accuracy per simulation, while amortized methods excel when many posterior evaluations are needed. Flow-based methods consistently outperform methods based on mixture density networks. Our benchmark provides a foundation for systematic progress in SBI research and is available as an open-source Python package.

Citation

@inproceedings{lueckmann2021benchmarking,
  title={Benchmarking Simulation-Based Inference},
  author={Lueckmann, Jan-Matthis and Boelts, Jan and Greenberg, David S and Gon{\c{c}}alves, Pedro J and Macke, Jakob H},
  booktitle={International Conference on Artificial Intelligence and Statistics (AISTATS)},
  pages={343--351},
  year={2021},
  organization={PMLR}
}