Tom Donnelly

Systems Engineer, JMP Statistical Discovery LLC
“Active Learning with Bayesian Optimization: A Modern Framework for Faster Testing”

Session Materials: Link

Speaker Bio: 

Tom Donnelly works as a Systems Engineer for JMP Statistical Discovery supporting users of JMP software in the Defense and Aerospace sector. He has been actively using and teaching Design of Experiments (DOE) methods for the past 40 years to develop and optimize products, processes, and technologies. Donnelly joined JMP in 2008 after working as an analyst for the Modeling, Simulation & Analysis Branch of the US Army’s Edgewood Chemical Biological Center – now DEVCOM CBC. There, he used DOE to develop, test, and evaluate technologies for detection, protection, and decontamination of chemical and biological agents. Prior to working for the Army, Tom was a partner in the first DOE software company for 20 years where he taught over 300 industrial short courses to engineers and scientists. Tom received his PhD in Physics from the University of Delaware.

Abstract: 

Design of Experiments (DoE) is frequently relied upon to conduct Test and Evaluation (T&E) to ensure systems work as intended, manage acquisition risks, and confirm technical performance, operational effectiveness, suitability, and survivability in realistic environments.   To maintain the balance of an experimental design we almost always end up running conditions where we fully expect (already know?) the system will work as intended. What if we could sequentially experiment while focusing testing on the regions of greater interest while choosing factor conditions that are likely to make performance metrics meet requirements?

Industrial R&D has begun to use Active Learning methods like Bayesian Optimization to speed up testing in the development of new products and processes.  These approaches to sequential experimentation promise greater efficiency while being more approachable to scientists, engineers, and testers.  The approach can leverage historical data to inform the initial model, support modifying factor ranges during sequential experimentation, and choose among candidate trials the ones that best satisfy goals.

Generalizing Bayesian optimization (Bayes opt.) to real world complex problems involving multiple responses has proven challenging because in its standard formulation the Bayes opt. approach is inherently limited to a single response. In this tutorial we review the basics of Gaussian Process regression modeling and the standard approach to Bayes opt. We then introduce the generalization to multiple responses via the Bayesian Desirability framework. We will demonstrate the efficiency and approachability of the technique using new capabilities in JMP Pro 19.

In general, the reduced testing under an Active Learning approach like Bayesian Optimization will not yield as robust a predictive model covering as large an operational envelope as would be obtained using a response surface DOE, but one should acquire data in the more important test regions more quickly.  The data collected during the Active Learning testing can easily be augmented to support a more robust model.