Caleb King

Research Statistician Developer, JMP Statistical Discovery LLC
“MaLT: Machine-Learning-Guided Test Design & Fault Localization of Complex Software Systems”

Session Materials: Link

Session Recording: Link

Speaker Bio: 

Caleb King is a Senior Research Statistician Developer in the DOE & Reliability group at JMP. He received is PhD in Statistics from Virginia Tech in 2015, after which he spent three years as a statistician at Sandia National Laboratories before transitioning to JMP. His areas of expertise include design of experiments, accelerated testing, and reliability analysis.

Abstract: 

Software testing is essential for the reliable and robust development of complex software systems. Consider a critical flight software system such as the Traffic Alert and Collision Avoidance System (TCAS), which must operate correctly across a wide range of scenarios to ensure aircraft safety. TCAS software depends on many interacting input parameters, leading to a combinatorial explosion of possible scenarios. Given the high cost of test execution and fault diagnosis, exhaustive testing is infeasible and so a combinatorial testing and machine-learning-based testing approach is desirable. We outline in this talk a holistic machine-learning-guided test case design and fault localization (MaLT) framework, which combines efficient combinatorial testing techniques with probabilistic machine learning methods to accelerate the testing and fault diagnosis of complex software systems. MaLT consists of three steps: (i) the construction of a suite of test cases using a covering array for initial testing, (ii) when test outcomes are available, the investigation of posterior root cause probabilities via a Bayesian fault localization procedure, then (iii) the use of such Bayesian analysis to guide selection of subsequent test cases via active learning. The proposed MaLT framework can thus facilitate efficient identification and subsequent diagnosis of software faults with limited test runs.