Invited Abstracts

Total Invited Abstracts: 15
Show Invited Abstracts
# Name / Org Type Abstract Title Theme Abstract Add to Speakers
1 Terril Hurst
Senior Engineering Fellow,Raytheon Technologies		 Breakout
 A Decision-Theoretic Framework for Adaptive Simulation Experiments Design of Experiments We describe a model-based framework for increasing effectiveness of simulation experiments in the Read More presence of uncertainty. Unlike conventionally designed simulation experiments, it adaptively chooses where to sample, based on the value of information obtained. A Bayesian perspective is taken to formulate and update the framework’s four models. A simulation experiment is conducted to answer some question. In order to define precisely how informative a run is for answering the question, the answer must be defined as a random variable. This random variable is called a query and has the general form of p(theta | y), where theta is the query parameter and y is the available data. Examples of each of the four models employed in the framework are briefly described below: 1. The continuous correlated beta process model (CCBP) estimates the proportions of successes and failures using beta-distributed uncertainty at every point in the input space. It combines results using an exponentially decaying correlation function. The output of the CCBP is used to estimate value of a candidate run. 2. The mutual information model quantifies uncertainty in one random variable that is reduced by observing the other one. The model quantifies the mutual information between any candidate runs and the query, thereby scoring the value of running each candidate. 3. The cost model estimates how long future runs will take, based upon past runs using, e.g., a generalized linear model. A given simulation might have multiple fidelity options that require different run times. It may be desirable to balance information with the cost of a mixture of runs using these multi-fidelity options. 4. The grid state model, together with the mutual information model, are used to select the next collection of runs for optimal information per cost, accounting for current grid load. The framework has been applied to several use cases, including model verification and validation with uncertainty quantification (VVUQ). Given a mathematically precise query, an 80 percent reduction in total runs has been observed. Add to Speakers
2 Kathy Ortega
System Integration Test & Evaluation Engineering Manager,Northrop Grumman Aerospace Systems		 Breakout
 Leveraging IBM Jazz Capabilities to Optimize T&E Processes Test and Evaluation Methods for Emerging Technology Many Test & Evaluation (T&E) processes within the Aerospace and Defense industry, Read More specifically the autonomous weapon systems area, are outdated, inefficient, and prone to user error. Not only this, but there is also lack of customer visibility and the Digital Thread is not maintained throughout. Current test planning, procedure development, peer review, test execution, metrics collection and defect reporting processes rely on a wide array of fragmented tools. Incorporating a single-location integrated suite of online tools is greatly beneficial to both defense contractors and customers alike. The online suite of tools that presents a huge opportunity for modernization and tool conglomeration is IBM Jazz. In this Breakout session, the Quality Management and Change and Configuration Management areas of IBM Jazz will be explored. More to Follow. Add to Speakers
3 J. Michael Gilmore
Research Staff Member,Institute for Defense Analyses		 Breakout
 Applying Design of Experiments to Cyber Testing Design of Experiments We describe a potential framework for applying DOE to cyber testing and provide an example of its Read More application to testing of a hypothetical command and control system. Add to Speakers
4 Joel Williamsen
Research Staff Member,Institute for Defense Analyses		 Breakout
 DebProp: Orbital Debris Collision Effects Prediction Tool for Satellite Constellations Analysis Tools and Techniques Based on observations gathered from the IDA Forum on Orbital Debris (OD) Risks and Challenges Read More (October 8-9, 2020), the Director, Operational Test and Evaluation (DOT&E) needed first order predictive tools to evaluate the effects of orbital debris on mission risk, catastrophic collision, and collateral damage to DoD spacecraft and other orbital assets – either from unintentional or intentional [Anti-Satellite (ASAT)] collisions. This lack of modeling capability hindered the ability of DOT&E to evaluate the risk to operational effectiveness and survivability of individual satellites and large constellations, as well as risks to the overall use of space assets in the future. This presentation describes an IDA-derived technique (DebProp) to evaluate the debris propagating effects of large, trackable debris (>5 cm) or antisatellite weapons colliding with satellites within constellations. Using a Starlink-like satellite as a case study and working with Stellingwerf Associates, IDA researchers modified the Smooth Particle Hydrodynamic Code (SPHC) to output a file format that is readable as an input file for predicting orbital stability or debris re-entry for thousands of created particles, and predict additional, short term OD induced losses to other satellites in the constellation. By pairing this technique with SatPen (an Excel-based tool for determining the probability and mission effects of >1mm OD impacts and penetration on individual satellites with ORDEM 3.1 as an input, supplemented with typical damage prediction equations to support mission loss predictions), IDA can conduct long-term debris growth studies). Add to Speakers
5 Joel Williamsen
Research Staff Member,IDA		 Breakout
 SatPen: Orbital Debris Risk Prediction Tool for Constellation Satellites Analysis Tools and Techniques Based on observations gathered from the IDA Forum on Orbital Debris (OD) Risks and Challenges Read More (October 8-9, 2020), DOT&E needed first order predictive tools to evaluate the effects of orbital debris on mission risk, catastrophic collision, and collateral damage to DoD spacecraft and other orbital assets – either from unintentional or intentional [Anti-Satellite (ASAT)] collisions. This lack of modeling capability hindered the ability of DOT&E to evaluate the risk to operational effectiveness and survivability of individual satellites and large constellations, as well as risks to the overall use of space assets in the future. This presentation describes an IDA-derived Excel-based tool (SatPen) for determining the probability and mission effects of >1mm orbital debris impacts and penetration on individual satellites in low Earth orbit (LEO). Using a Starlink-like satellite as a case study and NASA’s ORDEM 3.1 orbital debris environment as an input, supplemented with typical damage prediction equations to support mission loss predictions, IDA estimated the likelihood of satellite mission loss. By pairing this technique with DebProp (another IDA tool described at this forum for evaluating the debris propagating effects of large, trackable debris or antisatellite weapons colliding with satellites within constellations), IDA can predict additional, short term OD induced losses to other satellites in the constellation, and conduct long-term debris growth studies. Add to Speakers
6 Joel Williamsen
Research Staff Member,IDA		 Breakout
Publish		 Orbital Debris Effects Prediction Tool for Satellite Constellations Analysis Tools and Techniques Based on observations gathered from the IDA Forum on Orbital Debris (OD) Risks and Challenges Read More (October 8-9, 2020), DOT&E needed first-order predictive tools to evaluate the effects of orbital debris on mission risk, catastrophic collision, and collateral damage to DOD spacecraft and other orbital assets – either from unintentional or intentional [Anti-Satellite (ASAT)] collisions. This lack of modeling capability hindered DOT&E’s ability to evaluate the risk to operational effectiveness and survivability of individual satellites and large constellations, as well as risks to the overall use of space assets in the future. Part 1 of this presentation describes an IDA-derived Excel-based tool (SatPen) for determining the probability and mission effects of >1mm orbital debris impacts and penetration on individual satellites in low Earth orbit (LEO). IDA estimated the likelihood of satellite mission loss using a Starlink-like satellite as a case study and NASA’s ORDEM 3.1 orbital debris environment as an input, supplemented with typical damage prediction equations to support mission loss predictions. Part 2 of this presentation describes an IDA-derived technique (DebProp) to evaluate the debris propagating effects of large, trackable debris (>5 cm) or antisatellite weapons colliding with satellites within constellations. IDA researchers again used a Starlink-like satellite as a case study and worked with Stellingwerf Associates to modify the Smooth Particle Hydrodynamic Code (SPHC) in order to predict the number and direction of fragments following a collision by a tracked satellite fragment. The result is a file format that is readable as an input file for predicting orbital stability or debris re-entry for thousands of created particles, and predict additional, short-term OD-induced losses to other satellites in the constellation. By pairing these techniques, IDA can predict additional, short-term and long-term OD-induced losses to other satellites in the constellation, and conduct long-term debris growth studies. Add to Speakers
7 ZHI WANG
Data Scientist Contractor,Bayer Crop Science		 Breakout
No Publish		 Estimating the time of sudden shift in the location or scale of ergodic-stationary process Analysis Tools and Techniques Autocorrelated sequences arise in many modern-day industrial applications. In this paper, our focus Read More is on estimating the time of sudden shift in the location or scale of a continuous ergodic-stationary sequence following a genuine signal from a statistical process control chart. Our general approach involves “clipping” the continuous sequence at the median or interquartile range (IQR) to produce a binary sequence, and then modeling the joint mass function for the binary sequence using a Bahadur approximation. We then derive a maximum likelihood estimator for the time of sudden shift in the mean of the binary sequence. Performance comparisons are made between our proposed change point estimator and two other viable alternatives. Although the literature contains existing methods for estimating the time of sudden shift in the mean and/or variance of a continuous process, most are derived under strict independence and distributional assumptions. Such assumptions are often too restrictive, particularly when applications involve Industry 4.0 processes where autocorrelation is prevalent and the distribution of the data is likely unknown. The change point estimation strategy proposed in this work easily incorporates autocorrelation and is distribution-free. Consequently, it is widely applicable to modern-day industrial processes. Add to Speakers
8 Gina Sigler & Alex (Mary) McBride
Statistician,Scientific Test and Analysis Techniques Center of Excellence (STAT COE)		 Tutorial
No Publish		 Survey Dos and Don’ts Analysis Tools and Techniques How many surveys have you been asked to fill out? How many did you actually complete? Why those Read More surveys? Did you ever feel like the answer you wanted to mark was missing from the list of possible responses? Surveys can be a great tool for data collection if they are thoroughly planned out and well-designed. They are a relatively inexpensive way to collect a large amount of data from hard to reach populations. However, if they are poorly designed, the test team might end up with a lot of data and little to no information. Join the STAT COE for a short tutorial on the dos and don’ts of survey design and analysis. We’ll point out the five most common survey mistakes, compare and contrast types of questions, discuss the pros and cons for potential analysis methods (such as descriptive statistics, linear regression, principal component analysis, factor analysis, hypothesis testing, and cluster analysis), and highlight how surveys can be used to supplement other sources of information to provide value to an overall test effort. DISTRIBUTION STATEMENT A. Approved for public release; distribution is unlimited. CLEARED on 5 Jan 2022. Case Number: 88ABW-2022-0003 Add to Speakers
9 Ryan Norman
Chief Data Officer,Test Resource Management Center		 Breakout
No Publish		 TRMC Big Data Analytics Investments & Technology Review Analysis Tools and Techniques To properly test and evaluate today’s advanced military systems, the T&E community must utilize Read More big data analytics (BDA) and techniques to quickly process, visualize, understand, and report on massive amounts of data. This tutorial/presentation/TBD will inform the audience how to transform the current T&E data infrastructure and analysis techniques to one employing enterprise BDA and Knowledge Management (BDKM) that supports the current warfighter T&E needs and the developmental and operational testing of future weapon platforms. The TRMC enterprise BDKM will improve acquisition efficiency, keep up with the rapid pace of acquisition technological advancement, and ensure that effective weapon systems are delivered to warfighters at the speed of relevance – all while enabling T&E analysts across the acquisition lifecycle to make better and faster decisions using data previously inaccessible or unusable.  This capability encompasses a big data architecture framework – its supporting resources, methodologies, and guidance – to properly address the current and future data needs of systems testing and analysis, as well as an implementation framework, the Cloud Hybrid Edge-to-Enterprise Evaluation and Test Analysis Suite (CHEETAS). In combination with the TRMC’s Joint Mission Environment Test Capability (JMETC) which provides readily-available connectivity to the Services’ distributed test capabilities and simulations, the TRMC has demonstrated that applying enterprise-distributed BDA tools and techniques to distributed T&E leads to faster and more informed decision-making – resulting in reduced overall program cost and risk. Add to Speakers
10 Daniel A. Timme
PhD Candidate,Florida State University		 Breakout
Publish		 Nonparametric multivariate profile monitoring using regression trees Test and Evaluation Methods for Emerging Technology Monitoring noisy profiles for changes in the behavior can be used to validate whether the process is Read More operating under normal conditions over time. Change-point detection and estimation in sequences of multivariate functional observations is a common method utilized in monitoring such profiles. A nonparametric method utilizing Classification and Regression Trees (CART) to build a sequence of regression trees is proposed which makes use of the Kolmogorov-Smirnov statistic to monitor profile behavior. Our novel method compares favorably to existing methods in the literature. Add to Speakers
11 Keyla Pagán-Rivera
Research Staff Member,Institute for Defense Analyses		 Tutorial
No Publish		 Case Study on Applying Sequential Methods in Operational Testing Design of Experiments Sequential methods concerns statistical evaluation in which the number, pattern, or composition of Read More the data is not determined at the start of the investigation but instead depends on the information acquired during the investigation. Although sequential methods originated in ballistics testing for the Department of Defense (DoD), it is underutilized in the DoD. Expanding the use of sequential methods may save money and reduce test time. In this presentation, we introduce sequential methods, describe its potential uses in operational test and evaluation (OT&E), and present a method for applying it to the test and evaluation of defense systems. We evaluate the proposed method by performing simulation studies and applying the method to a case study. Additionally, we discuss some of the challenges we might encounter when using sequential analysis in OT&E. Add to Speakers
12 James Warner
Computational Scientist,NASA Langley Research Center		 Breakout
No Publish		 Machine Learning for Uncertainty Quantification: Trusting the Black Box Analysis Tools and Techniques Adopting uncertainty quantification (UQ) has become a prerequisite for providing credibility in Read More modeling and simulation (M&S) applications. It is well known, however, that UQ can be computationally prohibitive for problems involving expensive high-fidelity models, since a large number of model evaluations is typically required. A common approach for improving efficiency is to replace the original model with an approximate surrogate model (i.e., metamodel, response surface, etc.) using machine learning that makes predictions in a fraction of the time. While surrogate modeling has been commonplace in the UQ field for over a decade, many practitioners still remain hesitant to rely on “black box” machine learning models over trusted physics-based models (e.g., FEA) for their analyses. This talk discusses the role of machine learning in enabling computational speedup for UQ, including traditional limitations and modern efforts to overcome them. An overview of surrogate modeling and its best practices for effective use is first provided. Then, some emerging methods that aim to unify physics-based and data-based approaches for UQ are introduced, including multi-model Monte Carlo simulation and physics-informed machine learning. The use of both traditional surrogate modeling and these more advanced machine learning methods for UQ are highlighted in the context of applications at NASA, including trajectory simulation and spacesuit certification. Add to Speakers
13 Elizabeth Claassen
Research Statistician Developer,JMP Statistical Discovery		 Tutorial
No Publish		 Mixed Models: A Critical Tool for Dependent Observations Analysis Tools and Techniques The use of fixed and random effects have a rich history. They often go by other names, including Read More blocking models, variance component models, nested and split-plot designs, hierarchical linear models, multilevel models, empirical Bayes, repeated measures, covariance structure models, and random coefficient models. Mixed models are one of the most powerful and practical ways to analyze experimental data, and investing time to become skilled with them is well worth the effort. Many, if not most, real-life data sets do not satisfy the standard statistical assumption of independent observations. Failure to appropriately model design structure can easily result in biased inferences. With an appropriate mixed model we can estimate primary effects of interest as well as compare sources of variability using common forms of dependence among sets of observations. Mixed Models can readily become the most handy method in your analytical toolbox and provide a foundational framework for understanding statistical modeling in general. In this course we will cover many types of mixed models, including blocking, split-plot, and random coefficients. Add to Speakers
14 Christina Heinich
AST, Data Systems,NASA		 Tutorial
No Publish		 An Introduction to Data Visualization Special Topics Data visualization can be used to present findings, explore data, and use the human eye to find Read More patterns that a computer would struggle to locate. Borrowing tools from art, storytelling, data analytics and software development, data visualization is an indispensable part of the analysis process. While data visualization usage spans across multiple disciplines and sectors, most never receive formal training in the subject. As such, this tutorial will introduce key data visualization building blocks and how to best use those building blocks for different scenarios and audiences. We will also go over tips on accessibility, design and interactive elements. While this will by no means be a complete overview of the data visualization field, by building a foundation and introducing some rules of thumb, attendees will be better equipped for communicating their findings to their audience. Add to Speakers
15 Lindsey Butler
Research Staff Member,Institute for Defense Analyses		 Breakout
No Publish		 A New Method for Planning Full-Up System-Level (FUSL) Live Fire Tests Design of Experiments Planning Full-Up System-Level (FUSL) Live Fire tests is a complex process that has historically Read More relied solely on subject matter expertise. In particular, there is no established method to determine the appropriate number of FUSL tests necessary for a given program. We developed a novel method that is analogous to the Design of Experiments process that is used to determine the scope of Operational Test events. Our proposed methodology first requires subject matter experts (SMEs) to define all potential FUSL shots. For each potential shot, SMEs estimate the severity of that shot, the uncertainty of that severity estimate, and the similarity of that shot to all other potential shots. We developed a numerical optimization algorithm that uses the SME inputs to generate a prioritized list of FUSL events and a corresponding plot of the total information gained with each successive shot. Together, these outputs can help analysts determine the adequate number of FUSL tests for a given program. We illustrate this process with an example on a notional ground vehicle. Future work is necessary prior to implementation on a program of record. Add to Speakers

Contributed Abstracts

Total Contributed Abstracts: 18
Show Contributed Abstracts
# Name / Org Type Abstract Title Theme Abstract Add to Speakers
1 Lauren H. Perry
Senior Project Engineer,The Aerospace Corporation		 Presentation Trust Throughout the Artificial Intelligence Lifecycle Test & Evaluation Methods for Emerging Technology AI and machine learning have become widespread throughout the defense, government, and commercial Read More sectors. This has led to increased attention on the topic of trust and the role it plays in successfully integrating AI into highconsequence environments where tolerance for risk is low. Driven by recent successes of AI algorithms in a range of applications, users and organizations rely on AI to provide new, faster, and more adaptive capabilities. However, along with those successes have come notable pitfalls, such as bias, vulnerability to adversarial attack, and inability to perform as expected in novel environments. Many types of AI are data-driven, meaning they operate on and learn their internal models directly from data. Therefore, tracking how data were used to build data properties (e.g., training, validation, and testing) is crucial not only to ensure a high-performing model, but also to understand if the AI should be trusted. MLOps, an offshoot of DevSecOps, is a set of best practices meant to standardize and streamline the end-to-end lifecycle of machine learning. In addition to supporting the software development and hardware requirements of AI-based systems, MLOps provides a scaffold by which the attributes of trust can be formally and methodically evaluated. Additionally, MLOps encourages reasoning about trust early and often in the development cycle. To this end, we present a framework that encourages the development of AI-based applications that can be trusted to operate as intended and function safely both with and without human interaction. This framework offers guidance for each phase of the AI lifecycle, utilizing MLOps, through a detailed discussion of pitfalls resulting from not considering trust, metrics for measuring attributes of trust, and mitigations strategies for when risk tolerance is low.
2 Gregory J. Hunt
Assistant Professor,William & Mary		 Presentation Everday Reproducibility Data Management / Reproducible Research Modern data analysis is typically quite computational. Correspondingly, sharing scientific and Read More statistical work now often means sharing code and data in addition writing papers and giving talks. This type of code sharing faces several challenges. For example, it is often difficult to take code from one computer and run it on another due to software configuration, version, and dependency issues. Even if the code runs, writing code that is easy to understand or interact with can be difficult. This makes it difficult to assess third-party code and its findings, for example, in a review process. In this talk we describe a combination of two computing technologies that help make analyses shareable, interactive, and completely reproducible. These technologies are (1) analysis containerization, which leverages virtualization to fully encapsulate analysis, data, code and dependencies into an interactive and shareable format, and (2) code notebooks, a literate programming format for interacting with analyses. This talks reviews both the problems at the high-level and also provides concrete solutions to the challenges faced. In addition to discussing reproducibility and data/code sharing generally, we will touch upon several such issues that arise specifically in the defense and aerospace communities.
3 William Raymond Whitledge
Research Staff Member,IDA		 Poster Session Analysis Apps for the Operational Tester Analysis Tools and Techniques In the acquisition and testing world, data analysts repeatedly encounter certain categories of data, Read More such as time or distance until an event (e.g., failure, alert, detection), binary outcomes (e.g., success/failure, hit/miss), and survey responses. Analysts need tools that enable them to produce quality and timely analyses of the data they acquire during testing. This poster presents four web-based apps that can analyze these types of data. The apps are designed to assist analysts and researchers with simple repeatable analysis tasks, such as building summary tables and plots for reports or briefings. Using software tools like these apps can increase reproducibility of results, timeliness of analysis and reporting, attractiveness and standardization of aesthetics in figures, and accuracy of results. The first app models reliability of a system or component by fitting parametric statistical distributions to time-to-failure data. The second app fits a logistic regression model to binary data with one or two independent continuous variables as predictors. The third calculates summary statistics and produces plots of groups of Likert-scale survey question responses. The fourth calculates the system usability scale (SUS) scores for SUS survey responses and enables the app user to plot scores versus an independent variable. These apps are available for public use on the Test Science Interactive Tools webpage https://new.testscience.org/interactive-tools/.
4 Terril Hurst
Senior Engineering Fellow,Raytheon Technologies		 Presentation A Decision-Theoretic Framework for Adaptive Simulation Experiments Design of Experiments This paper describes a framework for increasing effectiveness of high-performance computing (HPC) to Read More support decision-making in the presence of uncertainty intrinsic to queries, models, and simulation results. Given a mathematically precise query, the framework adaptively chooses where to sample. Unlike conventionally designed simulation experiments, which specify beforehand where to sample, the framework optimally schedules sampling predicated upon four interconnected models: (a) the surrogate model, e.g., a continuous correlated beta process, which globally estimates the response using beta distributions; (b) a value model, e.g., mutual information, for estimating the benefit of candidate runs to answering the query; (c) a cost model for predicting time to execute candidate runs, possibly from multi-fidelity simulation options; and (d) a grid state model. Runs are chosen by maximizing information per cost. A Bayesian perspective is taken to formulate and update each of these models as simulation results arrive for use in the iterative run-selection and scheduling phase. For a precisely stated query, up to an 80 percent reduction in total runs has been observed. The paper illustrates use of the framework with simple examples. A simulation experiment is conducted to answer some question. In order to define precisely how informative a run is for answering the question, the answer must be defined as a random variable. This random variable is called a query and has the general form of p(theta | y), where theta is the query parameter and y is the available data. Example models employed in the framework are briefly described below: 1. The continuous correlated beta process model (CCBP) estimates the proportions of successes and failures using beta-distributed uncertainty at every point in the input space. It combines results using an exponentially decaying correlation function. The output of the CCBP is used to estimate value of a candidate run. 2. The mutual information model quantifies uncertainty in one random variable that is reduced by observing the other one. The model quantifies the mutual information between any candidate runs and the query , thereby scoring the value of running each candidate. 3. The cost model estimates how long future runs will take, based upon past runs using, e.g., a generalized linear model. A given simulation might have multiple fidelity options that require different run times. It may be desirable to balance information with the cost of a mixture of runs using these multi-fidelity options. 4. The grid state model, together with the mutual information model, are used to select the next collection of runs for optimal information per cost, accounting for current grid load. The framework has been applied to multiple use cases involving a variety of queries: (a) assessing compliance with a performance requirement, (b) sensitivity analysis to system input factors, (c) design optimization in the presence of uncertainty, and (d) calibration of simulations using field data, i.e. model verification and validation that includes quantifying uncertainty (VVUQ). The paper describes several aspects that emerge when applying the framework to each of these use cases.
5 Neil Ashton
WW Principal CFD Specialist Solution Architect, HPC,Amazon Web Services		 Presentation Cloud Computing for Computational Fluid Dynamics (CFD) in T&E Test & Evaluation Methods for Emerging Technology In this talk we’ll focus on exploring the motivation for using cloud computing for Computational Read More Fluid Dynamics (CFD) for Federal Government Test & Evaluation. Using examples from automotive, aerospace and manufacturing we’ll look at benchmarks for a number of CFD codes using CPUs (x86 & Arm) and GPUs and we’ll look at how the development of high-fidelity CFD e.g. WMLES, HRLES, is accelerating the need for access to large scale HPC. The onset of COVID-19 has also meant a large increase in the need for remote visualization with greater numbers of researchers and engineering needing to work from home. This has also accelerated the adoption of the same approaches needed towards the pre- and post-processing of peta/exa-scale CFD simulation and we’ll look at how these are more easily accessed via a cloud infrastructure. Finally, we’ll explore perspectives on integrating ML/AI into CFD workflows using data lakes from a range of sources and where the next decade may take us.
6 kazu okumoto
CEO,Sakura Software Solutions (3S) LLC		 Presentation A revolutionary approach to software reliability modeling for software quality assurance Analysis Tools and Techniques This tutorial introduces a revolutionary approach to software reliability modeling for software Read More quality assurance. It is an integration of traditional software reliability modeling with recent advances in computing science. Software reliability models have been based on a single mathematical curve, i.e., either an S-shaped or an exponential curve, to represent a defect detection process. We have developed an innovative method for automatically generating multiple curves to accurately represent a defect detection process by identifying inflection points. It’s a piece-wise application of well-known exponential statistical models based on a non-homogeneous Poisson process. It is a good representation of the complex nature of current software development and test processes. With the advancement in computing science the innovative algorithm has been implemented in Python to run in a cloud environment. This approach enables the analytics to run in a real-time environment and share the results with other project team members. And it becomes more practical for use. In addition to the defect detection process, we also address the importance of defect closure process. In practice, we need to allocate development resources to fix software defects or bugs. We have developed a method for predicting software defect closure curve based on the defect detection curve. The difference between the detection curve and the closure curve is called the defect open curve, which represents the number of defects to be fixed. It’s an important measure for software quality at a delivery date. By combining the detection and open curves the project management will be able to balance the development and test resource allocation for a given delivery date. Next, we address early defect prediction without actual defect data during a planning phase. For this purpose, we use development and test effort data which are good representation of software complexity. We then analyze the relationship between defects detected and effort spent from previous releases. Our approach is not only technically sound but also useful for real software development industries to deliver a high-quality software product. Key steps encompassing from concept and proof of concept (prototype & trials) to productization of the innovative algorithm have been demonstrated. We have developed an online tool, called STAR. Its presentation of the output is focused on visualization using charts and tables. It will help project managers to quantitatively strike a balance between software delivery deadlines and quality. STAR can be used for internal test defects or acceptance test defects. It also includes early defect prediction without actual data using planning data such as development and test effort data during a planning period. It’s applicable to various software projects ranging from small-scale to large-scale development. In addition, STAR interactively provides quality impact of corrective actions such as a delay of delivery date or additional development resources. Analytics are all based on zero-touch automation algorithm. User input is basically delivery dates and defect data plus planning data as an option. The tutorial will cover an online demonstration of STAR as time permits.
7 Hyoshin Kim
,North Carolina State University		 Poster Session Risk Comparison and Planning for Bayesian Assurance Tests Design of Experiments Designing a Bayesian assurance test plan requires choosing a test plan that guarantees a product of Read More interest is good enough to satisfy consumer’s criteria but not ‘so good’ that it causes producer’s concern if they fail the test. Bayesian assurance tests are especially useful because they can incorporate previous product information in the test planning and explicitly control levels of risk for the consumer and producer. We demonstrate an algorithm for efficiently computing a test plan given desired levels of risks in binomial and exponential testing. Numerical comparisons with the Operational Characteristic (OC) curve, Probability Ratio Sequential Test (PRST), and a simulation-based Bayesian sample size determination approach are also considered.
8 kazuhira okumoto
CEO,Sakura Software Solutions (3S) LLC		 Poster Session Live demo of a revolutionary online tool for software quality assurance Analysis Tools and Techniques This poster session presents a revolutionary online tool, STAR, for software quality assurance. STAR Read More implements a world-leading prediction method of software defects with user friendly interface and visualization. Key measurements used as input are defect detected date and closed date with severity and impacted component for each defect during the internal test and customer deployment periods. It will automatically generate output such as how many more defects are expected by delivery, how many more defects are expected after deployment, how many defects are still open at delivery. It will enable software development companies to quantitatively strike a balance between delivery deadlines and quality. With a direct connection to customer’s fault database, it will automatically extract the data from the database and update prediction in real time. The results can be shared with other project members via the online access. A live demonstration will show a user input upload process and the interpretation of STAR output using demo data sets. The defect data for demonstration has been generated based on over 40 years of experience working with real projects. STAR can be used for internal test defects or acceptance test defects. It’s applicable to various software projects ranging from small-scale to large-scale development. It contains several output views with actual vs. prediction: Executive summary of current quality assessment and predicted quality metrics at delivery, defect arrival & closure trends, defects by severity & component, release over release view, prediction stability over time. STAR also includes early defect prediction without actual data using planning data such as development and test effort data during the planning period. In addition, it also provides quality impact of corrective actions interactively. Examples of corrective action are delay of delivery date and additional developers. Analytics are all based on zero-touch automation algorithm. User input is basically delivery dates and defect data plus planning data as an option. This poster session will be helpful for the software reliability community for both academia and industry practitioners. Academia will be able to understand the need for making the current reliability modeling more practical. Practitioners will be able to understand the power of the online analytics tool, STAR, and begin to collect right sets of data from their own development projects. It will allow them to focus on developing quality improvement plans. An earlier version of this tool has been extensively used at Nokia.
9 James Brownlow
mathematical statistician,USAF		 Presentation Analysis of Target Location Error using Stochastic Differential Equations Analysis Tools and Techniques This paper presents an analysis of target location error (TLE) based on the Cox Ingersoll Ross (CIR) Read More model. In brief, this model characterizes TLE as a function of range based the stochastic differential equation model dX(r) = a(b-X(r))dr + sigma *sqrt(X(r)) dW(r) where X(t) is TLE at range r, b is the long-term mean (terminal) of the TLE, a is the rate of reversion of X(r) to b, sigma is the process volatility, and W(t) is the standard Weiner process. Multiple flight test runs under the same conditions exhibit different realizations of the TLE process. This approach to TLE analysis models each flight test run as a realization the CIR process. Fitting a CIR model to multiple data runs then provides a characterization of the TLE system under test. This paper presents an example use of the CIR model. Maximum likelihood estimates of the parameters of the CIR model are found from a collection of TLE data runs. The resulting CIR model is then used to characterize overall system TLE performance as a function of range to the target as well as the asymptotic estimate of long-term TLE.
10 Andrew Mastin
Operations Research Scientist,Lawrence Livermore National Laboratory		 Presentation Sparse Models for Detecting Malicious Behavior in OpTC Analysis Tools and Techniques Host-based sensors are standard tools for generating event data to detect malicious activity on a Read More network. There is often interest in detecting activity using as few event classes as possible in order to minimize host processing slowdowns. Using DARPA’s Operationally Transparent Cyber (OpTC) Data Release, we consider the problem of detecting malicious activity using event counts aggregated over five-minute windows. Event counts are categorized by eleven features according to MITRE CAR data model objects. In the supervised setting, we use regression trees with all features to show that malicious activity can be detected at above a 90% true positive rate with a negligible false positive rate. Using forward and exhaustive search techniques, we show the same performance can be obtained using a sparse model with only three features. In the unsupervised setting, we show that the isolation forest algorithm is somewhat successful at detecting malicious activity, and that a sparse three-feature model performs comparably. Finally, we consider various search criteria for identifying sparse models and demonstrate that the RMSE criteria is generally optimal.
11 Sean Fiorito
short stop,VSTRAT, LLC		 Speed Session / Poster Testing Abstract Notice Design of Experiments Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore Read More et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum
12 Poornima Madhavan
Capability Steward for Social/Behavioral Sciences,MITRE		 Presentation A Framework to Guide Human-Systems Integration and Resilience in Sociotechnical Systems Test and Evaluation Methods for Emerging Technology The goal of this work is to support technology development teams in building advanced technologies Read More and transitioning them into existing systems in a way that strengthens the adaptivity and resilience of high-consequence work systems, including military, healthcare, air traffic, petrochemical, and utility systems. These work systems are complex: They feature a large variety of interacting factors; the variety and interactions produce emergent conditions that can challenge or exceed the system’s capability envelope. Furthermore, in these systems there is constant potential for excessive demands, outages and malfunctions, anomalies, threats such as ransomware, and crises. Given the variety and unpredictability of operating conditions as well as the high cost of failure, these work systems must be capable of adapting and evolving, i.e., demonstrate resilience. In today’s work systems, humans are the actuators and guides of system adaptation and evolution. For humans to effectively guide these processes, the system must have certain features, referred to as system resilience sources, which tend to be inherent in all resilient complex systems. We have developed the “Transform with Resilience during Upgrades to Socio-Technical Systems” (TRUSTS) Framework to guide system modernization and advanced-technology acquisition. The framework specifies complex-system resilience sources, allowing system stakeholders to identify modernization strategies that improve system resilience and enabling technology developers to design and transition technologies in ways that contribute to system resilience. Currently, we are translating the framework’s system resilience sources into engineering methods and tools. This presentation will provide an overview of the TRUSTS Framework and describe our efforts towards integrating it into systems development practice.
13 David Spalding
Research Staff Member,Institute for Defense Analyses		 Presentation Method for Evaluating Bayesian Reliability Models for Developmental Testing Analysis Tools and Techniques For analysis of military Developmental Test (DT) data, frequentist statistical models are Read More increasingly challenged to meet the needs of analysts and decision-makers. This is largely due to tightening constraints on test resources and schedule that reduce quantity and increase complexity of test data. Bayesian models have the potential to address this challenge. However, although there is a substantial body of research on Bayesian reliability estimation, there appears to be a paucity of Bayesian applications to issues of direct interest to DT decision makers. Due to user unfamiliarity with the characteristics and data needs of Bayesian models, the potential for such models appears to be unexploited. To address this deficiency, the purpose of this research is to provide a foundation and best practices for use of Bayesian reliability analysis in DT. This is accomplished by establishing a generic structure for systematically evaluating relevant statistical Bayesian models. First, it identifies reliability issues for DT programs using a structured poll of stakeholders combined with interviews of a selected set of Subject Matter Experts. Secondly, candidate solutions are identified in the literature and, thirdly solutions matched to issues using criteria designed to evaluate the capability of a solution to improve support for decision-makers at critical points in DT programs. The matching process uses a model taxonomy structured according to decisions at each DT phase, plus criteria for model applicability and data availability. The end result is a generic structure that allows an analyst to identify and evaluate a specific model for use with a program and issue of interest. This work includes example applications to models described in the statistical literature.
14 Anthony Sgambellone
,Huntington Ingalls Industries		 Presentation Building Bridges: a Case Study of Assisting a Program from the Outside Special Topics STAT practitioners often find ourselves outsiders to the programs we assist. This session presents Read More a case study that demonstrates some of the obstacles in communication of capabilities, purpose, and expectations that may arise due to approaching the project externally. Incremental value may open the door to greater collaboration in the future, and this presentation discusses potential solutions to provide greater benefit to testing programs in the face of obstacles that arise due to coming from outside the program team. DISTRIBUTION STATEMENT A. Approved for public release; distribution is unlimited. CLEARED on 5 Jan 2022. Case Number: 88ABW-2022-0002
15 Daniel Ries
,Sandia National Laboratories		 Speed Session / Poster Exploring the behavior of Bayesian adaptive design of experiments Design of Experiments Physical experiments in the national security arena, including nuclear deterrence, are often Read More expensive and time-consuming resulting in small sample sizes which make it difficult to achieve desired statistical properties. Bayesian adaptive design of experiments (BADE) is a sequential design of experiment approach which updates the test design in real time, in order to optimally collect data. BADE recommends ending experiments early by either concluding that the experiment would have ended in efficacy or futility, had the testing completely finished, with sufficiently high probability. This is done by using data already collected and marginalizing over the remaining uncollected data and updating the Bayesian posterior distribution in near real-time. BADE has seen successes in clinical trials, resulting in quicker and more effective assessments of drug trials while also reducing ethical concerns. BADE has typically only been used in futility studies rather than efficacy studies for clinical trials, although there hasn’t been much debate for this current paradigm. BADE has been proposed for testing in the national security space for similar reasons of quicker and cheaper test series. Given the high-consequence nature of the tests performed in the national security space, a strong understanding of new methods is required before being deployed. The main contribution of this research was to reproduce results seen in previous studies, for different aspects of model performance. A large simulation inspired by a real testing problem at Sandia National Laboratories was performed to understand the behavior of BADE under various scenarios, including shifts to mean, standard deviation, and distributional family, all in addition to the presence of outliers. The results help explain the behavior of BADE under various assumption violations. Using the results of this simulation, combined with previous work related to BADE in this field, it is argued this approach could be used as part of an “evidence package” for deciding to stop testing early due to futility, or with stronger evidence, efficacy. The combination of expert knowledge with statistical quantification provides the stronger evidence necessary for a method in its infancy in a high-consequence, new application area such as national security. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
16 Takayuki Iguchi
PhD Student,Florida State University		 Speed Session / Poster Profile Monitoring via Eigenvector Perturbation Test and Evaluation Methods for Emerging Technology Control charts are often used to monitor the quality characteristics of a process over time to Read More ensure undesirable behavior is quickly detected. The escalating complexity of processes we wish to monitor spurs the need for more flexible control charts such as those used in profile monitoring. Additionally, designing a control chart that has an acceptable false alarm rate for a practitioner is a common challenge. Alarm fatigue can occur if the sampling rate is high (say, once a millisecond) and the control chart is calibrated to an average in-control run length (ARL0) of 200 or 370 which is often done in the literature. As alarm fatigue may not just be annoyance but result in detrimental effects to the quality of the product, control chart designers should seek to minimize the false alarm rate. Unfortunately, reducing the false alarm rate typically comes at the cost of detection delay or average out-of-control run length (ARL1). Motivated by recent work on eigenvector perturbation theory, we develop a computationally fast control chart called the Eigenvector Perturbation Control Chart for nonparametric profile monitoring. The control chart monitors the l_2 perturbation of the leading eigenvector of a correlation matrix and requires only a sample of known in-control profiles to determine control limits. Through a simulation study we demonstrate that it is able to outperform its competition by achieving an ARL1 close to or equal to 1 even when the control limits result in a large ARL0 on the order of 10^6. Additionally, non-zero false alarm rates with a change point after 10^4 in-control observations were only observed in scenarios that are either pathological or truly difficult for a correlation based monitoring scheme.
17 Paul Fanto
Research Staff Member,Institute for Defense Analyses		 Presentation Bayesian Reliability Methods for Developmental Testing of a Generic Complex System Analysis Tools and Techniques Improving the statistical methods used to predict and assess defense system reliability in Read More developmental test (DT) programs would benefit Department of Defense (DoD) acquisition. Current methods for reliability planning tend to produce high reliability goals that are difficult to achieve in practice. Moreover, in many current applications, traditionally used frequentist methods for reliability assessment generally do not combine information across DT segments and tend to produce large uncertainty intervals of limited use to program decision makers. Much recent work has demonstrated the advantages of using Bayesian statistical methods for planning and assessing system reliability. This work investigates the application of Bayesian reliability assessment methods to notional DT lifetime data. The notional data are generated with the Bayesian reliability growth planning methodology of (Wayne 2018), and the Bayesian assessment methods are based on the methodology of (Wayne and Modarres 2015). The system under test is assumed to be a generic complex system, in which the large number of individual failure modes are not described individually. This work explores the sensitivity of the Bayesian results to the choice of the prior distribution and amount of DT data available. Furthermore, this work compares the Bayesian results for the reliability point estimate, uncertainty interval, and probability of passing a demonstration test to analogous results from traditional reliability assessment methods. Defining point estimates of key reliability metrics, in particular the mean time before system failure (MTBF), is discussed. The effect of relaxing the assumption of a generic complex system and attributing failures to individual failure modes is considered. Finally, the results of this study are compared with recent case studies that apply Bayesian reliability assessment methods. Wayne, Martin. 2018. “Modeling Uncertainty in Reliability Growth Plans.” 2018 Annual Reliability and Maintainability Symposium (RAMS). 1-6. Wayne, Martin, and Mohammad Modarres. 2015. “A Bayesian Model for Complex System Reliability Growth Under Arbitrary Corrective Actions.” IEEE Transactions on Reliability 64: 206-220.
18 Victoria Sieck
Deputy Director / Assistant Professor of Statistics,Scientific Test & Analysis Techniques Center of Excellence (STAT COE) / Air Force Institute of Technology (AFIT)		 Presentation A Framework for Using Priors in a Continuum of Testing Analysis Tools and Techniques A strength of the Bayesian paradigm is that it allows for the explicit use of all available Read More information—to include subject matter expert (SME) opinion and previous (possibly dissimilar) data. While frequentists are constrained to only including data in an analysis (that is to say, only including information that can be observed), Bayesians can easily consider both data and SME opinion, or any other related information that could be constructed. This can be accomplished through the development and use of priors. When prior development is done well, a Bayesian analysis will not only lead to more direct probabilistic statements about system performance, but can result in smaller standard errors around fitted values when compared to a frequentist approach. Furthermore, by quantifying the uncertainty surrounding a model parameter, through the construct of a prior, Bayesians are able to capture the uncertainty across a test space of consideration. This presentation develops a framework for thinking about how different priors can be used throughout the continuum of testing. In addition to types of priors, how priors can change or evolve across the continuum of testing—especially when a system changes (e.g., is modified or adjusted) during phases of testing—will be addressed. Priors that strive to provide no information (reference priors) will be discussed, and will build up to priors that contain available information (informative priors). Informative priors—both those based on institutional knowledge or summaries from databases, as well as those developed based on previous testing data—will be discussed, with a focus on how to consider previous data that is dissimilar in some way, relative to the current test event. What priors might be more common in various phases of testing, types of information that can be used in priors, and how priors evolve as information accumulates will all be discussed.