DATAWorks Agenda

April 26th Agenda

7:30 AM – 8:45 AM

Registration / Check-in

9:00 AM – 12:00 PM: Parallel Sessions

Room A



Virtual Session
Short Course 1, Part 1
  • Using R Markdown & the Tidyverse to Create Reproducible Research

    Dr.Justin Post (NCSU)

  • Materials URL:

Room B



Virtual Session
Short Course 2, Part 1
  • Operational Cyber Resilience in Engineering and Systems Test

    Dr.Peter Beling (Virginia Tech)

  • Materials:

    Materials URL:

Room C

Short Course 3, Part 1
  • Categorical Data Analysis

    Dr.Chris Franck (Virginia Tech)

  • Materials URL:

Room D



Virtual Session
Short Course 4, Part 1
  • A Practitioner’s Guide to Advanced Topics in DOE

    Dr.Drew Landman (Old Dominion University)

  • Materials URL:

12:00 PM – 1:00 PM

Lunch

1:00 PM – 4:00 PM: Parallel Sessions

Room A



Virtual Session
Short Course 1, Part 2
  • Using R Markdown & the Tidyverse to Create Reproducible Research

    Dr.Justin Post (NCSU)

  • Materials URL:

Room B



Virtual Session
Short Course 2, Part 2
  • Operational Cyber Resilience in Engineering and Systems Test

    Dr.Peter Beling (Virginia Tech)

  • Materials:

    Materials URL:

Room C

Short Course 3, Part 2
  • Categorical Data Analysis

    Dr.Chris Franck (Virginia Tech)

  • Materials URL:

Room D



Virtual Session
Short Course 4, Part 2
  • A Practitioner’s Guide to Advanced Topics in DOE

    Dr.Drew Landman (Old Dominion University)

  • Materials URL:

April 27th Agenda

7:30 AM – 8:30 AM

7:30 AM – 8:30 AM

Registration / Check-in

8:30 AM – 8:40 AM

Room A+B

Virtual Session

8:30 AM – 8:40 AM

Opening Remarks

Norton Schwartz

(IDA)


8:40 AM – 9:20 AM

Room A+B

Virtual Session

8:40 AM – 9:20 AM

Keynote 1

Wendy Masiello

(President, Wendy Mas Consulting LLC)


Materials:


9:20 AM – 10:00 AM

Room A+B

Virtual Session

9:20 AM – 10:00 AM

Keynote 2

Jill Marlowe

(Digital Transformation Officer, NASA)


Materials:


10:00 AM – 10:20 AM

10:00 AM – 10:20 AM

Break

10:20 AM – 12:20 PM: Parallel Sessions

Room A



Virtual Session
Session 1A: T&E for AI, ML, and HMT
Session Chair: Chad Bieber, JHU/APL
  • Test and Evaluation Framework for AI Enabled Systems

    Brian Woolley (Joint Artificial Intelligence Center)

  • Materials:

  • Test & Evaluation of ML Models

    Anna Rubinstein (MORSE Corporation)

  • Materials:

  • A Systems Perspective on Bringing Reliability and Prognostics to Machine Learning

    Tyler Cody (Virginia Tech National Security Institute)

  • Materials:

  • Topological Modeling of Human-Machine Teams

    Jay Wilkins and Caitlan Fealing (IDA)

  • Materials:

Room B



Virtual Session
Session 1B: Modeling & Simulation
Session Chair: Elizabeth Gregory, NASA
  • What statisticians should do to improve M&S validation studies

    John Haman (IDA)

  • Materials:

  • A Decision-Theoretic Framework for Adaptive Simulation Experiments

    Terril Hurst (Raytheon Technologies)

  • Materials:

  • Enabling Enhanced Validation of NDE Computational Models and Simulations

    William C. Schneck, III (NASA LaRC)

  • Materials:

  • USE OF DESIGN & ANALYSIS OF COMPUTER EXPERIMENTS (DACE) IN SPACE MISSION TRAJECTORY DESIGN

    David Shteinman (Industrial Sciences Group)

  • Materials:

Room C

Session 1C: Statistical Methods
Session Chair: Tom Donnelly, JMP
  • Safe Machine Learning Prediction and Optimization via Extrapolation Control

    Tom Donnelly and Laura Lancaster (JMP Statistical Discovery LLC)

  • Materials:

  • Applications of Equivalence Testing in T&E

    Sarah Burke (The Perduco Group)

  • Materials:

  • Computing Statistical Tolerance Regions Using the R Package ‘tolerance’

    Derek Young (University of Kentucky)

  • Materials:

  • Kernel Regression, Bernoulli Trial Responses, and Designed Experiments

    John Lipp (Lockheed Martin, Systems Engineering)

  • Materials:

Room D



Virtual Session
Mini-Tutorial 1: Data Integrity For Deep Learning Models
Session Chair: Doug Ray, US Army CCDC Armaments Center
  • Data Integrity For Deep Learning Models

    Victoria Gerardi and John Cilli (US Army, CCDC Armaments Center)

  • Materials URL:

12:20 PM – 1:30 PM

12:20 PM – 1:30 PM

Lunch

1:30 PM – 3:00 PM: Parallel Sessions

Room A



Virtual Session
Session 2A: Reproducible Research
Session Chair: Eric Chicken, Florida State University
  • Everyday Reproducibility

    Gregory J. Hunt (William & Mary)

  • Materials:

  • From Gripe to Flight: Building an End-to-End Picture of DOD Sustainment

    Benjamin Ashwell (IDA)

  • Materials:

  • Using the R ecosystem to produce a reproducible data analysis pipeline

    Andrew Farina (United States Military Academy)

  • Materials:

    Materials URL:

Room B



Virtual Session
Session 2B: Trust in AI/ML
Session Chair: Elizabeth Gregory, NASA
  • Trust Throughout the Artificial Intelligence Lifecycle

    Lauren H. Perry (The Aerospace Corporation)

  • Materials:

  • Let’s stop talking about “transparency” with regard to AI

    David Sparrow and David Tate (IDA)

  • Materials:

  • Machine Learning for Uncertainty Quantification: Trusting the Black Box

    James Warner (NASA Langley Research Center)

  • Materials:

Room C

Speed Session 1
Session Chair: Denise Edwards, IDA
  • Analysis Apps for the Operational Tester

    William Raymond Whitledge (IDA)

  • Materials:

  • Quantifying the Impact of Staged Rollout Policies on Software Process and Product Metrics

    Lance Fiondella (University of Massachusetts Dartmouth)

  • Materials:

  • Estimating the time of sudden shift in the location or scale of ergodic-stationary process

    Zhi Wang (Bayer Crop Science)

  • Materials:

  • Analysis of Target Location Error using Stochastic Differential Equations

    James Brownlow (USAF)

  • Materials:

  • Data Science & ML-Enabled Terminal Effects Optimization

    John Cilli (Picatinny Arsenal)

  • Materials:

  • Predicting Trust in Automated Systems: Validation of the Trust of Automated Systems Test

    Caitlan Fealing (IDA)

  • Materials:

  • Exploring the behavior of Bayesian adaptive design of experiments

    Daniel Ries (Sandia National Laboratories)

  • Materials:

  • T&E of Responsible AI

    Rachel Haga (IDA)

  • Materials:

  • Next Gen Breaching Technology: A Case Study in Deterministic Binary Response Emulation

    Eli Golden (US Army DEVCOM Armaments Center)

  • Materials:

Room D



Virtual Session
Mini Tutorial 2: Mixed Models
Session Chair: Keyla Pagan-Rivera, IDA
  • Mixed Models: A Critical Tool for Dependent Observations

    Elizabeth Claassen (JMP Statistical Discovery)

  • Materials URL:

3:00 PM – 3:20 PM

3:00 PM – 3:20 PM

Break

3:20 PM – 4:50 PM: Parallel Sessions

Room A



Virtual Session
Session 3A: Cybersecurity Methods and Concepts
Session Chair: Pete Mancini, IDA
  • Applying Design of Experiments to Cyber Testing

    J. Michael Gilmore (IDA)

  • Materials:

  • Sparse Models for Detecting Malicious Behavior in OpTC

    Andrew Mastin (Lawrence Livermore National Laboratory)

  • Materials:

  • Adversaries and Airwaves – Compromising Wireless and Radio Frequency Communications

    Mark Herrera and Jason Schlup (IDA)

  • Materials:

Room B



Virtual Session
Session 3B: Sustainment in the DoD
Session Chair: Benjamin Ashwell, IDA
  • An Introduction to Sustainment: The Importance and Challenges of Analyzing System Readines

    V. Bram Lillard and Megan L. Gelsinger (IDA)

  • Materials:

  • M&S approach for quantifying readiness impact of sustainment investment scenarios

    Andrew C. Flack, Han G. Yi (IDA (OED))

  • Materials:

  • Taming the beast: making questions about the supply system tractable by quantifying risk

    Joseph Fabritius and Kyle Remley (IDA)

  • Materials:

Room C

Speed Session 2
Session Chair: Alyson Wilson, NCSU
  • Assurance Techniques for Learning Enabled Autonomous Systems which Aid Systems Engineering

    Christian Ellis (Army Research Laboratory / University of Mass. Dartmouth)

  • Materials:

  • Optimal Designs for Multiple Response Distributions

    Brittany Fischer (Arizona State University)

  • Materials:

  • Machine Learning for Efficient Fuzzing

    John Richie (USAFA)

  • Nonparametric multivariate profile monitoring using regression trees

    Daniel A. Timme (Florida State University)

  • Materials:

  • Profile Monitoring via Eigenvector Perturbation

    Takayuki Iguchi (Florida State University)

  • Materials:

  • Risk Comparison and Planning for Bayesian Assurance Tests

    Hyoshin Kim (North Carolina State University)

  • Materials:

  • Utilizing Machine Learning Models to Predict Success in Special Operations Assessment

    Anna Vinnedge (United States Military Academy)

  • Materials:

  • Convolutional Neural Networks and Semantic Segmentation for Cloud and Ice Detection

    Prarabdha Ojwaswee Yonzon (United States Military Academy (West Point))

  • Materials:

  • Bayesian Estimation for Covariate Defect Detection Model Based on Discrete Cox Proportiona

    Priscila Silva (University of Massachusetts Dartmouth)

  • Materials:

  • Combining data from scanners to inform cadet physical performance

    Nicholas Ashby (United States Military Academy)

  • Materials:

Room D



Virtual Session
Mini-Tutorial 3: Introducing git for reproducible research
Session Chair: Matthew Avery, IDA
  • Introducing git for reproducible research

    Curtis Miller (IDA)

  • Materials:

5:00 PM – 7:00 PM

Café

5:00 PM – 7:00 PM

Reception and Poster Session

April 28th Agenda

7:30 AM – 8:30 AM

7:30 AM – 8:30 AM

Registration / Check-in

8:30 AM – 8:40 AM

Room A+B

Virtual Session

8:30 AM – 8:40 AM

Opening Remarks

Bram Lillard

(IDA)


8:40 AM – 9:20 AM

Room A+B

Virtual Session

8:40 AM – 9:20 AM

Keynote 3

Nickolas Guertin

(Director, Operational Test & Evaluation, OSD/DOT&E)


9:30 AM – 10:30 AM Panel Session

Room A+B



Virtual Session

Featured Panel: Evolving Data-Centric Organizations

  • Panelist 1

    Heather Wojton (IDA)

  • Panelist 2

    Laura Freeman (Virginia Tech)

  • Panelist 3

    Jane Pinelis (Joint Artificial Intelligence Center)

  • Panelist 4

    Calvin Robinson (NASA)

  • Moderator

    Matthew Avery (IDA)

10:30 AM – 10:50 AM

10:30 AM – 10:50 AM

Break

10:50 AM – 12:20 PM: Parallel Sessions

Room A



Virtual Session
Session 4A: Statistical Engineering
Session Chair: Pete Parker, NASA
  • Building Bridges: a Case Study of Assisting a Program from the Outside

    Anthony Sgambellone (Huntington Ingalls Industries)

  • Materials:

  • STAT and UQ Implementation Lessons Learned

    Kelsey Cannon (Lockheed Martin)

  • Materials:

  • An Overview of NASA’s Low Boom Flight Demonstration

    Jonathan Rathsam (NASA Langley Research Center)

  • Materials:

Room B



Virtual Session
Session 4B: Big Data and Cloud-Based Tools for T&E
Session Chair: Breeana Anderson, IDA
  • TRMC Big Data Analytics Investments & Technology Review

    Edward Powell (Test Resource Management Center)

  • Cloud Computing for Computational Fluid Dynamics (CFD) in T&E

    Neil Ashton (Amazon Web Services)

  • Materials:

  • Leveraging Data Science and Cloud Tools to Enable Continuous Reporting

    Timothy Dawson (AFOTEC Detachment 5)

  • Materials:

Room C

Session 4C: Machine Learning and Dynamic Programming Topics
Session Chair: Jay Dennis, IDA
  • Legal, Moral, and Ethical Implications of Machine Learning

    Alan B. Gelder (IDA)

  • Materials:

  • Forecasting with Machine Learning

    Akshay Jain (IDA)

  • Materials:

  • Structural Dynamic Programming Methods for DOD Research

    Mikhail Smirnov (IDA)

  • Materials:

Room D



Virtual Session
Mini-Tutorial 4: Survey Dos and Don’ts
Session Chair: Brian Vickers, IDA
  • Survey Dos and Don’ts

    Gina Sigler & Alex (Mary) McBride (Scientific Test and Analysis Techniques Center of Excellence (STAT COE))

  • Materials:

12:20 PM – 1:30 PM

12:20 PM – 1:30 PM

Lunch

1:30 PM – 3:00 PM: Parallel Sessions

Room A



Virtual Session
Session 5A: Analysis Tools for T&E
Session Chair: Denise Edwards, IDA
  • Orbital Debris Effects Prediction Tool for Satellite Constellations

    Joel Williamsen (IDA)

  • Using Sensor Stream Data as Both an Input and Output in a Functional Data Analysis

    Thomas A Donnelly (JMP Statistical Discovery LLC)

  • Materials:

  • Measuring training efficacy: Structural validation of the Operational Assessment of Training

    Brian Vickers (IDA)

  • Materials:

Room B



Virtual Session
Session 5B: DOE Applications
Session Chair: Dominik Alder, Lockheed Martin
  • Experiment Design and Visualization Techniques for an X-59 Low-boom Variability Study

    William J Doebler (NASA Langley Research Center)

  • Materials:

  • Case Study on Applying Sequential Methods in Operational Testing

    Keyla Pagán-Rivera (IDA)

  • Materials:

  • A New Method for Planning Full-Up System-Level (FUSL) Live Fire Tests

    Lindsey Butler (IDA)

  • Materials:

Room D



Virtual Session
Mini-Tutorial 5: Data Visualization
Session Chair: Jay Wilkins, IDA
  • An Introduction to Data Visualization

    Christina Heinich (NASA)

  • Materials:

3:00 PM – 3:20 PM

3:00 PM – 3:20 PM

Break

3:20 PM – 4:20 PM: Parallel Sessions

Room A



Virtual Session
Session 6A: Applications of Deep Learning and Monte Carlo Analysis
Session Chair: Tom Donnelly, JMP
  • Deep learning aided inspection of additively manufactured metals

    Brendan Croom (JHU Applied Physics Laboratory)

  • Materials:

  • Applications for Monte Carlo Analysis within Job Shop Planning

    Dominik Alder (Lockheed Martin, Program Management)

  • Materials:

Room B



Virtual Session
Session 6B: Bayesian Statistics
Session Chair: John Haman, IDA
  • A Framework for Using Priors in a Continuum of Testing

    Victoria Sieck (Scientific Test & Analysis Techniques Center of Excellence (STAT COE) / Air Force Institute of Technology (AFIT))

  • Materials:

  • Method for Evaluating Bayesian Reliability Models for Developmental Testing

    Paul Fanto and David Spalding (IDA)

  • Materials:

Room D



Virtual Session
Session 6D: Disease and Mental Health
Session Chair: Joe Warfield, JHU/APL
  • Stochastic Modeling and Characterization of a Wearable-Sensor-Based Surveillance Network

    Jane E. Valentine (Johns Hopkins University Applied Physics Laboratory)

  • Materials:

  • The Mental Health Impact of Local COVID-19 Cases Prior to the Mass Availability of Vaccine

    Zachary Szlendak (IDA)

  • Materials:

4:20 PM – 4:40 PM

Room A+B

Virtual Session

4:20 PM – 4:40 PM

Awards

4:40 PM – 4:50 PM

Room A+B

Virtual Session

4:40 PM – 4:50 PM

Closing Remarks

Alyson Wilson

(NCSU)