April 27th Agenda
7:30 AM – 8:30 AM | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
7:30 AM – 8:30 AM Check-in 8:30 AM – 8:40 AM
|
![]() Room A+B ![]() Virtual Session ![]() Virtual Session 8:30 AM – 8:40 AM Opening Remarks The Honorable Nickolas H. Guertin (OSD/DOT&E) Nickolas H. Guertin was sworn in as Director, Operational Test and Evaluation (DOT&E) on December 20, 2021. A Presidential appointee confirmed by the United States Senate, he serves as the senior advisor to the Secretary of Defense on operational and live fire test and evaluation of Department of Defense weapon systems. Mr. Guertin has an extensive four-decade combined military and civilian career in submarine operations; ship construction and maintenance; development and testing of weapons, sensors, combat management products including the improvement of systems engineering; and defense acquisition. Most recently, he has performed applied research for government and academia in software-reliant and cyber-physical systems at Carnegie Mellon University’s Software Engineering Institute. Over his career, he has led organizational transformation, improved competition, and increased application of modular open-system approaches, prototyping, and experimentation. He has also researched and published extensively on software-reliant system design, testing, and acquisition. He received a Bachelor of Science in Mechanical Engineering from the University of Washington and an MBA from Bryant University. He is a retired Navy Reserve Engineering Duty Officer, was Defense Acquisition Workforce Improvement Act (DAWIA) certified in Program Management and Engineering, and is also a licensed Professional Engineer (Mechanical). Mr. Guertin is involved with his community as an Assistant Scoutmaster and Merit Badge Counselor for two local Boy Scouts of America troops, and is an avid amateur musician. He is a native of Connecticut and now resides in Virginia with his wife and twin children 8:40 AM – 9:20 AM
|
![]() Room A+B ![]() Virtual Session ![]() Virtual Session 8:40 AM – 9:20 AM Keynote 1 Col Sacha Tomlinson (United States Space Force) Colonel Sacha Tomlinson is the Test Enterprise Division Chief for Space Training and Readiness Command (STARCOM), Peterson Space Force Base, Colorado. She is also the STARCOM Commander’s Deputy Operational Test Authority. In these two roles, she is responsible for establishing STARCOM’s test enterprise, managing and executing the continuum of test for space systems and capabilities. Colonel Tomlinson’s background is principally in operational test, having served two assignments in the Air Force Test and Evaluation Center first as the Test Director for the Space Based Infrared System’s GEO 1 test campaign, and later as AFOTEC Detachment 4’s Deputy Commander. She also had two assignments with the 17th Test Squadron in Air Combat Command, first as the Detachment 4 Commander, responsible for testing the Eastern and Western Range Launch and Test Range Systems, and later commanding the 17th Test Squadron, which tested major sustainment upgrades for Space systems, as well as conducting Tactics Development & Evaluations and the Space Weapon System Evaluation Program. 9:20 AM – 10:00 AM
|
![]() Room A+B ![]() Virtual Session ![]() Virtual Session 9:20 AM – 10:00 AM Keynote 2 Mr. Peter Coen (NASA) Peter Coen currently serves as the Mission Integration Manager for NASA’s Quesst Mission. His primary responsibility in this role is to ensure that the X-59 aircraft development, in-flight acoustic validation and community test elements of the Mission stay on track toward delivering on NASA’s Critical Commitment to provide quiet supersonic overflight response data to the FAA and the International Civil Aviation Organization. Previously, Peter was the manager for the Commercial Supersonic Technology Project in NASA’s Aeronautics Research Mission, where he led a team from the four NASA Aero Research Centers in the development of tools and technologies for a new generation of quiet and efficient supersonic civil transport aircraft. Peter’s NASA career spans almost 4 decades. During this time, he has studied technology integration in practical designs for many different types of aircraft and has made technical and management contributions to all of NASA’s supersonics related programs over the past 30 years. As Project Manager, he led these efforts for 12 years. Peter is a licensed private pilot who has amassed nearly 30 seconds of supersonic flight time. 10:00 AM – 10:20 AM
|
![]() 10:00 AM – 10:20 AM Break 10:20 AM – 12:20 PM: Parallel Sessions Room A ![]()
|
Session 1A: Cybersecurity in Test & Evaluation Session Chair: Mark Herrera, IDA
Room B ![]()
|
Session 1B: Validation of Statistical Software and Self-Validated Ensemble . . . Session Chair: Tom Donnelly, JMP Statistical Discovery
Room C
|
Session 1C: Statistical and Systems Engineering Applications in Aerospace Session Chair: Melissa Hooke
Room E ![]()
|
Mini-Tutorial 1 Session Chair: George Khoury, IDA
12:20 PM – 1:30 PM
|
![]() 12:20 PM – 1:30 PM Lunch 1:30 PM – 3:00 PM: Parallel Sessions Room A ![]()
|
Session 2A: Statistical Engineering Session Chair: Peter Parker, NASA Langley Research Center
Room B ![]()
|
Session 2B: Situation Awareness, Autonomous Systems, and Digital Engineering in . . . Session Chair: Elizabeth Green, IDA
Room C
|
Students & Fellows Speed Session 1 Please note many of the Speed Sessions will also include a poster during the Poster Session Session Chair: Alyson Wilson, NC State
Room D
|
Contributed Session 1 Session Chair: Elizabeth Gregory, NASA Langley Research Center
Room E ![]()
|
Mini-Tutorial 2 Session Chair: Caleb King, JMP Statistical Discovery
3:00 PM – 3:20 PM
|
![]() 3:00 PM – 3:20 PM Break 3:20 PM – 4:50 PM: Parallel Sessions Room A ![]()
|
Session 3A: Methods & Applications of M&S Validation Session Chair: Benjamin Ashwell, IDA
Room B ![]()
|
Session 3B: Software Quality Session Chair: Sushovan Bhadra, UMASS Dartmouth
Room C
|
Students & Fellows Speed Session 2 Please note many of the Speed Sessions will also include a poster during the Poster Session Session Chair: Jim Starling, United States Military Academy at West Point
Room D
|
Contributed Session 2 Session Chair: Jim Warner, NASA Langley Research Center
Room E ![]()
|
Mini-Tutorial 3 Session Chair: Jason Sheldon, IDA
1 Data Management for Research, Development, Test, and EvaluationIt is important to manage Data from research, development, test, and evaluation effectively. Well-managed data makes research more efficient and promotes better analysis and decision-making. At present, numerous federal organizations are engaged in large-scale reforms to improve the way they manage their data, and these reforms are already effecting the way research is executed. Data management effects every part of the research process. Thoughtful, early planning sets research projects on the path to success by ensuring that the resources and expertise required to effectively manage data throughout the research process are in place when they are needed. This interactive tutorial will discuss the planning and execution of data management for research projects. Participants will build a data management plan, considering data security, organization, metadata, reproducibility, and archiving. By the conclusion of the tutorial, participants will be able to define data management and understand its importance, understand how the data lifecycle relates to the research process, and be able to build a data management plan.
Heather Wojton (IDA)
Heather Wojton is the Director, Research Quality and Chief Data Officer for IDA, a role she assumed in 2021. In this position, Heather provides strategic leadership, project management and direction for the corporation’s data strategy. She is responsible for enhancing IDA’s ability to efficiently and effectively accomplish research and business operations by assessing and evolving data systems, data management infrastructure and data-related practices. She also oversees the quality management processes for research projects, including the research product publication process and the technical review process. Heather joined IDA in 2015 as a member of the research staff. She is an expert in quantitative research methods, including test design and program evaluation. She held numerous research and leadership roles before being named an assistant director of a research division. As a researcher at IDA, Heather led IDA’s test science research program that facilitates data-driven decision-making within the Department of Defense by advancing statistical, behavioral, and data science methodologies and applying them to the evaluation of defense acquisition programs. Heather’s other accomplishments include advancing methods for test design, modeling and simulation validation, data management and curation, and artificial intelligence testing. In this role, she worked closely with academic and Defense Department partners to adapt existing test design and evaluation methods and develop novel methods where gaps persisted. Heather has a doctorate in experimental psychology from the University of Toledo and a bachelor’s degree in research psychology from Marietta College, where she was a member of the McDonough International Leadership Program. She is a graduate of the George Washington University National Security Studies Senior Management Program and the Maxwell School National Security Management Course at Syracuse University.
Matthew Avery (IDA)
Matthew Avery is an OED Assistant Director and part of OED’s Sustainment group. He represents OED on IDA’s Data Governance Council and acts as the Deputy to IDA’s Director of Data Strategy and Chief Data Officer, helping craft data-related strategy and policy. Matthew spearheads a Sustainment group effort to develop an end-to-end model to identify ways to improve mission-capable rates for the V-22 fleet. Prior to joining Sustainment, Matthew was on the Test Science team. As the Test Science Data Management lead, he helped develop analytical methods and tools for operational test and evaluation. He also led OED’s project on operational test and evaluation of Army and Marine Corps unmanned aircraft systems. In 2018-19 Matthew served as an embedded analyst in the Pentagon’s Office of Cost Assessment and Program Evaluation, where among other projects he built state-space models in support of the Space Control Strategic Portfolio Review. Matthew earned his PhD in Statistics from North Carolina State University in 2012, his MS in Statistics from North Carolina State in 2009, and a BA from New College of Florida in 2006. He is a member of the American Statistical Association. 5:00 PM – 7:00 PM: Parallel Sessions Café
|
Poster Session and Reception
|
April 28th Agenda
7:30 AM – 8:30 AM | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
7:30 AM – 8:30 AM Check-in 8:30 AM – 8:40 AM
|
![]() Room A+B ![]() Virtual Session ![]() Virtual Session 8:30 AM – 8:40 AM Opening Remarks Bram Lillard (IDA) V. Bram Lillard assumed the role of Director of the Operational Evaluation Division (OED) in early 2022. In this position, Bram provides strategic leadership, project oversight, and direction for the division’s research program, which primarily supports the Director, Operational Test and Evaluation (DOT&E) within the Office of the Secretary of Defense. He also oversees OED’s contributions to strategic studies, weapon system sustainment analyses, and cybersecurity evaluations for DOD and anti-terrorism technology evaluations for the Department of Homeland Security. Bram joined IDA in 2004 as a member of the research staff. In 2013-14, he was the acting science advisor to DOT&E. He then served as OED’s assistant director in 2014-21, ascending to deputy director in late 2021. Prior to his current position, Bram was embedded in the Pentagon where he led IDA’s analytical support to the Cost Assessment and Program Evaluation office within the Office of the Secretary of Defense. He previously led OED’s Naval Warfare Group in support of DOT&E. In his early years at IDA, Bram was the submarine warfare project lead for DOT&E programs. He is an expert in quantitative data analysis methods, test design, naval warfare systems and operations and sustainment analyses for Defense Department weapon systems. Bram has both a doctorate and a master’s degree in physics from the University of Maryland. He earned his bachelor’s degree in physics and mathematics from State University of New York at Geneseo. Bram is also a graduate of the Harvard Kennedy School’s Senior Executives in National and International Security program, and he was awarded IDA’s prestigious Goodpaster Award for Excellence in Research in 2017. 8:40 AM – 9:20 AM
|
![]() Room A+B ![]() Virtual Session ![]() Virtual Session 8:40 AM – 9:20 AM Keynote 3 The Honorable Christine Fox (Johns Hopkins University Applied Physics Laboratory) The Honorable Christine Fox currently serves as a member of the President’s National Infrastructure Advisory Council, participates on many governance and advisory boards, and is a Senior Fellow at the Johns Hopkins University Applied Physics Laboratory. Previously, she was the Assistant Director for Policy and Analysis at JHU/APL, a position she held from 2014 to early 2022. Before joining APL, she served as Acting Deputy Secretary of Defense from 2013 to 2014 and as Director of Cost Assessment and Program Evaluation (CAPE) from 2009-2013. As Director, CAPE, Ms. Fox served as chief analyst to the Secretary of Defense. She officially retired from the Pentagon in May 2014. Prior to her DoD positions, she served as president of the Center for Naval Analyses from 2005 to 2009, after working there as a research analyst and manager since 1981. Ms. Fox holds a bachelor and master of science degree from George Mason University. 9:30 AM – 10:30 AM Panel Session Room A+B
|
Featured Panel: AI Assurance Session Chair: Chad Bieber
This panel discussion will bring together an international group of AI Assurance Case experts from Academia and Government labs to discuss the challenges and opportunities of applying assurance cases to AI-enabled systems. The panel will discuss how assurance cases apply to AI-enabled systems, pitfalls in developing assurance cases, including human-system integration into the assurance case, and communicating the results of an assurance case to non-technical audiences. This panel discussion will be of interest to anyone who is involved in the development or use of AI systems. It will provide insights into the challenges and opportunities of using assurance cases to provide justified confidence to all stakeholders, from the AI users and operators to executives and acquisition decision-makers.
10:30 AM – 10:50 AM
|
![]() 10:30 AM – 10:50 AM Break 10:50 AM – 12:20 PM: Parallel Sessions Room A ![]()
|
Session 4A: Transforming T&E to Assess Modern DoD Systems Session Chair: Kristen Alexander, DOT&E
Room B ![]()
|
Session 4B: Artificial Intelligence Methods & Current Initiatives Session Chair: Brian Vickers, IDA
Room C
|
Session 4C: Applications of Machine Learning and Uncertainty Quantification Session Chair: Jonathan Rathsam, NASA Langley Research Center
Room D
|
Spotlight on Data Literacy Session Chair: Matthew Avery, IDA
Room E ![]()
|
Mini-Tutorial 4 Session Chair: Gina Sigler, STAT COE
12:20 PM – 1:30 PM
|
![]() 12:20 PM – 1:30 PM Lunch 1:30 PM – 3:00 PM: Parallel Sessions Room A ![]()
|
Session 5A: Methods and Tools at National Labs Session Chair: Karl Pazdernik, Pacific Northwest National Laboratory
Room B ![]()
|
Session 5B: Applications of Bayesian Statisics Session Chair: Victoria Sieck, STAT COE
Room C
|
Session 5C: Methods and Tools for T&E Session Chair: Sam McGregor, AFOTEC
Room E ![]()
|
Mini-Tutorial 5 Session Chair: John Haman, IDA
3:00 PM – 3:20 PM
|
![]() 3:00 PM – 3:20 PM Break 3:20 PM – 4:20 PM: Parallel Sessions Room A ![]()
|
Session 6A: Tools for Decision-Making and Data Management Session Chair: Tyler Lesthaeghe, University of Dayton Research Institute
Room B ![]()
|
Session 6B: Methods for DoD System Supply Chain and Performance Estimation Session Chair: Margaret Zientek, IDA
Room C
|
Session 6C: Statistical Methods for Ranking and Functional Data Types Session Chair: Elise Roberts, JHU/APL
Room E ![]()
|
Mini-Tutorial 6 Session Chair: Elizabeth Gregory, NASA Langley Research Center
4:20 PM – 4:40 PM
|
![]() Room A+B ![]() Virtual Session ![]() Virtual Session 4:20 PM – 4:40 PM Awards Wilks Award Winner to Be Announced, Student Winners to be Announced, Winner To Be Announced () Wilks Award Winner to Be Announced, Student Winners to be Announced, Winner To Be Announced () Wilks Award Winner to Be Announced, Student Winners to be Announced, Winner To Be Announced () 4:40 PM – 4:50 PM
|
![]() Room A+B ![]() Virtual Session ![]() Virtual Session 4:40 PM – 4:50 PM Closing Remarks General Norty Schwartz (U.S. Air Force, retired / Institute for Defense Analyses) Norton A. Schwartz serves as President of the Institute for Defense Analyses (IDA), a nonprofit corporation operating in the public interest. IDA manages three Federally Funded Research and Development Centers that answer the most challenging U.S. security and science policy questions with objective analysis leveraging extraordinary scientific, technical, and analytic expertise. At IDA, General Schwartz (U.S. Air Force, retired) directs the activities of more than 1,000 scientists and technologists employed by IDA. General Schwartz has a long and prestigious career of service and leadership that spans over 5 decades. He was most recently President and CEO of Business Executives for National Security (BENS). During his 6-year tenure at BENS, he was also a member of IDA’s Board of Trustees. Prior to retiring from the U.S. Air Force, General Schwartz served as the 19th Chief of Staff of the U.S. Air Force from 2008 to 2012. He previously held senior joint positions as Director of the Joint Staff and as the Commander of the U.S. Transportation Command. He began his service as a pilot with the airlift evacuation out of Vietnam in 1975. General Schwartz is a U.S. Air Force Academy graduate and holds a master’s degree in business administration from Central Michigan University. He is also an alumnus of the Armed Forces Staff College and the National War College. He is a member of the Council on Foreign Relations and a 1994 Fellow of Massachusetts Institute of Technology’s Seminar XXI. General Schwartz has been married to Suzie since 1981. |