There is a fundamental misalignment between current approaches to designing and executing verification and validation (V&V) strategies and the nature of intelligent systems. Current V&V approaches rely on the assumption that system behavior is preserved during a system’s lifetime. However, intelligent systems are developed so that they evolve their own behavior during their lifetime; such is the purpose of AI. This misalignment makes existing approaches to designing and executing V&V strategies ineffective for systems that embed AI. For example, it will be no longer sufficient to complete developmental V&V in the laboratory and assume that the behavior will be replicated in an operational environment. In this presentation, I provide a systems-theoretic explanation for (1) why AI learning capabilities create a unique and unprecedented family of systems, and (2) why current V&V methods and processes may not be fit-for-purpose in the context of systems with high autonomy. Making a paradigm shift in the practice of V&V is necessary, I conclude by delineating a set of theoretical advances and process transformations that could support such a shift.
The history of artificial intelligence and machine learning is replete with ebbs and flows of enthusiasm as technology has advanced, making new methods possible or revitalizing previously discarded methods. While “black-box” machine-learning techniques are currently commercially successful, a more recent development, called causal inference, has emerged, due to efforts by computer scientists such as Judea Pearl and Adnan Darwiche. The difference between the two approaches lies in the open nature of the latter, enabling/requiring analysts to pose probabilistic models based upon their evolving understanding of cause-effect relationships within an engineered system. In Pearl’s words: “You are smarter than your data.”
This presentation describes the basics and benefits of causal inference, which is founded upon a combination of propositional logic and Bayesian analysis. Causal diagrams and Bayesian networks are illustrated, which, when developed collaboratively using simple protocols, enable quantifying uncertainty for making decisions. Causal inference is based on a combination of prior knowledge/understanding and the analysis of data obtained either from observational studies (for example, reliability) or designed simulation and real-world experiments. Successes and challenges are highlighted for adopting causal inference, including the need to update skills to reason using the associated mathematics and protocols.
As AI and machine learning-based subsystems and components become more prevalent in complex systems, the importance of training data is driving a new specialty within the systems engineering discipline of requirements engineering. Traditionally, system requirements included functional and non-functional requirements, including the “-ilities” such as reliability, dependability, durability, sustainability, and others. For systems that include machine learning capabilities we propose additional discipline of data requirements.
Successful data requirements engineering is necessary to prevent headline-making AI failures, which appear mysterious and frightening to both engineers and the public, but which can often be traced to simple problems in the engineering of the data used to train the system. The best algorithm cannot overcome poor training data.
The systems engineer must be concerned with a new kind of system requirements--data requirements. Data requirements specify how much, and what kind of data must be made available to the AI subsystem for training and testing. Systems engineers must learn enough about the environment in which the system will operate, and about the situations it could encounter to fully specify the data required to successfully train it. More than the machine learning developer or even the data scientist, it is the systems engineer who will be aware of the system of which the AI subsystem is a part, and the larger context of the entire system in the environment where it will operate. In this session, we will take a closer look at the role of data in AI systems and how systems engineers will need to learn to deal with it effectively. We will use the framework of effective requirements engineering to show how data requirements can be elicited, analyzed, validated, and implemented in the data design.
This presentation provides a research area for exploration when it comes to increasing the operationalization success of AI-based systems. This research area, though at its inception, opens up a new case of systems engineering for AI (SE4AI) where one shall consider bringing a systems engineering discipline, Enterprise Architecture (EA), into AI and machine learning (ML) system-based designs. Many AI-based projects fail to be operationalized to the field or even if they do, many do not meet the intended objectives of the original design. We get to explore the tip of the iceberg in this session as to how we can potentially increase this operationalization success of such projects or system designs as systems engineers. This is a case of exploring what happens when as big picture thinkers, we want to design systems that must be deployed in the era of ML operations (MLOps) and how we could increase the success of such endeavors