Automated testing methods for condition analysis and exploration

Inventors

Gauf, Bernard • TOPALIS, Apostolos • Harris, Jacob

Assignees

Innovative Defense Technologies Ltd

Abstract

An embodiment of the present invention is directed to a novel approach of applying Machine Learning, statistical methods and/or other algorithms to identify associations of input conditions and values with results of requirements, measures of performance assessments, and/or other indications. These associations may be provided to an analyst, system designer, other recipient and/or receiving system or component to inform of input conditions and values that uncover system sensitivities.

Core Innovation

The invention provides a condition analysis and exploration workflow that defines a set of input conditions and value ranges to drive execution of one or more automated tests. It initiates execution of a series of test runs using the set of input conditions and the value ranges on an automated testing suite of tools, where the automated testing suite of tools provides automated test and retesting in a virtual test environment. The execution records test results that are used for subsequent generation of assessments.

The workflow generates assessments relating to one or more requirements and measures of performance based on the test results. It records, via the memory component, the assessments and the set of input conditions, including the executed condition values. It then executes an iterative optimization algorithm that analyzes an objective function to derive a next set of condition values.

The iterative optimization algorithm performs iterations driven by a sampling scheme or a shape of the objective function. Responsive to the execution of the iterative optimization algorithm, the system automatically identifies one or more relationships between the set of input conditions and the test results. The relationships are characterized as associations/relationships derived from the automated test and retesting results, including relationship modeling using logistic regression, mutual information, and association rules, with support for different relationship complexities such as a single-condition threshold and multi-condition/disjoint ranges.

The document further states that the workflow incorporates ATRT and its components as the execution and analysis environment, including a Test Manager, Analysis Manager, Test Strategy Manager, and Persistent Configuration Manager. It also describes a graphical user interface that can display condition coefficient importance and real-time test results, and it supports recording and using assessed data for iterative selection of next condition values within the condition analysis and exploration workflow.

Claims Coverage

The document contains two independent claims (a system and a method) that define the overall condition analysis and exploration workflow. Together, they recite a sequence of inventive features: defining input conditions and ranges, running automated test and retesting in a virtual test environment, generating requirement/performance assessments, recording those assessments with the input conditions, using an iterative optimization algorithm guided by a sampling scheme or objective-function shape to derive next condition values, and automatically identifying relationships between input conditions and test results.

Across the independent system and method claims, the core claim coverage is directed to combining virtual-environment automated test and retesting driven by defined input condition value ranges, generation and recording of requirement and performance assessments tied to executed condition values, iterative optimization of the objective function guided by a sampling scheme or objective-function shape, and automated identification of relationships between input conditions and test results.

Stated Advantages

Documented Applications

No documented applications found