Method and system for anthropomorphic interaction and automation of computer systems
Inventors
Gauf, Bernard • Bindas, Scott • Stubbs, William R. • MANN, Joshua
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-10572821-B1
Publication Date
2020-02-25
Expiration Date
Abstract
According to an embodiment of the present invention, a computer implemented system that automates development, maintenance and execution of procedures to autonomously interact with one or more external devices, comprises: an input configured to receive interaction data and to detect state data from an external computer system, the user interaction data comprising GUI data; a memory component configured to store the interaction data, the state data and relationship data between objects, events and resultant states where an event represents an interaction with the external computer system and where a resultant state represents a state resulting from an interaction; a semantic processor configured to interpret the interaction data into semantic objects and develop a system model, using a learning algorithm, based on the semantic objects, the state data and the relationship data; and an execution processor configured to execute tasks and roles accounting for environmental perturbations and system randomness.
Core Innovation
The invention automates development, maintenance and execution of procedures to autonomously interact with one or more external devices. Interaction data and state data are received from an external computer system, where user interaction data comprises GUI data and the state data represents current properties of the external computer system. An input detects the state data and the system stores the interaction data, the state data, and relationship data between objects, events and resultant states.
A semantic processor interprets the interaction data into semantic objects and develops a system model using a learning algorithm. The system model comprises a state graph, an event graph and an object model, where the state graph represents a detected status of the external computer system and the event graph represents relationships between semantic objects, events, and resulting external computer system state. A planning engine uses the state graph and the event graph to identify a current state as an existing state or a new state not previously encountered and to calculate one or more paths between the current state and a desired state.
An execution processor executes a plurality of tasks and a plurality of roles in which tasks and roles are defined in relation to semantic objects and external system state. Each task defines a set of interactions with one or more semantic objects, includes a set of starting states defining an event, and includes a set of ending states defining a goal. Each role represents a set of instructions specific to a scope of operation for a specific type of user and is associated with a corresponding set of unique tasks, and the execution processor derives a combination of workflow, rules, events and states required by a given task or role while employing a respective engine for each task or role.
Claims Coverage
The document contains two independent claims: one directed to a computer implemented system and one directed to a computer implemented method. Each independent claim shares a common core of receiving GUI-based interaction and state data, interpreting the data into semantic objects with a learning algorithm to build a system model, using a planning engine to compute paths from a current state to a desired state, and executing role- and task-defined interactions using workflow, rules, events and states with respective engines.
Autonomous procedure execution from GUI interaction and detected state data
A computer implemented system that automates development, maintenance and execution of procedures to autonomously interact with one or more external devices by receiving interaction data and detecting state data from an external computer system, where the user interaction data comprises GUI data and the state data represents current properties of the external computer system; and storing interaction data, state data and relationship data between objects, events and resultant states.
Learning-based semantic system model with state graph and event graph path planning
A semantic processor interprets the interaction data into semantic objects and develops a system model using a learning algorithm based on semantic objects, the state data and the relationship data, where the system model comprises a state graph, an event graph and an object model; and a planning engine uses the state graph and the event graph to identify a current state as an existing state or a new state not previously encountered and to calculate one or more paths between the current state and a desired state.
Role- and task-defined execution using workflow, rules, events and states with respective engines
An execution processor executes a plurality of tasks and a plurality of roles, where each task defines a set of interactions with one or more semantic objects with starting states defining an event and ending states defining a goal, and each role represents instructions for a scope of operation for a specific type of user and is associated with unique tasks; and the execution processor derives responsive to the system model a combination of workflow, rules, events and states required by a given task or role and employs a respective engine for each task or role.
Method for autonomous procedure execution using semantic system model and role/task planning and execution
A computer implemented method that automates development, maintenance and execution of procedures to autonomously interact with one or more external devices using an interactive interface by receiving interaction data and state data from an external computer system, where the user interaction data comprises GUI data and the state data represents current properties; storing interaction data, state data and relationship data between objects, events and resultant states; interpreting interaction data into semantic objects and developing a system model using a learning algorithm that comprises a state graph, an event graph and an object model; and using a planning engine to identify a current state and calculate one or more paths to a desired state; then executing a plurality of tasks and a plurality of roles by deriving responsive to the system model a combination of workflow, rules, events and states required by a given task or role and employing a respective engine for each task or role.
Across both independent claims, the inventive contribution is the combination of GUI interaction and detected state data acquisition and storage of relationships, learning-algorithm-based interpretation into semantic objects and a system model with state graph and event graph for path calculation, and autonomous execution of tasks and roles where the execution processor derives workflow, rules, events and states and uses respective engines to realize the planned interactions.
Stated Advantages
Autonomously interact with one or more external devices by automating development, maintenance and execution of procedures.
Identify a current state as an existing state or a new state not previously encountered and calculate one or more paths between the current state and a desired state.
Execute tasks and roles by deriving a combination of workflow, rules, events and states required by a given task or role while employing a respective engine for each task or role.
Documented Applications
No documented applications found
Interested in licensing this patent?