Mission modeling planning, and execution module (M2PEM) systems and methods
Inventors
Scrapper, Chris • Droge, Greg N • Xydes, Alexander L • de la Croix, Jean-Pierre • Rahmani, Amir • Vander Hook, Joshua • Lim, Grace
Assignees
California Institute of Technology • US Department of Navy
Publication Number
US-11599109-B2
Publication Date
2023-03-07
Expiration Date
2039-05-06
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Abstract
Methods and systems for accomplishing a mission using a plurality of unmanned vehicles can include graphically describing the mission tasks at a graphical user interface (GUI) using Business Process Model Notation (BPMN), and translating the graphical description into extensible machine language (XML) formatted robot operating system (ROS) instructions, which can be understood by each of the plurality of unmanned vehicles with a translator. An execution engine transmits the XML ROS instructions to a respective local controller on the respective unmanned vehicle. The BPMN graphical descriptor symbols can allow for planning of a mission by an end user that does not have expertise in the ROS domain, and that does not have an understanding of the ROS construct. The execution engine can provide feedback back to the GUI regarding mission execution. Based on the feedback, the graphical description can be modified while the mission is being accomplished.
Core Innovation
The invention pertains to methods and systems for accomplishing a mission using a plurality of unmanned vehicles by graphically describing mission tasks at a graphical user interface (GUI) using Business Process Model Notation (BPMN), and translating this graphical description into formatted robot operating system (ROS) instructions in an extensible machine language (XML) format. These instructions can be understood by the unmanned vehicles and executed to carry out the mission. The BPMN graphical descriptor symbols allow an end user without expertise in ROS to effectively plan missions, and the execution engine provides feedback to the GUI to allow modification of the mission during execution.
The problem being solved is the complexity and inaccessibility of coordinating and controlling multiple unmanned vehicles, especially heterogeneous ones, for mission execution. Traditional mission design methods require programming knowledge in the vehicle's operating systems or limit the ability to specify complex sequences, parallel operations, and decision logic. Existing solutions either lack the ability for an expert user without software programming experience to design and modify missions or cannot effectively coordinate teams of unmanned vehicles across multiple domains. The invention addresses the disconnect between mission planning and execution by providing a user-friendly, graphical, reconfigurable mission planning framework that integrates with ROS and supports real-time interaction.
Claims Coverage
The patent includes two independent claims covering methods and systems for mission accomplishment using unmanned vehicles with graphical mission design, ROS instruction translation, execution, and control via a Multi-Operator Control Unit (MOCU). The main inventive features highlight user interaction, translation, execution, and dynamic mission management.
Graphical mission description using BPMN enabling direct user interaction
The method employs a graphical user interface that uses Business Process Model Notation (BPMN) systems allowing end users to make decisions through pop-up alert insets during mission execution, facilitating direct interaction with a plurality of unmanned vehicles.
Translation of BPMN graphical descriptions into formatted ROS instructions
Graphical mission descriptions are translated into robot operating system (ROS) instructions formatted as XML mission files that can be input into unmanned vehicles, bridging user-friendly planning with machine-executable instructions.
Execution engine transmitting ROS instructions and managing mission
An execution engine transmits the formatted ROS instructions to the unmanned vehicles for mission execution. The engine manages tokens representing each activity’s progress, including creation and consumption or cancellation of tokens corresponding to activity lifecycle.
Mission control and modification through Multi-Operator Control Unit (MOCU)
At any time during execution, the MOCU can stop the mission, allow addition or modification of BPMN elements, and start a new mission, enabling dynamic mission updates and control without requiring programming expertise.
Verification of graphical mission descriptions using formal methods
The system includes verification of the graphical mission description using a Simple Process Meta Language Interpreter (SPIN) by comparing linear temporal logic (LTL) mission specifications to Process Meta Language translations of the ROS instructions, ensuring mission correctness and validity.
Use of static resource allocation and XML scripts defining tactical behaviors
XML scripts specify whether a tactical behavior is a vehicle or payload task, defining which behaviors to call, resource usage, and parameters, facilitating structured and precise behavior invocation during execution.
Graphical user interface employs color gradients to indicate activity status
The GUI incorporates color-based gradients to visually represent the status of mission activities, aiding users in monitoring mission progress and active tasks.
Together, these inventive features provide a comprehensive framework enabling graphical mission planning, formal verification, dynamic execution control, and intuitive user interaction for coordinating multiple unmanned vehicles using standardized ROS instructions.
Stated Advantages
Allows mission planning and control by users without expertise in ROS or programming.
Provides a graphical and user-friendly method to define, modify, and monitor complex multi-vehicle missions.
Supports parallel sequencing, hierarchical decomposition, decision logic, and dynamic mission updates during execution.
Integrates formal verification tools to ensure mission correctness before execution.
Enables coordination of heterogeneous unmanned vehicles across multiple domains with standardized communication.
Documented Applications
Coordinated control of teams of heterogeneous unmanned vehicles including unmanned aerial, surface, and underwater vehicles.
Military unmanned system mission planning and execution, such as route traversal, area search, formation control, and explosive hazard detection and inspection.
Operation of unmanned vehicle teams during complex missions with real-time user decision input using Multi-Operator Control Units.
Verification and validation of mission logic for unmanned vehicle missions prior to and during execution.
Interested in licensing this patent?