Robust useful and general task-oriented virtual assistants
Inventors
Srivastava, Biplav • LAKKARAJU, KAUSIK • VENKATARAMANAN, REVATHY • PALLAGANI, VISHAL • KHANDELWAL, VEDANT • YIP, HONG YUNG
Assignees
Publication Number
US-12067983-B2
Publication Date
2024-08-20
Expiration Date
2042-04-06
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Abstract
The disclosure deals with a system and method for improved general task-oriented virtual assistants (VAs). The presently disclosed framework incorporates discovery of knowledge from online sources to accomplish tasks (open world), user-specific knowledge for personalization, and domain-specific knowledge for context adaptation to recommend and assist the users over procedural tasks such as cooking and Do-it-Yourself (DIY) tasks. The approach also focuses on content curation for fault-tolerant execution to ensure the end goal is reached despite common failures.
Core Innovation
The invention provides a robust, general task-oriented virtual assistant (VA) framework that dynamically discovers and curates instructions from online sources, adapts to user-specific knowledge for personalization, and integrates domain-specific knowledge for context adaptation. This system is designed to assist users in procedural tasks such as cooking and do-it-yourself (DIY) projects, emphasizing content curation for fault-tolerant execution to ensure that tasks are completed successfully, even in the presence of common failures.
The patent addresses shortcomings in existing virtual assistants, particularly their inability to flexibly cover a wide range of topics, dynamically update instructions based on user preferences, detect and recover from failures beyond simple retry or skip actions, and deliver personalized, human-centric interactions. The disclosed approach proactively monitors task execution, intervenes when necessary, and replans steps to recommend recovery options based on observed failures, adjusting over time as more interactions and user behaviors are observed.
Key components of the system include a multimodal pipeline for assimilating diverse user inputs, an orchestrator for workflow selection, a knowledge base housing both personalized and domain-specific knowledge graphs, an executor for delivering instructions, and a plan manager for generating or retrieving primary and recovery plans. The VA leverages both audio and visual modalities, operates based on defined workflow paths, and continually updates its recommendations and recovery solutions, making task execution both successful and personalized.
Claims Coverage
The patent contains three independent claims, each outlining inventive features centered on methods, systems, and computer-readable media for robust, dynamic, and personalized task-oriented virtual assistants.
Dynamic virtual assistant for task completion with failure recovery and multiple workflow paths
A method using at least one or more processors programmed to: - Access information specifying a user-specified task. - In response, cause a virtual assistant to: - Discover instructions from online data sources to achieve the task. - Assist the user or monitor the user's execution of the instructions. - Observe failures of instruction execution. - Help recover from action and instruction level failures until the task is complete. - Enable the virtual assistant to operate according to at least one of several specific path workflows, both with and without personalization, with or without pre-existing recovery plans, as described in the patent.
Non-transitory computer-readable storage medium for automated virtual assistant task execution and recovery
A non-transitory computer-readable storage medium storing processor-executable instructions that, when executed, cause at least one or more computer processors to: - Access information specifying a user-specified task. - Cause a virtual assistant to: - Discover instructions from online data sources for achieving the task. - Assist or monitor execution by the user. - Observe failures and help recover from failures until completion. - Support operation along at least one defined workflow path, with or without personalization and with dynamic recovery planning, as recited in the claim.
System for virtual assistant task management with online instruction discovery and recovery planning
A system comprising at least one or more computer processors programmed to: - Access information for at least one user-specified task. - Cause a virtual assistant to: - Discover relevant instructions online. - Assist or monitor user performance of instructions. - Detect failures and help recover from these failures until the task is finished. - Configure the virtual assistant to operate according to at least one workflow path, adapted for personalization and recovery, as outlined in the patent.
The inventive features center on virtual assistants that dynamically mine instructions from online sources, monitor and assist users in task execution, detect and respond to failures with recovery plans, and operate via a range of workflow paths accommodating personalization and domain adaptation.
Stated Advantages
The system dynamically discovers and analyzes new content over time, reconciling existing information with user preferences for improved recommendations.
It estimates failure probabilities for tasks and curates recovery solutions, supporting robust fault-tolerant execution that ensures end goals are reached even after failures.
The approach allows for proactive monitoring, intervening only when necessary, and provides multimodal guidance to enhance user understanding during task execution.
Personalization is achieved by learning user preferences and health conditions through ongoing interactions, resulting in tailored recommendations.
The system curates domain-specific knowledge graphs to support content curation and context adaptation.
The assistant provides dynamic recommendations during task execution, improving user experience and reducing boredom by suggesting alternative ways to perform tasks.
Human-centric values are incorporated, enabling the assistant to sense psychological attributes, destress users, and provide health-centric inputs as needed.
Documented Applications
Assisting users in procedural cooking tasks with dynamic recipe discovery, adaptation, and recovery support.
Guiding users through do-it-yourself (DIY) tasks, including failure detection and instruction adjustment.
Recommending solutions and alternatives in home improvement domains, such as tool or ingredient substitution.
Interested in licensing this patent?