3D gaze control of robot for navigation and object manipulation

Inventors

ZHANG, Xiaoli • Li, Songpo

Assignees

Colorado School of Mines

Abstract

Disclosed herein are devices, methods, and systems for controlling a robot or assistive device that allows the robot or device to find and manipulate objects in a real world environment. The disclosed devices, methods, and systems may, in many cases, receive control input through monitoring, tracking, and analyzing the 3D gaze of a user/controller. Using the described 3D eye tracking, a user can directly and naturally look at an object of interest in the real world, while the system monitors, tracks, and records the user's 3D gaze position. This information, in many cases, is then translated into input commands for the robot or assistive.

Core Innovation

The invention relates to creating control output signals from visual information by monitoring one or more eyes of a user gazing at a target object. The system determines and measures movement of the eye and pupil, analyzes the movement to identify intentional movement and unintentional movement, and translates the intentional movement into data describing three dimensional movement of the user's gaze. The intentional movement data are translated into an output signal, thereby creating an output signal.

The disclosed approach estimates each eye’s visual axis from pupil and glint features extracted from a 3D binocular, head-mounted eye-tracking system. It computes a 3D gaze point in space using visual axis intersection or the shortest distance between visual axes, and derives a gaze vector and gaze distance. This produces three dimensional gaze data suitable for forming control output signals.

The disclosure addresses the “Midas touch problem” by distinguishing intentional manipulatory gaze from scanning using fuzzy-logic/neural-network fusion for analyzing intentional versus unintentional eye movements. The resulting 3D gaze data are translated into robot/assistive-device control via rule-based navigation and object-manipulation strategies, including a Mamdani-type fuzzy controller.

Claims Coverage

The independent claim covers a method that converts monitored eye and pupil movement into control output signals by distinguishing intentional versus unintentional movement and translating intentional movement into three-dimensional gaze data and then into an output signal. Dependent claims further narrow how the three-dimensional gaze metrics are computed and how the analysis and use of the output signal are performed, including fuzzy logic and transferring the output signal to a robot or assistive device.

Overall claim coverage centers on generating control output signals from eye and pupil movement by explicitly identifying intentional versus unintentional movement, translating intentional movement into three-dimensional gaze data, optionally using fuzzy logic for the intentional/unintentional analysis, and transferring the resulting output signal to a robot or assistive device.

Stated Advantages

Documented Applications