Communication sessions using object information

Inventors

Peebler, Bradley W. • da Veiga, Alexandre

Assignees

Apple Inc

Abstract

Various implementations use object information to facilitate a communication session. Some implementations create a dense reconstruction (e.g., a point cloud or triangular mesh) of a physical environment, for example, using light intensity images and depth sensor data. A less data-intensive object information is also created to represent the physical environment for more efficient storage, editing, sharing, and use. In some implementations, the object information includes object attribute and location information. In some implementations, a 2D floorplan or other 2D representation provides object locations and metadata (e.g., object type, texture, heights, dimensions, etc.) provide object attributes. The object location and attribute information may be used, during a communication session, to generate a 3D graphical environment that is representative of the physical environment.

Core Innovation

The invention provides a communication session with a second device via wireless communication, where the first device receives object information indicating attributes and locations of objects in a physical environment in which the second device is located. The attributes and locations are determined using sensor data captured by one or more image sensors in the physical environment, and the first device generates three-dimensional (3D) visual representations of the objects using the attributes of the objects in the physical environment.

The invention determines 3D positions for the visual representations of the objects in a coordinate space of a graphical environment using the locations of the objects in the physical environment. In accordance with the determined positions, the invention generates a graphical environment that includes the generated 3D visual representations of the objects at 3D positions within the 3D graphical environment based on the locations of the objects in the physical environment, and during the communication session, a first view of the graphical environment at the first device and a second view of the physical environment at the second device are presented simultaneously.

The disclosed approach further supports structuring object information for the graphical environment using spatial representations such as a two-dimensional (2D) floorplan showing object locations relative to a horizontal plane, and object attributes such as heights above a horizontal plane determined using a depth sensor. It also supports object types determined using a semantic segmentation machine learning model, and multi-fidelity generation of visual representations in which lower-fidelity output is generated using first object information and higher-fidelity output is generated using additional second object information.

Claims Coverage

The document includes three independent claims that share the same core workflow: wireless session, receiving sensor-determined object information, generating 3D visual representations, mapping them to a coordinate space, and presenting simultaneously a graphical environment view and a physical-environment view. Across the independent claims, the inventive features focus on creating a graphical 3D environment based on object attributes and locations derived from sensor data, together with simultaneous dual-view presentation.

Across all independent claims, the inventive coverage centers on generating a 3D graphical environment from sensor-determined object attributes and object locations communicated during a wireless session, mapping objects into a graphical coordinate space, and presenting a first graphical view and a second physical-environment view simultaneously; independent claim variants express this as a non-transitory computer-readable medium, a first system including specific sensors and components, and a processor-implemented method.

Stated Advantages

Documented Applications