Document search for document retrieval using 3D model

Inventors

Flagg, Cristopher • Frieder, Ophir

Assignees

Georgetown University

Abstract

Technologies are described for reconstructing physical objects which are preserved or represented in pictorial records. The reconstructed models can be three-dimensional (3D) point clouds and can be compared to existing physical models and/or other reconstructed models based on physical geometry. The 3D point cloud models can be encoded into one or more latent space feature vector representations which can allow both local and global geometric properties of the object to be described. The one or more feature vector representations of the object can be used individually or in combination with other descriptors for retrieval and classification tasks. Neural networks can be used in the encoding of the one or more feature vector representations.

Core Innovation

The problem addressed is reconstructing a three-dimensional (3D) model for an object from one or more images of the object, where the images include one or more epipolar views, and then using that 3D representation to perform matching for retrieval and identification. The disclosed approach generates the 3D model as a point cloud rather than another form, and derives one or more feature vector representations of the object from the point cloud. The resulting feature vector representations are then matched to one or more other feature vectors.

A key element is the 3D point cloud construction constrained by epipolar views through selecting a point on a first view, projecting a vector from the selected point into a model space, projecting the vector onto at least one other epipolar view, and determining candidate points on the vector for the point cloud. This enables the point cloud to be produced using relationships among multiple epipolar views and a model space tied to the projected vector. The disclosed selection and candidate-point determination form the geometric basis for the point cloud representation.

Another core element is using the generated feature vector representations for document retrieval and similarity-based identification. After generating the one or more feature vector representations of the object, the method matches them to one or more other feature vectors and identifies one or more documents having one or more feature vectors that most closely match the one or more feature vector representations generated for the object. The document retrieval is therefore grounded in comparing the object’s feature vectors to feature vectors associated with documents.

Claims Coverage

The independent claims are directed to a method performed by one or more computing devices and a non-transitory computer-readable medium storing instructions for executing the same method. Across these independent claims, the core inventive features include point-cloud-based 3D model generation from epipolar views with epipolar-view-constrained point selection and projection into model space, followed by feature-vector matching to identify documents with closely matching feature vectors.

The independent claims cover producing a point cloud 3D model from epipolar views using selection of a point on a first epipolar view, projecting a vector into a model space, projecting onto other epipolar views, and determining candidate points for the point cloud. They further cover generating feature vector representations for the object and matching those vectors to other feature vectors to identify documents whose feature vectors most closely match.

Stated Advantages

Documented Applications