Conflation of geospatial points of interest and ground-level imagery
Inventors
De, Debraj • Gurav, Rutuja • Fan, Junchuan • Thakur, Gautam
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12412375-B2
Publication Date
2025-09-09
Expiration Date
Abstract
Techniques are described that include accessing information about points of interest and images of scenes within the area of interest; encoding the information about each scene image as a respective scene-image vector; encoding the information about each point of interest as a respective point-of-interest vector; constructing a joint semantic graph having nodes and edges by (i) attributing to each node a respective point-of-interest vector or a respective scene-image vector, (ii) determining semantic distances between pairs of point-of-interest vectors, pairs of scene-image vectors, and pairs formed from a point-of-interest vector and a scene-image vector, and (iii) connecting each node with respective edges to a predetermined number of nearest-neighbor nodes having respective vectors with lowest semantic distances to each other. The constructed joint semantic graph can be used to enrich and/or clean the information about the points of interest and/or the images of scenes within the area of interest.
Core Innovation
A predictive data-conflation system accesses information about points of interest within an area of interest and information about scene images within the area of interest, where each point of interest comprises at least a point-of-interest category and each scene image comprises M scene tags with respective scene-tag probabilities. The system encodes each scene image as a scene-image vector with M coefficients that correspond one-to-one with the scene tags, while retaining only the N-highest scene-tag probabilities and setting remaining coefficients to zero. The system also encodes each point-of-interest category as a point-of-interest vector with M coefficients based on scores of word similarity between the point-of-interest category and the scene tags.
The system constructs a joint semantic graph comprising nodes and edges by attributing to each node either a point-of-interest vector or a scene-image vector. It determines semantic distances between pairs of point-of-interest vectors, between pairs of scene-image vectors, and between pairs formed from a point-of-interest vector and a scene-image vector. The system then connects each node with edges to at least some predetermined number of nearest-neighbor nodes K having vectors with the K-lowest semantic distances to each other, forming a semantic nearest-neighbor proximity structure in the joint semantic graph.
A scalable graph neural network is trained on split subgraphs to predict semantic edges/links between point-of-interest and image entities, using dot-product edge scoring and a binary cross-entropy loss. The described implementation includes a Johannesburg, South Africa use case and supports predictive semantic data fusion for enriching and/or cleaning point-of-interest and image data, including location and QA applications.
Claims Coverage
The independent claim covers a memory-encoded semantic-graph construction that fuses point-of-interest categories with probabilistically tagged scene images using vectors and semantic distances, and connects nodes to K nearest neighbors. It includes three core inventive features centered on vector encoding from tag probabilities and word similarity, and joint semantic graph construction via semantic distance-based nearest-neighbor edges.
Probabilistic scene-tag vector encoding with top-N retention
Encoding each scene image as a respective scene-image vector with M coefficients in one-to-one correspondence with the scene tags, where coefficients are the respective scene-tag probabilities for a predetermined number of coefficients N<M corresponding to N-highest scene-tag probabilities and zero for remaining (M−N) coefficients.
Point-of-interest category vector encoding via word similarity to scene tags
Encoding each point of interest as a respective point-of-interest vector with M coefficients in one-to-one correspondence with the scene tags, where the coefficients are respective scores of a word similarity between the point-of-interest category and the scene tags.
Joint semantic graph by semantic distances and K-lowest nearest-neighbor edges
Constructing a joint semantic graph comprising nodes and edges by attributing each node a respective point-of-interest vector or a respective scene-image vector, determining semantic distances between pairs of point-of-interest vectors, pairs of scene-image vectors, and pairs formed from a point-of-interest vector and a scene-image vector, and connecting each node with edges to at least some of a predetermined number of nearest-neighbor nodes K having respective vectors with the K-lowest semantic distances to each other.
The claim set centers on building a joint semantic graph where scene images are encoded using M scene-tag probabilities with N-highest retention, point-of-interest categories are encoded using word-similarity scores to the same scene tags, and edges are created by computing semantic distances and connecting each node to K nearest-neighbor nodes with lowest semantic distances.
Stated Advantages
Enriching and/or cleaning point-of-interest/image data.
Supporting location and QA applications.
Documented Applications
Johannesburg, South Africa use case for predictive semantic data conflation between points of interest and scene images.
Downstream support for location and QA applications using the semantic data conflation outputs.
Interested in licensing this patent?