System and method for matching products and determining spreads and plugs
Inventors
Savvides, Marios • Zhu, Chenchen • Chen, Fangyi • Ahmed, Uzair • Tao, Ran
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Assignees
Carnegie Mellon UniversityCarnegie Mellon University is a global research institution based in Pittsburgh, Pennsylvania, recognized for interdisciplinary education, research, and innovation in science, engineering, arts, technology, and social sciences. The university leads advancements in artificial intelligence, robotics, digital health, and performing arts. Located in a technology-driven and culturally rich city, CMU powers real-world impact through research centers, industry engagement, workforce training, and initiatives that shape regional and global communities.
Carnegie Mellon University is a global research institution based in Pittsburgh, Pennsylvania, recognized for interdisciplinary education, research, and innovation in science, engineering, arts, technology, and social sciences. The university leads advancements in artificial intelligence, robotics, digital health, and performing arts. Located in a technology-driven and culturally rich city, CMU powers real-world impact through research centers, industry engagement, workforce training, and initiatives that shape regional and global communities.
Publication Number
US-12536769-B2
Publication Date
2026-01-27
Expiration Date
Abstract
Disclosed herein is a system and method for matching products detected in an image of a shelf. The match or non-match of the products is then used to make a determination that the products are correctly positioned on the shelf of if the positioning of the products represents a plug or spread situation.
Core Innovation
An autonomous retail inventory monitoring camera system obtains an image of a shelf, including panoramic shelf images, and identifies a first and second object from the image. The system performs size match using a number of pixels in one or more dimensions with depth pixel applied to each pixel to adjust for different depths, and performs color match using average pixel values from corresponding portions of the first and second objects.
When the size and color match, the system uses a deep learning convolution neural network to extract features from portions of the image representing the first and second objects using a trained feature extractor. A classifier determines whether the first and second objects match if differences between the extracted features fall within a predetermined distance threshold, and the trained feature extractor is trained on multiple views of each object with associated identifying information.
The document further describes associating products to shelf labels using a product/shelf label association approach and determining inventory conditions, including out-of-stock, plug, and spread conditions, based on match results and shelf region membership. It also describes an enhanced misplacement detection approach using a learned product library with a deep feature extractor trained on prime IDs, feature matching to identify misplaced products, and dynamic enrollment of new or seasonal products into the library using zero-shot or low-shot enrollment based on feature distances and library updates.
Claims Coverage
The independent claim covers 8 inventive features centered on object matching for shelf images, with dependent claims refining shelf/region handling, shelf-label association, plug/spread reporting, and optical character recognition tied to writing comparison.
Depth-adjusted size match for shelf objects
Performing a size match between the first and second objects using a number of pixels in one or more dimensions of the first and second objects, wherein a depth pixel from a depth sensor has been applied to each pixel in the image comprising the first and second objects to adjust for different depths.
Average-value color match using corresponding portions
Performing a color match between the first and second objects using average pixel values from corresponding portions of the first and second objects.
Deep neural convolution feature extraction with distance-threshold classification
When the size and color match, using a deep learning convolution neural network to extract features from portions of the image representing the first and second objects using a trained feature extractor, and determining, using a classifier, that the first and second objects match if the differences between the features extracted from the first object and features extracted from the second object fall within a predetermined distance threshold, wherein the trained feature extractor is trained on multiple views with associated identifying information and the classifier is a deep neural network trained to compare extracted features.
Shelf image and regions of interest with bounding for objects
Obtaining an image of a shelf and detecting regions of interest in the shelf image, where each detected region bounds an image of an object.
Shelf label detection with region-of-interest association
Detecting one or more shelf labels in an image of a shelf and associating a region of interest with each detected shelf label.
Spread reporting based on matched objects not in the same region of interest
Determining that first and second objects match, verifying they are not in the same region of interest, and reporting a spread situation.
Plug reporting based on non-matching objects within the same region of interest
Determining that first and second objects do not match, that they are located in the same region of interest, and reporting a plug situation.
Optical character recognition for writing comparison
Performing optical character recognition on detected text within image portions for the first and second objects and determining whether the associated writings match.
Across the independent claim and refinements, the covered inventive aspects center on depth-adjusted size matching, average-value color matching, and deep neural network feature extraction plus predetermined-distance-threshold classification for object matching, with dependent claim logic extending shelf/region handling, shelf-label association, plug/spread reporting, and OCR-based writing comparison.
Stated Advantages
Documented Applications
No documented applications found
Interested in licensing this patent?