Semantic multimodal search of engineering documents

Inventors

Jahjah, Karl-Alexandre • GARDNER, MARC-ANDRÉ • LAPOINTE, Marc-André • Rausch-Larouche, Evan • Asselin, Louis-Philippe

Assignees

Bentley Systems Inc

Abstract

In example embodiments, a semantic multimodal search function is provided in an engineering application for searching engineering documents. In an indexing phase, the application indexes a library of engineering documents to build an embedding database. For each of the engineering documents, a region detector extracts regions that each correspond to a different mode of technical or engineering data, and a set of ML models is used to generate embeddings that represent the semantic significance of technical or engineering data in each of the regions. In a query phase, the application receives search input and uses the region detector to extract regions that each correspond to a different mode of input. The application uses the ML models to generate embeddings that represent the semantic significance of input in each of the regions, and then performs a vector search between these embeddings and those in the embedding database.

Core Innovation

The invention describes a semantic multimodal search system for searching engineering documents in an embedding database by splitting search input into a plurality of regions, where each region corresponds to a different mode of input. The plurality of regions includes at least one region that corresponds to schematic input or computer aided design (CAD) model input and at least one other region that corresponds to text input or image input. A region detector of the engineering application extracts the regions from the search input.

For each region, the system generates one or more query embeddings that represent the semantic significance of the input in that region using a plurality of machine learning (ML) models. The ML models are adapted to produce first query embeddings representing semantic significance of schematic input or CAD model input and second query embeddings representing semantic significance of text input or image. In the indexing or document embedding phase, the system generates and stores document embeddings that represent semantic significance of technical or engineering data in respective regions of engineering documents.

During the query phase, the system compares the one or more query embeddings to document embeddings of an embedding database maintained in a library of engineering documents and selects one or more engineering documents having closest document embeddings. Search results are then displayed based on the selected one or more engineering documents. The approach further uses an aggregation ML model concept to form main embeddings from temporary embeddings and can generate embeddings for non-text modes by creating text descriptions of semantic significance.

Claims Coverage

The partial content includes four independent claims. Across these claims, the inventive features revolve around region-based multimodal input splitting, mode-adapted ML query embeddings, comparison to document embeddings in an embedding database, selection of closest documents, and display of search results.

Across the independent claims, the core coverage is a region-detector-driven multimodal search method/device that extracts schematic/CAD and text/image regions, uses mode-adapted ML models to generate query embeddings representing semantic significance, compares them to precomputed document embeddings in an embedding database to select closest engineering documents, and displays the selected results; dependent claim coverage further specifies aggregation from temporary embeddings, text-description handling for non-text modes, distance-based selection, and query-refinement scoring and reordering.

Stated Advantages

Documented Applications

No documented applications found