Crack detection, assessment and visualization using deep learning with 3D mesh model
Inventors
Wu, Zheng Yi • Kalfarisi, Rony • Soh, Ken
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12347038-B2
Publication Date
2025-07-01
Expiration Date
Abstract
In various example embodiments, techniques are provided for crack detection, assessment and visualization that utilize deep learning in combination with a 3D mesh model. Deep learning is applied to a set of 2D images of infrastructure to identify and segment surface cracks. For example, a Faster region-based convolutional neural network (Faster-RCNN) may identify surface cracks and a structured random forest edge detection (SFRED) technique may segment the identified surface cracks. Alternatively, a Mask region-based convolutional neural network (Mask-RCNN) may identify and segment surface cracks in parallel. Photogrammetry is used to generate a textured three-dimensional (3D) mesh model of the infrastructure from the 2D images. A texture cover of the 3D mesh model is analyzed to determine quantitative measures of identified surface cracks. The 3D mesh model is displayed to provide a visualization of identified surface cracks and facilitate inspection of the infrastructure.
Core Innovation
The invention relates to acquiring a set of two-dimensional (2D) images of infrastructure and using a trained Mask Region-Based Convolutional Neural Network (Mask-RCNN) executing on one or more electronic devices to identify surface cracks in the 2D images and segment the identified surface cracks. The Mask-RCNN produces segmented 2D images having pixels divided into crack pixels that are part of identified surface cracks and non-crack pixels that are not part of the identified surface cracks.
Geometry information is extracted by a photogrammetry process executing on the one or more electronic devices from the segmented 2D images produced by the Mask-RCNN, and a three-dimensional (3D) mesh model of the infrastructure is generated from the geometry information. Textures are extracted from the segmented 2D images produced by the Mask-RCNN and draped on the 3D mesh model as a texture cover to form a textured 3D mesh model of the infrastructure.
The identified surface cracks are indicated in the textured 3D mesh model by showing the crack pixels in the texture cover with a contrasting color. An analysis process determines one or more quantitative measures of identified surface cracks in pixels based on the texture cover of the textured 3D mesh model, determines a conversion factor between pixels and a distance in the infrastructure, converts the pixel measures to quantitative measures for the infrastructure by applying the conversion factor, and displays in a user interface the textured 3D mesh model visualization and an indication of the quantitative measures for the infrastructure.
The document further describes training of the Mask-RCNN using transfer learning and fine-tuning with a crack image dataset, and refining the texture-cover processing by fusing the textures to produce a fused texture cover depicting the surface of the infrastructure without any overlap. The described workflow includes storing the textured 3D mesh model and storing an indication of the one or more quantitative measures of identified surface cracks for the infrastructure.
Claims Coverage
The document provides two independent claims: a method claim and a non-transitory electronic device-readable medium claim. Across the independent claims, there are six main inventive features.
Mask-RCNN crack identification and crack-pixel segmentation
Identify surface cracks in the 2D images and segment the identified surface cracks in the 2D images by a trained Mask Region-Based Convolutional Neural Network (Mask-RCNN) executing on one or more electronic devices, to produce segmented 2D images having pixels divided into crack pixels that are part of identified surface cracks, and non-crack pixels that are not part of the identified surface cracks.
Photogrammetry geometry extraction into a 3D mesh model
Extract geometry information from the segmented 2D images produced by the Mask-RCNN and generate a three-dimensional (3D) mesh model of the infrastructure from the geometry information, by a photogrammetry process executing on the one or more electronic devices.
Texture extraction and draping into a textured 3D mesh model
Extract textures from the segmented 2D images produced by the Mask-RCNN and drape the textures on the 3D mesh model as a texture cover to form a textured 3D mesh model of the infrastructure.
Contrasting-color crack indication on the texture cover
Indicate the identified surface cracks in the textured 3D mesh model by showing the crack pixels in the texture cover of the textured 3D mesh model with a contrasting color.
Pixel-based quantitative crack measures from the texture cover
Determine, by an analysis process executing on the one or more electronic devices, one or more quantitative measures of identified surface cracks in pixels based on the texture cover of the textured 3D mesh model.
Pixel-to-distance conversion to infrastructure quantitative measures
Determine a conversion factor between a number of pixels in the texture cover of the textured 3D mesh model and a distance in the infrastructure; convert the one or more quantitative measures of identified surface cracks in pixels to one or more quantitative measures for the infrastructure by applying the conversion factor.
Across the independent claims, the core claimed combination is the use of a trained Mask-RCNN to segment surface cracks into crack pixels and non-crack pixels in 2D infrastructure images, followed by photogrammetry to generate a textured 3D mesh where crack pixels are indicated on a texture cover with a contrasting color. The claims then compute pixel-based quantitative crack measures, convert those pixel measures using a pixel-to-distance conversion factor into infrastructure quantitative measures, and display or store the resulting indications.
Stated Advantages
Documented Applications
No documented applications found
Interested in licensing this patent?