Description
Brickinspector leverages advanced deep learning techniques to identify Lego® bricks in pictures.
Brickinspector is a stand-alone software solution that visually identifies Lego® bricks in images. Those recognitions can be used to determine the missing bricks to have a complete Lego set, the bricks that do not pertain to that set, or to easily build your Lego® bricks inventory.
Technology
The Artificial Intelligence module uses a convolutional neural network to recognize the bricks from the images. To train such models, usually a large set of manually annotated images is required. Due to the large catalog of Lego® bricks, this is an unfathomable task.
To solve this problem and easily scale this application to any Lego® product, Tramacsoft GmbH has developed a synthetic rendering pipeline to automatically train the network from 3D CAD models rendered as photo realistic images with perfect ground truth, without any human effort.
Examples of synthesized images using the rendering pipeline under diverse backgrounds, lighting conditions and poses
Moreover, a small set of manually annotated images (less than 10 images is enough) can be used to finetune the network and increase the recognition rate up to 97%. If more images are provided, the more the recognition rate will improve.
Detailed information of this technology can be found freely accessible journal published on the Sensors special issue: Applications of Video Processing and Computer Vision Sensor II
Brickognize: Applying Photo-Realistic Image Synthesis for Lego Bricks Recognition with Limited Data
In addition, and as stated in the paper, Tramacsoft GmbH is providing testing datasets of manually annotated images. One dataset containing images in a controlled environment where pictures are taken with controlled light and camera position, and another dataset in an uncontrolled environment with diverse backgrounds, lighting and camera poses. The datasets are annotated following COCO format.
Download Dataset
Other applications of the technology
This technology, although developed for Lego® bricks recognition, is not limited to this application. It can be easily applied to many industrial applications where CAD models are available. Photorealistic synthetic images with perfect ground truth can be automatically generated for them, reducing the human effort of manual annotation, and obtaining high recognition rates.