5/5 - (2 votes)

 About a project

Industry: Drone inspection
Location: San Jose, California, USA
Partnership period: The first phase was in 2019, the second phase of development started in October 2020 and is currently ongoing
Team size: 6 experts
Software product: A web tool with AI/ML functionality
Expertise delivered: Software development from scratch; partial decision-making; product vision and product development; testing, research, and development of AI/ML;

Goals

  • Developing an innovative AI/ML-powered web tool for the drone inspection industry with the specific functionality
  • Dealing with multiple AI-related tasks and conducting all required research to build the solution
  • Filling the gap in the drone inspection industry and potentially saving millions of dollars for companies

Solutions

  • Researching and developing from scratch several versions of a web tool to find out what fits the project the best
  • Setting up the communication between the back-end, front-end, and AI according to the COCO standard and project-specific features
  • Modifying the existing framework for the usage of a hierarchical dataset in the project, which was justified by the industry
  • The integration of orthophoto and 3D model functionality for users
  • Using image and camera metadata combined with geometry formulas to deal with the image grouping challenge

Results

  • Our team created an MVP of a web tool that automates the part of the inspection process, making the product truly innovative for the industry
  • We received 100% accuracy in the image grouping task
  • We managed to achieve a 2-3x cost cut on infrastructure for the client thanks to building the back-end part from the ground up and migrating to Postgres from Mongo

Tech Stack

  • Infrastructure: AWS (EC2, Batch, ECR, Cloud Watch, SES), ELK stack, Grafana, Prometheus
  • Back-End: Spring, Hibernate, PostgreSQL, Docker, OpenDroneMaps
  • Front-End: Angular 11, TypeScript, Three.js, Konva.js, Saas, RxJS, Jasmine
  • ML tools: PyTorch, MMDetection, OpenCV

Methodology, Tools

  • Scrum
  • Jira, Confluence
  • GitHub

Overview

As part of another project, we have built a strong relationship with the client for over a decade and helped with the consolidation of business assets for BlackHawk Network.

Our client was interested in a new industry and decided to create a startup that will be able to introduce a web tool with Artificial Intelligence and Machine Learning functionality for drone inspections. The client stepped up to be the CEO, hired industry experts as advisors, and contacted SPD Group for software development services, as trusted and proven partners. Our team played a major role in product vision, in addition to developing back-end, front-end, and AI/ML functionality.

During the development process, the number of involved experts changed per demand. However, the core team consists of:

  • Back-End Developer
  • Front-End Developer
  • Two AI/ML Experts
  • Project Manager
  • Delivery Manager

We had multiple iterations of the project. In this case study, we will focus on the latest iteration that started in October 2020 and aims to monitor the condition of solar panels and power lines.

Business Goals

The main business goal of this project was the creation of a platform, a web tool for drone inspections, that would allow:

  • to save the data from the inspections, including layouts of the objects that were photographed by the drones
  • to mark out objects on the pictures. If the solar panels were inspected, for example, the user should be able to mark the broken ones
  • to generate reports
  • to detect the objects and defects automatically thanks to AI/ML/Computer Vision module

“The biggest highlight of this project is the implementation of Artificial Intelligence and Machine Learning. Previously, all image processing and analysis from drone inspections have been done manually. Automating this process would save billions of dollars for the companies and will become a true paradigm shift in the industry. Expert analysis of the market showed that there are no similar solutions at the moment, but demand for this functionality is very high.”
—Oleksandr Boyko, Delivery Manager from SPD Group

This solution could be used both by separate inspectors and companies of any size. The web platform was chosen as the best way to achieve our goals. Currently, we don’t plan to develop standalone or mobile versions; however, we might consider it in the future.

Technical Challenges

The biggest challenge for us and for our client was the fact that there was no similar solution on the market, so we were breaking new ground and needed a lot of research, as well as trial and error to achieve our goals and build a required solution from scratch.

From the technical standpoint, AI-related tasks were the most challenging in this project:

  • Adjusting ML pipeline for complex hierarchical datasets
  • Creating 3D reconstruction from multiple images
  • Developing an algorithm that allows to group images capturing the same object. For example a building, that appears on multiple images

Solutions

“I joined the team on the latest iteration of the project. We already had a working prototype from a previous iteration but had to rebuild everything according to the wishes of our client. We completely redesigned the part of the data structure responsible for storing annotation data to make it compatible with the COCO format. Simultaneously, we developed an entirely new AI module, that would be able to provide coordinates of points on the picture, instead of masks.

The front-end part was also completely rebuilt, with the addition of hotkeys to improve the user experience. As a result, in the first few months, we got an entirely new part of the system, where the communication between back-end, front-end, and AI was conducted according to COCO standard and project-specific features.”
—Yevhenii Kukhol, Lead Back-End Developer from SPD Group

The usage of a hierarchical dataset was justified by the industry. Drone inspection of power lines, as one of the chosen directions, requires increased accuracy for AI to detect tons of small objects.

Simply put, a hierarchical dataset means that each annotation may have parents and children. For instance, precise object detection in the power line industry requires the AI system to separate pole and cross arm from the background, and be able to find objects on separated elements. To achieve that, our ML module should produce and consume data considering hierarchy to be able to understand that the pole contains a cross arm, and they are two different objects from the same hierarchy. There were no existing frameworks or models that can consume and produce annotation data in a hierarchical way, so our team decided to modify an existing framework for this goal.

In our web tool, users provide photos captured by drones, with the object of interest shown from different perspectives. In order to achieve more responsiveness, our team was requested to allow users to get an orthophoto and 3D model for their facilities. Having a 3D reconstruction is quite easy to get an orthophoto, so we started with that feature. We conducted an investigation and discovered an open-source project with the required functionality and successfully integrated it.

Finally, we have to figure out how to deal with the image grouping challenge. When you capture several objects from different perspectives, you have location metadata (longitude, latitude, and altitude) and camera pitch and direction. We decided to use image and camera metadata combined with geometry formulas to detect what exact objects the camera captured.

After that, we conducted an investigation on clusterization algorithms to improve the accuracy of the previous solution. We expected that the client was providing us data in a format where all images were related to the same object, were located in the same directory. Our team decided to get information about objects and related images from the directory structure, and we received 100% accuracy.

“It is also important to mention that our team built a back-end part from the ground up for the latest iteration of the project and migrated to Postgres from Mongo. The decision for migration was made due to the fact that the data model was very suitable for relative structure and most of our developers prefer to work with relative databases. After rebuilding CI/CD and the deployment model, we managed to cut the costs for infrastructure for the client 2-3x!”
—Yevhenii Kukhol, Lead Back-End Developer from SPD Group

Results

Our team has successfully completed an MVP and some additional features. The project is in the pre-sale stage. Even without AI/ML module, we managed to create a convenient web tool that surpasses competitor tools and allows users to upload data, effectively mark it and generate reports. The implementation of AI/ML, which allows automating a part of the inspection process, makes our product truly innovative for the industry.

We are forming the user testing group at the moment. The only people who actually tested the product and provided valuable feedback are drone inspection experts that were involved in the development process.

Currently, the project is focused on solar panels and power lines, but the list of industries could be easily expanded in the future. Since the platform works with pictures, it would be possible to cover any industry that can benefit from it. For example, it could be Healthcare, as the platform would be able to detect certain objects on x-ray shots, as well as learn and improve based on datasets.

“All decisions during the development process and tech stack choices were made with the increase of the development speed or significant benefits in the future in mind. Among the insights we got during the process, is the fact that complex features should be developed first. In AI/ML projects you should prioritize R&D, and then design the web tool. During the evolution of this project and research of AI/ML components, the understanding of a web tool changed multiple times, and we had to adapt accordingly.”
—Yevhenii Kukhol, Lead Back-End Developer from SPD Group