5/5 - (3 votes)

About the project

Industry: Drone inspection
Location: San Jose, California, the US
Partnership period: The first phase took place in 2019, the second phase of the development effort started in October 2020 and is currently ongoing
Team size: 6 experts
Software product: A web tool with AI/ML functionality
Expertise delivered: Software development from scratch; partial decision-making; product vision and product development; testing, research, and development of AI/ML.

Goals

  • Developing an innovative AI/ML-powered web tool for the Drone Inspection industry with business niche-specific functionality.
  • Dealing with multiple AI-related tasks and conducting all the required research to build the solution.
  • Filling the gap in the Drone Inspection industry and potentially saving millions of dollars for many companies.

Solutions

  • Researching and developing from scratch several versions of a web tool to find out what fits the project the best.
  • Setting up the communication between the back-end, front-end, and AI in accordance with the COCO standard and project-specific features.
  • Modifying the existing framework for the use of a hierarchical dataset in the project, in response to the market demand in the industry.
  • The integration of orthophoto and 3D model functionality.
  • Using image and camera metadata combined with geometry formulas to deal with the image grouping challenge.

Results

  • Our team created an MVP of a web tool that automates the part of the inspection process, making the product truly innovative for the Drone Inspection industry.
  • We have achieved 100% accuracy in the image grouping task.
  • We have managed to achieve a 2-3x cost-cut on the infrastructure for the client by building the system’s back-end from the ground up and migrating it from Mongo Postgres.

Don't have time to read?

Book a free meeting with our experts to discover how we can help you save time and money.

Book a Meeting

Tech Stack

  • Infrastructure: AWS (EC2, Batch, ECR, Cloud Watch, SES), ELK stack, Grafana, Prometheus
  • Back-End: Spring, Hibernate, PostgreSQL, Docker, OpenDroneMaps
  • Front-End: Angular 11, TypeScript, Three.js, Konva.js, Saas, RxJS, Jasmine
  • ML tools: PyTorch, MMDetection, OpenCV

Methodology, Tools

  • Scrum
  • Jira, Confluence
  • GitHub

Overview

Prior to the project’s commencement, we had been cooperating with the client for over a decade as part of another project and built a strong relationship with the client for over a decade. More specifically, we helped them consolidate their business assets for the BlackHawk Network.

Our client was interested in a new industry and decided to create a startup that would be able to introduce a web tool for drone inspection with Artificial Intelligence and Machine Learning functionality. One of the client’s Executives became the startup’s/project’s CEO. They hired industry experts as advisors and approached SPD Group as trusted and proven partners for software development services. Our team played a major role in defining the product vision and developing the back-end, front-end, and AI/ML functionality.

During the development process, the number of experts involved with the project varied in accordance with the project demands. However, our core project team consists of:

  • Back-End Developer
  • Front-End Developer
  • Two AI/ML Experts
  • Project Manager
  • Delivery Manager

The project had multiple iterations. In this case study, we will focus on the most recent iteration that started in October 2020 and aims to monitor the condition of solar panels and power lines.

Business Goals

The main business goal of this project was the creation of a platform, a web tool for drone inspections, that would allow:

  • to save the data from the inspections, including the layouts of the objects that were photographed by the drones.
  • to mark out objects on the pictures. If the solar panels were inspected, for example, the user should be able to mark the broken ones.
  • to generate reports.
  • to detect the objects and defects automatically by means of the AI/ML/Computer Vision module.

“The biggest highlight of this project is the implementation of Artificial Intelligence and Machine Learning. Previously, all image processing and analysis from drone inspections were performed manually. Automating this process would save billions of dollars for the companies and will become a true paradigm shift in the industry. Expert analysis of the market showed that there are no similar solutions at the moment, but demand for this functionality is very high.”
—Oleksandr Boyko, Delivery Manager from SPD Group

This solution could be used both by individual inspectors and companies of any size. The web platform was chosen as the best way to achieve our goals. Currently, we are not planning to develop standalone or mobile versions, but we might consider this in the future.

Technical Challenges

The biggest challenge for us and for our client was the fact that there was no similar solution on the market. We were breaking new ground and had to make a great deal of research and numerous experiments to achieve our goals and build the required solution from scratch.

From the technical standpoint, the following AI-related tasks were the most challenging ones in this project:

  • Adjusting ML pipeline for complex hierarchical datasets.
  • Creating 3D reconstruction from multiple images.
  • Developing an algorithm that allows one to group images that contain the same object. For example, a building that appears on multiple images.

Need help with software development services?

Book a free meeting with our experts to find out how we can help you to build your project according to your business vision.

Book a Meeting

Solutions

“I joined the team on the project during its most recent iteration. We already had a working prototype from the previous iteration but had to rebuild everything in accordance with the wishes of our client. We completely redesigned the part of the data structure responsible for storing annotation data to make it compatible with the COCO format.

Simultaneously, we developed an entirely new AI module that would be able to provide the coordinates of points on the picture, instead of masks.
The front-end part was also completely rebuilt, with the addition of hotkeys to improve the user experience. As a result, in the first few months, we received an entirely new part of the system, where the communication between back-end, front-end, and AI was performed in accordance with the COCO standard and project-specific features.”
—Yevhenii Kukhol, Lead Back-End Developer, SPD Group.

Using a hierarchical dataset was dictated by the industry’s demands. Drone inspection of power lines, as one of the chosen directions, requires an increased accuracy for AI to detect a vast number of minor objects.

Simply put, a hierarchical dataset means that each annotation may have parents and children. For instance, precise object detection in the Power Line industry requires that the AI system separate the pole and cross arm from the background. It must also be able to identify the same object in different images.

To achieve this, our ML module must produce and consume data taking into consideration the hierarchy in order to be able to determine whether a pole contains a cross arm and whether they are two different objects from the same hierarchy.

Unfortunately, none of the existing frameworks or models could consume and produce annotation data in a hierarchical way. Because of this, our project team decided to modify an existing framework for this purpose.

In our web tool, users provide photos, captured by drones. The object of interest is shown from different perspectives. In order to achieve greater responsiveness, our team was requested to allow users to get an orthophoto and 3D model for their facilities.

With 3D reconstruction, it is quite easy to obtain an orthophoto, so we started by creating this feature. We made research, discovered an open-source project with the required functionality, and successfully integrated our product with this functionality.

Finally, we had to figure out how to deal with the image grouping challenge.

When you capture several objects from different perspectives, you receive location metadata (longitude, latitude, and altitude) and the camera pitch and direction. We decided to use the image and camera metadata combined with geometry formulas to precisely identify the objects the camera captured.

After that, we conducted an investigation on clusterization algorithms to improve the accuracy of the previous solution. We expected the client to provide us with data in a format where all the images were related to the same object and located in the same directory. Our team decided to derive information on objects and the related images from the directory structure, having achieved 100% accuracy here.

“It is also important to mention that our team built the back-end part from the ground up for the latest iteration of the project and migrated to Postgres from Mongo. The decision for migration was made due to the fact that the product’s data model was very suitable for relative structure and most of our developers prefer to work with relative databases. After rebuilding CI/CD and the deployment model, we managed to cut the costs for infrastructure for the client by a factor of 2-3!”
—Yevhenii Kukhol, Lead Back-End Developer from SPD Group

Results

Our team has successfully completed the MVP and some additional features. Currently, the project is in the pre-sale stage. Even without the AI/ML module, we have managed to create a convenient web tool that surpasses competitor tools and allows users to upload data, effectively mark it, and generate reports. The implementation of AI/ML, which allows one to automate part of the inspection process, makes our product truly innovative for the Drone Inspection industry. At the moment, we are engaged in forming the user testing group.

Currently, the project is focused on solar panels and power lines, but the list of potentially interested industries could easily grow in the future. Since the platform works with images, it is possible to cover any industry that can benefit from it. For example, it could be Healthcare, as the platform could be able to detect certain objects on x-ray shots, as well as learn and improve based on datasets.

“All decisions during the development process and tech stack choices were made with the increase of the development speed or significant benefits in the future in mind. Among the insights we got during the process is the fact that complex features should be developed first. In AI/ML projects, you should prioritize R&D, and then design the web tool. During the evolution of this project and research of AI/ML components, our vision of a web tool changed multiple times, and we had to adapt accordingly.”

—Yevhenii Kukhol, Lead Back-End Developer, SPD Group

Do you have questions? Ask our experts!

Contact our experts to get a free consultation and time&budget estimate for your project.

Contact Us