- Industry: Finance
- Team size: 3 experts
- Technologies: Data Crawling, fastText, gensim, kNN, Clusterization, Text Classification
- Services: A visual representation of сompany clusters based on similarity and user-defined criteria
- Expertise delivered: Custom Software Development, support
Don't have time to read?
Book a free meeting with our experts to discover how we can help you save time and money.Book a Meeting
SPD Group implemented this project for a leading financial data provider that covers venture capital, private equity, and public markets worldwide. Prior to the commencement of the project, we had been their partner for over 13 years, having started as a small development team and gradually grown into the company’s principal full-cycle technology provider.
Our client made market research and learned their competitors already had a solution similar to the market map solution they had in mind. They requested us to refine the product vision they had and implement the product as a separate service with a visual interface, flexible functionality, and some additional features.
The client had a database with descriptions of companies and their verticals. Each company is assigned a number of keywords. To implement the solution we were tasked with developing, our project team needed to clusterize these descriptions, verticals, and keywords. The clusterization process had to be repeated numerous times, as each of the companies was operating in several different areas. Each of the industries and verticals had its own hierarchy with multiple levels. The hierarchies needed to be made adjustable in accordance with the required degree of detail.
Our experts needed to turn descriptions of companies into vectors and clusterize them using Machine Learning algorithms. Naming each of the groups correctly posed an additional challenge. As manual processing would have taken up a great deal of time, choosing and adjusting the right ML algorithms was the only option.
The second challenge we encountered was the need to select a suitable hierarchy for the industry codes, sectors, and verticals. Additionally, not all the companies had a full or accurate enough description, making it difficult to group them correctly.
Need help with software development services?
Book a free meeting with our experts to find out how we can help you to build your project according to your business vision.Book a Meeting
This project was tackled by the DataDev team that was responsible for ML solutions. The DataDev Team was composed of Team Lead Yuriy Batora and two Data Scientists — Oleksii Shashliuk and Denys Stupak.
“There is a large number of companies doing different things that can be split by both industry and vertical. ‘Industries’ is a basic classification, like Healthcare, Aircraft, or Banking. ‘Verticals’ is a logic-based classification of businesses with customer-defined custom parameters. For example, a client may be searching for drones, which is a sub-industry related to Aircraft, so drones or quadcopters are a ‘vertical” of Aircraft. Each industry has a certain number of verticals. So, our Market Map Solution allows searching not only by industry but by vertical too. This significantly broadens the selection of businesses.”
– Oleksii Shashliuk, Data Scientist
When the user is looking for an opportunity to invest only by industry, they can miss out on some not-so-obvious search results of companies, categorized by vertical. The main business value of the Market Maps solution is its ability to group businesses in relation to what they actually do at the moment, and not by the industry they represent.
“Basically, the solution consists of three Machine Learning-driven functions— vectorization, clusterization, and naming of the resulting clusters. For vectorization, we had tried fastText, Word2Vec, and Doc2Vec. We opted for fastText as the best option. The problem with clusterization is deciding the right number of clusters. We dealt with this by trial and error. Our current way of clusterization is great, but there is room to grow here. The naming of the clusters is a problem because we have around 600,000 keywords and 400,000 of them can be used only once.”
– Oleksii Shashliuk, Data Scientist
Our project team has applied the best practices of Artificial Intelligence to solve the challenges they were faced with. They have developed a solution that is capable of handling all three Machine Learning tasks. As the project is growing, we are now planning to make some updates to optimize the processes.
The Market Map Solution was developed within a span of 9 months. Currently, it is a fully functional service with more than 2 million companies in the database.
The project is in its fourth iteration. It is currently being supported by our DataDev team. They provide a daily refresh of the companies in the database. For example, when 1,000 new companies emerge,the DataDev team sorts them in accordance with the existing hierarchies. However, a complete rebuild of the hierarchies takes place after a certain amount of time has passed or a number of companies and industries have been added. This is done in order to provide a more accurate classification. Users can access the Market Map service via a website interface. They can customize their request the way they want, including the ability to edit the results for their convenience. This is what the interface of the application looks like:
36% of the users who have used the Companies and Deals (C&D) search capability, also used this feature. Market Maps were created in 12% of all the C&D searches. Corporate users loved the service, as in 17.5% of the C&D searches market maps were created.
“This is excellent and very helpful. I think we can turn around mapping for each of the reps on either Top 20 or current 9 touch cadence targets.”
– Payroll services company
“We work with an application of SaaS company. We want to show them the ecosystem, who the players are. We want to use your market maps to get a good starting place and add companies to it… My main usage in this is to slice it up by segments and then I’ll move companies around. And then I’d download.”
– Consulting firm
“Visually it is very appealing. If we were diligencing an opportunity we could use it to get a feel for the direct competitors of the company and also a couple sectors over see if there any players with deep pockets and the ability to quickly transition into the map of the company we’re looking at, this would be a helpful tool.”
– VC Firm
ARE YOU INTERESTED IN DEVELOPING ML-BASED SOFTWARE SOLUTION?
Contact our experts to get a free consultation and time&budget estimate for your project.Contact Us