BigHub Blog - Read, Discover, Get inspired.

BigHub has a longstanding partnership with a major international logistics firm, during which it has successfully implemented a diverse range of data projects. These projects have encompassed a variety of areas, including data engineering, real-time data processing, cloud and machine learning-based applications, all of which have been designed and developed to enhance the logistics company's operations, including warehouse management, supply chain optimization and the transportation of thousands of packages globally on a daily basis.

In 2022, BigHub was presented with a new challenge: to aid in the implementation of a system for the early detection of suspicious fraudulent shipments entering the company's logistic network. Based on the client's pilot solution, which had been developed and tested using historical data, BigHub improved the algorithms and deployed them in a production environment for real-time evaluation of shipments as they entered the transportation network. The initial pilot solution was based on batch evaluation, but the requirement for our team was to create a REST API that could handle individual queries with a response time of less than 200 milliseconds. This API would be connected to the client's network, where further operations would be carried out on the data.

‍

‍

The proposed application is designed with a high-level architecture, as illustrated in the accompanying diagram. The core of the system is the REST API, which is connected to the client's network to receive and process queries. These queries are subject to validation and evaluation, with the results then returned to the end user. The data layer serves as the foundation for the calculations, as well as for the training of models and pre-processing of feature tables. The evaluation results are also stored in the data layer to facilitate the production of summary analyses in the reporting layer. The MLOps layer manages the lifecycle of the machine learning model, including training, validation, storage of metrics for each model version and making the current version of the model accessible via the REST API. To achieve this, the whole solution leverages a variety of modern data technologies, including Kubernetes, MLFlow, AirFlow, Teradata, Redis and Tableau.

During the development of the system our team needed to address several challenges that include:

• Setup and scaling of the REST API to handle a high volume of queries (260 queries from 30 parallel resources per second) in real-time, ensuring it is ready for global deployment.
• Optimizing the evaluation speed of individual queries, through the use of low-level programming techniques, to reduce the time from hundreds of milliseconds to tens of milliseconds.
• Managing the machine learning model lifecycle, including automated retraining, deployment of new versions into API, monitoring of quality and notifications, to ensure reliable long-term performance.
• Implementing modifications on the run - our agile approach ensured flexibility and allowed quick and successful changes to the ongoing project for the satisfaction of both parties and better results.

‍

Tenth in Central Europe and sixth in the Czech Republic. These are the results of the Deloitte Technology Fast 50 CE 2021 competition, which compares the growth of registered technology companies over the previous four years, from 2017 to 2020. This year, 19 companies from the Czech Republic made it to the CE Top 50, top ten winners included FTMO, DoDo, Driveto, or DataSentics, and our BigHub! was one of them. We are very grateful to be ranked among such great companies.

With an overall growth of 1 795%, we close the top ten companies in CE. In addition, 139 local companies applied this year so it was possible to compile a ranking of 50 top Czech technology companies, where we ended up in sixth place. ”The experience from previous years shows that thanks to the Fast 50 programme, companies manage to find interesting opportunities, change their ways and keep growing,” says Jiří Sauer, CE Technology Fast 50 Programme Leader.

‍

‍

The dominance of the Czech tech scene was impossible to overlook. We have 6 representatives in the European top 10 and 19 domestic tech leaders in the European top 50. At the same time, this year's edition also recorded the highest growth figures, with the average number of companies registered growing by 2 278%.

The ceremony took place in the Archa Theatre and the evening was hosted by moderator Tomáš Studeník. All Czech and Central European companies that were awarded in the Deloitte Technology Fast 50 program, you can find on this page.

‍