Markup of the dataset: quality or economy?

Aim

To mark up more than 12 thousand images for training a neural network and do not go beyond the project budget.

Context:

Data markup may not always be outsourced, especially if the data and markup criteria change over the course of the project. For one of these orders, where data quality and budget constraints played a key role, ML refined a universal data markup platform. A convenient and multi-functional interface for managing label teams, as well as a financial "engine" with automatic calculations of coefficients and payment, allowed you to mark up more than 12,000 images in time and stay within the project budget.

Solution

To create an application that evaluates the condition of the skin from image from the smartphone camera, it was necessary to mark up a large number of photos with different marking zones, as well as skin types and a variety of appearance and condition options. Data markup is necessary for training neural networks. The quality of the markup determines the performance and success of the application in the market, as well as the accuracy of the forecasts that the neural network gives. As we are talking about assessing the skin condition of real users, accuracy and error-free is the most important indicator.

To protect the data, the customer wanted to implement the entire cycle of work on the same platform with one contractor, which was the Marketing Logic team. The MarkLab service was chosen as the platform. The platform initially had all the tools for marking up and working with lablers and project managers. The main task and "pain" of the customer was that the budget was strictly limited and to pay for work with flaws or inaccuracies, which certainly occurs when marking up such volumes of data, it means to go beyond the budget.

In order to eliminate this possibility, we have developed an additional financial scoring module specifically for this project, which helps managers evaluate the quality of each labler's work, sets a scoring mark, and also allows you to set up payment for each image based on the quality of its execution: the cost of marking each image directly depends on the score set by the system for this work. Thus, the budget allocated for the markup was distributed to the required number of images and divided into segments of acceptable quality with a gradation of payment. The step of increasing the cost allowed not to go beyond the budget and at the same time provides the label team with motivation to improve the quality of the markup.

What does it look like in the customer interface? All lablers have their own points and coefficients, which are adjusted as statistics accumulate and depend on the level of quality, accuracy and speed of work. Algorithms for evaluating work that automatically determine the percentage of completed tasks, the speed of work, "look" at accuracy and accuracy, and set a scoring score. Based on these indicators, as well as the established maximum level of payment, the fee of each label is calculated. If the minimum amount is accumulated, the labler can request its transfer to the card. The module performs all the calculations itself, providing the manager with a report to check. Such automation on the first project saved us and the customer a lot of time and effort, because the team included more than 300 people, and it would be more difficult to calculate the contribution of each person manually and transfer money from the point of view of labor costs and operational activities.

The project manager for each user sees all the objects assigned to it and their status, sees the entire dataset and the " heat map " of the markup of the entire data pool, in order to quickly assess the degree of readiness of the project and identify complex or problematic data blocks or work stages. And the initially calculated cost allowed us to stay within the agreed budget.

Result

The customer received the requested quality of work and high-quality material for training the "neural network". After spending additional effort on refinement, we received an upgrade of the MarkLab platform with a scoring and financial module. The platform allows you to conduct data markup projects in a comprehensive manner within its contour: form teams, manage them, upload, segment datasets, distribute tasks, upload reports and datasets at any stage, and perform all financial accruals and calculations.