This task takes either the sample images or the related masked images to build the Neuronal Network. The training is a loop in which, in any cycle, 2 parameters are computed: the training loss and the validation loss.
The progress of training can be evaluated by comparing the training loss with the validation loss. During training, both values should decrease before reaching the minimal value, which should not change significantly with further cycles. Comparing the validation loss development with the training loss can give insights into the model’s performance. If both training and validation loss values are still decreasing, this indicates that training is still necessary. If the validation loss suddenly increases again, while the training loss decreases towards zero, it usually means that the network is overfitting to the training data.

Basically, the training is based on math operations. These operations are repetitive and time-consuming and can easily be parallelized. The usage of GPU resources improves the training performance in reducing the total time. Working with the CPU only, a complex training can take 7 to 10 days of work, while using the GPU the total time may reduce to mere hours (10 to 12).