A method of dividing clinical data set for medical image AI training
Dezhong Zheng, Yuanyuan Yang, Wentao Li · 2020
In the AI training, the data set is always divided into training set and test set at random, but the clinical image data from hospitals is different from the public data set. The division of public data set is reasonably divided and evenly distributed after many experiments. Accurate understanding of the data distribution directly affects the training model quality. So we proposed a new method of dividing clinical data set based on distance metric learning of the Gaussian mixture model to obtain more reasonable data set divisions. The distance metric learning based on deep neural network, first embeds data into a new metric space, then in the metric space uses in-depth mining based on data characteristics, calculates the distance between samples, finally compares the differences. The method can accurately know the data distribution characteristics to a certain extent. Under the condition of understanding the data distribution characteristics, more reasonable divisions can be obtained. That can greatly affect the accuracy and generalization performance of the models we trained.