Analysis of nailfold capillaroscopy images with artificial intelligence: Data from literature and performance of machine learning and deep learning from images acquired in the SCLEROCAP study

Objective: To evaluate the performance of machine learning and then deep learning to detect a systemic scleroderma (SSc) landscape from the same set of nailfold capillaroscopy (NC) images from the French prospective multicenter observational study SCLEROCAP. Methods: NC images from the first 100 SCLEROCAP patients were analyzed to assess the performance of machine learning and then deep learning in identifying the SSc landscape, the NC images having previously been independently and consensually labeled by expert clinicians. Images were divided into a training set (70 %) and a validation set (30 %). After features extraction from the NC images, we tested six classifiers (random forests (RF), support vector machine (SVM), logistic regression (LR), light gradient boosting (LGB), extreme gradient boosting (XGB), K-nearest neighbors (KNN)) on the training set with five different combinations of the images. The performance of each classifier was evaluated by the F1 score. In the deep learning section, we tested three pre-trained models from the TIMM library (ResNet-18, DenseNet-121 and VGG-16) on raw NC images after applying image augmentation methods. Results: With machine learning, performance ranged from 0.60 to 0.73 for each variable, with Hu and Haralick moments being the most discriminating. Performance was highest with the RF, LGB and XGB models (F1 scores: 0.75–0.79). The highest score was obtained by combining all variables and using the LGB model (F1 score: 0.79 ± 0.05, p < 0.01). With deep learning, performance reached a minimum accuracy of 0.87. The best results were obtained with the DenseNet-121 model (accuracy 0.94 ± 0.02, F1 score 0.94 ± 0.02, AUC 0.95 ± 0.03) as compared to ResNet-18 (accuracy 0.87 ± 0.04, F1 score 0.85 ± 0.03, AUC 0.87 ± 0.04) and VGG-16 (accuracy 0.90 ± 0.03, F1 score 0.91 ± 0.02, AUC 0.91 ± 0.04). Conclusion: By using machine learning and then deep learning on the same set of labeled NC images from the SCLEROCAP study, the highest performances to detect SSc landscape were obtained with deep learning and in particular DenseNet-121. This pre-trained model could therefore be used to automatically interpret NC images in case of suspected SSc. This result nevertheless needs to be confirmed on a larger number of NC images.