Multimodal Skin Cancer Prediction: Integrating Dermoscopic Images and Clinical Metadata with Transfer Learning

Skin cancers exist as the most pervasive cancers in the world; to increase the survival rates, early prediction has become more predominant. Many conventional techniques frequently depend on visual review of clinical information and dermoscopic illustrations. In recent technological developments, the enthralling algorithms of combining modalities are used for increasing diagnosis accuracy in deep learning.

Our research proposes a multi-faceted approach for the prediction of skin cancer that incorporates clinical metadata with dermoscopic visuals. The pre-trained convolutional neural networks, like EfficientNetB3, were used for dermoscopic images along with transfer learning techniques to excavate some of the visual attributes in this study. Moreover, TabNet was used for processing the clinical metadata, including age, gender, and medical history. The features obtained from both fusion techniques were integrated to enhance the prediction accuracy. The benchmark datasets, like ISIC 2018, ISIC 2019, and HAM10000, were used to assess the model.

The proposed multi-faceted system achieved 98.69% accuracy in the classification of skin cancer, surpassing the model that used dermoscopic snapshots with clinical data. The convergence of images with clinical metadata has substantially enhanced prediction resilience, demonstrating the importance of multimodal deep learning in skin lesion diagnosis.

This research focused mainly on the efficiency of integrating dermoscopic visuals and clinical information using transfer learning for skin cancer prediction. The proposed system offers a promising tool for improving diagnostic accuracy, and further research could explore its application in other medical fields requiring multimodal data integration.