Synthetic Data Augmentation for Multi-Class Skin Lesion Classification: A Hybrid EfficientNet-Vision Transformer Framework with Explainable AI
DOI:
https://doi.org/10.67556/zjyqke62Keywords:
Skin lesion classification, HAM10000, stable diffusion, synthetic augmentation, EfficientNet, Vision Transformer, Grad-CAM, class imbalance, explainable AIAbstract
The research explores the utilization of Stable Diffusion models for synthetic image augmentation aimed at generating an improved dataset to enhance the performance of deep learning models in diagnosing skin diseases. The study utilizes the HAM10000 dataset, which consists of seven skin lesion categories exhibiting severe class imbalance, to fine-tune a class-conditional Stable Diffusion model for minority-class image synthesis. The resultant synthetic images, evaluated at a Fréchet Inception Distance (FID) of 98.0, are combined with real training images to train a dual-branch hybrid architecture,SkinHybrid,fusing EfficientNet and Vision Transformer (ViT) representations. An ablation study demonstrates that incorporating Stable Diffusion-generated samples improves the weighted F1-score from 84.1% to 97.0%. Post-hoc explainability via Grad-CAM confirms that the model attends to clinically relevant dermoscopic morphology
##plugins.themes.default.displayStats.downloads##
References
Agarwal, S., & Mahto, A. K. (2025). Skin cancer classification: Hybrid CNN-transformer models with KAN-based fusion. arXiv. https://doi.org/10.48550/arXiv.2508.12484
Akrout, M., Gyepesi, B., Holló, P., Poór, A., Kincső, B., Solis, S., Cirone, K., Kawahara, J., Slade, D., Abid, L., Kovács, M., & Fazekas, I. (2024). Diffusion-based data augmentation for skin disease classification: Impact across original medical datasets to fully synthetic images. In Deep Generative Models - MICCAI 2023 Workshop, Lecture Notes in Computer Science, 14533. Springer. https://doi.org/10.48550/arXiv.2301.04802
Azizi, S., Kornblith, S., Saharia, C., Norouzi, M., & Fleet, D. J. (2023). Synthetic data from diffusion models improves ImageNet classification. arXiv. https://doi.org/10.48550/arXiv.2304.08466
Chaturvedi, S. S., Gupta, K., & Prasad, P. S. (2019). Skin Lesion Analyser: An efficient seven-way multi-class skin cancer classification using MobileNet. arXiv. https://doi.org/10.48550/arXiv.1907.03220
Datta, S. K., Shaikh, M. A., Srihari, S. N., & Gao, M. (2021). Soft-attention improves skin cancer classification performance. arXiv. https://doi.org/10.48550/arXiv.2105.03358
Dhariwal, P., & Nichol, A. (2021). Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 34. https://doi.org/10.48550/arXiv.2105.05233
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An image is worth 16 x 16 words: Transformers for image recognition at scale. International Conference on Learning Representations. https://doi.org/10.48550/arXiv.2010.11929
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639), pp. 115-118. https://doi.org/10.1038/nature21056
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In Advances in Neural Information Processing Systems, 27. https://doi.org/10.48550/arXiv.1406.2661
Haque, M. M., Akter, R., Akib, A. S. M. A. S., & Hasib, A. (2026). A deep learning approach for automated skin lesion diagnosis with explainable AI. arXiv. https://doi.org/10.48550/arXiv.2601.00964
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, 30. https://doi.org/10.48550/arXiv.1706.08500
Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 33. https://doi.org/10.48550/arXiv.2006.11239
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2980-2988). https://doi.org/10.1109/ICCV.2017.324
Ma, X., Shan, J., Ning, F., Li, W., & Li, H. (2023). EFFNet: A skin cancer classification model based on feature fusion and random forests. PLOS ONE, 18(10), e0293266. https://doi.org/10.1371/journal.pone.0293266
Nguyen, V. D., Bui, N. D., & Do, H. K. (2022). Skin lesion classification on imbalanced data using deep learning with soft attention. Sensors, 22(19), 7530. https://doi.org/10.3390/s22197530
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., & Rombach, R. (2023). SDXL: Improving latent diffusion models for high-resolution image synthesis. arXiv. https://doi.org/10.48550/arXiv.2307.01952
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10684-10695). https://doi.org/10.48550/arXiv.2112.10752
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 618-626). https://doi.org/10.48550/arXiv.1610.02391
Sung, H., Ferlay, J., Siegel, R. L., Laversanne, M., Soerjomataram, I., Jemal, A., & Bray, F. (2021). Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: A Cancer Journal for Clinicians, 71(3), pp. 209-249. https://doi.org/10.3322/caac.21660
Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning (pp. 6105-6114). https://proceedings.mlr.press/v97/tan19a.html
Tschandl, P., Codella, N., Akay, B. N., Argenziano, G., Braun, R. P., Cabo, H., Gutman, D., Halpern, A., Helba, B., Hofmann-Wellenhof, R., Lallas, A., Lapins, J., Longo, C., Malvehy, J., Paoli, J., Puig, S., Rinner, C., Soyer, H. P., Sinz, C., & Kittler, H. (2019). Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: An open, web-based, international, diagnostic study. The Lancet Oncology, 20(7), pp. 938-947. https://doi.org/10.1016/S1470-2045(19)30333-X
Rosendahl, C., & Kittler, H. (2018). The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5, 180161. https://doi.org/10.1038/sdata.2018.161
