MyBlogPosts
Back to Blogs

MIDAS Skin Cancer Detection!

A Full-Stack DeepLearning and AI enhanced Application

Published
July 2025
Reading time
6 min
Topics
DeepLearning · MachineLearning · Medical Imaging · Healthcare
Sections
7
On this page (7)
  1. What and Why Multi-Modal Intelligent Dermoscopy Analysis(MIDAS)?
  2. Tech Stack
  3. Dataset Overview
  4. Model Architecture(BaseLine)
  5. Improved Model Architecture
  6. Tech&logic behind the model
  7. timm and Optuna

What and Why Multi-Modal Intelligent Dermoscopy Analysis(MIDAS)?

The goal of the MIDAS V1 project is to create an intelligent diagnostic tool that will help medical professionals detect a variety of skin lesions, from benign moles to potentially fatal diseases like melanoma. The unique feature of our method is its multimodal design: MIDAS includes crucial patient metadata, such as age, sex, and lesion location, in addition to dermoscopy images. In order to provide a more comprehensive and contextually aware analysis, this combination of clinical and visual data attempts to mirror how dermatologists evaluate cases in their daily work.

Using the clinical context provided by patient data as well as the visual patterns present in the lesion images, our system is trained to classify seven different skin conditions, acknowledging the importance of early and accurate detection of skin cancer. Compared to single-modality systems, MIDAS provides a more accurate, sophisticated, and clinically relevant assessment by combining these two sources via a bidirectional attention mechanism. Our ultimate objective is to create a decision-support tool that promotes prompt interventions in dermatological care and increases diagnostic confidence.

Tech Stack

  • React
  • Tailwindcss
  • Python
  • Pytorch
  • FastAPI
  • Docker
  • Typescript

Dataset Overview

Our project is based on the HAM10000 dataset (Human Against Machine with 10,000 training images), a popular and carefully selected set of dermoscopic photos meant for the classification of common pigmented skin lesions. Clinically significant metadata, including lesion ID, image ID, diagnosis (dx), diagnosis type (dx_type), patient age, sex, and lesion location, are included with each of the 10,015 high-quality dermoscopy images. We found 57 missing values during the preliminary analysis, the majority of which were in the age column. During preprocessing, these were handled properly. Although PAD-UFES-20, a secondary dataset, was taken into consideration for inclusion, it was not included in this phase because of a large amount of missing data and to guarantee a controlled and targeted scope for model development.

Model Architecture(BaseLine)

Model Architecture(BaseLine)

The Multi-Modal Intelligent Dermoscopy Analysis System (MIDAS_V1) is a specialized model developed for the detailed analysis of skin lesions. Its core strategy involves the integration of two key types of information: the visual data from dermoscopy images and the contextual data from patient records. By uniting the visual evidence with relevant clinical details like patient age and the lesion’s location, the system can perform a more thorough analysis than is possible when considering either type of information in isolation. The process starts with the Image Encoder, which handles the visual analysis. This component uses the EfficientNet-B0 model, a highly capable convolutional neural network. Having been pre-trained on a vast collection of images, the encoder excels at identifying and extracting important visual features. It examines the dermoscopy image for key characteristics like texture, color variations, and structural patterns, ultimately producing a detailed numerical summary that captures the lesion’s appearance. Working in parallel, the Metadata Encoder evaluates the patient’s accompanying information. This part of the system is a neural network tailored to interpret tabular data, such as the patient’s demographic details and the anatomical location of the lesion. Its purpose is to find meaningful patterns within this information and convert the patient’s background details into a useful numerical format. In doing so, it supplies the analysis with essential clinical context. In the final stage, the Fusion and Classification Head unites the outputs from the previous two components. The numerical summaries from both the Image Encoder and Metadata Encoder are joined together, creating a single, comprehensive profile of the case. This unified profile is then delivered to the classifier, which is the final decision-making part of the network. The classifier evaluates the combined information to arrive at a final determination about the lesion. This integrated method ensures the system’s conclusion is based on the full context of the case.

Improved Model Architecture

Improved Model Architecture

For our advanced model, we designed a new architecture called MIDAS_V1, which improves upon our baseline by combining image and metadata features more effectively.During experiments, we can easily test different backbones because the model uses a dynamic image encoder (backbone) that was chosen from popular architectures using the timm library. The model learns the relationship between the two inputs by processing metadata independently through a small feedforward network and then fusing it with image features using a cross-attention module.

We use the Optuna framework to find the optimal configuration. It assists us in automatically determining the optimal image backbone and optimizing important hyperparameters such as attention dimensions, learning rate, and dropout rate. By modifying its structure and settings in response to validation results, our model becomes more adaptable and better suited to the dataset.

Tech&logic behind the model

In building MIDAS, we relied heavily on deep learning techniques to handle both dermoscopy images and patient metadata in a unified manner. For the image analysis part, we used the PyTorch Image Models (timm) library, which provides a rich collection of pre-trained deep neural networks such as EfficientNet, ResNet, and Vision Transformers. By leveraging timm, we could dynamically test different state-of-the-art backbones without having to manually implement each one, making the experimentation process more efficient. These backbones, pre-trained on large-scale datasets, gave the model a strong ability to extract meaningful features like texture, color, and structural patterns from skin lesion images. On the metadata side, a separate lightweight neural network was designed to interpret clinical variables such as age, sex, and lesion location. To combine the two modalities, we introduced a cross-attention fusion mechanism, which aligns image-derived features with metadata representations, mimicking the way dermatologists consider both visual and clinical context when making diagnoses. Beyond architecture, we employed Optuna for hyperparameter optimization, tuning key parameters like learning rate, dropout, and attention dimensions to adapt the model effectively to the HAM10000 dataset. The pipeline also integrated class balancing strategies and data augmentation to address the inherent skew in lesion categories. Importantly, we used interpretability techniques like Grad-CAM to visualize which parts of the lesion the model attended to, ensuring transparency and trust in the predictions. This deep learning strategy, with timm at its core for flexible image backbone selection, enabled MIDAS to evolve from a 72% baseline accuracy to nearly 85%, while delivering more balanced and clinically reliable predictions

timm and Optuna

The timm (PyTorch Image Models) library and Optuna are two powerful tools that greatly simplify modern deep learning experimentation. Timm provides a rich collection of pre-trained vision models—ranging from classic CNNs like ResNet and EfficientNet to more advanced architectures such as Vision Transformers—making it easy to experiment with different backbones without needing to implement them from scratch. These pre-trained models offer strong feature extraction capabilities, which can be fine-tuned to specialized datasets such as medical images. On the other hand, Optuna is an automatic hyperparameter optimization framework designed to efficiently search large parameter spaces. Using a sampling-and-pruning strategy, it dynamically explores learning rates, dropout ratios, layer dimensions, and other configurations to identify the best-performing setup. When combined, timm and Optuna enable rapid iteration over both architecture and hyperparameters: timm ensures flexibility in choosing powerful backbones, while Optuna systematically fine-tunes the model for optimal performance, reducing the trial-and-error effort typically required in deep learning research.