A medical image analysis and education platform with deep learning models, Grad-CAM visualization, and AI-powered explanations. Built with PyTorch, FastAPI, and React.
π Live Demo: medlens-two.vercel.app
This project provides:
- REST API for medical image classification with Grad-CAM visualization
- AI-generated explanations using Claude vision API
- Analyze Mode: Upload images for diagnosis with visual + written explanations
- Learn Mode: Unlimited quiz for practicing diagnosis with instant feedback
- Dashboard: Track learning progress with accuracy charts and weak area identification
- Extensible architecture for adding new classification models
| Model | Task | Classes | Accuracy | AUC |
|---|---|---|---|---|
brain_tumor |
MRI Classification | Glioma, Meningioma, Pituitary, No Tumor | 99% | - |
pneumonia |
Chest X-Ray | Normal, Pneumonia | 88% | 0.958 |
bone_fracture |
X-Ray Fracture Detection | Fractured, Not Fractured | 98% | 0.998 |
retinal_oct |
Retinal OCT Analysis | CNV, DME, Drusen, Normal | 99% | - |
π Click on any model below to view training curves, confusion matrix, and Grad-CAM visualizations.
π§ Brain Tumor MRI Classification (99% accuracy)
4-class classification of brain MRI scans.
| Metric | Value |
|---|---|
| Test Accuracy | 99% |
| Classes | Glioma, Meningioma, Pituitary, No Tumor |
Training Progress
Confusion Matrix
Grad-CAM Visualizations
π« Pneumonia Detection (88% accuracy, 0.958 AUC)
Binary classification from chest X-rays.
| Metric | Value |
|---|---|
| Test Accuracy | 88% |
| AUC Score | 0.958 |
| Classes | Normal, Pneumonia |
Training Progress
Confusion Matrix
ROC Curve
Grad-CAM Visualizations
𦴠Bone Fracture Detection (98% accuracy, 0.998 AUC)
Binary classification of X-rays across all anatomical regions.
| Metric | Value |
|---|---|
| Test Accuracy | 98.22% |
| AUC Score | 0.998 |
| Classes | Fractured, Not Fractured |
Training Progress
Confusion Matrix
ROC Curve
Grad-CAM Visualizations
ποΈ Retinal OCT Analysis (99% accuracy)
4-class classification of retinal OCT scans for eye disease detection.
| Metric | Value |
|---|---|
| Test Accuracy | 99.38% |
| Classes | CNV, DME, Drusen, Normal |
Training Progress
Confusion Matrix
Grad-CAM Visualizations
medical-image-classifier/
βββ api/
β βββ app/
β β βββ main.py # FastAPI application
β β βββ models/
β β β βββ base.py # Base classifier interface
β β β βββ brain_tumor.py # Brain tumor classifier
β β β βββ pneumonia.py # Pneumonia classifier
β β β βββ bone_fracture.py # Bone fracture classifier
β β β βββ retinal_oct.py # Retinal OCT classifier
β β βββ utils/
β β βββ gradcam.py # Grad-CAM visualization
β β βββ llm.py # Claude LLM integration
β βββ scripts/
β β βββ generate_explanations.py # Batch generate explanations
β β βββ generate_overlays.py # Batch generate Grad-CAM overlays
β βββ weights/ # Model weights (not tracked in git)
β βββ Dockerfile
β βββ fly.toml
β βββ requirements.txt
βββ frontend/
β βββ src/
β β βββ pages/ # HomePage, AnalyzePage, LearnPage, DashboardPage
β β βββ components/ # React components
β β β βββ shared/ # Navigation
β β β βββ quiz/ # Quiz components
β β βββ hooks/ # Custom hooks
β β βββ utils/ # API, sample data, dashboard data
β βββ public/
β β βββ samples/ # Sample images and pre-cached overlays
β βββ package.json
β βββ vite.config.js
βββ notebooks/
β βββ brain_tumor_classifier.ipynb
β βββ pneumonia_classifier.ipynb
β βββ bone_fracture_classifier.ipynb
β βββ retinal_oct_classifier.ipynb
βββ assets/ # README images
All models use EfficientNet-V2-S pretrained on ImageNet with custom classification heads:
- Input: 224x224 RGB images
- Backbone: EfficientNet-V2-S (frozen early layers)
- Classifier: Dropout(0.3) β Linear(1280, 512) β ReLU β Dropout(0.15) β Linear(512, num_classes)
Visualizations generated by backpropagating target class scores and computing weighted activation maps from the last convolutional layer.
Image-specific explanations generated using Claude Haiku vision API, describing anatomical findings and Grad-CAM focus areas.
- Analyze Mode: Model selection, sample gallery, drag-and-drop upload, Grad-CAM visualization with opacity slider, AI explanations
- Learn Mode: Unlimited quiz, instant feedback with Grad-CAM, pre-cached explanations, session tracking
- Dashboard: Accuracy stats, per-model charts, weak area identification, session history
- Mobile responsive design
| Model | Dataset |
|---|---|
| Brain Tumor | Brain Tumor MRI Dataset |
| Pneumonia | Chest X-Ray Pneumonia |
| Bone Fracture | Fracture Multi-Region X-ray |
| Retinal OCT | Retinal OCT Images |
Full API documentation available at medical-image-api.fly.dev/docs
| Method | Endpoint | Description |
|---|---|---|
GET |
/health |
Health check |
GET |
/models |
List available models |
POST |
/predict/{model_name} |
Classification |
POST |
/predict/{model_name}/gradcam |
Classification with Grad-CAM |
POST |
/explain/{model_name} |
AI-generated explanation |
cd api
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
# Add weights to weights/
# Add ANTHROPIC_API_KEY to .env
python -m uvicorn app.main:app --reload --port 8000API docs: http://localhost:8000/docs
cd frontend
npm install
# Create .env.local for local development
echo "VITE_API_URL=http://localhost:8000" > .env.local
npm run devFrontend: http://localhost:5173
MIT
- Brain Tumor MRI Dataset by Masoud Nickparvar
- Chest X-Ray Pneumonia Dataset by Paul Mooney
- Bone Fracture Dataset by B. Madushani Rodrigo
- Retinal OCT Dataset by Kermany et al.
- EfficientNet-V2 by Google Research
- Grad-CAM by Selvaraju et al.













