Chest X-Ray Diagnosis with Joint Report Generation and Lesion Localization Using Multimodal Vision-Language Models

Lung diseases remain a major global health concern, and chest X-ray imaging is a critical tool for early diagnosis and treatment. To support radiologists, automatic computeraided diagnosis (CAD) systems must detect abnormalities and accurately localize them within the image. However, developing such systems faces key challenges: annotated bounding box labels are scarce, and many models lack clear interpretability and visual grounding to support clinical trust. To address these gaps, we propose a unified multimodal diagnostic framework that combines a DenseNet-121 classifier with Grad-CAM-based pseudo-labeling and a powerful vision-language model (Qwen2.5-VL-7B-Instruct). The DenseNet classifier is trained on pneumonia and pneumothorax cases from the MIMIC-CXR-JPG dataset, with Grad-CAM used to extract pseudo bounding boxes by thresholding activation maps. These pseudo-labels, combined with limited expert annotations, supervise the fine-tuning of Qwen2.5-VL, enabling it to generate structured radiology reports alongside spatial coordinates for pathological findings. Experimental results demonstrate that incorporating pseudo-labeled data significantly boosts localization performance, achieving 26.09% recall at IoU ≥ 0.25 compared to 4.35% using real labels alone. Meanwhile, report generation quality remains strong, with BLEU-4 and ROUGE-L scores comparable to state-of-the-art systems and with faster inference than models like LLaVA. This work illustrates how Grad-CAM-based pseudo-labeling and multimodal vision-language models can enhance interpretability and efficiency, providing a practical pathway toward integrated, explainable AI for chest X-ray diagnosis.