RoI-MedCap: Region of Interest-Based Medical Image Captioning with Multi-Stream Connector

Medical image captioning has gained significant attention due to the rapid advancements in Artificial Intelligence. However, existing research primarily focuses on global image captioning, lacking a mechanism for Region of Interest (RoI)-based captioning where users can specify an area and receive a caption centered on that specific region. In this paper, we propose a novel architecture with a vision encoder, a connector, and a Large Language Model (LLM) to generate captions for medical images with integrated RoI. We introduce a Multi-Stream Connector (MSC) to project visual features from a vision encoder to a representation that helps the LLM to generate captions centered on a specified region of an image indicated by a bounding box. We aim to generate captions with three aspects including the modality and structure, RoI analysis and lesion findings in RoI, and local-global relationship denoting impacts of findings in RoI to other regions. To achieve this goal, MSC incorporates three Cross Attentions focusing on three different aspects of generated captions. Our extensive experiments demonstrate that our method is well capable of generating captions highly aligned with human judgement, compared to existing related methods. The source code is available at https://github.com/alshahriarrubel/RoI-MedCap.