SwinTF3D: A Lightweight Multimodal Fusion Approach for Text-Guided 3D Medical Image Segmentation
By: Hasan Faraz Khan, Noor Fatima, Muzammil Behzad
Potential Business Impact:
Lets doctors find body parts using words.
The recent integration of artificial intelligence into medical imaging has driven remarkable advances in automated organ segmentation. However, most existing 3D segmentation frameworks rely exclusively on visual learning from large annotated datasets restricting their adaptability to new domains and clinical tasks. The lack of semantic understanding in these models makes them ineffective in addressing flexible, user-defined segmentation objectives. To overcome these limitations, we propose SwinTF3D, a lightweight multimodal fusion approach that unifies visual and linguistic representations for text-guided 3D medical image segmentation. The model employs a transformer-based visual encoder to extract volumetric features and integrates them with a compact text encoder via an efficient fusion mechanism. This design allows the system to understand natural-language prompts and correctly align semantic cues with their corresponding spatial structures in medical volumes, while producing accurate, context-aware segmentation results with low computational overhead. Extensive experiments on the BTCV dataset demonstrate that SwinTF3D achieves competitive Dice and IoU scores across multiple organs, despite its compact architecture. The model generalizes well to unseen data and offers significant efficiency gains compared to conventional transformer-based segmentation networks. Bridging visual perception with linguistic understanding, SwinTF3D establishes a practical and interpretable paradigm for interactive, text-driven 3D medical image segmentation, opening perspectives for more adaptive and resource-efficient solutions in clinical imaging.
Similar Papers
Adaptive Transformer Attention and Multi-Scale Fusion for Spine 3D Segmentation
CV and Pattern Recognition
Helps doctors see spine details better in scans.
Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models
CV and Pattern Recognition
Helps doctors understand 3D body scans better.
Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
CV and Pattern Recognition
Helps doctors understand body scans better.