Computer Science > Computer Vision and Pattern Recognition
Title:ORGANIC3D: Grounding 3D CT Vision Language Models in Organ Level Image Text Coupling
Abstract:Vision language models (VLMs) have achieved strong performance in 3D computed tomography (CT) classification, but high accuracy does not reflect whether their predictions rely on the correct anatomical evidence. We introduce ORGANIC3D, a framework for evaluating and improving organ level image text alignment through controlled anatomical interventions. Our benchmark assesses 12 vision encoder checkpoints and 7 generative VLMs using matched patient comparisons, organ erasure, lesion replacement, and laterality perturbations, revealing substantial alignment failures even among top performing models. To learn organ level image text correspondence, ORGANIC3D aligns each finding sentence with the anatomy aware image features and trains on edited scans to anchor this alignment in the organ's image evidence. We further extend this framework to generative VLMs, promoting evidence dependent question answering and report generation. Trained exclusively on public data, ORGANIC3D achieves SOTA classification AUROC on CTRATE, Merlin, and the heldout RADChestCT dataset and reaches 77% laterality accuracy, compared with at most 66% for prior models. Its generative VLM matches or exceeds all evaluated publicly released VLMs on visual question answering, achieves the highest report accuracy, and retracts 96 to 99% of target organ findings after organ removal, versus a maximum of 69% for other models. Together, these results establish ORGANIC3D as a framework for testing whether 3D CT vision language models ground their predictions in anatomical evidence and for improving organ level alignment alongside predictive accuracy.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
|---|---|
| Cite as: | arXiv:2611.23529 [cs.CV] |
| (or arXiv:2611.23529v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23529 arXiv-issued DOI via DataCite (pending registration) |
Submission history
Access Paper:
- HTML (experimental)
Additional Features
- Audio Summary
Current browse context:
References & Citations
- Google Scholar
- Semantic Scholar
BibTeX formatted citation
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.