NA-MIC Project Weeks

Back to Projects List

Multimodal tumor classification of RICE in Kapaana - integrating VLMs

New contrast-enhancing lesions following treatment of intracranial tumors may reflect either true tumor recurrence or radiation-induced contrast enhancements (RICE). Distinguishing between these entities remains unreliable but is critical for subsequent treatment decisions.
We have developed and trained a multimodal deep learning model (RICE-Net) that achieves good performance in differentiating RICE from tumor recurrence using longitudinal MRI data (e.g., post-intervention, post-pseudoprogression) in combination with radiation treatment plans.
RICE-Net is currently integrated into Kapaana, providing a structured environment for processing imaging and radiotherapy data and enabling inference on these inputs within a clinical routine.
However, relevant clinical information, including pathology and radiology reports as well as medication plans, is not yet integrated, although it could substantially improve model performance and clinical utility. Corresponding multimodal extensions are currently under development and planned for integration into a new Kapaana workflow.im

However, such models cannot yet be integrated into Kapaana, as the platform currently does not support text processing or the integration of large language and vision–language models.

Extend Kapaana to support inference with vision-language models.
Enable ingestion, parsing, and tokenization of PDF-based clinical documents for use as model inputs.
Create the technical basis for multimodal models that combine imaging, textual, and structured clinical data.

Develop the necessary infrastructure within Kapaana for deployment and inference of LLM/VLM models.
Implement robust pipelines for extracting and tokenizing text from clinical documents and aligning them with imaging data.
Integrate textual and tabular clinical information with longitudinal MRI and radiation data in a unified multimodal model.
Evaluate the extended model on retrospective cohorts with respect to performance, robustness, and interpretability.
Prepare the system for subsequent prospective evaluation and use in interdisciplinary tumor board settings.

No response

No response