M3H: Multimodal Multitask Machine Learning for Healthcare

danelchen

医疗多模态多任务机器学习

摘要
Artificial intelligence holds promise to fundamentally enhance healthcare. Developing an
integrated many-to-many framework leveraging multimodal data for multiple tasks is essential to
unifying modern medicine. We introduce M3H, an explainable Multimodal Multitask Machine
Learning for Healthcare framework that consolidates learning from tabular, time-series, language,
and vision data for supervised binary/multiclass classification, regression, and unsupervised
clustering. M3H encompasses an unprecedented range of medical tasks and problem domains and
consistently outperforms traditional single-task models by on average 11.6% across 40 disease
diagnoses from 16 medical departments, three hospital operation forecasts, and one patient
phenotyping task. It features a novel attention mechanism balancing self-exploitation (learning
source-task), and cross-exploration (learning cross-tasks), and offers explainability through a
proposed TIM score, shedding light on the dynamics of task learning interdependencies. Its
adaptable architecture supports easy customization and integration of new data modalities and
tasks, establishing it as a robust, scalable solution for advancing AI-driven healthcare systems.

https://arxiv.org/pdf/2404.18975