Together with Lasse Hansen and Riccardo Fusaroli from Aarhus University, I am working on developing text-, audio-, and combined text-and-audio models for language-based inference of psychiatric disorders in a multimodal and multiclass settings.
We have engineered a number of baseline models (using XGBoost on text and audio features) as well as transformers-based architectures, and trained them to predict clinical diagnoses for a cohort of individuals diagnosed with ASD, schizophrenia or major depressive disorders and matched controls. In our forthcoming manuscript, we show that performance in multiclass settings decreases significantly compared to binary (diagnosis vs. control) prediction problems, highlighting the need for more research (and larger datasets!) aimed at improving the specificity and the real-world clinical utility of language- and voice-based diagnostic approaches.
We also show that ensemble approaches (text + audio) can improve specificity in multiclass settings, efficiently leveraging information from multiple modalities at a low computational cost.