Fusion of Multi-Modal Neuroimaging Data and Association With Cognitive Data





Citation of Original Publication

M. D. LoPresto, M. A. B. S. Akhonda, V. D. Calhoun and T. Adali, "Fusion of Multi-Modal Neuroimaging Data and Association With Cognitive Data," 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), Rhodes Island, Greece, 2023, pp. 1-5, doi: 10.1109/ICASSPW59220.2023.10193147.


© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.



Efforts to develop a deeper understanding of the human brain benefit from the joint analysis (fusion) of multiple sources of data to exploit the complementary information in each modality. A number of Blind Source Separation (BSS) techniques have been developed for data-driven analysis and fusion of neuroimaging data, but only recently have fusion frameworks emerged that consider cognitive data alongside the neuroimaging data. In our approach, we first apply transposed Independent Vector Analysis (tIVA) across three neuroimaging modalities to extract subject covariations that are then associated with cognitive data. The tIVA step allows full interaction of the neuroimaging modalities to discover cross-modality relationships. We cluster the data based on the subject covariations to find the cognitive scores that offer significant discrimination between each pair of clusters. Our approach allows full interaction of multiple neuroimaging modalities and makes direct association with the cognitive data, identifying the Brief Assessment of Cognition in Schizophrenia Composite Score (BACS CS), Hopkins Verbal Learning Test (HVLT) and the Neuropsychological Assessment Battery (NAB) Mazes Test as being strongly associated with the cross-modality connections.