Generating fMRI Images of Neural Activations using Ranked Voxel Mask and Volumetric Conditional GAN

Abstract

We propose a novel approach of estimating brain activity for a stimulus as measured by fMRI using a volumetric conditional Generative Adversarial Network (GAN) model. Reliable predictions of neural responses are invaluable in understanding the systematic processes involved in processing a language by the brain. To understand how the human brain encodes the information necessary to process a language, we can start by modeling a computational system that can imitate the brain activity in response to a stimulus. Recent advancements in GAN models have shown significant improvements in generating data for semi-supervised learning, especially for denoising and segmentation. Also, 3D convolution neural networks have proven to be highly effective in addressing computer vision problems consisting of 3D volumes including neural segmentation which is a challenging task considering that the fMRI image of the brain consists of slices along the horizontal, sagittal, and coronal axis that are different from each other. We investigate the ability for 3D conditional GANs to estimate fMRI response in response to noun stimulus. We use the signal to noise ratio to generate the latent space and noun/verb co-occurrences as static embeddings for the conditional generator. Our evaluation employs a stability scoring function for ranking voxels using signal to noise ratio in the training fMRI images based on the 3D spatial signal to noise ratio of voxel response. We use these ranked voxels to mask the output of the volumetric GAN to localize the regions of the brain activity. We present quantitative and qualitative analysis demonstrating that the proposed method can estimate neural response with greater accuracy than random chance, and we further compare the performance of the proposed algorithm against a traditional linear regression technique.