Presentation

Search Abstracts | Symposia | Slide Sessions | Poster Sessions | Poster Slams

Semantic decoding of continuous language from non-invasive brain recordings

Poster A67 in Poster Session A, Thursday, October 6, 10:15 am - 12:00 pm EDT, Millennium Hall
Also presenting in Poster Slam A, Thursday, October 6, 10:00 - 10:15 am EDT, Regency Ballroom

Jerry Tang¹, Amanda LeBel¹, Shailee Jain¹, Alexander Huth¹; ¹University of Texas at Austin

A brain-computer interface that decodes continuous language from non-invasive recordings would have many scientific and practical applications. Currently, however, decoders that reconstruct continuous language use invasive recordings from surgically implanted electrodes, while decoders that use non-invasive recordings can only identify stimuli from among a small set of letters, words, or phrases. We introduce a non-invasive decoder that reconstructs continuous natural language from cortical representations of semantic meaning recorded using functional magnetic resonance imaging (fMRI). To overcome the low temporal resolution of fMRI, we used a Bayesian approach that combines a neural network language model and a voxel-wise encoding model. The language model generates linguistically coherent word sequences, and the encoding model predicts how the brain would respond to each sequence. Our decoder then identifies the most likely stimulus by comparing the predicted brain responses to the recorded brain responses. Given novel brain recordings, our decoder generates intelligible word sequences that recover the meaning of perceived speech, imagined speech, and even silent videos, demonstrating that a single language decoder can be applied to a range of semantic tasks. To study how language is represented across the brain, we tested the decoder on different cortical networks, and found that natural language can be separately decoded from multiple cortical networks in each hemisphere. To test whether decoder predictions are modulated by attention, we instructed subjects to attend to a different speaker for each repeat of a multi-speaker stimulus, and found that the decoder selectively reconstructs the attended stimulus. Finally, as brain-computer interfaces should respect mental privacy, we tested whether successful decoding requires subject cooperation. We found that decoders trained on cross-subject data performed substantially worse than decoders trained on within-subject data, suggesting that subject cooperation is necessary to train the decoder. Further, we found that subjects could consciously resist decoding of perceived language by performing a different cognitive task, suggesting that subject cooperation is also necessary to apply the decoder. Our results demonstrate that continuous language can be decoded from non-invasive brain recordings, enabling future multipurpose brain-computer interfaces.

Topic Areas: Computational Approaches, Control, Selection, and Executive Processes