Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Decoding personal mental images with an fMRI model of sentence semantics
Poster C1 in Poster Session C, Friday, October 25, 4:30 - 6:00 pm, Great Hall 4
Andrew Anderson1, Leonardo Fernandino1, Bill Gross1, Hernan Rey1, Jeff Binder1; 1Medical College of Wisconsin
People simulate personal experiences through imagination and understand descriptions of other peoples’ experiences via language. Although both processes are anchored in experience, it remains unclear whether autobiographical simulations and language comprehension are encoded in common brain systems. For instance, automated meta-analyses and popular renditions of the brain's language or semantic network often exclude medial cortical zones that underpin autobiographical simulations, whereas other fMRI studies have implicated these medial cortices in encoding semantic features of language. However, demonstrating semantic feature encoding in medial cortices does not entail that the same features underlie the content of autobiographical imagery. We hypothesized that this would be the case and that the encoding of autobiographical-imagery and sentence semantics would overlap in a shared neural feature space. To test the above hypothesis, we reanalyzed a mental imagery fMRI dataset: Outside the scanner fifty participants read twenty generic scenario cues (e.g. party, exercising, wedding) and vividly imagined themselves personally experiencing each scenario. Because different people have different experiences, it was reasonable to expect that they would bring different mental images to mind for the same cue. Participants rated each mental image on 20 sensory, motor, affective, social, cognitive and spatiotemporal features of experience (0-6 scale). Participants then underwent fMRI as they re-imagined the same scenarios in random order on written prompt. Preprocessing produced a single fMRI volume for each mental image per person. We then evaluated whether participants’ personal feature ratings could be decoded from the mental imagery fMRI data using a pre-trained cross-participant decoder built from a separate sentence reading fMRI data set. Such decoding would not be possible if autobiographical imagery and sentence semantics were encoded in separate neural systems or different feature spaces. The sentence fMRI data was collected from 14 different participants who read 240 sentences. The sentences were third person and 3-9 words long, e.g. “The child broke the glass”. Sentence semantics were modeled via crowd-sourced ratings of the same 20 features above. The semantic feature decoder was trained using ridge regression to map the entire cross-participant sentence fMRI dataset to reconstruct the 20 crowdsourced ratings for corresponding sentences. We then applied the pre-trained semantic feature decoder to reconstruct personal feature ratings from fMRI scans of mental images. We found that: (1) 11/20 feature ratings could be reconstructed from fMRI data in “autobiographical” medial cortices. This was evaluated by computing the correlation between reconstructed and observed features. (2) The reconstructed features reflected individual-differences in participants’ ratings. This was evaluated by selecting participant pairs and correlating their rating reconstructions with both their own and the other participants’ observed ratings, then testing which correlation coefficient was greatest. Participant pairs were discriminated with 70% accuracy, p<1e-4 (permutation test). Repeating this with individual feature vectors revealed that social- and speech-related features contributed most to discrimination. These results: (1) Indicate that experiential content of language semantics is encoded in the same cortical areas with the same representational codes as autobiographical simulation. (2) Demonstrate “zero-shot” decoding of personal mental states from fMRI data.
Topic Areas: Meaning: Lexical Semantics, Computational Approaches