Presentation
Search Abstracts | Symposia | Slide Sessions | Poster Sessions | Poster Slams
Decoding the time course of acoustic, lexical, syntactic and semantic features with MEG during story listening
Poster B67 in Poster Session B and Reception, Thursday, October 6, 6:30 - 8:30 pm EDT, Millennium Hall
Shaonan Wang1,3, Jiajun Zhang1, Chengqing Zong1,2; 1Institute of Automation, Chinese Academy of Sciences, 2CAS Center for Excellence in Brain Science and Intelligence Technology, 3New York University
[INTRODUCTION] Language comprehension requires construction of multiple levels of representations, including perception (e.g., acoustic), lexical (e.g., frequency), syntax (e.g., parsing strategy), semantic (e.g., social), etc. Previous studies have identified cortical locations of neural activity related to these different levels of representations. However, the temporal sequence of processing from sensory input to language comprehension remains unclear. Here we address two research questions: 1) what type of features are encoded by the MEG signal; 2) how these encoded features emerge over time? [METHODS] Magnetoencephalography (MEG) activity from 12 Mandarin-speaking adults was recorded during listening of ~6 hours of stories. Here we consider 17 word-by-word features including 1 acoustic feature (i.e., audio envelope), 2 lexical features (i.e., word frequency and word length), 7 syntactic features (i.e., word position in a sentence, content word indicator, depth in a tree, distance from head word, node counts from top-down, bottom-up and left-corner parsing strategies), and 7 semantic features (i.e., social, vision, action, emotional arousal, emotional polarities, time and space with scores ranging from 1 to 7 which were crowdsourced to 30 participants). We conducted a data-driven decoding method which deciphers a specific feature from the MEG sensor data at each time point. To eliminate the effect of correlations among different features, we used the back-to-back ridge regression model to evaluate the accuracy of the joint decoding models in predicting each modeled feature. Group level significant results were calculated by using a one-sample permutation cluster test. [RESULTS] First, the acoustic feature was reliably decodable from a sequence of neural responses unfolding before ~100ms and after ~260ms after word onset. Second, lexical features were mainly decodable between ~180ms to ~460ms after word onset. Third, syntactic features of content word, tree depth, top-down and bottom-up parsing could be successfully decoded at a very long-time window from ~0ms to ~800ms after word onset; word position could be decoded at ~510ms; left-corner parsing was decodable around 290ms and 710ms; distance from head word was decodable around 240ms and 730ms. Lastly, semantic features of social and action could be discriminated around 450ms and 310ms respectively; vision feature was decodable during multiple small time-windows at 80ms, 250ms, 350ms, 450ms, 550ms; emotional arousal was decodable from 200ms to 430ms. We didn’t find significant effects of the time feature, space feature and emotional polarities. [CONCLUSIONS] We successfully decipher from MEG activity multiple levels of representations comprising lower-level auditory and lexical features to higher-level syntactic and semantic. The acoustic feature has a huge impulse on brain responses at each word onset, while lexical features are activated and reactivated during the whole-time window. This implies that lower-level features would have influences on word processing even if higher-level features are activated. Interestingly, syntactic features are encoded in a long time-window, while semantic features are only encoded at a certain time point, indicating different mechanism of syntactic and semantic processing.
Topic Areas: Computational Approaches, Meaning: Lexical Semantics