Presentation
Search Abstracts | Symposia | Slide Sessions | Poster Sessions | Lightning Talks
Neural encoding of distinct linguistic hierarchies across different languages
There is a Poster PDF for this presentation, but you must be a current member or registered to attend SNL 2023 to view it. Please go to your Account Home page to register.
Poster B32 in Poster Session B, Tuesday, October 24, 3:30 - 5:15 pm CEST, Espace Vieux-Port
Hongbin Qing1, Jixing Li1; 1City University of Hong Kong
Introduction. Language comprehension involves understanding a hierarchy of linguistic units, including phonemes, words, phrases, sentences, and paragraphs. The encoding mechanism of these units during speech comprehension has been a subject of ongoing debate. Hasson et al. (2015) proposed a hierarchy along the temporal-parietal axis, with increasing temporal receptive windows supporting the processing of phonemes and syllables to higher-level units such as words and phrases. However, limited research has been conducted in cross-language contexts. In the present study, we investigated the neural encoding of three languages (Chinese, English, French) across three linguistic hierarchies (word, sentence, paragraph) by scanning participants with functional magnetic resonance imaging (fMRI) while they listened to a naturalistic story. Methods. We used the openly-available Le Petit Prince fMRI Corpus (LPPC; Li et al., 2022), where 49 English, 35 Chinese, and 28 French native speakers listened to the audiobook The Little Prince in their respective native languages. We first applied voxel-wise inter-subject correlation (ISC; Hasson et al., 2004) to select the voxels that exhibited the highest correlations among subjects within each language group. We then utilized the Multilingual GPT model (mGPT; Shliazhko et al., 2022) to construct the word-, sentence-, and paragraph-level embeddings of the story in three languages. To estimate how these semantic features at different levels were represented in the brain across the three languages, we trained a regularized ridge regression model using the leave-one-out strategy (Huth et al., 2016). The model's performance was assessed by computing the correlation between the predicted responses and the actual blood-oxygen-level-dependent (BOLD) signals. At the group level, we performed non-parametric bootstrap tests on the prediction results of the three groups with 1,000 bootstraps to identify the cerebral voxels significantly responding to different linguistic units (p<0.05). Finally, we employed Multidimensional Scaling (MDS), a dimensionality reduction technique, to visualize the temporal dynamics of the semantic features at different levels. The fMRI signals derived from the significant voxels identified by the regression model were reduced to a 2-dimensional space using MDS. These reduced signals were then plotted against time to illustrate the temporal dynamics of plot-tracking in the three languages. Results and conclusion. We identified three distinct clusters within the left temporal lobe, spanning from more anterior to more posterior regions. These clusters corresponded to the semantic features at the word, sentence, and paragraph levels, respectively. This distribution pattern was found to be consistent across all three languages. The temporal dynamics of these semantic features revealed a higher degree of similarity in tracking sentence- and paragraph-level information across the three languages. This suggests a commonality in how these higher-level linguistic units are processed and represented in the brain, irrespective of the specific language being processed.
Topic Areas: Multilingualism, Computational Approaches