Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Role of Gesture in Second Language Comprehension: an fMRI study
Poster A55 in Poster Session A - Sandbox Series, Thursday, October 24, 10:00 - 11:30 am, Great Hall 4
This poster is part of the Sandbox Series.
Jiaxin Yan1, Victoria-Anne Flood1, Motoaki Sugiura2, Hyeonjeong Jeong1; 1Graduate School of International Cultural Studies, Tohoku University, Sendai, Japan, 2Department of Human Brain Science, IDAC, Tohoku University, Sendai, Japan
Language and gesture are closely intertwined to facilitate communication and comprehension. Previous research on first language (L1) has shown that gestures enhance comprehension, especially in complex contexts, involving sensorimotor brain areas (Cuevas et al., 2019). Recent behavioral studies on second language (L2) have indicated that gestures also improve L2 comprehension (Lin, 2021; Zhang et al., 2023). Two types of gestures, iconic and metaphoric, have been extensively examined. Iconic gestures, with their visual characteristics, help L2 learners understand and remember content by concretizing abstract concepts (Macedonia et al., 2011). Metaphoric gestures convey abstract meanings physically, mirroring cognitive processing of abstract language, and helping learners grasp more complex, intangible ideas (Steines et al., 2021). Both types provide visual and contextual cues that support language processing. Despite these insights, specific neural mechanisms underlying the enhancement of L2 comprehension through gestures remain underexplored. Our study aims to investigate these mechanisms by examining the brain activity in L2 learners exposed to speech with these gestures. We focus on sensorimotor and language-related brain regions, examining differences between L1 and L2 as well as iconic and metaphoric gestures. Following a similar design to Steines et al. (2021) and our pilot test, we selected 160 English sentences that have been reviewed and videotaped by two English native speakers. Japanese raters assessed each video for comprehensibility, imageability, and naturalness. These were translated into Japanese and videotaped by two Japanese actors, resulting in 320 video stimuli (160 in L1 and 160 in L2) lasting 5 seconds each. Stimuli consisted of four conditions: L1 with and without gesture, L2 with and without gesture. We manipulated gestures to express various target words, including nouns, verbs, adjectives, and multiword expressions, classifying them as abstract or concrete using concreteness scores from Brysbaert et al. (2014), to explore how gestures represent linguistic elements and impact comprehension. We will recruit 40 right-handed native Japanese speakers learning English as their L2. They will undergo two 20-minute fMRI sessions for L1 and L2, with stimuli counterbalanced across conditions. Using event-related paradigms, videos for each condition will be presented in a randomized order, followed by a comprehension probe question to test participants’ attention. After the fMRI sessions, we will conduct memory tests to evaluate how well participants remember words associated with gestures, assessing differences between conditions. We expect the with-gesture condition to show faster reaction times in comprehension probe questions and a higher accuracy rate in memory tests compared to the without-gesture condition, particularly in L2, as gestures compensate for linguistic limitations and enhance comprehension and memory of the content. We predict that fMRI results will show increased brain activity in the with-gesture condition in areas integrating gesture and speech, and in sensorimotor and language comprehension-related areas. We will also analyze differences in activation levels between iconic and metaphoric gestures, and between L1 and L2. Our findings will provide a deeper understanding of how different types of gestures facilitate language processing in L2 communication.
Topic Areas: Multilingualism, Multisensory or Sensorimotor Integration