Presentation
Search Abstracts | Symposia | Slide Sessions | Poster Sessions | Lightning Talks
The dynamic modulation of multimodal cues in real-world language comprehension
Poster C107 in Poster Session C, Wednesday, October 25, 10:15 am - 12:00 pm CEST, Espace Vieux-Port
Gabriella Vigliocco1, Ye Zhang1, Antonia Jordan-Barros1, Diego Frassinelli2; 1University College London, 2University of Konstanz
Face-to-face communication contains rich multimodal cues: gestures serve pragmatic, semantic (meaningful gestures) or emphatic (beats) functions. Previous EEG studies have shown that both L1 and L2 comprehenders integrate and process gestures during comprehension (Özyürek, 2014 for review). Zhang et al. (2021) found that gestures and discourse-level linguistic information were dynamically integrated in the same N400 time-window. Meaningful gestures reduced N400 effects for less predictable words in context, suggesting that they were used to predict the meaning of upcoming words. However, beat gestures co-occurring with less predictable words were associated with larger N400 amplitudes. Here, we assess whether effects of gesture are also observed in naturalistic stimuli where the comprehender is not the addressee (i.e., participants watching video-clips of two adults talking about various objects). This contrasts with previous studies in which participants watched video-clips of an individual speaker (typically an actor/actress). 30 English-speaking participants watched 36 videos (mean duration = 55.61s, mean number of words = 128) of dyadic conversations between friends extracted from the ECOLANG corpus (Gu et al., in preparation) while 32-channel EEG data was recorded. The linguistic predictability of the speaker’s words in the video-clips was quantified using semantic surprisal (negative log-transformed conditional probability of a word given its preceding context) computed using a bigram model trained on the ENCOW14-AX English corpus (Frank et al., 2015; Zhang et al., 2021). We annotated representational, pragmatic, beat, and pointing gestures. Lexical affiliates (i.e., words whose meaning was represented by the gesture) were coded for representational and pointing gestures (i.e., meaningful gestures). Words were coded for beat or pragmatic gestures if they co-occurred with these gestures. For each content word, we calculated the N400 as the averaged ERPs from all electrodes in the 300-600ms time-window (following Zhang et al., 2021). In a linear mixed-effects analysis (LMER), we predicted N400 based on surprisal, meaningful, beat, and pragmatic gestures. Control variables were baseline, word length, word order, and the relative electrode positions (X, Y, Z coordinates). Higher surprisal words induced more negative N400 (β=-0.040, t=-10.186, p<.001) as did words accompanied by beat gestures (β=-0.005, t=-3.440, p<.001); while meaningful gestures reduced the N400 (β=0.015, t=12.324, p<.001). Moreover, higher surprisal words combined with any type of gestures resulted in more negative N400: meaningful gestures (β=-0.014, t=-8.694, p<.001), beat gestures (β=-0.012, t=-5.953, p<.001), and pragmatic gestures (β=-0.011, t=-10.388, p<.001). Thus, in naturalistic contexts where a comprehender simply observes a conversation, multimodal cues are integrated with linguistic cues during processing, in line with previous studies (Zhang et al., 2021). In contrast to Zhang et al. (2021) who reported N400 reduction for high surprisal words accompanied by meaningful gestures, here higher surprisal words combined with any type of gestures resulted in more negative N400, indexing increased processing difficulty. This suggests that different neural and cognitive mechanisms engage in multimodal integration when observing (rather than participating as addressee) in conversation (see De Felice et al., 2021; Rice and Redcay, 2016).
Topic Areas: Signed Language and Gesture, Speech Perception