Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Deciphering parallel and hierarchical processes in multisensory gesture-speech integration across diverse semantic states
Poster Session A - Sandbox Series, Thursday, October 24, 10:00 - 11:30 am, Great Hall 3 and 4
This poster is part of the Sandbox Series.
wanying zhao1,2, Jiawang Yang1,2, Yanni Cui1, Zhouyi Li1,2, Yi Du1,2,3,4; 1Institute of Psychology, Chinese Academy of Sciences, Beijing, China., 2Department of Psychology, University of Chinese Academy of Sciences, Beijing, China., 3CAS Center for Excellence in Brain Science and Intelligence Technology, Shanghai, China 200031, 4Chinese Institute for Brain Research, Beijing, China 102206
Introduction: In natural communication, messages are frequently conveyed through misaligned multimodal sources, encompassing both verbal speech and non-verbal gestures. Previous studies have demonstrated that preceding gestures facilitate top-down semantic predictions of the upcoming speech. However, a comprehensive understanding of how sensorimotor gestures and verbal speech converge and distill into coherent concepts, and how subsets of these signals transform, remains elusive. Quantifying the information from both sources and their interaction using information-theoretic metrics of entropy, mutual information and information gain, the present study traces the neural responses with High-temporal event-related potentials as gesture precedence speech of various semantic states. Methods: This study manipulated gesture and speech semantic states across nine paradigms, totaling 1368 pairs. Gesture priming on speech was induced by presenting speech after gesture offset. Gesture and speech information were quantified separately using entropy metrics. Bidirectional influence was measured with information gain, convergence with unit information gain, and integration with mutual information. Thirty participants indicated voice gender while fixating on a screen. Representational Similarity Analysis (RSA) correlated neural responses with gesture entropy, speech entropy, and mutual information to explore integration dynamics. Cluster-based permutation tests identified temporal differences in signal transformation during integration by correlating amplitudes with information gains. Results: The study identified distinct neural structures for integrating gesture and speech. Gesture processing occurred earlier (0-250 ms), while speech processing showed a late positive component (650-684 ms). Integration, seen in semantic congruency, happened during N400 (442-542 ms). Temporal patterns correlated with gesture entropy (r=-0.0252, p=.001), speech entropy (r=0.0237, p=.002), and mutual information (r=0.0226, p=.003), indicating modulation by gesture-speech information. For gesture information gain, significance was noted only with balanced gesture-speech information. Speech influenced gesture early (N1, r=-0.51, p=.027), later with increased information (LPC, r=-0.51, p=.027). Significance in speech information gain was observed when auditory stimuli reached semantic states. Early effects (N1-P2) were seen in speech IP conditions, while later effects (N400) emerged during speech after_IP conditions. In both cases, clusters tended to occur earlier with increased gesture semantics. Convergence of gesture-speech information, indexed by unit information gain, was significant except in one condition. Convergence occurred if either source reached semantic states, with clusters （spanning from 200-500ms）revealing delayed responses as the information of either source increased. Conclusion: We present distinct neural engagements during the self-contribution, bi-directional influence, convergence, and integration of gesture-speech information. (1) The top-down influence of gesture interacts with the bottom-up processing of speech, leading to a unified comprehension where both processes are progressively engaged. (2) Throughout this interaction, there exists a bi-directional influence between gesture and speech, albeit in distinctly different manners. While gesture influences speech after speech has reached its semantic stage, speech influences gesture when there is equilibrium between gesture and speech information. (3) The integration of gesture and speech information, exemplified by the amalgamation of both sources of information, occurs when there is equilibrium between gesture and speech information. Conversely, the convergence of gesture and speech information, demonstrated by unit information gain, takes place when either gesture or speech reaches its semantic stage.
Topic Areas: Meaning: Lexical Semantics, Multisensory or Sensorimotor Integration