Presentation

Mariel Gonzales¹, Kristina Backer¹, Antoine Shahin¹; ¹University of California, Merced

The Audiovisual Time-Flow Illusion (TFI) is a novel phenomenon in spoken language perception, whereby illusory “pausing” or “skipping” of the visual stimuli is perceived in temporally intact visual speech merged with temporally altered auditory speech. When brief pauses are inserted in or brief portions are excised from an auditory speech stream merged with an intact, congruent visual stream (A-pause/V-norm, A-skip/V-norm), individuals perceive the visual modality as “pausing” or “skipping,” respectively. Our previous behavioral study demonstrated that an illusory “pause” in the visual modality was perceived 35.4% of the time in the A-pause/V-norm condition, which is significantly different from the percentage of an illusory “pause” perceived in the unchanged A-norm/V-norm condition (5.3%, p < 0.001). Similarly, an illusory “skip” in the visual modality was perceived 48% of the time in the A-skip/V-norm condition, which is significantly different from the percentage of an illusory “skip” perceived in the unchanged A-norm/V-norm condition (2.2%, p < 0.001). Interestingly, when the stimulus manipulations were reversed—that is, when brief pauses were inserted in or brief portions were removed from the visual stream, while the auditory stream was kept intact (A-norm/V-pause, A-norm/V-skip)—the illusion was perceived significantly below chance level, < 19%. Taken together, these findings suggested that the auditory modality sets the pace of the visual modality during audiovisual integration of natural speech. To examine the neural mechanisms underlying the TFI, we performed an EEG experiment in which individuals were presented with TFI stimuli (A-pause/V-norm, A-skip/V-norm). Based on data from two subjects, results revealed the presence of alpha power bursts directly following each “pausing” and “skipping” abruption in the auditory speech stream, substantiated by a marked increase in alpha activity (8-12 Hz) approximately 400 to 500 ms following each abruption onset compared to the baseline unchanged condition. Given an n of 2, we cannot firmly conclude whether this alpha is auditory or visual in nature, however, the enhanced alpha power suggests a resetting of ongoing alpha activity to abruption, which may have triggered the illusion. In natural speech, mouth movements typically precede sound production in speech. The visual modality thus has a role in predicting acoustic speech signals as they unfold. The current findings suggest that the visual modality realigns its pace with that of the auditory modality in order to preserve its predictive impact on audiovisual speech processing.

Topic Areas: Perception: Speech Perception and Audiovisual Integration, Speech Perception