Presentation

Search Abstracts | Symposia | Slide Sessions | Poster Sessions | Poster Slams

Perception of Audiovisual Pairings Among Heritage Speakers of Spanish

Poster B30 in Poster Session B and Reception, Thursday, October 6, 6:30 - 8:30 pm EDT, Millennium Hall

Zunaira J. Iqbal¹, Antoine J. Shahin¹, Kristina C. Backer¹; ¹University of California, Merced

INTRODUCTION. Visual information can readily influence our perception of speech, as exemplified by the famous McGurk illusion (McGurk & MacDonald, 1976). It has also been found that bilinguals are more influenced by visual mouth movements than their monolingual counterparts when comprehending speech in their second language (Marian et al., 2018). However, it is not well understood how the presence of visual information influences speech perception in bilingual individuals who speak two languages that categorize certain phonemes differently. Here we investigated how the presence of visual mouth movements influence Spanish-English heritage speakers’ percepts along a /va/-/ba/ continuum. The phonemes /v/ and /b/ were selected since Spanish phonology maps the orthographic /v/ onto the /b/ phoneme, whereas English makes a clear distinction between these two phonemes. METHODS. The auditory speech continuum from /va/ to /ba/ comprised five consonant vowel (CV) stimuli. The video stimuli consisted of a speaker producing either /va/ or /ba/. The 5 auditory CVs were paired with each video, producing 10 possible audiovisual combinations (5 CVs x 2 videos). Spanish-English bilingual participants (n=27) were presented with Auditory-Only and Audiovisual trials and reported what they perceived on each trial. Since Spanish phonology maps /v/ onto /b/, we hypothesized that Spanish-English bilinguals would perceive /ba/ more often than /va/, even towards the /va/ end of the /va/-/ba/ continuum. We also hypothesized that the presence of visual mouth-shape information would bias participants’ perception of speech tokens along the continuum towards the visually conveyed phoneme. RESULTS. Overall, the CV stimulus step (i.e., along the /va/-/ba/ continuum), but not the AV condition (i.e., visual-va, visual-ba, or auditory-only), significantly affected perception (p < 0.001). Collapsing across AV conditions, participants perceived /ba/ more often as the continuum progressed from /va/ towards /ba/, with the most /ba/ responses toward the /ba/-end of the continuum. There was also a significant interaction between the CV stimulus step and AV condition (p = 0.009). This interaction was driven by more /ba/ responses occurring when auditory stimuli at the /va/-end of the continuum were paired with visual-ba, compared to visual-va or the auditory-only condition. CONCLUSION. These results suggest that the presence of visual-mouth movements play a significant role in altering bilinguals’ speech perception, such that participants will often rely upon the viseme (i.e., visual mouth shape) when reporting what they heard. Follow-up experiments will further explore this effect and compare across Spanish-English bilinguals and English monolinguals to characterize any group differences due to language experience. Additionally, we will investigate whether individual differences in language history demographics among bilingual listeners influence the extent to which visual information modulates their auditory speech perception.

Topic Areas: Multilingualism, Perception: Speech Perception and Audiovisual Integration