Presentation

Alexandra Emmendorfer^1,2, Lara Banovac³, Judith Holler^1,2; ¹Donders Institute for Brain, Cognition, and Behaviour, ²Max Planck Institute for Psycholinguistics, ³Tilburg University

In conversation, speakers coordinate speaking turns in a highly efficient manner, with most turn gaps lying in the range of 0 – 200 ms for question-response sequences (Stivers et al., 2009). These short intervals indicate that speakers must begin preparing their response before the end of the ongoing utterance, with experimental evidence corroborating this (Barthel et al., 2017; Barthel & Levinson, 2020; Bögels et al., 2015; 2020; Sjerps & Meyer, 2015). But studies to date have investigated responding to unimodal, speech-only utterances. The most common environment for conversation, however, is face-to-face interaction, where speaking turns are embedded in a rich visual infrastructure. One prominent visual signal is speaker gaze, but its potential effect on early response planning is entirely unknown. While some studies suggest that perceiving direct gaze captures attention and thus may delay response planning, others suggest it may speed it up (Burra & Kerzel, 2021). We here test these hypotheses. Corpus analyses of 5967 questions revealed that in the majority of questions, speaker gaze was directed at the addressee for the entire duration of the question, while 19.8% of questions overlapped with gaze shifts. The mean onset of the gaze shift away from the addressee was 184ms after question onset, and gaze returned to the addressee before the end of the question in the majority of questions. Turn gaps were shorter in questions with static direct gaze (median: 280 ms), compared to questions with gaze shifts, where questions that ended with averted gaze had longer turn gaps (median: 476 ms) than those that ended with direct gaze (median: 347.5 ms). As gaze was typically directed to the addressee within the first half of the question, it did not appear to be a reliable indicator of turn end, thus suggesting that its function in conversation may lie more in signaling that a response is expected at all (Stivers & Rossano, 2010). We developed two experiments (online behavioral, EEG) to experimentally test the influence of speaker gaze on response time and EEG correlates of response preparation. In both behavioral and EEG experiments, participants respond to 240 polar questions (yes/no) by means of button press. Questions are asked by an avatar, whose gaze was manipulated in three conditions: static direct (SD), dynamic direct (DD), and dynamic averted (DA). In the SD condition, the avatar’s gaze remains directed at the participant for the duration of the question. In both dynamic conditions (DD and DA), the avatar’s gaze started from an intermediate starting point (15° averted) and either shifted toward the participant (DD) or was further averted (DA) at the beginning of the question. In line with the corpus data, we hypothesize our manipulation to affect response times, with static direct gaze resulting in the fastest, and dynamic averted gaze the slowest responses. EEG analyses will examine the lateralized readiness potential (LRP), providing an indicator of how early the hypothesized differences in response preparation arise. Data collection will launch in June, and we expect to present preliminary results at the SNL meeting.

Topic Areas: Perception: Speech Perception and Audiovisual Integration, Signed Language and Gesture