Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Introducing A Real-Time Measure of Comprehension During Natural Speech Listening
Poster Session B, Friday, October 25, 10:00 - 11:30 am, Great Hall 3 and 4
Irmak Ergin1, Jill Kries1, Shiven Gupta, Laura Gwilliams1; 1Stanford University
Speech comprehension has been described as an effortless, robust process; yet, in real- world contexts, it is common for a listener to misunderstand, or to fail to derive meaning entirely. To experimentally measure speech comprehension, researchers have used post-hoc measures such as comprehension questions, self-ratings, and summarization. These measures fail to capture the time resolution of comprehension, which emerges dynamically as speech unfolds. Challenges in behaviorally measuring real-time comprehension have led to an impoverished description of the neural processes supporting online comprehension. We designed and tested a novel way of measuring real-time speech comprehension during naturalistic listening. We built a slider device that synchronizes with behavioral and neural recording software, and provides millisecond read-out. Fourteen native English participants listened to audiobook segments while providing continuous comprehension ratings using the slider. To inject variability in comprehension success, speech segments were presented 1-5 times the original speed. We tested the time-resolved slider against three established methods of assessing speech comprehension: (i) 10-point scale rating; (ii) a multiple choice question accuracy; (iii) a written summary. We evaluated the cosine similarity between the segment and written summary using vector embeddings from GloVe and BERT. Additionally, we obtained working memory and auditory acuity measures using the Digit Span and Digit-In-Noise tasks respectively, to account for by-subject variability in comprehension scores not due to comprehension per se. To validate our time-resolved measure against traditional post-hoc measures, we used Mixed Effects Linear Regression with real-time comprehension as the independent variable, three post hoc measures as fixed effects, and working memory and auditory acuity as random slopes over subject. We found significant effect of self-rating and written summary similarity, and non-significant trend for multiple-choice accuracy. By-subject measures were not significant predictors. We found that multiple-choice accuracy was consistently above chance, even when the participant reported understanding nothing, which we attribute to difficulty in designing question sets that are independent from story context and real-world knowledge. Furthermore, we found that written summaries were significantly more similar to later portions of the spoken segment than earlier portions, which highlights a bias limitation of this approach. Overall, our findings demonstrate the validity of our novel time-resolved comprehension measure. They suggest that it is possible to derive an online behavioral measure of speech comprehension in real-time - the lack of which has been a major limitation of naturalistic language studies. This overcomes numerous limitations of static post-hoc assessments, including recency bias for summarization, and challenges of multi-choice question design. We propose that this continuous speech comprehension measure can be effectively integrated with neuroimaging techniques, offering a valuable tool for future research on dynamic processes during naturalistic listening.
Topic Areas: Methods,