Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Comprehending semantic and syntactic anomalies from an LLM versus human interlocutor: An ERP study
Poster Session C, Friday, October 25, 4:30 - 6:00 pm, Great Hall 3 and 4
Xiaohui Rao1, Hanlin Wu1, Zhenguang Cai1; 1The Chinese University of Hong Kong
As large language models (LLMs) increasingly engage in human-like conversations, it is crucial to know how people comprehend language produced by LLMs compared to humans. When processing sentences, people would evaluate the input against their world, lexical-semantic and grammatical knowledge, with semantic anomalies eliciting an N400 effect and syntactic anomalies eliciting a P600 effect. The N400 and P600 effects can be modulated by our expectations about the interlocutor’s identity (e.g., Hanulíková & Carreiras, 2015; Van Berkum et al., 2008). However, it remains unclear whether people have similar expectations when interacting with LLM interlocutors and with human ones. LLMs are known for their superb grammatical competence, but may sometimes subject to illusion, producing nonsensical responses (Rawte et al., 2023). If so, people may exhibit reduced sensitivity to semantic anomalies (a smaller N400 effect) and increased sensitivity to syntactic anomalies (a larger P600 effect) when interacting with LLM interlocutors compared to human interlocutors. To address this question, we conducted two ERP experiments to investigate how people process semantic anomalies (Experiment 1) and syntactic anomalies (Experiment 2) when they were introduced that these sentences are produced by an LLM versus a human interlocutor. Experiment 1 used 40 semantically coherent/anomalous Chinese sentences (e.g., 小玲调完闹钟/月亮就睡觉了. Xiaoling set the clock/moon and went to bed), with 80 semantically coherent fillers, presented word by word at a fixed rate, followed by a sentence sensibility judgement task. Linear mixed-effects models showed a main effect of Sentence Type at the critical word (e.g., clock/moon), an N400 effect to semantic anomalies (p<.0001). There was a significant interaction between Sentence Type and Interlocutor Type (p<.05), such that participants exhibited a reduced N400 effect when interacting with perceived LLM versus human interlocutors. This indicates people are less sensitive to semantic anomalies produced by LLMs, potentially due to expectations of LLMs’ higher chance of illusion. Experiment 2 was similar but focusing on syntactic anomalies, using 40 syntactically well-formed/anomalous sentences (e.g., 昨天/明天，小明吃过汉堡了，今天就不了.Yesterday/Tomorrow Xiaoming has eaten a hamburger, so today he doesn’t want to), with 80 well-formed fillers. Results showed a main effect of Sentence Type at the critical segment (e.g., has eaten) (p<.05) and an interaction with Interlocutor Type (p<.05), such that a P600 effect was only observed for syntactic anomalies from LLMs, not humans. This suggests insensitivity to syntactic anomalies from humans but increased sensitivity to those from LLMs, possibly because people don’t expect LLMs to produce syntactically deviant sentences and are more tolerant to humans in this regard. Our findings provide novel insights into how people process language produced by LLM interlocutors compared to human interlocutors. The reduced N400 to semantic anomalies and increased P600 to syntactic anomalies potentially reflect people’s awareness of LLMs’ potential hallucinations and near-perfect grammatical competence, respectively. This aligns with previous findings that people’s expectations about their interlocutors can modulate the N400 and P600 effects.
Topic Areas: Meaning: Discourse and Pragmatics, Meaning: Lexical Semantics