Presentation

Search Abstracts | Symposia | Slide Sessions | Poster Sessions | Poster Slams

Transformer language models partly predict retrieval interference effects

Poster B61 in Poster Session B and Reception, Thursday, October 6, 6:30 - 8:30 pm EDT, Millennium Hall

Tzu-Yun Tung¹, Jonathan R. Brennan¹; ¹University of Michigan

INTRODUCTION: Transformer language models successfully predict facilitatory interference effects for agreement processing [7]. We ask here whether they can capture the interaction between retrieval interference and expectations [2]. Surprisal values from GPT2 [1, 6] are compared against human EEG data for Mandarin noun-phrase (NP) ellipsis. We find that interference effects on the P600 are modulated by target predictability, but this modulation is not captured by GPT2. METHODS: Mandarin NP ellipsis requires retrieving an antecedent NP upon encountering a number-classifier sequence with matching semantic features. We manipulate the target antecedent NP to be either a semantic match (Grammatical) or mismatch (Ungrammatical) to the classifier. For example, the classifier CL-Jian (件) categorizes shirts and luggage ([+Jian, -Ben]), and CL-Ben (本) categorizes books ([+Ben, -Jian]). We modulate interference using a distractor NP that is either a semantic match (High Interference) or mismatch (Low Interference) to the classifier. Lastly, the lexical-semantic expectation of the antecedent is modulated by varying the main verb to be either compatible with both the antecedent and distractor NP (Low Expectation), or only the antecedent (High Expectation). For example, “bring” is compatible with “shirt”, “luggage” and “book”, but “wear” is only compatible with “shirt”. 44 sentence sets with this 2x2x2 design were presented with 104 fillers using RSVP to N=30 Mandarin speakers during EEG recording. Data were epoched −300 to 1000 ms around the critical number-classifier sequence, and artifacts removed with ICA and visual inspection (0.1%–19% removal rate). Statistical analysis was conducted on centro-posterior electrodes from 650–800 ms with a non-parametric permutation test [4]. Surprisal, -log2(Pr(w | context), of the critical word was computed from Chinese GPT2. RESULTS: EEG data show a three-way interaction between grammaticality, interference and expectation. In Ungrammatical conditions, the P600 was decreased in High vs. Low Interference, when Expectation was High only (678–748 ms; p < .05). We thus replicate the facilitatory interference [8] with an additional modulation by expectation. GPT2 shows only a two-way interaction between grammaticality and interference (F(1, 43) =36.74, p < .001), without an Expectation modulation (F(1, 43) = 1.77, p = 0.19). In Ungrammatical conditions, High Interference shows lower surprisal than Low Interference, regardless of Expectation (Low-High = 0.876, p < .01). No reliable differences surfaced for Grammatical conditions (Low-High = 0.028, p = 0.99). Indeed, a single-trial analysis found no reliable contribution of GPT2 surprisal to the P600 response, consistent with e.g. [3]. CONCLUSION: Although Transformer language models predict facilitatory interference effect in Mandarin ellipsis processing, they do not capture the modulation of expectation on interference. Such modulation is observed with EEG data and follows from a cue-based retrieval theory with preferential cue-weighting [5]. Specifically, facilitatory interference only surfaces when the preferred structural cues are neutralized by prediction error caused by a highly predictable antecedent mismatching the retrieval cues. The across-the-board facilitatory interference effect by GPT2 suggests a lack of such cue preference. Supporting figures, tables, and references are available at: https://drive.google.com/file/d/1ms_egozDfuluvEUDtO2w0Uw9r2bkv_-Q/view?usp=sharing

Topic Areas: Syntax, Computational Approaches