Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Comparing LLM layerwise activation with EEG dynamics during language comprehension with RSA
Poster B7 in Poster Session B, Friday, October 25, 10:00 - 11:30 am, Great Hall 4
Jonathan Brennan1, Maria Figueiredo1, Gabriel Mora1; 1University of Michigan
INTRODUCTION: Large Language Models (LLMs) have emerged as a compelling in silico model for aspects of human language processing. Statistical alignment between human brain signals and model internal states provide evidence supporting some shared representational properties between these systems. Representational Similarity Analysis (RSA) is one tool to probe those internal states by: (i) comparing model internal activations for each word in some linguistic input, yielding a matrix M of pairwise similarities, (ii) repeating that step for human neural activations yielding pairwise similarity matrix N, and (iii) comparing the similarity of M to N as a measure of model-to-human alignment. Here we use RSA to test the strong hypothesis that that layerwise activation in a LLM align monotonically with the temporal dynamics of human neural responses recorded with EEG. METHODS: Data come from an open EEG dataset of N=33 English users who listened to a 12 minute audiobook while signals were recorded from 61 electrodes at 500 Hz (0.1-200 Hz pass band; linked mastoid reference). Data are epoched at word onset and eye blink artifacts removed with ICA; epochs/channels with excessive noise are marked with the autoreject algorithm. We compute word-to-word pairwise spatial correlation between epochs at 12 timepoints (-0.1-1 s). The model is GPT2, a pre-trained 12-layer transformer which was presented with the same story text as the human participants. We compute word-by-word pairwise correlation between activations at each layer, after accounting for epochs excluded in the human data. Finally, we compare the cosine similarity of the human and model correlation matrices, yielding a time-series of RSA values per layer per dataset; these are submitted to a hierarchical bayesian regression for statistical analysis; <timepoint,layer> pairs are statistically reliable if the 95% credibility interval excludes zero. RESULTS: Reliable increases in similarity relative to baseline are observed beginning around 200 milliseconds in model layer 6; similarity values then increase across layers 7 through 10 and then decline, matching previous reports with MEG. Interestingly, similarity increases across a wide time-window, with even later layers showing high similarity at early time-points. A hint of layer-to-time alignment emerges when examining correlation peaks, which shift from ~200 ms in layers 6 and 7 to ~500 ms by layer 10, although peak analysis is notoriously noisy. CONCLUSIONS: RSA analysis shows similarity in layer-wise activation in GPT2 and human EEG signals, confirming prior reports with fMRI, MEG, and ECoG. This similarity emerges in middle layers of the networks and spans a broad timewindow from 200 to >800 ms with little evidence that earlier layers better correspond, on the whole, to human responses at earlier timepoints. Different dynamics between model and human EEG signals point to different organizational principles for the representational spaces in these systems which plausibly reflect architectural differences that pose limits on how well LLMs can aid in understanding human neural systems. Human neural language circuits, for example, combine top-down and bottom-up mechanisms, whereas transformer LLMs like GPT2 are solely feed-forward.
Topic Areas: Computational Approaches, Syntax and Combinatorial Semantics