Presentation

Jin Dou¹, Andrew Anderson¹, Edmund Lalor¹; ¹University of Rochester

The human brain understands natural speech at rates of around 120–200 words per minute. A well-known signature of this process is the N400 electrophysiological component – a prominent centroparietal negativity (N) that peaks 400ms after word onset. The time course of the N400 is very reliable and has been fleshed out over thousands of studies. Indeed, the N400 is so well established that one of its base assumptions – that peak response is time-locked to word onset time – often goes unquestioned. However, this strict time-locking of brain responses to word onsets may be more an experimental convenience than a biological reality. This is because different words take different times to recognize: consider “at” and “atmosphere”, and word recognition is constrained by context: consider “The at” vs “The atmosphere”. Discovering and predicting such putative non-linearities in word recognition is especially important for building accurate brain models of natural speech comprehension. To this end, we present initial work on a neurophysiological model estimating non-linear processing dynamics hidden beneath the natural speech N400. We trained the new non-linear modelling approach on three pre-collected natural speech comprehension EEG datasets with a total of 73 participants and 1.75 hours speech. Following previous work, we modeled the N400 amplitude using lexical surprisal, estimated using a cutting-edge deep neural network language model (GPT-2). This approach assumes the N400 reflects the signal computed when the brain misanticipates an upcoming word’s identity. However, unlike previous models that linearly fit EEG data with surprisal impulses time-aligned to word onsets, we fit EEG to a response distribution spanning 1.5 secs. Critically, this enables predicted EEG response profiles to vary between words, rather than being constantly the same. To fit response distributions we optimized a deep learning network to predict natural speech EEG from a time-series of lexical surprisal entries. The network contains two consecutive modules. The first module maps lexical surprisal onto the response distribution layer via non-linear layers. The second module implements a linear “temporal response function” that maps the predicted lexical surprisal response distribution to EEG data. The network was optimized using backpropagation of RMS error of EEG prediction across both modules. To evaluate the non-linear model, we contrasted how accurately it could predict a testing EEG dataset with the traditional approach of modelling surprisal as impulses at word onset. The non-linear model on average yielded 16% improvement of prediction accuracies (signedrank=640, p-value=1e-4,n=73). To interpret the non-linear model, we examined how estimated peak response times covaried with lexical surprisal. We found that unexpected words have delayed responses relative to word onset that could not be accounted by factors such as word length (partial r = 0.46, controlling word length). This suggests brain responses to surprising words may be magnified not only in amplitude but also in time. More generally these results provide early evidence that: (1) the natural speech N400 is underpinned by non-linear dynamics and thus may be less rigid than thought. (2) Non-linear N400 dynamics can be learnt from natural speech EEG using artificial neural networks.

Topic Areas: Computational Approaches, Meaning: Lexical Semantics