Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Modeling the trajectory of language processing in neural state space
Poster Session C, Friday, October 25, 4:30 - 6:00 pm, Great Hall 3 and 4
Zhengwu Ma1, Jixing Li; 1City University of Hong Kong
Introduction. In recent years, dynamical system models have been increasingly adopted to explain neural population dynamics involved in functions such as decision-making, timing, and motor control (refer to Vyas et al., 2020 for a review). Language processing can similarly be conceptualized within a dynamical systems framework, where the lexicon is seen as regions within a state space, and grammar functions as attractors or repellers that constrain movement within this space (Elman, 1995). As demonstrated by Elman (1995), a simple recurrent neural network (RNN) can learn to differentiate lexical items as distinct regions within the hidden unit state space. The sequential dependencies among words in sentences are reflected by movement through this space as the network processes each word sequentially. These dependencies are encoded in the network’s weights, mapping inputs (the current state plus a new word) to subsequent states. These weights effectively implement grammatical rules that facilitate the processing of well-formed sequences, enabling the network to form valid predictions about upcoming words. The generality of these rules means that the network weights form attractors in the state space, allowing the network to respond appropriately to novel inputs, such as unfamiliar words in known contexts. Since Elman’s initial proposal of the language system as a dynamical system, empirical studies to validate this hypothesis have been limited. In this study, we adopt a dynamical systems approach to model sequential dependencies among words in sentences by tracking movement through the neural state space during naturalistic listening. Methods. We used a previously published magnetoencephalography (MEG) dataset where 36 participants (21 females, mean age = 24.2 years, SD = 10.4 years) listened to a 12-minute narrative in the scanner. We utilized LLaMA2 (Touvron et al., 2023), a widely-used open-source LLM, to extract the word embeddings for each word in the stimuli, along with the probability of each word given the preceding context. We then performed ridge regression to correlate these embeddings with source-localized MEG data within a left-lateralized language network. Subsequently, we averaged the MEG data across subjects, time-locked to each word from the significant spatiotemporal clusters. We reduced the dimensionality of both the word embeddings and the MEG data for each word to a two-dimensional state space using Principal Component Analysis (PCA). Finally, we used differential equations to model the temporal evolutions as the LLM and the brain process each word sequentially. Results. Our preliminary results revealed distinct clusters where auxiliary verbs, degree adjectives, and adverbs occupy different regions in the neural state space. Additionally, the transitions between words within the neural state space for each sentence, reflected as displacements in the neural state space, demonstrate similar patterns for similar grammatical rules. These findings provide initial evidence supporting Elman’s (1995) hypothesis of a dynamic systems approach to language processing in the brain.
Topic Areas: Computational Approaches, Speech Perception