Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Parameter-Efficient Tuning Outperforms Fine-Tuning in Aligning Language Models with Brain Activity
Poster Session D, Saturday, October 26, 10:30 am - 12:00 pm, Great Hall 3 and 4
Jingyuan Sun1, Xinpei Zhao2, Marie-Francine Moens3; 1KU Leuven, 2Insititute of Automation, Chinese Academy of Sciences
Tuning pretrained large language models (LLMs) has emerged as a highly effective strategy for solving diverse natural language understanding (NLU) tasks, often achieving empirical performance levels that closely approximate human performance. Traditionally, fine-tuning (FT) has been the primary method for adapting LLMs to specific tasks by adjusting their pretrained weights. While this approach can lead to superior task performance, it is resource-intensive, requiring substantial computational power and storage capacity. To address these challenges, various parameter-efficient tuning (PET) methods have been developed. These methods preserve the pretrained parameters while integrating additional ones to tailor the model to specific tasks. Advanced PET methods significantly reduce the computational demands of tuning while delivering performance levels comparable to traditional FT. Currently, FT and PET are the principal strategies for adapting LLMs to downstream tasks. Despite the successes of tuning LLMs in numerous NLU tasks, there is ongoing debate about whether these models truly understand the nuances of language or rely on surface-level heuristics. Prior research has used human brain recordings to compare the representations derived from untuned LLMs with those generated by the human brain during language processing. This approach involves comparing model-generated representations with those captured from the brain using advanced imaging techniques, such as functional magnetic resonance imaging (fMRI). A significant correlation between these sets of representations, referred to as brain alignment, suggests a meaningful overlap in how language is represented by both the LLM and specific brain regions during comprehension. Studies have shown that untuned pretrained language models can predict activity across broad areas of the brain involved in language understanding. However, the alignment of tuned LLMs, including those adjusted via FT or PET, with human brain activity remains largely unexplored. Our study addresses this gap by exploring the connections between PET and FT of LLMs in relation to brain mechanisms through neural encoding, aiming to predict neural responses to linguistic stimuli. We selected four LLM architectures, each in two parameter sizes, resulting in a total of eight models. Unlike prior work, our models were tuned for causal language modeling on texts that human subjects were exposed to during neural response recordings, ensuring both LLMs and humans engaged with identical material. The tuning involved full FT and three PET methods: LoRA, AdaLoRA, and IA3. After tuning, we used these models to create embeddings for the stimuli, which were then used in regression models to predict neural responses recorded by fMRI from two publicly accessible datasets. Our findings reveal that embeddings from PET-tuned models consistently outperform those from FT-tuned models in accurately predicting the brain's language network, a pattern that remains stable across changes in learning rate and model size. Interestingly, PET methods can sometimes achieve better brain alignment than even the original, untuned models. We also observe that overly specific stimuli can lead to embeddings that do not accurately predict brain activation patterns, even if these are the focal points for subjects during neuroimaging studies.
Topic Areas: Methods, Meaning: Lexical Semantics