Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Online summarization of the past information in the human brain during natural language comprehension
Poster Session C, Friday, October 25, 4:30 - 6:00 pm, Great Hall 3 and 4
Jiahao Jiang1, Tengfei Zhang1, Shaonan Wang2,3, Xinran Xu1, Chunming Lu1; 1State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China, 2State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China, 3School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
One striking ability of the human brain is to comprehend the current input by incorporating past information, even for long stimuli that lasts for hours. Although the large language models (LLM) that simulate functions of the human brain can employ all (or long) information in prior contexts, this is unlikely to be the case for the human brain due to our limited cognitive resource. Thus, here we hypothesized that online summarization of the past information is the potential mechanism of our brains in integrating previous context into the current input. We predicted that the high-order brain areas might compress information based on meaningful boundaries (“Summary at Boundary”, SB), rather than on fixed amounts of stimuli or fixed length of time (“Summary at Fixed amount or time”, SF). To test the above hypothesis, we analyzed the Sherlock dataset (Chen et al., 2017). In this dataset, 17 participants were requested to watch a video and then freely recall as detailed as possible, while fMRI data were collected. There were 998 annotations (mean = 3.9s) describing the content of each video segment, and they were divided into 50 events according to meaningful boundaries (SB). As a comparison, we evenly divided these annotations every 20 segments (SF). After retrieving summaries for each chunk with GPT-4, we concatenated SB/SF summaries or equivalent length of previous annotations (“No Summary”, NS) to every annotation, and obtained contextualized semantic representations with a pretrained LLM Mistral-7B. With these three sets of representations, we separately built encoding models for each brain parcel with 10-fold linear regression, and obtained encoding performance by averaging correlation coefficients in test sets, which reflected how well each parcel represented the information. Finally, the encoding performance of each parcel was compared between SB, SF and NS conditions by paired T-test. We also tested the functional role of summarization by conducting Spearman correlation between encoding performance and the precisely recall ratio of participants (available in the dataset). The results showed that the encoding performance of SB was significantly better than NS in precuneus and visual cortex, and better than SF additionally in left MTG. For SF condition, however, there was no parcels encoded better than SB, and only one parcel encoded better than NS in right MTG. Moreover, encoding performance of SB and SF in precuneus was significantly associated with precisely recall ratio among participants, whereas NS did not. These results can be replicated with the traditional GPT-2 model. In conclusion, we found evidences that compressing previous contexts at meaningful boundaries by summarization is probably the mechanism of how human brain efficiently encode mass information. Participants that better integrated previous summaries could better memorize the content in detail, which indicated the functional role of online summarization.
Topic Areas: Meaning: Discourse and Pragmatics, Computational Approaches