Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Computational operations of phrasal composition in the brain
Poster Session D, Saturday, October 26, 10:30 am - 12:00 pm, Great Hall 3 and 4
Shaonan Wang1, Songhee Kim2, Jeffrey Binder2, Liina Pylkkänen3; 1Institute of Automation Chinese Academy of Sciences, 2Medical College of Wisconsin, 3New York University
Specifying the computational operations underlying phrasal composition is crucial for developing neurobiological theories of language. While our understanding of when and where basic phrasal composition affects neural activity has matured in recent years (Pylkkänen, 2019), we still lack detailed computational characterizations. Additionally, research has not yet connected basic composition with theories of distributed semantic features of single concepts (Binder et al, 2016). This work bridges the gap by asking: Are combinatory operations primarily determined by linguistic relations (e.g., Verb+Noun vs. Adj+Noun), experiential features of concepts, or both? Also, what composition operation does the brain use to combine individual features of concepts (e.g., social vs. sensory)? To investigate how the brain selects compositional operations for combining concepts, we used magnetoencephalography (MEG) and human-annotated semantic features with a novel semantic encoding approach. Our stimulus set included 216 phrases across six linguistic relations: ScalarAdj+Noun ("small cake"), IntersectiveAdj+Noun ("green cake"), Verb+Noun ("eat cake"), HasNounNoun ("glitter cake"), ForNounNoun ("birthday cake"), and MadeOfNounNoun ("corn cake"). The 107 component words and phrases were rated by participants for 65 semantic features (Binder et al, 2016). Thirty native English speakers participated in a MEG experiment with a semantic matching task. With the ratings data, we first examined how each feature’s phrasal rating is derived from those of words, assuming the following basic composition operations - Addition: (w1 + w2)/2; Multiplication: (w1 × w2)/6; Word1: w1; Word2: w2. Particularly, we calculated the 'composition error' (i.e., predicted rating from the rules – observed rating), which turns out to be mediated by both linguistic relations and semantic features (Wang et al, submitted). Then, using MEG data, we: i) calculated the projection matrix from feature ratings to brain activations using linear regression; ii) generated predicted activations for features combined by one of four composition operations; and iii) identified brain regions and time scales with activations closely matching the predictions via spatiotemporal clustering. Our findings show that composition operations depend on both linguistic relations and semantic features. For instance, intersective Adjective-Noun phrases primarily use additive operations for attention-related features and are influenced by the first word’s color attributes. Different brain regions perform distinct functions, all of which together help in creating meaning. For example, when processing a pattern-verb pair, the left medial parietal lobe is inclined towards addition, while the right insular lobe prefers multiplication. This division of labor aids in the understanding of verb-noun phrases. Activation in most feature-relation pairs is rapid, starting at 200ms for 'Word1' operations—faster than the 300ms onset for 'Word2', indicating immediate semantic processing post-stimulus. Some regions, like the inferior frontal lobe, specialize in specific combinations such as Adjective-Noun, while others show no activation for certain operations. This highlights a distributed and specialized neural architecture for phrase composition, designed to optimize the processing of linguistic relations and experiential features per the demands of specific tasks. This study bridges linguistically guided research on composition and theories of the distributed semantic feature space, suggesting that the neural implementation of composition reflects both abstract linguistics relations and experiential semantic features.
Topic Areas: Syntax and Combinatorial Semantics, Meaning: Lexical Semantics