Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Capturing Invariant Neural Processes for Aphasia Rehabilitation Using Transfer Learning and Group Dynamics
Poster Session B, Friday, October 25, 10:00 - 11:30 am, Great Hall 3 and 4
Aditya Singh1, Tessy Thomas1, Nitin Tandon1; 1UT Health
Recent advancements in neural recording technologies have revolutionized language neurobiology and motor speech, enabling brain-computer interfaces (BCI) that restore speech functions. However, real-time speech decoding models, customized to individual data from the intact sensorimotor cortex, struggle to generalize, particularly for aphasia patients with disrupted language networks. To address this, we developed deep learning models capable of integrating information across patients, tasks, and anatomical constraints. This approach allows us to use transfer learning techniques to enhance individual patient data performance by leveraging a healthy cohort’s dataset with better coverage, more training data, and intact motor-speech networks. To prove the reliability of this deep learning architecture, we aimed to robustly predict target words for a single-word naming production task in an aphasic patient with extensive sensorimotor lesions and dysfluent speech. We trained a seq2seq phoneme decoder on sEEG broadband gamma activity from variable electrode coverage constrained by clinical epilepsy monitoring and compared it to a word-level SVM classifier. Tested on utterances of single words with at most 3 phonemic errors during articulation, the phonemic decoder achieved 67% word decoding accuracy, comparable (p=0.21) to human observer performance at 74% for predicting the correct word based on hearing the patient’s overt speech. The SVM word classifier was significantly worse at 49% accuracy (p<0.05), with chance performance being 12% accuracy. Additionally, the seq2seq model could predict words held out from the training set at 35% accuracy by reconstructing held-out words at the phonemic level, a feat impossible for a word-based classifier. We developed a group variant of the seq2seq model, simultaneously training a shared recurrent layer on neural datasets from six individuals with normal language functions and dense sensorimotor cortex coverage. We then leveraged transfer learning techniques to bridge these learned shared latent dynamics into the decoder for the aphasic patient. Pre-articulatory word decoding performance was significantly improved using this technique (p<0.05) compared to training solely on the patient's dataset. However, the most significant improvement in decoding performance was for trials where the patient could not respond, wherein by utilizing this group-level pre-trained recurrent layer fine-tuned to the subject's own dataset, we achieved a 20% performance increase in decoding the semantically cued target word. Here, we show the ability to retain a latent feature space through subject-invariant pre-training that can be fine-tuned for datasets with limited information. By leveraging multi-site, multi-subject cortical activity, models are initialized on a flexible set of neural codes, improving performance for patients with dysfunctional language networks or brain lesions and offering a robust approach to creating a neural state shunt for speech rehabilitation. This comprehensive methodology not only advances BCI design and neural decoding techniques but also has profound implications for how we understand the neurobiology of language, how to capture invariant neural processes at a single trial level for group models, and how to translate these scientific findings to give a patient the ability to talk fluently again.
Topic Areas: Speech Motor Control, Phonology