Presentation

Sushmita Sadhukha¹, Saskia B.J. Koch², Margot Mangnus², Kamren Khan¹, Jana Bašnáková², Ivan Toni², Arjen Stolk^1,2; ¹Dartmouth College, ²Radboud University

People communicate rich ideas with remarkable flexibility and ease, delivering appropriate signals in spite of the irregularities and shifts in context that are characteristic of our everyday social interactions— why are some people good at this while others struggle? For instance, deficits in communication and social interaction are among the core diagnostic criteria for Autism Spectrum Disorder (ASD), even though the precise sources of these deficits remain largely unknown. In an attempt to address this core issue, empirical studies have largely focused on how autistic individuals process social stimuli isolated from the context of interaction with others. Here, we plan to look at autistic communication through the lens of the conceptual alignment framework (Stolk et al., 2016), and test whether the deficits arise from a reduced ability to align communicative context with an interlocutor. To this end, we analyze communicative behaviors evoked during live social interaction with recently developed context-sensitive neural networks. We build on a body of behavioral and neuroimaging studies which use a controlled, yet open-ended non-verbal communication game (de Ruiter et al., 2010). In the game, two players are instructed to interact on a digital game board with a 3x3 matrix layout and jointly reproduce target configurations of two geometric shapes on a trial-by-trial basis. This paradigm is particularly notable because communicators do not have access to contextual priors or communicative conventions (i.e., linguistic expressions, gestures, body language, etc.), providing privileged access to the full communicative context and history. Furthermore, the game creates experimental conditions which require communicators to 1) invent new communicative movements (i.e. signals) on the fly, 2) coordinate with each other to impute a shared meaning into these signals, and 3) modify the signals to meet the demands of the changing context (determined by trial type and difficulty), and thereby the specific goals of the interaction. Under these circumstances, it was shown that autistic individuals struggle to rapidly find relevant context for each other’s signals (Wadge et al., 2019). We use deep learning models, specifically a transformer architecture (Vaswani et al., 2017), to closely examine 1) the abovementioned signal dynamics 2) contextual dynamics of communication, herein defined as shifts in signal dependency structures across time, and explore how these dynamics interact with the signals being produced. By deploying Transformers iteratively, we aim to precisely define contextual dynamics generated by communicators. Our process is threefold: we 1) iteratively train models on progressively expanding interaction histories of behaviors from the same individual; 2) transform network embeddings into a dependency structure based on the distances of signals in the embedding space; and 3) correlate the dependency structures across consecutive interactions, generating a metric of contextual dynamics, namely interpersonal alignment of signals’ embeddings at each communicative turn. In sum, using this computational tool on precisely quantified neurotypical and autistic interactions, our study will characterize how people navigate the contextual dynamics of communication.

Topic Areas: Computational Approaches, Methods