Interplay of Amodal and Modal Encodings Underlying Directional Space-Metric Associations

01/01/2024 to 31/12/2026

The modal or amodal origin of directional space-metric associations (SMAs) – as revealed, for example, by the spatial-numerical associations of response codes (SNARC) effect – is currently under debate. Modal encodings imply culturally-grounded, long-term associations of particular stimuli with space – such as the mental number line, which extends from left to right in western cultures. Amodal encodings, on the other hand, refer to the manner how entities, including numbers, are temporally encoded ordinally in working memory (WM) – where the order is associated with a virtual, left-to-right axis in western cultures. The results from our joint project have shown that both long term associations (including number-line-like effects and distinct stimuli, such as the letter ‘A’ on the left side) and WM content play significant roles. Moreover, in our studies in virtual realities (VR), we have discovered that the spatial associations strongly depend on which spatial axes are task-relevant. Meanwhile, we have developed a resourceful event-predictive inference (REPI) system, which models the cognitive dynamics that unfold during the involved perceptual and decision making processes. REPI offers explicit computational and algorithmic explanations about how SMAs may affect stimulus processing and response behavior across modal and amodal representations. However, many open questions remain, particularly concerning the precise nature of the modal and amodal interactions, their selective task-relevant spatial associations, and the modularized modelling of the interaction dynamics. In the present proposal, we thus follow up on this prior work. In WP 1, we will examine the logic behind the dominant role of stimulus locations, hand arrangements, and response targets in further VR studies. We will also look at the (potential) limits of generalization of the amodal ordinal working memory (WM) process by using other ordinal stimuli – not only numbers and letters. In WP2, we will systematically investigate whether the role of WM changes as a function of individual stimuli, which would indicate that WM content is not as amodal as postulated. In WP3, we will further challenge the amodality hypothesis of WM content by systematically varying the modalities in the encoding and classification phase. Finally, in WP4, we will look at the developmental trajectories in a cross-sectional experiment to find out how modal and amodal encodings of SMAs develop over time. In all four work packages, the results (as well as previous and other available results) will be modelled with the REPI framework. We thus hope to be able to offer a complete computational and algorithmic explanation of SMAs and their effects on reaction times, error rates, and decision making dynamics both across trials and over development.