Perception of Facial Emotional Expressions and Touch in Virtual Face-to-Face Interaction
Ville Johannes Harjunen · Työväentutkimus Vuosikirja · 2019
Emotional expressions as manifested in facial movements, voice, and touch are a crucial part of face-to-face interaction. The majority of existing neuroscientific research on emotional expressions concerns the perception of unimodal emotional cues, such as facial emotional expressions. In natural face-to-face interaction, however, emotions are often expressed as compounds of facial, tactile, prosodic, and postural cues. How the brain processes such multimodal emotional information remains poorly understood. The aim of the current dissertation is to investigate how emotional expressions conveyed consecutively via face and touch are integrated in perceptual processing and decision-making. Four studies were conducted to measure event-related brain potentials (ERPs) and autonomic nervous system responses to simulated touches and facial emotional expression stimuli. The first two studies used virtual reality to investigate how a virtual agent’s facial emotional expressions influenced the way the agent’s subsequent touch was perceived (Study I) and whether the receiver’s individual characteristics influenced this visuo-tactile affective modulation (Study II). Touch perception was measured using self-reports, somatosensory-evoked potentials (SEPs), and cardiac orienting responses (ORs), and the individual characteristics were indexed by behavioural inhibition system sensitivity (BIS) and gender. Study III investigated whether receiving a touch influenced the processing of a subsequent emotional face picture presented on the computer screen. Here, face-evoked ERPs, ORs, and facial electromyography were measured. Finally, the Study IV examined whether a virtual agent’s touch and emotional facial expressions influence receivers’ decision-making and offer-related ORs in an economic decision-making game. Additionally, the study examined whether the receivers’ behavioural inhibition/approach system (BAS/BIS) sensitivities and sensitivity to unfair treatment moderated persuasiveness of nonverbal cues. Study I revealed that happy, angry, and sad facial expressions resulted in amplified SEPs around 20–50 ms after touch onset, whereas in later latencies (250–650 ms), the angry facial expression amplified and the happy expression decreased the SEP amplitudes. In Study II, men with high BIS were found to perceive touch from male agents as especially intense if accompanied by happy, angry, of fearful facial expressions, and they showed pronounced cardiac OR to all the touches. Study III demonstrated that receiving a computer-generated touch did not modulate emotional face processing in any of the measured indices. Finally, in Study IV, people were found to accept unfair offers more often if the agent smiled or touched them before making the offer. The touch had a stronger persuasive influence in people with low sensitivity to unfairness and low approach tendency, whereas the effect of facial expressions was moderated by BIS. Altogether, the findings of the dissertation reveal that a sender’s facial emotional expressions modulate subsequent touch perception at a very early stage and that the modulation is based on different emotional information in different temporal stages. In addition, the findings suggest that motivational tendencies and gender influence the manner in which people perceive a sender’s multimodal emotional expressions and make decisions thereafter. These findings are valuable for basic research, but their implications extend also to the development of novel clinical interventions and social virtual reality applications.