Body and gestures - unimi.ithomes.di.unimi.it/~boccignone/GiuseppeBoccignone... · • Hand gestures are used in most cases • to regulate interactions (e.g., to yield the ... •

Body and gestures

Corso di Interazione uomo-macchina II !Prof. Giuseppe Boccignone!Dipartimento di InformaticaUniversità di [email protected]://boccignone.di.unimi.it/IUM2_2014.html

Non verbal behavior //Ekman taxonomy

• Affective/attitudinal/cognitive states (e.g. fear, joy, stress, disagreement, ambivalence and inattention)

• Emblems (culture-specific interactive signals like wink or thumbs up),

• Manipulators (actions used to act on objects in the environment or self- manipulative actions such as lip biting and scratching),

• Illustrators (actions accompanying speech such as finger pointing and raised eyebrows), and

• Regulators (conversational mediators such as the exchange of a look, palm pointing, head nods and smiles).

DeMeuse (1987)

Non verbal behavior //De Meuse taxonomy

DeMeuse (1987)

Non verbal behavior //De Meuse

• Hand gestures are used in most cases

• to regulate interactions (e.g., to yield the turn in a conversation),

• to communicate a specific meaning (e.g., the thumbs up gesture to show appreciation),

• to punctuate a discourse (e.g., to underline an utterance by rising the index finger), to greet (e.g., by waving hands to say goodbye), etc.

Social signal processing //using hand gestures

• In some cases hand gestures are performed unconsciously and they are interesting from an SSP point of view because they account for honest information , i.e., they leak cues related to the actual attitude of a person with respect to a social contex

• adaptors express boredom, stress and negative feelings towards others.

• usually displayed unconsciously and include self-manipulations

• (e.g., scratching, nose and ear touching, hair twisting), manipulation of small objects (e.g., playing with pens and papers), and self-protection gestures (e.g.,folding arms or rythmicly moving legs)

Social signal processing of EBL //using hand gestures

Non verbal behavior //postures and gestures

metabolismo

emozioni motivazioni

azione percezione

riflessione

!non verbale

ambiente

!visiva

Social signal processing of Body Language(BL)/ and Emotional Body Language (EBL)

• Neurobiology shows that both posture and body/ limb motions change with emotion expressed.

• Basic research also provides evidence that body gestures like head inclination, face touching, and shifting posture often accompany social affective states like shame and embarrassment .

• However, as indicated by researchers in the field as much as 90% of body gestures are associated with speech, representing typical social signals such as illustrators, emblems, and regulators

Social signal processing of BL/EBL• Body language comes in clusters of signals and postures, depending on the internal emotions and

mental states. Recognizing a whole cluster is thus far more reliable than trying to interpret individual elements.

• Aggressive body language: Showing physical threat.

• Attentive body language: Showing real interest.

• Bored body language: Just not being interested.

• Closed body language: Many reasons are closed.

• Deceptive body language: Seeking to cover up lying or other deception.

• Defensive body language: Protecting self from attack.

• Dominant body language: Dominating others.

• Emotional body language: Identifying feelings.

• Evaluating body language: Judging and deciding about something.

• Greeting body language: Meeting rituals.

• Open body language: Many reasons for being open.

• Power body language: Demonstrating one's power.

• Ready body language: Wanting to act and waiting for the trigger.

• Relaxed body language: Comfortable and unstressed.

• Romantic body language: Showing attraction to others.

• Submissive body language: Showing you are prepared to give in.

• Core patterns

• A number of core patterns can be identified that include clusters of body movements:

• Crossing, Expanding, Moving away, Moving forward, Opening, Preening, Repeating, Shaping, Striking and Touching

Social signal processing of BL/EBL

• Crossing

• involves moving parts of the body across one another or the body, in a defensive act.

• Crossing arms across the body.

• Crossing just the hands or wrists.

• Holding hands.

• Inter-twining the fingers.

• Crossing legs, either above the knee or at the ankles

Social signal processing of EBL //core patterns

• Expanding

• Making the body bigger says 'I am powerful' and is a typical male action. This warns other men not to attack and may indicate that the person is thinking of attacking. With women, expansion often also says 'I will protect you', demonstrating the basic partnership requirement to protect one's family from harm

• The body is made taller, wider and generally bigger by:

• Standing upright, as tall as possible.• Straightening the head.• Thrusting out the chin.• Flaring the nose.• Push out the elbows and arms, possibly sweeping out a wide

space.• Opening the hands.• Puffing out the chest.


• Moving away

• A person may retreat from the other person in a number of ways. These can be large movements or small signals where they move only slightly away. Defensive retreat is often coupled with other defensive acts including pulling in of arms and pulling down of head and body.

• Pulling back the head in fear, confusion or surprise.

• Pulling back arms or shoulders.

• Hollowing the chest, pulling it back.

• Turning away the head and, in extreme, showing the back.

• Stepping back.


• Moving forward

• can be an act of aggression and so signal anger.

• Other signals will indicate the actual intent (there are many romantic signals, for example). This can be used as method of emphasis as we move closer to gain attention and create a bond through which ideas are transmitted.

• Reaching forward with arms and hands, giving, grabbing or striking.

• Pushing the head forward.

• Thrusting the hips suggestively forward.

• Leaning forward.

• Stepping forward.


• Opening

• is a pattern of unfolding, removing protection and offering. Being open may be used in flirting :

• Raising the head from a chin-down position to looking forwards.

• Unfolding arms.• Holding open palms.• Spreading palms in an opening circular

move around from front to side.• Turning hands over from palm-down to

palms-up.• Pointing toes outwards, with splayed feet.• Standing with legs akimbo


• Preening

• is act of faked cleaning or tidying that is common in courtship rituals across the animal kingdom. birds pick at their feathers, chimps pick at fleas and humans tidy themselves to look good for their prospective partners. As an act in front of another person, preening says 'Look, I am making myself beautiful for you!' It may be combined with the look away -- look up flirting pattern. Actions include

• Straightening the tie or other clothes.

• Looking in a mirror.

• Curling lips to even out lipstick.

• Brushing imaginary lint off arms or legs.

• Patting down hair or combing it with the fingers.


• Repeating

• When a person is bored, they seek other things to do and a repetitive movement can provide a simple distraction. The stimulation of movement may also be seen in dancing and moving along to music. Curiously, repetition leads to a trance state which can be pleasurable, and which may explain why some people repeat actions Actions include

• Nodding or shaking the head.• Tapping the teeth.• Waggling the eyebrows.• Swinging the arms.• Clapping of hands.• Waving with hands.• Drumming of fingers.• Swinging or bouncing a leg.• Tapping of feet.• Rubbing the body in various places.• Waving the body back and forth.• Stroking various parts of the body.


• Shaping

• When people talk, they use their whole body to describe what they are talking about, shaping what they are saying as a reinforcement and emphasis of their words.

• What is being described is literally carved out what is from the air in front of the person

• Hands are the main implements, shaping the item being described.

• Arms may also be important, particularly when shaping something big. For something very big, the person may move the rest of their body, reaching up on the toes.

• Fingers can be used to shape something small.• Words may be shaped with the lips.


• Striking

• Striking (without actually hitting others) is usually an open act of aggression, saying 'I want to hit you!' and can be very intimidating, particularly if the person involved could clearly do damage. It is thus closely associated with anger.

• Hitting others is socially undesirable and legally forbidden. To handle anger, we thus displace it into a relatively harmless simulation (although this can still be scary for others).

• A small but rapid nod of the head can be a symbolic head-butt.• Wagging a finger in admonishment, symbolizing striking them with a

club.• Shaking an entire arm.• Jabbing a finger toward someone, as if prodding them.• Striking subtly sideways with an elbow.!

• Poking a finger into an open hand or onto a table.• Slapping a fist into an open hand or onto a table (this makes a good

noise).• Stamping the ground with a foot.


• Touching

• Touching oneself is often a sign of uncertainty or discomfort. It is as if the person is reassuring themselves, using their own hands in place of the hands of a non-present parent or friend.

• Touching can similarly be an affirmation of the identity. 'I can feel myself, therefore I exist!'

• When a person is stressed their muscles become tense and they may sweat and itch. They may thus rub the areas affected. Lying is often a stressful activity and thus rubbing can be an indicator. It can also mean the person is worried about something else or is just hot.

• Covering such as the mouth, nose, eyes and ears often means 'I do not want to use these' and indicates the person would rather be elsewhere or they are holding themselves back from potentially harmful action.

• Touching a friend affirms their identity and forms a physical bond. Holding them close emphasizes this.

• Touching other people with whom you are not comfortably familiar can be a sign of power ('I can break social rules and you can't do anything about it!').

• Touching varies greatly across cultures,


• Touching the cheek in surprise or horror 'Oh goodness!'• Stroking the chin whilst thinking.• Covering or touching the mouth to silently say 'I don't know what to say!' or 'I don't want

to speak'.• Finger to the lips to say the same thing or also 'Shh!'• Tapping the teeth in boredom or irritation.• Fingering the nose, when thinking.• Scratching the nose when lying.• Rubbing the nose in disagreement or discomfort.• Pinching the bridge of the nose in negative evaluation.• Tapping the nose to indicate 'this is a secret'.• Rubbing the eyes to say 'I don't want to see' or 'I want to be elsewhere' or otherwise as

an indicator of discomfort.• Putting a palm to the forehead to say 'Phew, that was close' or 'Oh no, that's terrible!'• Tapping the forehead with the palm or heel of the hand to say 'Oh I'm so stupid!'• Touching the forehead in salute.• Stroking the hair when flirting with others.• Touching arms or hands or other part of the body in self-comfort.• Clasping or touching fingers in an evaluative gesture.• Caressing bottom, hips, legs, thigh, knees or other area to say 'I'd like you to do this to

me...'• Rubbing the neck in discomfort.


• One of the main classifications of postural behaviours proposes three main criteria to assess the social meaning of postures.

1.The first criterion distinguishes between inclusive and non-inclusive postures and accounts for how much a given posture takes into account the presence of others.

• facing in the opposite direction with respect to others is a clear sign of non-inclusion.

Social signal processing of EBL //a smaller taxonomy

!

2.The second criterion is face-to-face vs. parallel body orientation and concerns mainly people involved in conversations.

• Face-to-face interactions are in general more active and engaging (the frontal position addresses the need of continuous mutual monitoring),

• while people sitting parallel to each other tend to be either buddies or less mutually interested.


!

!

3.Third criterion is congruence vs. incongruence: symmetric postures tend to account for a deep psychological involvement, while non-symmetric ones correspond to the opposite situation.

• The postural congruence is an example of a general phenomenon called chameleon effect or mirroring that consists of the mutual imitation of people as a mean to display affiliation and liking.

• Postural behaviour includes also walking and movements that convey social information such as status, dominance and affective state .


• Postural behaviour includes also walking and movements that convey social information such as status, dominance and affective state .


Social signal processing of EBL //Coulson: How Well Do Bodies Communicate Emotion?







• The results paint a complex picture of how well body posture communicates emotion.

• Disgust was not attributed to any posture by more than 50% of the sample, and fear and surprise were consistently attributed to only a small number of postures.

• Anger, sadness, and happiness, by contrast, were attributed to large numbers of postures, and some stimuli were attributed to the same emotion by 90% or more of the sample.

• For these three emotions, agreement rates are comparable to those obtained from static facial expressions, and replicate other findings suggesting they are accurately perceived from posture

• Surprise and happiness generated perceptually similar postures


• One possible explanation for this concerns the relative contributions of posture and movement to the attribution of emotion.

• Although static images can imply movement, and have been shown to activate neural systems associated with motion detection, attributions of certain emotions may be especially sensitive to motion cues, and static images may not represent an appropriate medium within which dynamically expressed emotions can be communicated


The neural basis of visual body perception

• The human body, like the human face, is a rich source of socially relevant information about other individuals.

• Evidence from studies of both humans and non-human primates points to focal regions of the higher-level visual cortex that are specialized for the visual perception of the body.

• These body-selective regions, which can be dissociated from regions involved in face perception, have been implicated in

• the perception of the self and the ‘body schema’,

• the perception of others’ emotions

• the understanding of actions

The neural basis of visual body perception //Evidence for body-selective neural mechanisms

• Non-human primates. Studies on macaque monkeys have revealed that some neurons in the inferior temporal cortex (IT) respond selectively to the shapes of human and monkey bodies and body parts

Closely!neighbouring regions of the superior temporal sulcus (STS) in the macaque are!selectively activated in functional MRI by images of faces (left), as compared with bodies,!fruits, artefactual objects and hands. !!This is also true for images of bodies (right), as!compared with faces, fruits, artefactual objects and hands.



Increasing responses to progressively more hand-like stimuli in a neuron from macaque inferotemporal cortex. !!Numbers reflect approximate relative magnitude of responses to different stimuli



Spike histograms and schematic stimuli from an investigation of a hand-selective neuron. !This neuron seems to be selectively tuned to the general form of the hand. The response to faces is low, ruling out a general response to all biological stimuli. The response to hand-like geometric stimuli is also low, demonstrating the specificity of the response to realistic hand shapes.


• Key findings from intracranial recordings in humans with epilepsy

Pre-surgical intracranial recordings from a patient with epilepsy reveal an electrode at the approximate location of!the extrastriate body area EBA (electrode 15; indicated by the yellow box) that is body-selective (red line) from 190 ms poststimulus,!relative to faces (blue line), tools (green line) and animals (black line)


• Body- and face-selective regions of the human occipito-temporal cortex, as revealed by fMRI

The extrastriate body area (EBA; shown!here in the right hemisphere of six subjects) is found in the posterior inferior temporal!sulcus. X values indicate Talairach coordinates for each slice plane


• Body- and face-selective regions of the human occipitotemporalcortex, as revealed by fMRI

Body- and face selective!regions of the human visual cortex, in a ventral view of the right hemisphere of one individual!!Bodies and faces activate similar, but not identical, regions of the fusiform gyrus (the fusiform body area (FBA), and fusiform!face area (FFA), respectively). !!Posterior to this region are nearby but distinct body selective!(EBA) and face-selective (occipital face area (OFA)) regions.



Responses of the functionally defined EBA to various stimuli, indicating the body selective response of this region.




• Both bodies and faces provide cues to the identity, emotions, intentions, age and gender of other people.

• Bodies and faces elicit globally similar patterns of evoked potentials, similar effects on neural activity when inverted or displaying emotion, adjacent or overlapping selective activity in the monkey STS, and closely overlapping fMRI responses in the posterior fusiform gyrus in humans

• Evidence from all of the methods reviewed here points to functional and anatomical distinctions between the neural systems involved in face and body processing

• The evidence for similar neural responses to bodies and faces raises the possibility, as yet largely untested, that body and face perception are closely functionally integrated


• The posterior fusiform gyrus, in which body and face representations are in close proximity, would be an ideal place for local interactions between face and body processing

• The close proximity and interaction of the FFA and FBA would be especially useful when cues from either the face or the body alone are not sufficient for recognition:

• when seeing someone from a distance, facial features alone might not be enough for identification. In this case, cues from the body, processed in the FBA, could inform and perhaps support face-selective processing in the FFA, and vice versa.

The neural basis of visual body perception //Functional role of body-selective regions

• EBA vs. FBA: What is computed by the neurons in these regions?

• Given their distinct locations in the visual system, there might also be dissociations between the EBA and the FBA on one or more aspects

• Methods that have been developed for extracting orientation information from primary visual cortex activity, or motion direction from hMT activity, could be adapted to determine whether and how the EBA and the FBA encode information about features such as gender, identity, posture or action

The neural basis of visual body perception //Interactions among body- and action-perception regions.

• What is the interrelationship among the brain regions that respond to static and dynamic bodies?

• EBA (and possibly the FBA) may provides static snapshots of body postures to action perception systems, for example, in the pSTS

• More generally, these regions might interact with the frontoparietal mirror areas involved in both the perception and production of actions

The neural basis of visual body perception //Evolutionary origins

• The problem of identifying and understanding conspecifics is not unique to humans or primates:

• Any social species must face this problem, although many will not solve it primarily with vision

• The evidence for body- and face-selective regions in monkeys and humans raises the question of whether other species exhibit similarly selective visual brain regions

• For example, there is now substantial evidence for face-specific neural systems in sheep.

• Further research in other species on the neural basis of the perception of conspecifics would shed light on the evolutionary origins of these cortical systems.

Body and gestures: emotional body language?

• Following the work of Darwin, which was the first to describe body expressions associated with emotions in animals and humans, there have been a number of studies on human body postures and gestures communicating emotions

• perception and display of body postures relevant to basic emotions including happiness, sadness,surprise, fear, disgust, and anger

• bodily expressions of felt and recognized basic emotions as visible in specific changes in arm movement, gait parameters, and kinematics.

Towards the neurobiology of emotional body language (EBL)

• People’s faces show fear in many different circumstances.

• However, when people are terrified, as well as showing emotion, they run for cover.

• When we see a bodily expression of emotion, we immediately know what specific action is associated with a particular emotion, leaving little need for interpretation of the signal, as is the case for facial expressions.

• Research on emotional body language is rapidly emerging as a new field in cognitive and affective neuroscience.


• Many valuable insights into human emotion and its neurobiological bases have been obtained from the study of facial expressions

• By comparison, the neurobiological bases of EBL are relatively unexplored

• Human and non-human primates are especially sensitive to the gestural signals made by other primates, and use these signals as guides for their own behaviour

• Fearful faces signal a threat, but do not provide information about either the source of the threat or the best way to deal with it

• By contrast, fearful body positions signal a threat and at the same time specify the action undertaken by the individuals fearing for their safety

• EBL consists of an emotion expressed in the whole body, comprising coordinated movements and often a meaningful action, and so prompts research to go beyond facial expressions and to consider issues of perception of movement and action, which have so far been researched in isolation and not specifically related to perception of EBL.

• Emotion and the amygdala:

• Current models of human emotion place the amygdala at the core of a network of emotional brain structures, involving the orbitofrontal cortex (OFC), anterior cingulate cortex (ACC), premotor cortex and somatosensory cortex

• The amygdala decodes the affective relevance of sensory inputs and initiates adaptive behaviours via its connections to the motor systems

• Fearful bodies activate the two main areas associated with the processing of facial expressions of fear — the amygdala and the right middle fusiform gyrus.

• When we see neutral, fearful or happy whole-body expressions with the facial expression blurred, fear images activate these areas



body expression of fear

neutral body posture

Activation shown is in response to the fearful bodies (yellow) in the fusiform face area (FFA). No!activation is seen for the neutral bodies (blue).



• EBL overshadows facial expressions:

• Recent studies show that we process bodies as rapidly as faces and perceive the same properties of both facial and body stimuli

• If bodies are recognized rapidly, it would be expected that

• EBL might influence face processing in the earliest stages of facial recognition, and

• even when attention is focused on the facial expression

• To investigate this possibility, facial expressions were combined with congruent or incongruent bodily expressions


• EBL overshadows facial expressions:

• Recent studies show that we process bodies as rapidly as faces and perceive the same properties of both facial and body stimuli

Participants had to judge the expression of faces that were accompanied by either a congruent or incongruent bodily expression. Categorization of facial expressions in the presence of an incongruent body emotion significantly reduces accuracy and increases observers’ reaction times

• Emotional contagion: the automatic spread of emotion and body movement to another person when an individual is viewing EBL

• Why emotional bodies are contagious?

• Fear is contagious, as are joy and sadness, but what is the underlying mechanism for this? Is it the emotion expressed? Perhaps it is the movement represented or implied? Or the action represented? Or is it a combination of all these factors?

• Functional brain imaging in participants passively viewing images of whole bodies shows activity in a complex of areas that are known for processing not only emotions and biological movement but also goal-directed actions

• significant fear related activation in response to EBL was also observed in areas that have a role in action representation and in other motor areas


• Bodies in movement.

• As movement is a key feature of EBL in the natural world, it probably contributes to the recognition of EBL, and may induce emotion with minimal input from the other aspects of EBL. How much of EBL recognition/perception is perception of movement?

• Biological movement provides important information about the nature of the object that is moving, the direction of movement or the meaning of the action, and this is observed even in the early stages of development

• Which comes first perception of movement or perception of emotion?


• Bodies in movement.

• There is already some evidence of the power of movement alone in the induction of emotion.

• The movement of an area of dots as a collective that is perceived as pleasant triggers amygdala and fusiform cortex activity, as well as activity in premotor cortex and sensorimotor areas


• Bodies in action:

• Both the fusiform body area and the EBA are sensitive to actions shown by the body.

• In real life, biological movements frequently consist of goal-directed actions

• seeing someone fleeing, observers describe the action rather than the movement — that is, the person is trying to reach the exit rather than is running away.

• The STS, parietal cortex and premotor cortex are activated during the perception of biological movements and of object-directed actions

• More recently, the discovery of neurons that encode complex movements and actions (mirror neurons):

• in contrast to animal models, in which the amygdala emotionally tunes the motor system, in humans it could be motor neuron activity that spreads to the amygdala


• A two-systems theory of EBL:

• The amygdala is a key piece in the orchestration of two separate emotional circuits,

• an automated reflex-like circuit that predominantly resides in subcortical structures

• and a cortically controlled circuit in the service of recognition and deliberation

• In higher organisms, both systems cooperate in decoding EBL signals and monitoring behaviour following an emotional signal provided by EBL

• Both systems are closely connected with a set of structures that support bodily sensations and bodily awareness


The primary network sustains the!rapid automatic perception of EBL and!preparation of adaptive reflexes.

In this system, affective stimulus input is decoded in

combination with pastexperience and memory. The main role of the second

system is to perceive EBL indetail and to compute the behavioural consequences!

of an emotion and to decide on a!

course of action in response to the stimulus; typically, the

action is determined by the stimulus eliciting the bodily reaction (for example, fear

expressed in running)


• This model represents how the brain reacts to emotional signals, there are similarities with existing models of how facial expressions are recognized, most directly with models based on two separate, relatively independent systems of cortically and subcortically based face processing

• the relationship between emotion and behaviour might depend on the specific emotion.

• For example, food-induced disgust is rigidly linked with specific motor activity, but this link is much more complex in the case of fear, in which the range of adaptive actions varies from running away from the sound of gunfire to freezing on the spot on seeing a snake

• EBL might not have the typical one-to-one relationship with specific emotions that has been assumed for basic facial expressions since the work of Ekman in the 1970s


Documents

Body and gestures - unimi.ithomes.di.unimi.it/~boccignone/GiuseppeBoccignone... · • Hand gestures are used in most cases • to regulate interactions (e.g., to yield the ... •