Abstract Affective RoboticsAlwin de Rooija, Joost Broekensb and Maarten Lamersa

aLeiden Institute of Advanced Computer Science, Leiden University, Niels Bohrweg 1, 2333 CA, Leiden, The Netherlands

bMan-Machine Interaction Group, Delft University of Technology, Mekelweg 4, 2628 CD, Delft, The Netherlands

Abstract. What form should happiness take? And how do we shape Disgust? This research investigates a fundamentally novel design approach to affective robotics design in which abstraction and the minimal requirements to perceive emotion expressions play a fundamental role. The core idea is that if we abstract away from the human body, and leave only the essential components relevant to convey emotions, this is sufficient to meet the cognitive requirements of human beings for affective interaction. As such, the novelty of our approach lies in the fact that it does not revolve around designing the anatomical body features used in emotion expression, e.g. facial expressions, body postures and arm gestures, but instead focuses on designing expressions based on the essential components of visual emotion recognition. Such a turnaround in design strategies requires a new design theoretical framework. To facilitate further exploration, this research unifies affective robotics, affective science and abstract art from a psychological perspective to develop design guidelines for what we call: abstract affective robotics.

Keywords: Robot, Design, Emotion, Expression, Recognition, Abstraction, Art

1. Introduction

In the last decade robotics research has seen the addition of research aimed at exploring the analogy of affect and its role within and among humans to develop robot cognition and behavior [Breazeal 2002, Canamero 2002, Gadahno 2003], and in particular to facilitate the functioning of robots in social environments [Fong et al 2003]. This aim is fundamental to a variety of techno-scientific research areas such as affective computing [Picard 1997], human-robot interaction [Feil-Safer & Matarić 2008] and social robotics [Fong et al 2003]. Such technology can be applied as real-world research platforms for theoretical affective and social science [Broekens 2010], but mostly thrives on the promise of practical application in for example healthcare [Bemelmans et al 2010, Broekens et al 2009], therapy [Dautenhahn et al 2002, Michaud & Théberge-Turmel 2002] and education [Saerbeck et al 2010]. Design strategies for affective robots focus on modeling robot appearance that matches the cognitive and perceptual requirements of humans for social and affective interaction, and have a particular focus on the design of anatomical features used in emotion expression and their configurational aspects, i.e. facial expressions, body postures and arm gestures. What is common to current design strategies for affective robots is that these employ varying degrees of realism and iconism and assume either an anthropomorphic or a zoomorphic basis for their design [Blow et al

2006, Fong et al 2003, Hegel et al 2009], while strikingly little experimental research exists on the possible role of abstraction and minimal requirements for such design approaches.

Recent research on visual emotion recognition offers substantial evidence that the recognition of emotion does not require the context nor configuration of a body or face per se [Aronoff 2006, Atkinson & Adolphs 2005, Gelder 1999, Lundqvist & Ohman 2004], instead the recognition of emotion expressions can rely on basic motion and form features alone [Aronoff 2006, Atkinson & Adolphs 2005, de Rooij et al 2011, Pavlova et al 2005]. This is complemented by a large body of experimental research on emotion attribution to abstract geometrical shapes [Aronoff et al 1992, Aronoff 2006, Collier 1996, de Rooij et al 2011, Larson et al 2007, Larson et al 2008, Pavlova et al 2005]. This motivates us to assume that in order to arrive at the minimal and abstract properties of conveying an emotion expression in the context of design, such a design strategy should focus on designing emotion expressions based on the essential components of visual emotion recognition instead of the anatomical features used in emotion expression. Such a conceptual turnaround in design strategies for affective robotics requires a novel and fundamentally different theoretical framework.

A similar but more general approach to a design strategy of such type can be found in abstract art. Abstract art focuses on finding minimal representations of visual information and ways to exploit these representations in relation to perception and sensation [Zimmer 2003]. Recently the psychology behind art has been taking shape, which states that one of the major contributions of the creative arts is to define and exploit essential components of visual information processing and as such has to reflect specific mechanisms of visual perception [Di Dio & Gallese 2009, Latto 2004, Ramachandran & Hirstein 1999, Zeki & Lamb 1994, Zimmer 2003]. Taking this psychological perspective on abstract art provides the scientific context to develop general practical and theoretical insights into designing artifacts that reflect essential components of visual emotion recognition and as such serves as an empirical basis to formalize abstract design.

Our research merges these psychological insights from abstract art with recent research in visual emotion recognition in the context of design strategies for affective robotics. As such, we aim to unify contemporary research into affective robotics, affective science and the psychology of abstract art to develop design guidelines and provide a scientific context for abstract affective robotics.

2. Abstraction

Abstraction can be defined as a strategy for the simplification of the concrete, which through a process of removing all its contingencies arrives at its essential universal aspects [Zimmer 2003]. From a psychological perceptive abstraction is a fundamental operation that is applied at many levels of cognition. It is a critical feature in the acquisition and processing of knowledge and it allows for reduced memory storage and rapid processing of information [Barsalou 2003, Hampton 2003]. As such, the process of abstraction allows us to reason with generalizations instead of being stuck to details [Zeki 2001], which in turn can also be a disadvantage [Hampton 2003]. Yet the question of how the human-brain forms abstraction is still a central problem in science [Barsalou 2003, Hampton 2003]. The way abstraction has been applied within the arts may hint on some of the functional perceptual mechanisms involved relevant to our research.

2.1 Abstract art

Early abstract art was motivated by the desire to formalize a minimal visual language of color, form, line and composition through a process of abstraction. In this process of abstraction the artist needs to disregard inessential information from the concrete visual world in order to represent the essential aspects of objects. This idea marks the divide from early explorations that aimed to represent the visual world through a process of increasing abstraction, by depicting reduced scenery, objects or people; to representing sensory experience, by exploring and exploiting those features that exhibit strong experiential qualities. In that sense non-representational abstraction is not less concrete than a realistic representation, instead it attempts at a concrete visualization of the mental representation [Zimmer 2003].

Many artists attempted to formalize their findings into compositional systems that relate basic perceptual properties to emotional and aesthetic effects. Artists have employed artistic methods from color and form studies, sketches and physical models to develop formalizations that allow them to research the experiential quality of media. Zeki & Lamb (1994) state that since art comes from the brain it has to obey the laws of visual processing. As such, these formalisms are developed by intuitively selecting salient properties of color, form and motion, and mechanisms to work with these properties [Zeki & Lamb 1994]. From that perspective the artists’ formal methods reflects brain mechanisms governing abstraction and perception. It is outside of the scope of this paper to do a full review on the wide variety of formalisms developed by artists, since these formalisms were not devised by scientific but by artistic method instead. Nevertheless we want to refer to some interesting examples as they can provide a starting point for future investigations: Point and Line to Plane [Kandinsky 1926], Die gegenstandslose Welt [Malevich 1927], Principles of neo-plastic art [van Doesburg 1925] and Neue gestaltung [Mondriaan 1925].

Figure 1: composition with red, yellow, blue and black (1921) by Piet Mondriaan

Figure 2: counter composition V (1924) by Theo van Doesburg

To illustrate it is interesting to quote Piet Mondriaan from a letter he wrote to H.P. Bremmer in 1914:

“I construct lines and color combinations on a flat surface, in order to express general beauty with the utmost awareness. Nature (or, that which I see) inspires me, puts me, as with any painter, in an emotional state so that an urge comes about to make something, but I want to come as close as possible to the truth and abstract everything from that, until I reach the foundation (still just an external foundation!) of things… I believe it is possible that, through horizontal and vertical lines constructed with awareness, but not with calculation, led by high intuition, and brought to harmony and rhythm, these basic forms of beauty, supplemented if necessary by other direct lines or curves, can become a work of art, as strong as it is true.”

We quote Mondriaan here because of the work in neuroaesthetics by Latto (Latto et al 2000, Latto & Russel-Duff 2002, 2004), who found that people both have a quicker response to, and also preferred the horizontal and vertical compositions in Mondriaan’s work over oblique or diagonal lines (figure 1), which is the case in the work of for example Theo van Doesburg (figure 2). Additionally, it was confirmed that the fluency of processing of vertical and horizontal lines is higher than that of diagonal lines, and that fluency yields an aesthetic response [Latto & Russel-Duff 2004, Reber et al 2004]. As such, Mondriaan’s intuitive approach brought him quite close to the neurobiological truth. This provides us with an interesting perspective on the role of visual abstraction within the arts. Taking such a psychological perspective on abstract art provides an empirical and theoretical basis to formalize abstract affective design in a scientific context.

2.2 Mechanisms

Following this perspective on abstract art we review fundamental design operations from a psychological perspective. We limit this review based on to the general applicability of these perceptual mechanisms as design operations to essential features (which we will later review in section 3.4), and their possible relation to affective quality, the affective valuation of objects and events. Though these mechanisms also relate to aesthetic quality, the evaluation of objects and events in terms of preference, liking and beauty, we do not focus on this aspect of abstract design in this review [for future reference see Latto 2004, Ramachandran & Hirstein 1999, Reber et al 2004, Winkielman et al 2007]. We distinguish four relevant mechanisms: isolation, the peak-shift principle, translation and multiplicity.

2.2.1 Isolation

A fundamental aspect of abstraction in visual art is to isolate a modality, e.g. colors, shapes, contours or movement, which convey the essence of a sensation or object in an image, and remove or reduce any other visual information that does not contribute specifically to conveying that essence. As such, isolation emphasizes essential features by purging the image of redundant information and therefore allocates attention to the essential features we normally have to search for [Ramachandran & Hirstein 1999, Zimmer 2003]. Hence the cliché: less is more.

This is well illustrated by the work of work of Matisse (figure 3), who shows that by using simple outlines, i.e. the absence or reduction of other structural and detailed visual information but contour, essential visual aspects are well communicated and understood. Leaving them out differentiates these essential features from the environment and guides attention towards it essential aspects more quickly and easily [Lehrer 2009, Ramachandran & Hirstein 1999].

Figure 3: La Gerbe (1953) by Henri Matisse.

The flipside of such abstraction is that by reducing an object to its essential features you reduce the amount of contextual information available as well. Research shows that indeed people make very specific and consistent attributes to representational and realistic pictures, but interpret abstract imagery more openly [Vessel & Rubin 2010].

2.2.2 Shifting Peaks

Ramachandran & Hirstein (1999) identify the peak-shift principle as one of the fundamental perceptual mechanisms behind abstract art but also artistic practice in general. This principle states that a human or animal will show a stronger response to a feature when it is exaggerated as opposed to a previously presented similar instance of that feature. For example, if a rat is trained to respond to a rectangle and avoid a square. When it is presented later with several rectangles, it will have the strongest response to the rectangle with the most elongated aspect ratio rather than to the rectangle most similar to the one that was used in training. In that sense it does not respond to the object, it responds proportionally to the underlying pattern.

The peak-shift principle can be seen as a possible method for visual abstraction and is therefore also fundamental to abstract art [Ramachandran 2001, Zimmer 2003]. In practice this starts by identifying the features that are essential to what the artist is trying to convey. Once identified there must be an underlying pattern to that essential feature. To identify this underlying pattern we need to determine an anchor point and a path away from that anchor point. This path should contain a continuum of the exact perceptual characteristics of the essential feature. If we travel along the path away from the anchor point, then in principle we increasingly exaggerate the essential characteristics. This exaggeration creates a starker contrast between similar instances of features (the anchor point), emphasizing the essential feature, thereby exploiting the relative nature of visual perception [Ramachandran & Hirstein 1999, Zimmer 2003]. Hence exaggeration leads to specification. As such an exaggerated feature stands out in terms of visual perception and easily captures attention. It has been shown that such a process increases visual information processing both in terms of speed and intensity, and in some cases manipulates visual information to the point that it turns out to be better recognizable than its realistic counterpart [Ramachandran & Hirstein 1999, Lehrer 2009]. Note that

isolation of a specific modality in combination with the peak-shift principle can strengthen its effect in that modality [Lehrer 2009].

The peak-shift principle is well illustrated by the work of caricaturists [Ramachandran & Hirstein 1999]. A caricaturist is trained to identify the essential features of for example the human face. If the caricaturist caricatures Barack Obama, the essential features, in this case the features that we use to distinguish between faces, of Barack are compared to those of the average face of all people. This average face becomes an anchor point, and the differences between essential features determine the path away from the anchor point. Perceptually this makes for a clearer Barack-experience. It is clear that the success of using the peak-shift principle in visual art is highly dependent on the anchor point and chosen path away from it. Naturally this phenomenon is not exclusive to caricatures or any other type of figurative depiction and thus can be applied to realistic as well as abstract and representational as well as non-representational features. In that sense one could say that Kazimir Malevich peak-shifts the essentials of form [Malevich 1915], Mark Rothko those of color [Rothko 1961] and Alexander Calder those of movement [Calder 1960].

In summary, the contribution of exaggeration using peak-shift principle is threefold: It can be applied as a general intensifier; a way to prevent misinterpretation as it is easier to distinguish between similar instances of essential features; and a way to allocate attention. In any case, the challenge remains how to identify essential features, the anchor point and the appropriate direction or path away from it.

2.2.3 Translation

Figure 4: Composition VIII (1923) by Wassili Kandinsky

Figure 5: Number 23 (1948) by Jackson Pollock

A different fundamental aspect of abstract art can be found in the way experiences natural to one modality can be conveyed using another. As any type of visual information the core experiential quality of visual information is mediated by essential features. Some features may be specific to one modality. Others however, have been shown to translate to different modalities as well. In other words, essential features specific to one modality can sometimes be mapped to different media. This suggests that essential features can mediate an experience or characteristic natural to one modality to another through translation. Such phenomena are fundamental to the work of amongst others

Wassili Kandinsky (figure 4) [Ward 2010]. Kandinsky was one of the first visual artists to venture into abstraction, but also went a step further. Kandinsky wanted his paintings to evoke the experience of sound through sight, a painted equivalent to the musical symphony that would not only stimulate the eyes but the ears as well through vision alone. Such vision-sound correspondences have been evidenced by Köhler (1947), who showed that there exist cases where sound is consistently translated to shape based on corresponding features. In his experiment he found that when using nonsense words, rounded vowels (bouba) translate to rounded shapes and sharp angular vowels (kiki) translate to angular shapes [Ramachandran & Hubbard 2003]. Similarly research on synaesthesia elaborates on such correspondences [Ramachandran & Hubbard 2001, Ramachandran & Hubbard 2003]. The flexibility of translation is well illustrated by the work of Jackson Pollock. Freedberg & Gallese (2007) showed that the emotional impact of the work is guided by the painterly evidence of the motion and gestures made (Figure 5), and as such translates movement to a static image. Do note that although features can be translated over modalities, it is not always clear whether its affective quality is retained.

2.2.4 Multiplicity

Another perceptual mechanism fundamental to abstract art comes in part from a recent development called generative art, and is termed multiplicity [Whitelaw 2007]. In most cases the quality of essential features is the focal point of abstract work. Multiplicity extends this by exploiting quantity as well. Multiplicity is characterized by the use of very simple features such as basic geometric primitives in very large multitudes. It is believed that the quality of this approach relies in part on the formation of gestalts (grouping) [Whitelaw 2007]. Complementary Aronoff (2006) evidences that the intensity of affective quality is in part determined by the amount of affectively meaningful features present (figure 6). So to rephrase: the more of a certain feature, the higher its intensity is perceived.

Figure 6: Process 9 (2006) by Casey Reas.

2.3 Composition

As discussed, one fundamental aspect of abstract art is to identify the essential features (which we will later see in section 3.4) and operate on these features in such a way that it conveys the intended experience (section 2.2). Another fundamental aspect of abstract art revolves around the question what perceptual mechanisms are relevant to combining essential features as combining (i.e. composing) different features can have a profound impact on the perception of these features. Hence following the same perspective on abstract art we distinguish two major complementary perceptual mechanisms relevant to composition: contrast and grouping.

2.3.1 Contrast

We can define contrast by the difference between features or modalities. This is relevant in composition because high contrast areas are shown to be more attention grabbing than homogeneous areas [Ramachandran & Hirstein 1999]. As such contrast can be used to accentuate essential features. Contrast often appears between dissimilar features that are close together, for example, aligning different textures next to each other will create a highly contrasting area where the textures meet. As such they create an edge, which automatically captures the eye’s attention. Next to allocating attention to specific contrasting regions, contrast is also of importance because of the relative nature with which features may be perceived. For example, a bright color may be perceived brighter if it is contrasted to a dark color. It is possible that contrast plays a role across modalities as well. This is well illustrated by the work of kinetic artist Alexander Calder, who wrote that motion was most efficiently represented by the juxtaposition of highly contrasting surfaces [Zeki & Lamb 1994]. To attain the desired contrast Calder limited the color use in his work to black, white and red, for which he claimed that red was the color that contrasts the most with black and white, while the use of other colors confuse the clarity of motion. Calder’s intuitive understanding is confirmed by findings in neuroscience that suggest that motion is indeed best perceived by contrasting surfaces as motion perception is sensitive to luminance [Zeki & Lamb 1994].

2.3.2 Grouping

As is contrast, grouping (and binding) is a perceptual mechanism that is also concerned with the discovery of objects. Gestalt psychology describes the perceptual mechanisms used to perceptually unify features into a group or into a whole. As such it is evident that combinations of features are interpreted in ways that are not predicted based on considering each feature in isolation. For example, features may group together when they are symmetrical to each other or configured in parallel, move in the same direction or change simultaneously, as is explained by the gestalt grouping principles. For an overview of the gestalt grouping principles we would like to refer to Wolfe et al (2005). The importance of grouping in relation to affect is shown by the work of Aronoff (2006) who evidenced in a study on 17th century Dutch art that people attributed emotions based on the amount of angular versus rounded forms found in a work of art. The angular forms were associated with negative emotions and rounded forms with positive emotions. Emotion attributions were in part based on the formation of angularity resulting from the configuration of gestures made or groups of people distributed over the painting, i.e. grouping. These were shown to have the same effect as those forms would if they were perceived singular and isolated [Aronoff 2006]. In other words, forms created by grouping also convey meaning and affective quality as if they would be singular. For further use of Gestalt psychology in abstract art and design we would like to refer to van Campen (1997) and Arnheim (1974) for an extensive overview.

2.5 Conclusion on Abstraction

The reviewed literature shows the basic perceptual mechanisms governing abstract design, it must however be noted that this involves highly experimental research. Additionally, we see a distinction between those specific perceptual mechanisms, and the role of essential features. Where essential features highly depend on the visual information that is conveyed, and the discussed perceptual mechanisms operate on such essential features in terms of attribution and attention. In general, keeping track of novel developments in the arts as well as neuroaesthetics and perception science may aid in future developments of our research.

3. Affect

Affect can be defined as the experiential quality common in emotion, mood, affective attitude and feeling. Affect is thought to be woven through the whole of cognition and as such plays a fundamental role in cognition [Damasio 1994, LeDoux 1998], perception [Atkinson & Adolphs 2005, Pessoa & Adolphs 2010] and behavior [Fischer & Manstead 2008]. Emotions are relatively short but intense episodes that influence attention, action selection, action preparation, memory, motivation and homeostasis. Emotions in principle are attributed to a causal process which can be of a perceptual, appraisal, and conceptual nature [Izard 2009]. In contrast, emotions differ from mood in the sense that mood is thought not to be consciously attributed and is a less intense but longer-term affective state. For example, a mood may arise when an emotion is repeatedly activated, accumulating into an overall affective state. Affective attitude refers to the learned affective valuation of objects and events. Whereas with feeling we refer to the actual inner experience of affect.

3.1 Theory of Emotion

Contemporary affective science distinguishes three major theories on the cognitive structure of emotion: discrete, continuous and componential. The discrete theory of emotion states emotions are distinct categories [Izard 2009]. Emotion categories can be described as a complex of expressive, physiological, behavioral and experiential characteristics. For example, joy could be described by exhibiting a happy facial expression, being physiologically highly aroused, exhibiting playful behavior and experiencing a good feeling. Many basic sets of emotion categories have been proposed, a famous example is Ekman’s six universal emotions: anger, disgust, fear, joy, sadness and surprise [Ekman 1993]. In contrast, the continuous approach states affect can be described along (orthogonal) dimensions [Russel 2003, Barrett 2006]. Here, typically a two-factor model is used with dimensions of pleasure/displeasure and arousal. Other dimensions have been proposed as well depending on the focus of research, for example pleasure, arousal and dominance to account for social emotions [Mehrabian 1980]. The componential theory states emotions are formed by the combined activation of different underlying cognitive and physiological components [Scherer et al 2001]. Typically, componential theory regards emotion as the result of an evaluation of the environment against personal relevance (cognitive appraisal theory). A famous example being Scherer’s stimulus checks, where the evaluation of an event in terms of appraisal dimensions (novelty, pleasantness, goal/ need conductiveness and coping potential), culminate into specific emotional episodes.

Focus About what? Emotion representation

Dimensional Categorical Componential

Social Theories emphasizing interaction: communication, social phenomena, and emotion expression

Pleasure, arousal, dominance

Pride, remorse, guilt, etc.

Psychological Theories emphasizing individuality: cognitive appraisal, feeling, mood, personality traits, attitude, emotion effects on cognition

Pleasure, arousal

Happy, sad, surprised, anger, fear

Appraisal variables, BDI

Biological Theories emphasizing species and individual adaptation: survival, learning, motivation, adaptive behavior

Hormones and homeostasis

Emotional behaviors (fight/flight)

Drives, needs, rewards

Physiology Theories emphasizing physiology, neurology and stress

Hormones and neuro-transmitters

Emotion-specific brain areas (modularity)

Neural architectures

Table 1: Overview of emotion theories along axes of focus and representation.

The particulars of emotion theories differ depending on the focus of research, which we can roughly divide in social, psychological, biological and physiological perspectives on emotion (Table 1). However, affective science is still very much in debate on the nature of emotions and complementing theoretical constructs. As such, a further review would not do justice to current developments. Instead we would like to refer to some of the major contemporary literary works in the field [Lewis et al 2008, Davidson et al 2009].

3.2 Visual Emotion Expression

An emotional episode can manifest itself physically in what is called an emotion expression, i.e. muscle and motor actions that visibly reveal ones internal emotional state, e.g. facial expressions [Matsumoto 2008], body postures and gestures [Wallbott 1998]. Emotion expressions play a fundamental role in interpersonal communication [Clark & Brissette 2003], e.g. non-verbal communication and efficiency (affective channel) [Fischer & Manstead 2008, LeDoux 1994] as well as inducing empathy and evoking prosocial behavior [Hoffman 2008], and can occur with or without being aware of the expression [Picard 1997]. Facial expressions are thought to communicate emotions by independent expressive muscles resulting in minute differences in the relative position of facial features (e.g. the eyebrows, nose, mouth) [Duchenne 1990]. Facial expressions are restricted by social display rules, which determine the specific actions and intensity of expressions appropriate in particular social situations [Matsomuto 1990]. Similarly in the case of emotion expressions by the body, emotions may result in specific muscle tensions that influence movement behavior and

postures [Lowen 1971], as well as the relative configurational aspects of anatomical features, e.g. head, shoulders, upper body, arms, hands and movement qualities [Wallbott 1998].

3.3 Visual Emotion Recognition

The role of visual emotion recognition is evident as it mediates the interpretation of emotion expressions [Atkinson & Adolphs 2005, Izard 2009, Russel 2003, Scherer et al 2001, Pessoa & Adolphs 2010]. Current research hypothesizes on the involvement of and interaction with of a variety of visual processing mechanisms, object recognition and salience, delegation of attention and the mechanisms that generate emotions [Pessoa & Adolphs 2010]. Particularly relevant to our work is the separation between two major processing mechanisms: the extraction of configurational information, i.e. the relative positions and limitations of anatomical features of the human body and face [Atkinson et al 2007, Lundqvist & Öhman 2004], and the extraction of emotionally salient features, i.e. emotionally essential features [Aronoff 2006, Lundqvist & Öhman 2004]. Though essentially a separate process, it is thought that when configurational information is present, it interacts with the extracted essential features, and weights them according to a specific hierarchical structure [Lundqvist & Öhman 2004]. Experimental evidence however shows that the involvement of configurational information is not a prerequisite for recognizing emotions. It merely overrides the effect of individual features on the condition that configurational information is available [Lundqvist & Öhman 2004]. As such, recognizing an emotion does not require the recognition of the face or human body, nor its configuration per se [Atkinson & Adolphs 2005, Aronoff 2006, de Gelder et al 1999, Lundqvist & Öhman 2004]. Recognizing emotions independent of configuration is evidenced to not only enable the recognition of emotion expressions from the human body and face, but also mediates the (mis)attribution of emotions to places and things. Such evaluations are mediated by essential features in visual emotion recognition [Aronoff 2006]. This claim is complemented by a large body of work on emotion attribution to simple geometrical shapes, that are shown to evoke specific emotion attributions, even when presented in isolation, and devoid of any reminiscence to the human body or face and its configuration, based solely on the presence of specific essential features of visual emotion recognition.

3.4 Essential Features

Following these findings, we review emotion attribution research and limit ourselves to research related to movement and form features derived from human bodily and facial expressions. The reviewed literature shows experimental evidence of emotional attribution independent of the context and configuration of the human body and face and in particular towards simple two dimensional shapes and features.

3.4.1 Movement

Literature suggests that motion is sufficient to evoke basic emotion attributions. Such attributions are shown to be automatic and consistent, and independent of form. We can distinguish motion described in kinematics, dynamics in terms of perceived instability and relative motion to play an essential role in visual emotion recognition.

Figure 7: Heider and Simmel (1944) type animation (reproduction)

A relatively large body of experimental work exists on social and emotional attributions to the movement and perceived interaction of simple geometrical shapes. These attributions are found to be automatic, involuntary and consistent [Scholl & Tremoulet 2000]. One of the first of such experiments was performed by Heider & Simmel (1944), who revealed that subjects attribute human-like role and intent to simple animated shapes based solely on their movement characteristics (figure 7). Subjects tend to describe such animations in terms of shapes vigorously chasing one another, getting ready for attack or being madly in love. These social attributions are mostly informed by the relative motion and orientation between objects independent of the form of the objects. Especially the features speed, direction [Scholl & Tremoulet 2000] and perceived goal-directedness [Dittrich & Lea 1994] play an important role. The emotional attributions are evoked through the perceived intentionality of the shapes. See Scholl & Tremoulet (2000) for a complete overview of such attributions and their key features.

Complementary, recent research shows that affect from movement can be described by movement traces, kinematic descriptors and especially specific combinations of kinematic descriptors. Research shows that the relative interpretation of kinematic features is more informative than the absolute interpretation of kinematic features [Lee et al 2007, Pollick et al 2001, Saerbeck & Bartneck 2010, Sawada et al 2003]. Especially average speed and movement time have been shown to be strong features in emotion attributions towards movement patterns [Pollick et al 2001, Sawada et al 2003]. In general the combination of movement features increased speed and jerkyness evoke attributes of anger, where fearful and sad movements are associated with smaller and slower movements [Pollick et al 2001, Rime et al 1985, Roether et al 2010, Sawada et al 2003]. Pollick et al (2001) add that fast and jerky movements may also evoke attributions of happiness, while Lee et al (2007) adds that movement smoothness is an indicator for pleasantness. Furthermore, Pollick et al (2001) state that energetic motion is characterized by shorter direction, acceleration, jerkiness and greater magnitudes of average and peak velocity. Complementary, Saerbeck & Bartneck (2010) found the level of acceleration can be used to predict the perceived arousal of a motion pattern, while Lee et al (2007) show activation is correlated to average velocity. There also exist cases where form features are translated to movement traces. For example, the angularity and curvature of movement traces are shown to evoke emotion attributions [Aronoff 2006, Bassili 1978]. Saerbeck & Bartneck (2010) add that valence is in part in the interaction of acceleration and the curvature of movement patterns. In general, attributions of emotion to the kinematics of movement remain steady independent of the

realism [Oatley & Yuill 1985, Rime et al 1985] and embodiment [Saerbeck & Bartneck 2010], which shows that movement can express affect as an isolated modality. In general, research shows that although valenced attributions are made towards essential kinematic features, kinematic features relate most strongly to the perceived arousal of expressions [Pollick et al 2001].

Additionally, movement, or implied movement described in terms of implied dynamics are is shown to play a role in emotion attribution. Pavlova et al (2005) found that the dynamics implied by a static image enables specific emotional attribution. The perceived instability is dependent on form on the perceptual level, but the emotional attributions made are consistent with the measure of perceived instability, and as such do not directly rely on form. Perceived instability increasingly entails attributions of negative emotions. This is especially true for negative emotions such a fear and suffering. Vertical orientation was found to evoke positive attributions, especially joy and surprise. It has been suggested that these findings correspond to the dynamics in bodily expression in terms of balance and imbalance. We tend to take upright postures and make upright movements when we are joyful or surprised. We lose our dynamic balance and fail to hold ourselves upright when we are fearful or suffering [Pavlova et al 2005]. As such, the implied dynamics are shown to have a strong effect on valence.

3.4.2 Form

Complementary to movement features, literature suggests an important role for form features in visual emotion recognition. Essential form features are evoke emotion attributions independent of movement. From literature, we distinguish the basic forms and form features angularity, roundness, V-shapes, asymmetry and size as playing a central role in emotion attribution.

Studies by Aronoff (2006) show the role of geometric features in emotion attribution. The primary geometric features of roundness, linearity and angularity, diagonality make a substantial contribution to the perception of emotional expression [Aronoff 2006]. Increasing the angularity of a contour or diagonality is shown to evoke increasingly negatively valenced, arousing and potent attributions, while a rounded contour and linearity were shown to elicit a more positive response, be less arousing and did not show an effect on potency. These features were also found to play a role in mis-attributions towards everyday objects and evoke emotional attributes when isolated, devoid of contextual information, invariant and hidden within gestalts (grouping) [Aronoff 2006]. There exists some evidence for the exception that curvature can also evoke negative valence, low arousal attributions [Collier 1996]. Additionally, contour features were shown to play a role in emotion attribution towards emotion categories of love, joy, and anger, fear, but not sadness [de Rooij et al 2011]. The role of contour was confirmed in several independent studies using simple geometrical patterns [Aronoff 2006, Aronoff et al 1992, Bar & Neta 2006, Bar & Neta 2007, Collier 1996, de Rooij et al 2011, Silvia & Barona 2009].

A special case has to be made for V-shapes and downward pointing triangles. Early experimentation on static geometric form and the perception of affect was done by Bassili (1978). His research first showed that facial contours expanded outwards forming rounded shaped movements on posing a happy expression, while when posing an angry face the dots retracted downward forming a V-shaped angle. V-shapes and downward pointing triangles were shown to receive significantly more negative attributions than similar shapes with an angular contour in a different orientation [Aronoff 2006], V-

shapes and downward pointing triangles are shown to elicit a specific threat response [Aronoff 1988, Larson 2007 et al, Larson et al 2008].

Locher & Nodine (1989) show that for simple geometrical patterns asymmetry elicits heightened arousal while symmetry elicits low arousal. His work shows a connection to the complexity of visual processing and the perception of arousal in general. It was shown that vertical symmetry is deemed more simple horizontal symmetry, horizontal is in turn deemed simpler to process than diagonal symmetry [Reber 2004]. Additionally, studies indicate that asymmetry may evoke valenced responses as well [de Rooij et al 2011].

Additionally, evidence exists that the size or height of people and animals increases the perceived dominance and social status [Ellis 1994]. Studies using simple geometrical patterns show that the surface area of a shape is associated with attributions of potency [Aronoff 2006]. This is found to be the case in gestalts (grouping) as well. Size was also shown to affect the perceived arousal of expressions [de Rooij et al 2011]. Additionally experimental evidence exists that shows the increased or decreased circumference, the circumference measured by connecting the outer points of a shape, is a good predictor for the maximum intensity of emotions, as well as the emotion concept surprise [de Rooij 2011].

It was found that from the form features contour, asymmetry and size, the contour features roundness and angularity affect emotion attribution most strongly, these are used to distinguish between whether an emotion is good or bad, pleasurable or displeasurable. Additionally, asymmetry and size were shown to affect attribution, but to a much lesser extent. Increased and decreased size complements the contour features by affecting the perceived arousal and dominance of an expression. As such, the combination of the contour features angularity or roundness and size is sufficient to distinguish between most basic emotional qualities [de Rooij et al 2011]. Additionally asymmetry was shown to have an inverse relation with the affective dimensions pleasure and dominance, on the prerequisite that the contour features angularity or roundness are dominant in distinguishing between pleasure and displeasure. On the condition that contour features affect attribution most strongly, asymmetry combined with roundness decreases the perceived pleasure, while combined with angularity, asymmetry decreases the perceived displeasure (or increases the pleasure) [de Rooij et al 2011]. In a similar way, asymmetry affects the perceived dominance, when contour features are dominant increased asymmetry lessens the perceived dominance, but also lessens the perceived submissiveness [de Rooij et al 2011]. Additionally, a specific downward orientation of asymmetry was found to evoke emotions of sadness, effectively overruling the effect of contour features [de Rooij et al 2011]. However on a general note, it is proposed that because of the dominant role of contour, the main contribution of essential form features lies with distinguishing between positive and negative.

3.4.3 Hierarchy and Dependence

The reviewed literature shows emotion attributions to essential features of motion and form. What is clear from literature is that essential motion features are thought to be sufficient for basic emotional attribution, as is the application of form features. The role of specific combinations of essential features is however unclear. Research suggests there is no fundamental difference between form and the change of form. As such, spatio-temporality is not likely to be essential in emotion attribution when no configuration is present [de Rooij et al 2011]. It has been speculated that when combining

form and movement, movement is used to distinguish between basic emotions, while form may complement motion and allows for the recognition of more complex emotional qualities [Atkinson et al 2007]. Such speculation is motivated by emotion attributions made towards moving shapes, where attributions are made based on their relative motion, independent of form [Heider & Simmel 1944]. These findings suggest a hierarchy among the two modalities movement and form. However, we could argue for an alternative interpretation. Kinematic features were shown to affect the perceived arousal more strongly [Pollick et al 2001], while form features are shown to affect pleasure the most [de Rooij et al 2011]. Therefore, movement and form may have a complementary relation. Such a proposed complementary relation is however pure speculation on our account. In general, to the knowledge of the authors no experimental evidence is available on the hierarchy and dependence among motion features expressed in kinematics and dynamics, the relation between kinematics and form features, and dynamics and form features with respect to emotion attribution.

3.5 Conclusion

Note that attribution research is very fragmented and leaves open some key questions relevant to our research. These involve the role of dynamics in terms of mass and force, and especially the hierarchy and dependence among modalities with respect to emotion attribution. Keeping track of developments in attribution research and visual emotion recognition can aid future development of our research.

4. Robots

Affective Robots are those autonomous agents that explore or exploit affect in analogy to the role of affect within human cognition, behavior or social interaction. As such, robotics research can benefit from affect in a variety of ways. Affect can improve the robustness of autonomous agents in terms of interactivity and adaptivity to function in dynamic and uncertain environments [Cañamero 2005], research has focused in particular on the role of simulated affect in action selection [Cañamero 2002, Murphy et al 2002], learning [Gadanho 2003, Velasquez 1998] and memory management [Dodd & Gutierrez 2005¸ Velasquez 1998]. However, a majority of research in affective robotics focuses on the design of robots that require affective properties to function in social environments [Fong 2003]. These robots not only require cognitive and behavioral models suitable for natural affective man-machine interaction, but also require a suitable morphology to communicate affect, i.e. a body that allows for emotional expressions [Fong et al 2003]. Equipping machines with expressive capabilities can facilitate natural and efficient communication [Picard 1997], induce an emphatic response [Picard 1997] and evoke prosocial behavior in the human user [Takeuchia & Hada 2006], and plays a fundamental role in attaining believability for robots [Rose et al 2008].

4.1 Embodiment

Contemporary techno-scientific research on the design of affective robots may draw on the theory of embodiment: the hypothesis that perceptual, motor and cognitive processes are for a large part determined by the form or morphology of the (human) body and its interactions with the world [Wilson 2002, Ziemke 2003]. This also holds for social and emotional processes [Barsalou et al 2003, Niedenthal et al 2005], e.g. the relation of visual emotion expression and visual emotion recognition.

With the emphasis on designing affective robots for social environments and interaction with humans, the theory of embodiment entails a design approach that aims to find novel ways to match the artificial body of the affective robot with the human cognitive and perceptual properties and requirements. In any case the theory of embodiment points out the importance of morphology in the design of affective robots.

4.2 Design strategies

Following this approach, design strategies for affective robots attempt to match the physical characteristics of an affective robot with the cognitive and perceptual properties and requirements necessary for affective man-machine interaction. Taking a user perspective, this involves the means by which the affective robot influences the user’s emotional and social information processing (e.g. emotion expressions) and the implications of design strategies on the attitude and expectations of the user towards affective robots. Inspired by Dautenhahn (2002) and Blow et al (2006), design strategies for affective robots and emotion expression can be described along axes of realism, iconism and abstraction (Figure 8). As of today little experimental research has been done on what the optimal design strategy is for affective and social robots [Broekens et al 2009].

Figure 8: Design space for affective robots (modeled after [Dautenhahn 2002, Blow et al 2006]). Abstract: Lovotics - Samani et al (2010), realistic: Repliee Q1 – Ishiguru (XXXX), iconic: Flobi – Lütkebohle et al (2010).

4.2.1 Realistic

The obvious is to equip robots with expressive features that are similar to those of humans and animals. It has been hypothesized that an increasing amount of realism yields and increasing amount of positive and empathic responses from humans to realistic affective robots [Mori 1970]. Such an approach draws on familiarity and attractiveness, which has influence on the willingness of people to interact with a robot [Hegel et al 2009]. An increased level of realism also increases the perceived affordance of an affective artifact as it leaves less to interpret, but also raises the expectations of its cognitive and interactive capabilities [Duffy 2003, Duffy 2008, Kiesler 2005]. These expectations rise upon encounter and shape the attitude and social behavior of the user towards the artifact. The difficulty for the realistic design approach is that realism itself becomes the focal point of the research and not necessarily the ability to convey affect in general, as it is assumed that believable emotional expression will be the by-product of good realistic design. Technically this is very challenging as the full embodiment of the artifact should be matched to that of the human or animal it is trying to emulate. As such realistic design is bound to match the design of the affective robot to the mental model, the specific concepts and characteristics the user already associates with a real human or animal. Deviation from the mental model has consequences for the quality of the interaction with and attitude towards an affective robot in terms of believability and acceptance [Goetz et al 2003, Kiesler 2005, Rose et al 2008].

A famous example of such a mismatch between a realistic human-like robot and the mental model of a user is described by the “uncanny valley” theory. The uncanny valley describes that the positive effects of increasing realism suddenly decrease upon nearing full realism and evoke attributes of revulsion [MacDorman et al 2009, Vinayagamoorthy et al 2005]. After the valley the positive effect again increases with increasing realism. This mismatch of the user’s mental model can be explained by; or the combination of different levels of realism results in something unnatural, or this combination of elements matches a different mental model, for example a diseased or deceased person. Solutions involve matching the robots appearance and behavior to the task it is used for [Goetz et al 2003], matching the realism of appearance and realism of movement [Vinayagamoorthy et al 2005], or a match in realism of face proportions and realism of texture [MacDorman et al 2009].

4.2.2 Iconic

A decrease in realism entails a generalization of the face and body and an emphasis on the particular communicative features [Blow et al 2006]. Specific information relevant to distinguish between different people is reduced, while isolating and emphasizing anatomical features sufficient social and affective interaction. Iconism therefore emphasizes the expression, or social content [Blow et al 2006]. Such a generalization allows us to project personality on the artifact and allows users to identify more easily with an affective robot [Blow et al 2006] and as such allows a user to create a mental model that matches the affective artifact through projection and identification. The iconic design approach relies heavily on knowledge from animation [van Breemen 2004] and may benefit from caricature as well [Mead & Matarić 2010]. Especially four features are important: the mouth, nose, eyes and eyebrows to represent the face, but also the line of pose to represent the body postures [Thomas 1995], as well as simplified configuration and movement of those features sufficient for expression. The decrease in realism may generate less initial expectations by the user of the artifact as opposed to if it were a full realistic design and more easily prevent a mismatch with the mental model because both the perceptual and design complexity is reduced considerably [Lütkebohle et al 2010]. Therefore the iconic approach is often favored for technical reasons.

4.2.3 Abstract

A design strategy for abstract affective robotics as we describe in this paper focuses on designing physical artifacts that reflect the essential components of visual emotion recognition [Section 3.4], i.e. the features most fundamental to the recognition of emotion expression and the perceptual mechanisms to operate on them for design. We want to make clear this approach differs fundamentally from the iconic and realistic approach at the representational level, as these approaches focus on modeling anatomical features used in emotion expression, and are as such dependent on the resemblance and configuration of the human body and face. Design principles for abstract affective robots focus on the minimal and essential representations of emotion recognition and use these to model emotion expression outside the context of the human body. Design principles are gleaned from visual information processing and focus on the general perception of essential features themselves. As such we speculate the abstract design approach is on the one hand a shortcut to the perception of emotion, as the essential features from visual emotion recognition we normally have to search for are emphasized and isolated, i.e. easy to perceive. On the other hand abstract design may fit the mental model of technological artifacts better because of its apparent simplicity [Dautenhahn 2002]. Hypothetically, users do not have to bother with the expectations we have of real humans or animals as we under-attribute the capabilities of abstract affective robots. This could entail that users can discover the affective quality of an abstract affective robot naturally during interaction, i.e. through interaction increasingly construe a fitting mental model of the artifact. However, the downside of abstract design could be that the perceived affordance of such designs is relatively low in social situations.

4.3 Related projects

The abstract design strategy has been applied marginally within the field of affective robotics. However, there exist some related projects within the robotics community that relate to our review on essential features. This mainly involves the usage of whole body movement independent of form to endow robots with a synthetic means of expressing emotion independent of their morphology [Barakova & Lourens 2009, Bethel & Murphy 2008, Maeda 1999, Saerbeck & Bartneck 2010]. Similarly Harris & Sharlin (2010) report on minimizing form and the usage of movements as an isolated modality for emotional expression. In further relation to essential features of emotion expression and robotics, form and spatiotemporal form is actively being researched in the field of self-reconfigurable or modular robotics [Stoy et al 2010], but has as of today not examined any means of emotional expression. To the knowledge of the authors the attempt closest to what we describe in this paper as abstract design is done by Samani et al (2010) in the Lovotics project. The Lovotics project aims to develop a bi-directional friendship-love relationship between the robot and the user. Design is used as a basis to engage users to explore the possibilities of such relationships. To uncover the essential factors needed for appearance and expression they have taken a user-centered design approach and found their aim is best reflected by using an oval form and by expressing affective behavior through rotation, vibration, tilt, variable height and variable color. Abstract design has however been explored a bit more in depth in other techno-scientific application areas than that of robotics. For further reference, these include affective evaluation methods [Laaksolahti et al 2009, Isbister et al 2006, Isbister et al 2007], communication technology [Liu et al 2003, Scheirer & Picard 2000, Stahl et al 2005], interface design [Sanches et al 2010, Scheirer & Picard 2000, Sengers et al 2002] and ambient displays [Boehner et al 2003, Mutlu et al 2006].

4.4 Conclusion on Robotics

A survey on related projects shows that there exists some work on exploiting movement and other minimal features as an expressive quality in mostly functional robots. Although this shows that some minor aspects of abstract design have been used, in general we can conclude that the abstract design approach to affective robots has received little attention. Our explanation of design strategies positions our research on design guidelines for abstract affective robots in the whole spectrum of affective robots and hints on some preliminary theoretical and practical implications of the abstract design approach concerning design theory and user attitude. Additionally, our review on affective robotics shows a largely unexplored and undeveloped niche for abstract affective robotics.

5. Design Guidelines for Abstract Affective Robots

The aim of our review on affective robotics, affective science and the science behind abstract art is to unify that research into a set of guidelines for the design of abstract affective robots. Based on this review we are able to distinguish two main design phases. The first design phase pertains the application of essential features from visual emotion recognition to the robot morphology. An abstract design should contain sufficient essential features and reflect the structure of visual emotion recognition such that emotions can be conveyed by the design. The second design phase pertains the design tools, or operations on these essential features, that affect the perception of essential features in terms of attribution and attention.

5.1 Features

Essential features are the fundamental building blocks of abstract affective design. The application of these features allows us to compose synthetic emotional expressions by combining different variations and combinations of these features into a robots morphology. In principle, essential features can be selected, implemented and matched to the intended expressive qualities and restrictions of the design. We provide a summary of our review on essential features in table 2 and 3. However, composing essential features into specific emotions may not be straightforward at all times. Visual emotion recognition poses a limitation on the interpretation of combined essential features by specific hierarchies and dependencies among features. As such, more complex designs must not only reflect the essential features but also the structure of visual emotion recognition in order to compose consistent synthetic emotion expressions.

Essential Feature Attribution Reference

Perceived instability negative emotions, fear, suffering

Pavlova et al 2005

Perceived stability positive emotions, joy, surprise Pavlova et al 2005

Acceleration perceived arousal Saerbeck & Bartneck 2010

Velocity activation Lee et al 2007

Smoothness of movement pleasure Lee et al 2007

Fast, jerky movement happiness Pollick et al 2001

Large, fast, jerky movement anger Pollick et al 2001, Sawada et al 2003, Rime et al 1985, Rime et al 1985

Small, slow movement sadness, fearful Pollick et al 2001, Sawada et al 2003, Rime et al 1985, Rime et al 1985

rounded movement trace Positive emotions Aronoff 2006, Saebeck & Bartneck 2010

Angular movement trace Negative emotions Aronoff 2006

Relative motion and perceived goal-directedness

social attributions, animacy, emotional attributions through perceived intentionality.

Dittrich & Lea 1994, Heider & Simmel 1944, Scholl & Tremoulet 2000

Table 2: Effects of essential movement features on emotion attribution.

Essential Feature Attribution Reference

Roundness, curvilinearity positive valence, lessened arousal, no effect on potency, happy, intimate, warmth

Aronoff 2006, Bar & Neta 2006, Collier 1996, de Rooij et al 2011, Silvia & Barona 2009

Roundness negative valence, low arousal, sadness

Collier 1996

Angularity, diagonality negative valence, heightened arousal, heightened potency, threat, coldness

Aronoff 2006, Bar & Neta 2006, Collier 1996, de Rooij et al 2011, Silvia & Barona 2009

V-shapes, downward Pointing triangles

threat Aronoff 2006, Aronoff et al 1988, Larson et al 2007, Larson et al 2008

Asymmetry heightened arousal, valence*, dominance*

De Rooij et al 2011, Locher & Nodine 1989

Symmetry low arousal, valence*, dominance*

De Rooij et al 2011, Locher & Nodine 1989

Size, height, surface area, circumference

Arousal, potency, surprise Aronoff 2006, de Rooij et al 2011

Table 3: Effects of essential form features on emotion attribution. *) These features were shown to have a negative relation with respect to emotion attribution (section 4.3.2).

For example, we could design a robot that can vary its contour in terms of roundness and angularity, and can increase and decrease its overall size. The provided tables 2 and 3 contain their effects with respect to emotion attribution. Based on this information we have designed a robot that can theoretically compose synthetic expressions by varying its contour features and its size where increasing its roundness and angularity affects the perception the robot in terms valence or pleasure, and varying the size of the robots expression in terms of perceived arousal and dominance. More specifically when for example roundness is complemented by increased size the expression will be perceived as positive, aroused and dominant. Such a description matches emotions such as joy and happiness [table 5, de Rooij et al 2011]. The application of contour features in combination with increased size proves to be straightforward.

Affect dimension roundness angularity asymmetry size

pleasure + x - x

displeasure x + - x

arousal x x x* +

dominance x x - +

submissiveness x x - +

Table 4: Example of the combined effects and relational aspects of essential form features with respect to the pleasure-arousal-dominance model taken from de Rooij et al (2011). Where a “+” denotes a positive relation to the affect dimension, “-” denotes a negative relation and “x” denotes no relation.*) results on the contribution of asymmetry to arousal are ambiguous (see de Rooij et al 2011).

Emotion category Roundness Angularity Asymmetry Size

Love + x - x

Joy + x - +

Surprise x x x +

Anger x + - +

Sadness x x +* -

Fear x + - x

Table 5: Example of the combined effects and relational aspects of essential form features with respect to emotion categories taken from de Rooij et al (2011). Where a “+” denotes a positive relation to the affect dimension, “-” denotes a negative relation and “x” denotes no relation.*) especially a specific orientation downward and downward-right in combination with asymmetry were shown to relate to sadness (see de Rooij et al 2011).

Now lets revise the design of the robot and add the capability of changing from symmetry to asymmetry. Here, the limitations posed by the structure of visual emotion recognition becomes more evident (table 4, table 5, de Rooij et al 2011). From table 3 we can see that asymmetry has been linked to increased arousal, and was also shown to affect valence. However, research on their combined effects shows that first, asymmetry has a negative relation with both contour and dominance and second asymmetry impacts pleasure much less than contour [de Rooij et al 2011], third, this effect is only true on the condition that contour features are the dominant factor in emotion attributions towards the robot. If this is not the case, a particular combination of downward and especially downward-right can overrule the effect of contour and evokes attributions of sadness [de Rooij et al 2011]. Additionally, asymmetry has a negative relation to dominance, this impact resembles the impact of increased size in strength. As such a several different combinations of asymmetry and size may be shown to evoke similar levels of perceived dominance.

Therefore, when designing an abstract affective robot, composing essential features into specific synthetic expressions can at times be straightforward. The structure of visual emotion recognition does however poses different restrictions on specific combinations of features, which affects their interpretation with respect to emotion attribution. This is dependent on what and how many essential features are combined. Though an example was shown based on previous work [de Rooij et al 2011], there are many open questions pertaining the hierarchy and dependence among essential features, and especially among combining essential features from different modalities. Therefore, additional attribution research on specific designs may be required. Also, for further information on the specific structure of emotion recognition, and as such the hierarchy and dependence among essential features with respect to emotion attribution, we refer to the references provided in table 2 and 3, as well as section 3.4.

5.2 Design tools

Design tools, or design operations are those mechanisms we can apply on essential features from visual emotion recognition that affect these essential features in terms of emotion attribution and differentiation. From our review the psychology behind abstract art we are able to distinguish six design tools that can be applied to those features: translation, isolation, exaggeration, multiplicity, grouping and contrast. Do note that that the design tools we propose here may be used, but do not

all have to be used. As such, the design guidelines show the possibilities of abstract design within an affective robotics design framework. The proposed methods for abstract affective design [section 2.2 and 2.3] are not neatly framed with respect to their effects concerning attribution and differentiation. Methods can have multiple relevant effects simultaneously. It is necessary that we are aware of this so we can take this into account when designing an abstract affective robot.

5.2.1 Attribution

Attribution revolves around managing the emotional quality when designing an abstract affective robot, i.e. essential features and the operations thereon that affect emotion attribution. Applying these design tools to essential features (table 2 and 3) affect the formation and intensity of essential features, and as such the emotional attributions towards them. As such, design tools that affect essential features with respect to emotion attribution provide flexibility in the essential features we use, how we combine them and how they are interpreted. We distinguish four major design tools that affect attribution: exaggeration, multiplicity, grouping and translation.

5.2.1.1 Translation

The design tool translation allows some more freedom in the usage of specific essential features, as we can translate essential features from one modality to another [section 2.2.3, table 6]. The advantage of translation can be that in the absence of ways to accurately meet in the physical requirements for features that evoke specifically desired emotional attributions, it may be possible to translate these essential features to another modality that is possible to implement, i.e. they may provide the design of robots with specific morphological restrictions with a wider vocabulary of emotion expressions. If we take the example of an abstract affective robot that has only motion kinematics to express emotions, and assume motion kinematics do not provide a sufficient emotional attribution of positive valence, but we can find this precise emotional attribution in the form feature roundness or curvature, translation shows it may be possible to translate this form feature to motion patterns, while retaining its affective quality. The question does remain which essential features can be translated while retaining their affective quality, and to what extent they will impact composing features with respect to hierarchy and dependence. Therefore, it is advised to verify designs using translation. To verify this may require empirical research for a particular design.

Essential Feature translation Reference

kinematics static image that reflects movement and kinematic traces

Freedberg & Gallese 2007

roundness movement traces Aronoff 2006, Bassili 1978, Saerbeck & Bartneck 2010

angularity movement traces Aronoff 2006, Bassili 1978

Table 6: Suggested translations of essential features that retain affect

5.2.1.2 Exaggeration

The design operation exaggeration provides us with a means to intensify the emotional attributions to essential features based on the peak shift principle [section 2.2.2]. The implications on the level of attribution are that emotional attributions towards specific features will be perceived relative to similar instances of those same features. The perceived distance and orientation from a specific instance of the feature towards the exaggerated version of that feature determines the impact on emotional attributions in that same direction. As such exaggeration provides a means to explore the space of emotion attributions in a directional way relative to a featural anchor point. For example, if we endow a robot with the capability to vary its perceived instability. Following attribution research (table 2 and 3) the perceived instability evokes a fearful reaction. Exaggeration along the path of perceived instability then intensifies the perception of fear relative to its previous instance.

Practically, the core question of exaggeration revolves around what the exact path of peak shifting is [section 2.2.2]. Note this is not a trivial question to answer because the path of exaggeration depends on the specific perceptual qualities of a feature. Continuing with our example robot, exaggeration would entail increasing the perception of instability which in turn increases the emotional attribution towards the feature. Perceiving instability may be dependent on the materials used, form and the composition of elements in relation to the earth’s gravitational pull. So in principle the correct path of exaggeration is always dependent on the specifics of the essential feature we are operating on. This requires research, either from literature or empirical. To start, we would like to refer to the references in table 2 and 3 for further details. Also note exaggeration is a design operation relating to attention as well [section 2.2.2 and 5.2.2.2]. Lastly, we recommend isolation [section 5.2.2.1] to precede exaggeration when designing an abstract affective robot.

5.2.1.3 Multiplicity

Another design operation that affects the emotional attribution towards essential features is multiplicity [section 2.2.4]. Multiplicity simply states that the quantity of essential features has an impact on emotion attribution. Typically, essential features of one type reinforce each other, so increasing quantity entails intensification of the accompanying emotional attribution. For example, if we build a robot that has a variable amount of angular features, and attribution research shows these features evoke a threat response, than increasing the amount of angular features the robot shows in an expression makes the expression more threatening. Note multiplicity is closely tied to composition and should therefore keep in mind the role contrast and especially grouping when applying this design operation [section 5.2.1.4 and 5.2.2.3].

5.2.1.4 Grouping

A key issue of composition in the context of abstract affective robots is to be aware of any gestalt effects [section 2.3.2]. The relevance of grouping is that combinations of features may perceptually form different features when put together according to the Gestalt grouping principles. The specifics of the Gestalt grouping principles can be found in Wolfe (2005). For example, let’s assume we have built a robot that has three rounded features that are positioned as a downward pointing triangle on the robots body. Rounded features are generally known to evoke positive emotion attributions while a downward triangle is known to evoke a threat response (table 2). According to the gestalt principles we can both perceive those three rounded features but also the square through grouping and binding mechanisms depending on the focus of the user. So this essentially means it depends on the users’ focus what essential feature will be perceived, and as such what emotion will be attributed. Grouping

principles become increasingly relevant for the design of abstract affective robots when the complexity of a design increases. And as such it becomes increasingly difficult for a designer to be aware of any Gestalt effects, i.e. the more features we combine the higher the chance for Gestalts will form the designer is not aware of when the robot is operating. More interestingly, the Gestalt principles can be used to design expressions based on essential features as well. For example, grouping is strongly related to the design operation multiplicity as well as it can be used as an additional operator for multiplicity to reinforce emotion attributions [section 5.2.1.3]. As such, Gestalt mechanisms such as grouping and binding are therefore important to take into account when designing an abstract affective robot.

5.2.2 Differentiation

Abstract affective robot designs inevitably have more features than only its essential features for emotion expression. A key question is how to guide attention towards the essential features and away from features that do not evoke emotion attributions. Justification for such a design step lies in the fact that if attention is not allocated to a feature it’s information will not be processed, or at least not processed sufficiently [Lavie 1995]. From a design theoretical perspective this becomes increasingly important when the morphological complexity increases. The design step differentiation provides tools to ensure this. The design operations involved are: isolation, exaggeration and contrast.

5.2.2.1 Isolation

The design tool isolation is applied to remove redundant features from the design in order to benefit the perception of features essential to emotion attribution [section 2.2.1]. This is accomplished by limiting the amount of essential features on one modality and removing or neutralizing any other, in terms of emotion attribution, redundant (visual) information. By removing or neutralizing inessential features essential features stand out and therefore attention is guided towards these essential features. In that sense isolation is a subtractive operation, i.e. it does not operate on essential features but purges the design from inessential information. Applying the essential features from table 2 and 3 should reduce the difficulty of applying isolation as it allows for a bottom-up design process. A simple heuristic is to limit the amount of features per modality, which reduces overall design complexity. Inevitably, there is a need to reduce other non-essential visual information on modalities that are not used for emotion expression. Here, design choices can be made at the level of attribution and at the level of perception. For example, if we develop a robot that has only movement to express emotions with, it will inevitably have form, texture and color as well. As such these modalities may guide attention away from the essential feature movement because they are interpreted as emotionally meaningful and bias attribution. In the case of form we can derive possible interpretations from table 3. Attention can also be guided away from essential features because those other modalities are for example highly contrasting and therefore draw attention. To prevent both issues we recommend using an affectively and perceptually neutral form, texture and color in this example. Also see section 5.2.2.3 for an alternative design solution related to contrast. Another contribution of isolation to design is that the perceptual clarity that comes with isolating essential features combats ambiguity as different expressions will become easily distinguishable in sequence. Lastly, we recommend isolation to precede other the other design tools discussed [section 5.2.1 and 5.2.2].

5.2.2.2 Exaggeration

The design operation exaggeration contributes to differentiation by guiding attention towards exaggerated features as they contrast with similar instances of essential features, making them more eye-catching [section 2.2.2]. For a detailed description and examples on the implications of exaggeration on implementation and attribution see section 5.2.1.2. The difference with isolation in the context of differentiation is that exaggeration directly operates on essential features, while isolation is a subtractive procedure and as such affects the overall design directly, but not the essential features themselves. Therefore the additional use of exaggeration over isolation is that it can be used to target specific features and guide attention to those features independent of changing other features of the robot. For example, if we design a very complex abstract affective robot, i.e. one with many features and expressive modalities, exaggeration can be used to force attention to specific essential features that make up a specific expression we want to convey. Secondly, this also entails that exaggeration can be used to combat ambiguity. This is not only helpful in a complex design, but may be particularly helpful when working with limited technological means. To illustrate, when we purposefully limit the modality movement of an abstract affective robot to only two speeds, fast and slow. This could be achieved by downgrading resolution. Sticking to peak shifting allows us to match the mechanics of the robot to the path of exaggeration [section 2.2.2] and thus allows for more specific and affectively consistent construction. Do note that in any case exaggeration affects attribution as well [section 5.2.1.2].

5.2.2.3 Contrast

When we combine different features, and make choices on the placement of those features, contrast may arise between them. There are implications for contrast between different modalities and different features. Firstly, contrast can be used to combine different modalities to specifically reinforce the perception of another. For example, let’s assume we have a robot that has several moving elements and can only use movement as an expressive modality. Then when we give each moving element different, but highly contrasting colors, this reinforces the perception of movement, increasing the ease and detail with which movement is perceived [section 2.3.1, Zeki 1994]. It is important to be aware of such effects for differentiation between modalities. This may require additional research into the neuroscience of visual perception. Also note this may be an alternative design choice when considering isolation as well [section 5.2.2.1]. The second implication of contrast relates directly to the placement of features as combinations of features may combine to form contrasting areas. This needs to be taken into account when designing a robot. Consider a case the combination of two essential features visually meet and form an edge. Edges are known to be highly contrasting and as such draw attention to them, away from the two essential features from our robot. Note contrast may arise not only contrast between static features, but also those on the temporal plane, examples include moving or changing elements contrasting other dynamic or static elements of the design. Practically, ensuring contrast arises only on essential features ensures attention is drawn to them. So in summary, depending on the design contrast distracts from or attracts towards, or reinforces the perception of the expressions we are trying to communicate.

6 Discussion and Conclusion

With this research we have investigated a fundamentally novel design approach for the morphological design of affective robots in which minimal requirements and abstraction play a central role. The core quality of our design approach is that it provides a novel way to match the robots morphology with the cognitive and perceptual requirements of a user for affective interaction. Our approach shifts the focus of design from modeling anatomical features and their dynamics used in bodily expressions (i.e. facial expressions, body postures and arm gestures) to physically modeling emotion expressions after the essential components and structure of visual emotion recognition.

A core challenge for abstract design is that it requires emotions to be conveyed outside the context and configuration of the human body. To do so requires both the identification of essential features of visual emotion recognition, their hierarchy and dependence with respect to emotion attribution and new ways to design and implement them consistent with theories of visual information processing. To develop such a theoretical framework we have reviewed emotion attribution research and adopted abstract art and the psychology behind it as an empirical basis for investigating abstract design. In addition, we have positioned abstract design in the whole spectrum of design strategies for affective robots and found little research has been done on the developing such artifacts. As such, we have attempted to unify research from affective robotics, affective science and the psychology behind abstract art to develop guidelines and provide scientific context for the design of abstract affective robots.

The developed guidelines consist of a two phase design process along which we have discussed specific design operations from the reviewed literature, as well as their practical implications in the context of robot design. The first design phase consists of the selection of essential features from visual emotion recognition. The second design phase consists of the possible application of design tools, these can be used to manipulate the essential features, which provides the designer with additional flexibility and tools to compose a suitable expressive robotic morphology that suits the design objective. In the first design phase features can be selected and combined to compose specific expressive capabilities for an affective robot. However, as we show there exist cases were application is straightforward, but also where application requires not only the presence of essential features, but also requires their composition to reflect the structure of visual emotion recognition. In the second design phase essential features can further be manipulated into specific designs. The proposed design tools can be distinguished by two major categories: attribution and attention. Attribution concerns essential features and the operations thereon that affect the emotional attributions towards them. Differentiation provides the design operations that ensure attention is guided to these essential features. Additionally, we have aimed to provide the designer with sufficient theoretical and practical information on the possibilities of designing and researching abstract affective robots.

Future research can take diverse directions. Attribution research already shows more research is needed on emotional attributions towards combinations of essential features, which are the fundamental building blocks of abstract affective robots. Also, the specific effects of design tools require further investigation. Additionally, user studies can guide future development by investigating the advantages and disadvantages of abstract affective robots with respect to acceptance and believability, and their perceived affordance, which could eventually lead to suitable practical applications and new application areas. Furthermore, the integration of other biological valuation processes, such as aesthetic valuation, with abstract affective design can lead to a more complete

theory of abstract design in general. Further development benefits on the advances made in research areas such as the science behind art, visual emotion recognition and neuroaesthetics. In turn developing abstract affective robotics may contribute theoretically to those fields as well.

References

1. Arnheim R (1974/1997) Art and visual perception: a psychology of the creative eye – the new edition, University of California Press

2. Aronoff J (2006) How we recognize angry and happy emotions in people, places and things. Cross-cultural research 40(1):83-105

3. Aronoff J, Barclay AM, Stevenson LA (1988) The recognition of threatening facial stimuli. Journal of personality and social psychology 54:647-655

4. Aronoff J, Woike BA, Hyman LM (1992) Which are the stimuli in facial displays of anger and happiness? Configural bases of emotion recognition. Journal of personality and social psychology 62(6):1050-1066

5. Atkinson AP, Tunstall ML and Dittrich WH (2007) Evidence for distinct contributions of form and motion information to the recognition of emotions from body gestures. Cognition 10:59-72

6. Atkinson AP, Adolphs R (2005) Visual emotion perception: mechanisms and processes. In: Feldman Barrett L, Niedenthal PM, Winkielman P (eds) Emotion and consciousness. The Guilford Press

7. Barsalou LW (2003) Abstraction in perceptual symbol systems. Philosophical transactions of the royal society of London: biological sciences 358:1177-1187

8. Barsalou LW, Niedenthal PM, Barbey A, Ruppert J (2003) Social embodiment. The psychology of learning and motivation 43:43-92

9. Bassili JN (1978) Facial motion in the perception of faces and emotional expression. Journal of experimental psychology: human perception and performance 4:373-379

10. Bar M, Neta M (2006) Humans prefer curved visual objects. Psychological science 17(8):645-648

11. Bar M, Neta M (2007) Visual elements of subjective preference modulate amygdale activation. Neuropsychologia 45:2191-2200

12. Barakova EI, Lourens T (2009) Expressing and interpreting emotional movements in social games with robots. Personal and ubiquitous computing 14(5):457-467

13. Bemelmans R, Gelderblom GJ, Jonker P, de Witte L (2010) The potential of Socially Assistive Robotics in care for Elderly, a systematic review. In: Proceedings of the 3rd international conference on human-robot personal relationships, Leiden, The Netherlands, June 2010

14. Bethel CL, Murphy RR (2008) Survey of non-facial/non-verbal affective expressions for appearance constrained robots. In: IEEE transactions on systems, man, and cybernetics – part c: applications and reviews 38(1):83-92

15. Blow M, Dautenhahn K, Appleby A, Nehaniv CL, Lee DC (2006) Perception of robot smiles and dimensions for human-robot interaction design. In: Proceedings of the 15th IEEE international symposium on robot and human interactive communication (RO-MAN 2006), pp 469-474

16. Boehner K, Chen M, Liu Z (2003) The vibe reflector: an emergent impression of collective experience. In: Proceedings of CHI 2003 workshop on providing elegant peripheral awareness

17. Breazeal CL (2002) Designing sociable robots. MIT Press

18. Breazeal CL, Picard P (2008) The role of emotion inspired abilities in Relational Robots. In: Parasuraman R, Rizzo M (eds) Neuroergonomics: The brain at work. Oxford University Press

19. Broekens J (2010) Modeling the experience of emotion. International journal on synthetic emotions 1(1):1-17

20. Broekens J, Heerink M, Rosendal H (2009) Assistive social robots in elderly care: a review. Gerontechnology 8(2):94-103

21. Calder A (1960) The star. University of Kentucky Art Museum, Lexington

22. Cañamero LD (2002) Designing emotions for activity selection in autonomous agents. In: Trappl R, Petta P, Payr S (eds) Emotions in humans and artifacts. MIT Press, pp 115-148

23. Cañamero L (2005) Emotion understanding from the perspective of autonomous robots research. Neural Networks 18:445-455

24. Cavanagh P (2005) The artist as neuroscientist. Nature 434:301-307

25. Clark MS, Brissette I (2009) Two types of relationship closeness and their influence on people’s emotional lives. In: Davidson RJ, Scherer KR, Goldsmith HH (eds) Handbook of affective sciences, Oxford University Press, pp 824-835

26. Clynes M (1977/1989) Sentics: The touch of the emotions. Prism Press

27. Collier GL (1996) Affective synesthesia: extracting emotion space from simple perceptual stimuli. Motivation and emotion 20(1):1-32

28. Damasio AR (1994) Descartes' error - emotion, reason and the human brain. Putnam

29. Dautenhahn K (2002) Design spaces and niche spaces of believable social robots. In: Proceedings of the 11th IEEE international workshop on robot and human interactive communication (RO-MAN 2002), pp 192-197

30. Dautenhahn K, Werry I, Rae J, Dickerson P, Stribling P, Ogden B (2002) Robotic playmates: analysing interactive competencies of children with autism playing with a mobile robot. In: Dautenhahn K, Bond A, Canamero L, Edmonds B (eds) Socially intelligent agents: creating relationships with computers and robots. Kluwer Academic Publishers, pp 117-124

31. Davidson RJ, Scherer KR, Goldsmith HH (eds) (2009) Handbook of affective sciences, Oxford University Press

32. de Gelder B, Vroomen J, Pourtois G, Weiskrantz L (1999) Non-conscious recognition of affect in the absence of striate cortex. Neuroreport 10:3759-3763

33. de Rooij A, Broekens J, Lamers M (2011) Geometrical salience in visual emotion recognition, manuscript in progress.

34. Di Dio C, Gallese V (2009) Neuroaesthetics: a review. Current opinion in neurobiology 19(6):682-687

35. Dittrich W, Lea S (1994) Visual perception of intentional motion. Perception 23:253-268

36. Duchenne de Boulogne GB, Cuthbertson RA (1990) The mechanism of human facial expression, Cambridge University Press

37. Duffy BR (2003) Anthropomorphism and the social robot. Robotics and autonomous systems 42:177-190

38. Duffy BR (2008) Fundamental issues in affective intelligent social machines. The open artificial intelligence journal 2:21-34

39. Ekman P (1993) Facial expression and emotion. American Psychologist 48(4):384-392

40. Ellis L (1994) The high and mighty among man and beast: how universal is the relationship between height (or body size) and social status?. In: Ellis L (ed) Social stratification and socioeconomic inequality: reproductive and interpersonal aspects of dominance and status Vol. 2., Praeger, pp 93-111

41. Feil-Safer D, Matarić MJ (2008) Human-robot interaction. In: Meyers RA (ed) Encyclopedia of Complexity and System Science. Springer

42. Feldman Barrett LD (2006) Solving the emotion paradox: categorization and the experience of emotion. Personality and social psychology review 10(1):20-46

43. Fischer AH, Manstead ASR (2008) Social functions of Emotion. In: Lewis M, Haviland-Jones JM, Feldman Barrett L (eds) Handbook of emotions third edition. The Guilford Press, pp 456-468

44. Freedberg D, Gallese V (2007) Motion, emotion and empathy in esthetic experience. Trends in cognitive sciences 11(5):197-203

45. Fong T, Nourbakhsh I, Dautenhahn K (2003) A survey of socially interactive robots. Robotics and autonomous Systems 42:143-166

46. Gadanho SC (2003) Learning behavior-selection by emotions and cognition in a multi-goal robot task. The journal of machine learning research 4:385-412

47. Gallese V (2005) Embodied simulation: from neurons to phenomenal experience. Phenomenology and the cognitive sciences 4:23-48

48. Goetz J, Kiesler S, Powers A (2003) Matching robot appearance and behavior to tasks to improve human-robot-cooperation. Proceedings of the 12th IEEE international workshop on robot and human interactive communication (RO-MAN2003), pp 55-60

49. Hampton JA (2003) Abstraction and context in concept representation. Philosophical transactions of the royal society of London: biological sciences 358:1251-1259

50. Harris J, Sharlin E (2010) Exploring emotive actuation and its role in human-robot interaction. In: Proceedings of the 5th ACM/IEEE international conference on human-robot interaction, pp 95-96

51. Hegel F, Lohse M, Wrede B (2009) Effects of visual appearance on the attribution of applications in social robotics. In: The 18th IEEE international symposium on robot human interactive communication (RO-MAN 2009), pp 64-71

52. Heider F, Simmel M (1944) An experimental study of apparent behavior. American journal of psychology 57:243-259

53. Hoffman ML (2008) Empathy and prosocial behavior. In: Lewis M, Haviland-Jones JM, Feldman Barrett L (eds) Handbook of emotions - third edition. The Guilford Press, pp 440-455

54. Isbister K, Höök K, Laaksolahti J, Sharp M (2007) The sensual evaluation instrument: developing a trans-cultural self-report measure of affect. International journal of human-computer Studies 65:315-328

55. Isbister K, Höök K, Laaksolahti J, Sharp M (2006) The sensual evaluation instrument: developing an affective evaluation tool. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 1163-1172

56. Izard CE (2009) Emotion theory and research: highlights, unanswered questions, and emerging issues. Annual review of psychology 60:1-25

57. Kandinsky W (1926) Point and Line to Plane. Dover Publications

58. Kiesler S (2005) Fostering common ground in human-robot interaction. In: Proceedings of the 14th IEEE international workshop on robot and human interactive communication (RO-MAN2005), pp 729-734

59. Köhler W (1929) Gestalt Psychology. Liveright

60. Laaksolahti J, Isbister K, Höök K (2009) Using the sensual evaluation instrument. Digital creativity 20(3):165-175

61. Larson C, Aronoff J, Sarinopoulos I, Zhu D (2008) Recognizing threat: a simple geometric shape activates neural circuitry for threat detection. Journal of cognitive neuroscience 21(8):1523-1535

62. Larson C, Aronoff J, Stearns J (2007) The shape of threat: simple geometric forms evoke rapid and sustained capture of attention. Emotion 7(3):526-534

63. Latto R (2004) Do we like what we see?. In: Malcolm G, Paton R (eds) Multidisciplinary studies of visual representations and interpretations. Elsevier, pp 343-356

64. Latto R, Brain D, Kelly B (2000) An oblique effect in aesthetics: homage to Mondriaan (1872-1944). Perception 29:981-987

65. Latto R, Russel-Duff K (2002) An oblique effect in the selection of line orientations by twentieth century painters. Empirical studies of the arts 20:49-60

66. Lee JH, Jin-Yung P, Nam TJ (2007) Emotional interaction through physical movement. In: Proceedings of the 12th international conference on human-computer interaction: intelligent multimodal interaction environments, pp 401–410

67. Lehrer J (2009) Unlocking the mysteries of the artistic mind. Sussex publishers LLC http://www.psychologytoday.com/articles/200907/unlocking-the-mysteries-the-artistic-mind. Accessed 27 march 2010.

68. Lewis M, Haviland-Jones JM, Feldman Barrett L (eds) Handbook of emotions – third edition, The Guilford Press

69. Liu H, Selker T, Lieberman H (2003) Visualizing the affective structure of a text document. In: CHI '03 extended abstracts on Human factors in computing systems, pp 740-741

70. Locher P, Nodine C (1989) The perceptual value of symmetry. Computers & mathematics with applications 17(4-6):475-484

71. Lowen A (1971) The language of the body, Macmillan

72. Lundqvist D, Esteves F, Öhman A (2004) The face of wrath: the role of features and configurations in conveying social threat. Cognition and emotion 18(2):161-182

73. Lundqvist D, Öhman A (2005) Caught by the evil eye: nonconscious information processing, emotion, and attention to facial stimuli. In: Feldman Barrett L, Niedenthal PM, Winkielman P (eds) Emotion and consciousness. The Guilford Press

74. Lütkebohle I, Hegel F, Schutz S, Hackel M, Wrede S, Wachsmuth S, Sagerer G (2010) The Bielefeld anthropomorphic robot head “Flobi”. In: 2010 IEEE international conference on robotics and automation (ICRA), pp 3384-3391

75. MacDorman KF, Green RD, Ho C, Koch CT (2009) Too real for comfort? Uncanny responses to computer generated faces. Computers in human behavior 25:695-710

76. Maeda Y (1999) Fuzzy rule expression for emotional generation model based on subsumption architecture. In: International conference of the North American Fuzzy Information Processing society (NAFIPS), pp 781-785

77. Malevich K (1915) Black circle. State Russian Museum, St. Petersburg

78. Malevich K, Wingler HM (1927/1980) Die gegenstandslose Welt, Mann, Berlin

79. Matsumuto D (1990) Cultural similarities and differences in display rules. Motivation and emotion 14(3):195-214

80. Matsumoto D, Keltner D, Shiota MN, O’Sullivan M, Frank M (2008) Facial expressions of emotion. In: Lewis M, Haviland-Jones JM, Feldman Barrett L (eds) Handbook of emotions – third edition. The Guilford Press, New York/ London, pp 211-234

81. Mead R, Matarić MJ (2010) Automated caricature of robot expressions in socially assistive human-robot interaction. The 5th ACM/IEEE international conference on human-robot interaction (HRI2010): workshop on what do collaborations with the arts have to say about HRI?

82. Mehrabian A (1980) Basic dimensions for a general psychological theory. OG&H Publishers, New York

83. Michaud F, Théberge-Turmel C (2002) Mobile robotic toys and autism. In: Dautenhahn K, Bond A, Canamero L, Edmonds B (eds) Socially intelligent agents - creating relationships with computers and robots. Kluwer Academic Publishers, pp 125-132

84. Mondriaan P (1921) Composition in red, yellow, blue and black. Gemeentemuseum, The Hague

85. Mondriaan P, Wingler H (1925/1998) Neue gestaltung. Mann

86. Mori M (1970) Bukimi no tani (uncanny valley). Energy 7:33–35

87. Mutlu B, Forlizzi J, Nourbakhsh I, Hodgins J (2006) The use of abstraction and motion in the design of social interfaces. In: Proceedings of the 6th conference on designing interactive systems, pp 251-260

88. Murphy RR, Lisetti CL, Tardif R, Irish L, Gage A (2002) Emotion-based control of cooperating heterogeneous mobile robots. IEEE transaction on robotics and automation 18(5):744-757

89. NiedenThal PM, Barsalou LW, Winkielman P, Krauth-Gruber S, Ric F (2005) Embodiment in attitudes, social perception, and emotion. Personality and social psychology review 9(3):184-211

90. Oatley K, Yuill N (1985) Perception of personal and interpersonal action in a cartoon film. British journal of social psychology 24:115-124

91. Pavlova M, Sokolov AA, Sokolov A (2005) Perceived dynamics of static images enables emotional attribution. Perception 34:1107-1116

92. Picard R (1997) Affective Computing. MIT Press

93. Pollick FE, Paterson HM, Bruderlin A, Sanford AJ (2001) Perceiving affect from arm movement. Cognition 82B:51-61

94. Pessoa L, Adolphs R (2010) Emotion processing and the amygdala: from a 'low road' to 'many roads' of evaluating biological significance. Nature reviews neuroscience 11:773-782

95. Ramachandran VS (2001) Sharpening up the “science of art”, an interview with Anthony Freeman. Journal of consciousness studies 8(1):9-29

96. Ramachandran VS, Hirstein W (1999) The Science of Art: A Neurological Theory of Aesthetic Experience. Journal of consciousness studies 6(6-7):15-51

97. Ramachandran VS, Hubbard EM (2001) Synaesthesia, A window into perception, thought and language. Journal of consciousness studies 8(12):3–34

98. Ramachandran VS, Hubbard EM (2003) Hearing colors, tasting shapes. Scientific American april:52-59

99. Reber R, Schwarz N, Winkielman P (2004) Processing fluency and aesthetic pleasure: is beauty in the perceiver’s processing experience?. Personality and social psychology review 8(4):364–382

100. Rimé B, Boulanger B, Laubin P, Richir M, Strookbants K (1985) The perception of interpersonal emotions originated by patterns of movement. Motivation and emotion 9(3):241-260

101. Roether CL, Omlor L, Giese MA (2009) Features in the recognition of emotions from dynamic bodily expression. In: Masson GS, Ilg UJ (eds) Dynamics of visual motion processing: neuronal, behavioral, and computational approaches. Springer, pp 313-340

102. Rose R, Scheutz M, Schermerhorn P (2008) Empirical investigations into the believability of robot affect. In: Proceedings of the AAAI spring symposium, AAAI Press

103. Rothko M (1961) Number 207 (Red over Dark Blue on Dark Gray), Berkley Art Museum and Pacific Film Archive, Berkley

104. Russel JA (2003) Core affect and the psychological construction of emotion. Psychological review 110(1):145-172

105. Saerbeck M, Bartneck C (2010) Attribution of affect to robot motion. Proceedings of the 5th ACM/IEEE international conference on human-robot interaction (HRI2010), Osaka, Japan

106. Saerbeck M, Schut T, Bartneck C, Janse MD (2010) Expressive robots in education – varying the degree of social supportive behavior of a robotic tutor. In: Proceedings of the 28th ACM conference on human factors in computing systems (CHI2010), pp 1613-1622

107. Samani HA, Cheok AD, Tharakan MJ, Koh J, Fernando N (2010) A design process for Lovotics. 3rd International conference on human-robot personal relationships, Leiden, The Netherlands

108. Sanches P, Höök K, Vaara E, Weymann C, Ferreira P, Peira N, Bylund M, Sjölinder M (2010) Mind the body! Designing a mobile stress management application encouraging personal reflection. In: ACM DIS 2010 - Designing interactive systems conference, Aarhus, Denmark

109. Sawada M, Suda K, Ishii M (2003) Expression of emotions in dance: relation between arm movement characteristics and emotion. Perceptual and motor skills 97:697-708

110. Scheirer J, Picard R (1999) Affective objects. MIT media laboratory perceptual computing section technical report 524

111. Scherer KR, Schorr A, Johnstone T (eds) (2001) Appraisal processes in emotion: theory, methods, research. Oxford University Press

112. Scholl B, Tremoulet P (2000) Perceptual causality and animacy. Trends in cognitive sciences 4:299-309

113. Sengers P, Liesendahl R, Magar W, Seibert C, Müller B, Joachims T, Geng W, Mårtensson P, Höök K (2002) The enigmatics of affect. In: Proceedings of the 4th conference on designing interactive systems: processes, practices, methods, and techniques, pp 87-98

114. Silvia P, Barona C (2009) Do people prefer curved objects? Angularity, expertise, and aesthetic preference. Empirical studies of the arts 27(1):25-42

115. Ståhl A, Höök K, Sundström P (2005) A foundation for emotional expressivity. In: Proceedings of the 2005 conference on designing for user experience, San Fransisco

116. Stoy K, Brandt D, Christensen DJ (2010) Self-reconfigurable robots, MIT press

117. Takeuchia Y, Hada T (2006) Human prosocial response to emotive facial expression of interactive agent. In: The 15th IEEE international symposium on robot and human interactive communication (ROMAN 2006), pp 680-685

118. van Breemen AJM (2004) Bringing robots to life: applying principles of animation to robots. In: Proceedings of the CHI 2004 workshop on shaping human-robot interaction, Vienna, Austria

119. van Campen C (1997) Early abstract art and experimental Gestalt psychology. Leonardo 30(2):133-136

120. van Doesburg T (1924/1969) Principles of neo-plastic art, Lund Humphries, Farnham

121. Velásquez JD (1998) Modeling emotion-based decision making. In: Emotional and Intelligent: The tangled knot of cognition - Papers from the 1998 AAAI fall symposium, pp 164-169

122. Vessel EA, Rubin N (2010) Beauty and the beholder: highly individual taste for abstract, but not real-world images. Journal for vision 10(2):1-14

123. Vinayagamoorthy V, Steed, Slater M (2005) Building characters: lessons drawn from virtual environments. In: Toward social mechanisms of android science: a cogsci 2005 workshop, pp 119-126

124. Ward O (2010) The man who heard his paintbox hiss. Telegraph media group limited. http://www.telegraph.co.uk/culture/art/3653012/The-man-who-heard-his-paintbox-hiss.html. Accessed 1 November 2010.

125. Whitelaw M (2007) More is more: multiplicity and generative art. http://teemingvoid.blogspot.com/2007/10/more-is-more-multiplicity-and.html. Accessed 07 June 2010

126. Wilson M (2002) Six views of embodied cognition. Psychonomic bulletin & review 9(4):625-636

127. Winkielman P, Schwarz N, Reber R, Fazendeiro TA (2000/2007) Affective and cognitive consequences of visual fluency: when seeing is easy on the mind. In: Batra R (ed) Visual persuasion, Sage Publications

128. Wolfe JM, Kluender KR, Levi DM (2005) Sensation and perception. Sinauer Associates Inc

129. Zeki S (2001) Artistic Creativity and the Brain. Science magazine 293:51-52

130. Zeki S, Lamb M (1994) The neurology of kinetic art. Brain 117:607-636

131. Ziemke T (2003) What’s that thing called embodiment?. In: Proceedings of the 25th annual conference of the cognitive science society, pp 1134-1139

132. Zimmer R (2003) Abstraction in art with implications for perception. Philosophical transactions of the royal society of London: biological sciences 358:1285-1291