Jiajie Zhang
School of Health Information Sciences
University of Texas at Houston
An updated version was published in Cognition and Pragmatics. See link below:
Zhang, J., & Patel, V. L. (2006, in press). Distributed cognition, representation, and affordance. Cognition & Pragmatics, 00, 000-000.
Gibson's (1972) ecological approach to perception is a radical departure from the conventional approach. It emphasizes the environmental information available in extended spatial and temporal pattern in optic arrays, for guiding the behaviors of animals, and for specifying ecological events. The ecological and conventional approaches differ in the following aspects.
• The Starting Point. For the conventional theory of perception, the starting point for perception is the retinal image. It is the stimulation of light on the retinal that provides information for visual perception. According to Gibson, however, the starting point is the ambient optic array. It is the structure in the light extended over space and time that provides direct information about the media, surfaces, substances, and events for an observer.
• The Level of Description. The perplexing lack of correlation between proximal stimulation and perception may well be due to the arbitrary physical dimensions that have been chosen to describe the proximal stimulus (Gibson, 1966). According to Gibson, the right level of describing perception is ecology, not physics or geometry, as adopted in the conventioanl theory of perception. Surfaces and edges are the subjects of ecological geometry, while planes and lines are the subjects of abstract geometry. For example, a surface is substantial, textured, and generally opaque; a plane is not. A surface can be perceived; a plane can only be visualized. The physics of photons and the biochemistry of photoreceptor can be used to explain how light is emitted and propagated and how receptors are stimulated, but not how the world is perceived. According to Gibson's ecological optics, the right way to describing perception is as the direct pickup of invariant properties in the optic array.
• Static Perception vs. Active perception. For the conventional theory, perception is a passive sensing of the retinal image. According to Gibson, however, perception is an act. Perception and action are seen as tightly interlocked and mutually constraining. Stimulation per se does not lead to perception, as evidenced by the perceptual experience in Ganzfeld. Movement is essential for seeing, and invariants of structure do not exist except in relation to variants.
• Mediated Perception vs. Direct Perception. According to the conventional theory of perception, perception is mediated by inferences, which are the transformations of mental representations. What one perceives depends not only on the stimulation of light, but also on the mental processes deployed in processing that inforamtion. It follows that perception is a computational process. According to Gibson, however, perception is not mediated by memory, nor by inference, nor by any other psychological processes in which mental representations are deployed. Perception is the direct pickup of invariants in the optic array. In addition, the invariants are sufficient to specify all objects and events in the organism's environment. No mental contents and processes are needed for perceiving. The end product of perception is not an internal representation of the environment; it is the direct pickup of the invariants in the environment.
• The Complimentarity of the Organism and the Environment. For the conventional theory, perception is the processing of the retinal image formed by the stimuli in the environment. It is a one-way perception. According to Gibson, however, perception of the environment is also the perception of the self. The environment and the organism are mutually constraining and complimentary. The environment implies the organism, and vice versa.
Gibson's ecological psychology was developed primarily for visual perception. However, the interest in Gibson's influential theory has often transcended the interest in perception alone. Even Gibson himself envisioned the implications of his ecological psychology for other domains of psychology and even for philosophy, especially in his theory of affordances, which will be analyzed in detail in the rest of this chapter. However, due to his radical hypothesis of direct perception and complete denial of internal representations, Gibson could not extend his great insights to the mainstream cognitive psychology, one of the greatest achivements in psychology in the last few decades. Neisser (1989) and Shepard (1984) were among the first who tried to reconcile Gibson's ecological psycholgoy that emphasizes the information in the environment and cognitive psychology that emphasizes internal representations. For example, Shepard (1984) studied the ecological constraints of internal representations. He argued that not only perceiving, but also imagining, thinking, and dreaming are all similarly guided by internalizations of long-enduring constraints in the external world. According to Shepard, instead of picking up the invariants that are wholly present in the sensory arrays, as a result of biological evolution and perceptual and cognitive learning, an organism is tuned to resonate to the invariants that are significant for it. Gibson's ecological psychology has also been used to develop a linguistic theory of situations (Barwise & Perry, 1983) and to study situated cognition and ecological visulization (Lewis, 1990b).
In addition to the extension to some theoretical issues of internal representations, Gibson's ecological psychology, especially his theory of affordances, has been elaborated in the areas of human factors and applied cognitive science. In the book The Psychology of Everyday Things, Norman (1988), by extending the meaning of affordances, systematical explored the roles of affordances, forcing functions, and natural mappings in the design of cognitive artifacts. Kirlik (1991), in the study of supervisory controls in dynamic and uncertain environments, built a process model of skilled human-environment interaction based on the theory of affordances. Gaver (19xx) also applied and extended Gibson's theory of affordances in the design of computer displays.
In this Chapter, I study affordances under the framework of distributed representations. First, I summarize Gibson's orginal definition of affordance. Second, I extend Gibson's affordance to domains that involve internal representations. Third, I develop a taxonomy of affordances. And finally, I explore the roles of distributed representations in bridging the conventional approach that emphasizes internal representations and the ecological approach that emphasizes the information in the environment.
Gibson's Theory of Affordances
According to Gibson (1977, 1979), the environment not only serves as the surfaces that separate substances from the medium in which the animals live, but also affords animals in terms of terrain, shelters, water, fire, objects, tools, animals, human displays, etc.; and there is not only information in light for the perception of the environment, but also information for the perception of what the environment affords. He proposed a radical hypothesis: the composition and layout of the surfaces in the environment constitute what they afford. Gibson's affordance has the following properties:
• Affordances provided by the environment are what it offers, what it provides, what it furnishes, and what it invites. The environment includes the medium, the substances, the surfaces and their layouts, the objects, places and hiding places, other persons and animals, and so on.
• The "values" and "meanings" of things in the environment can be directly perceived. "Values" and "meanings" are external to the perceiver.
• Affordances are relative to animals. They can only be measured in ecology, but not in physics.
• An affordance is an invariant.
• Affordances are holistic. What we perceive when we look at objects are their affordances, not their dimensions and properties.
• An affordance implies complementarity of the perceiver and the environment. It is neither an objective property nor a subjective property, and at the same time it is both. It cuts across the dichotomy of subjective-objective. Affordances only make sense from a system point of view.
Affordances as Distributed Representations
Gibson's insight of affordances is of great value and importance. It depicts a fundamental aspect of human cognition, that is, much information needed for perception and action is in the environment as invariants which can be picked up directly. Gibson's theory of affordances was developed primarily in the study of visual perception, and his affordances are basically for environmental things such as substances, media, layouts, events, etc. Gibson hinted that affordances are not only for visual peception, but also for biological (e.g., toxin and nutrition) and cultural (e.g., mail box) processes. However, he did not fully explore these implications. In this section, I use the theory of distributed representations to extend Gibson's affordances to a variety of domains and study the micro-structures of affordances.
Gibson himself made it clear that affordance is not solely the property of the environment; it is jointly determined by the environment and the organism. Affordances are the allowable actions specified by the environment coupled with the properties of the organism. In terms of the theory of distributed representations, affordances are distributed representations extended across the environment and the organism. The structures and information in the environment specifies the external representational space. The physical structures of the organism and the structures and mechanisms of internal biological, perceptual, and cognitive faculties specify the internal representational space. The external and internal representations together specify the distributed representational space: the affordance space (Figure 7.1). The external and internal representational spaces can be described by either constraints or allowable actions. Constraints are the negations of allowable actions. That is, the allowable actions are those satisfying the constraints, and the constraints set the range of the alllowable actions. If the external and internal representational spaces are described by constraints, then the affordances are the disjunction of the constraints of the two spaces (Figure 7.1A). If the external and internal representational spaces are described by allowable actions, then the affordances are the conjunction of the allowable actions of the two spaces (Figure 7.1B).
Figure 7.. The distributed representation of affordances. (A) The representational spaces are described by constraints. The affordance space is the disjunction of the external and internal representational spaces. (B) The representational spaces are described by possible actions. The affordance space is the conjunction of the external and internal representational spaces.
A Categorization of Affordances
Gibson's orginal affordances are basically those that are specified by the relations between the physical structures of the environment and the physique of the organism (e.g., chairs afford sitting for people). Two of the fundamental properties of affordances are the complimentarity of the environment and the organism and the direct, effortless pickup of affordances. These two properties exist not only at the level of organism's body structures, but also at many other levels. In this section, I extend Gibson's orginal affordances to phenomenon at other levels and develop a taxonomy of affordances.
Under the framework of distributed representations, affordances are distributed representations extended across external (the environment) and internal (the organism) representations. External representations belong to the environment; and internal representations belong to the organism. External representations can be at the levels of chemical processes, physical configurations, spatio-temporal layouts, and symbolic structures, which correspond to the levels for internal representations: biological mechanisms inside the body, the physique of the organism, perceptual systems, and cognitive representations.
Figure 7.2. The distributed representatons of affordances.
Biological Affordance
He talked about the affordances of food. For example, a healthy mushroom affords nutrition, while a toxic mushroom affords dying. This is at the level of biology. What Gibson did was to extend the concept of affordance inward to the inside of the body (biology).
Physical Affordance
Most of Gibson's examples of affordances belong to this category. The task is mainly constrained by the physical module. For example, the flat horizontal panel on a door can only be pushed. More examples of this type can be found in Norman (1988).
(1) Terrain features. An open environment affords locomotion in any direction over the ground, whereas a cluttered environjent affords locomotion only at opening. A path affords pedestrian locomotion form one place to another, between teh terrain features that prevent locomotion. An obstacle can be defined as an animal-=iszed object that affords collision and possible injury. Water margin, brink, step, slope.
(2) Shelters.
(3) Water.
(4) Fire.
(5) Object.
Perceptual Affordance
In this category, affordances are mainly provided by spatial mappings. For example, if the switches of the stovetop burners have the same spatial layout as the burners themselves, the switches provide affordances for controlling the burners (Norman, 1988).
Examples of this type include the pictorial signs for ladies' and men's restrooms.
(4a) Most criticisms on Gibson's ecological psychology is that he did not accept any notion of internal representations. Shiffrin and Schneider (197?) showed that many perceptual tasks which require conscious controls can be automated through practice. From the distributed cognition perspective, we can extend Gibson's invariants in the environment to the invariants in the mind. To perceive an affordance is to perceive the invariants distributed across internal minds and external environments, as long as the perception is direct, automatic, unconcious. Physical affordances are easy to conceive. Distributed affordances are not hard, either, if we view the probem from a global perspective.
Cognitive Affordance
Affordances of this type are provided by cultural conventions. For example, for traffic lights, red means "stop", yellow means "prepare to stop", and green means "go".
I am not saying that symbols are affordances. What I mean is that some symbolic representations CAN provide affordances. The varieties of affordances do not differ qualitatively. They are all on the same continuum. In other words, they are all the components under a single framework.
(3) Symbols are arbitrary. Well, only in some degree. Symbols are representations. So the form of a symbol matters. Therefore, some symbols can provide stronger affordances than others. Some external symbols are not arbitrary. They can provide affordance in the traditional sense. Arbitrary external symbols don not provide affordances at the beginning. But once they are internalized to form distributed symbols, they can provide affordances. In this case, the affordances are mainly internal. They might not be as good as physical affordances. But they still provide affordances. It is only a matter of degree. What I have in mind about symbols are more general. Let us take Don and Ed H.'s Object-symbol as an example. An Object-symbol is an artifact which is both the means of control and also the representation of the object state. A light switch is an Object-symbol. It not only controls the ON or OFF of a light bulb but also indicates whether the light is ON or OFF (up is on, down is off, at least for most cultures). Nevertheless, the light switch DOES provide affordance. The lever affords moving up or moving down, but not moving left or right.
Mixed Affordance
(2) Gibson himself did not completely deny internal representations. In fact, he acknowledged the importance of learning, memory, etc. Though the complimentarity in Gibson's theory is at the level of physical appearance (the coupling of the structure of a person's body and the structure of a chair), it can be extended further.
Many affordances are provided by a combination of more than one modules. For example, shoe ties afford tying shoes. This affordance is a conjunction of physical affordance and semantic affordance: the physical property of ties and the knowledge of how to make a tie.
For example, a mailbox, which is one of the examples used by Gibson, does not provide the affordance of mailing letters at all for a person who has no knowledge about postal systems. In this case, internal knowledge (internal symbol systems, if you like) is involved in constructing the affordance in a great degree.
The Continuum of Affordances
The above seven categories are only a preliminary list. There are many others, of course. These seven types of affordances are not equally effective. External affordances are more effective than than internal affordances because they do not need internal representations. Even among the external or internal affordances, some may be more effective than others. Their relative effectiveness is open for future research, which will be one of the topics of my future studies. They are all on the same continuum, though some might be at one pole and others at another pole.
(1) A problem (or task, including affordance) can be represented as a set of modules (though the use of "module" is open for discussions), some internal and some external. If the representation of a problem is mainly distributed externally, it provides stronger affordances (in the traditional sense). How much affordance a representation provides is just A MATTER OF DEGREE.
Varieties of Affordances (p36)
Why do we perceive invariants? Because invariants are the only psychologically signigicant information. Psychological scales, Gibson's invariants, ...
TOH internal and external poblem spaces. Conjunctions.
Psychological scales. Invariants. Fitting human visual system. Color and color blind. The invariants are what are perceived by humans. They are affordances.
The perplexing lack of correlation between proximal stimulation and perception may wellbe due tothe arbitrary physical dimensions that have been chosen to describe the proximal stimulus (Gibson, 1966).
Distributed cognition: representation. representations change processes. Thermodynamics: static vs dynamics.