Correlation and cause when inferring attentional guidance in the rainforest and beyond

A commentary on

Beyond correlation: do color features influence attention in rainforest?
by Frey, H., Wirz, K. T., Willenbockel, V., Betz, T., Schreiber, C., Troscianko, T., and Konig, P. (2011). Front. Hum. Neurosci. 5: 36. doi: 10. 3389/fnhum. 2011. 00036

Humans shift their eyes to sample the most important parts of the visual world. How do we know if it is worth looking at a certain location before we look there? An intuitively appealing answer to this question is that the brain may use a shortcut based on early visual processing to determine “ salient” parts of the scene that are likely to be worth fixating. In the last decade, research scrutinizing this proposal (and driven principally by the Saliency Map model as described by Itti and Koch, 2000 ) has given rise to something of a deadlock between demonstrations that fixation correlates with bottom-up saliency in free viewing ( Parkhurst et al., 2002 ), and experiments showing that eye guidance is dominated by top-down, cognitive control (e. g., Einhauser et al., 2008 ). The recent studies by Frey et al. (2011) take a novel approach that begins to address two major issues with such research.

The first issue concerns the causal role of certain features in eye guidance. There are systematic differences between fixated and non-fixated locations in natural images, with frequently looked-at regions tending to have higher luminance and chromaticity and to contain edges (e. g., Tatler et al., 2005 ). However, it does not necessarily follow that these features actually cause fixation. Instead, it might be that saccades are guided by top-down processes (e. g., a desire to look at objects), and that salient features merely coincide with such regions. In other words, saliency might be a useful shortcut for computational models while telling us nothing about what the brain is actually doing.

It is necessary, therefore, to move away from correlations between fixations and features and take an experimental approach (e. g., by manipulating the saliency of key objects: Foulsham and Underwood, 2007 ). Frey et al. (2011) modified the color content of their stimuli and report changes in fixation behavior with some modifications (eliminating red–green information) but not others (removing the blue–yellow channel). This is stronger evidence regarding the salient color features that are causally related to fixation. Frey et al. (2011) also take a different and ingenious approach by examining the impact of the observer – in this case their ability to perceive color in the normal way. If color features actually cause fixation in normal observers then color-blind individuals should not fixate in the same locations. Conversely, if color features merely coincide with semantic properties of the scene (e. g., because objects are colorful) then deuteranopes may behave the same. The fact that Frey et al. (2011) show both similarities (in the color features at fixation) and differences (in the timing of fixations to colored regions) shows that this is a powerful technique for pinning down causal influences of color in attentional guidance. Moreover, this avenue of research complements other work investigating the attention of atypical observers with object recognition problems who have normal low-level vision but lack higher-level scene semantics ( Foulsham et al., 2009 ).

Frey et al. (2011) also begin to address a second issue, which might be called an ecological problem: In what sorts of stimuli should we study visual attention? While traditionally most research has used simple stimuli, advances in modeling feature extraction in complex images means that there is now a large literature on eye movements in complex, color scenes. One of the advantages of using natural images is that they are associated with semantic information—gist and expectations about the environment to which humans are accustomed. Furthermore, the stimuli used by Frey et al. (2011) are arguably taken from the very same environment in which our trichromatic visual system evolved to identify ripening fruit in the rainforest. This offers the tantalizing prospect of linking the fundamentals of visual attention to both proximate mechanisms in the brain and distal causes of phylogeny and function in our evolutionary history. Converging evidence from correlational and experimental studies, from typical and atypical individuals, and from simple and more ecologically complex stimuli can therefore provide a more complete explanation of attention – in the rainforest and beyond.

References

Einhauser, W., Rutishauser, U., and Koch, C. (2008). Task-demands can immediately reverse the effects of sensory-driven saliency in complex visual stimuli. J. Vis. 8, 1–19.

Foulsham, T., Barton, J. J. S., Kingstone, A., Dewhurst, R., and Underwood, G. (2009). Fixation and saliency during search of natural scenes: the case of visual agnosia. Neuropsychologia 47, 1994–2003.

Foulsham, T., and Underwood, G. (2007). How does the purpose of inspection influence the potency of visual saliency in scene perception? Perception 36, 1123–1138.

Frey, H., Wirz, K. T., Willenbockel, V., Betz, T., Schreiber, C., Troscianko, T., and Konig, P. (2011). Beyond correlation: do color features influence attention in rainforest? Front. Hum. Neurosci. 5: 36. doi: 10. 3389/fnhum. 2011. 00036

Itti, L., and Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Res. 40, 1489–1506.

Parkhurst, D., Law, K., and Niebur, E. (2002). Modeling the role of salience in the allocation of overt visual attention. Vision Res. 42, 107–123.

Tatler, B. W., Baddeley, R. J., and Gilchrist, I. D. (2005). Visual correlates of fixation selection: effects of scale and time. Vision Res. 45, 643–659.