In their target article, Hulleman & Olivers (H&O) argue for a conceptual change in characterizing visual search efficiency. The classical view explains visual search times as a function of the number of stimuli in a display (i.e., set size). According to the critique by H&O, this perspective constrains the study of visual search to a scenario that requires clearly defined objects viewed during prolonged fixation. Moreover, they argue, the traditional approach falls short of incorporating results from a larger range of search conditions – including overt visual search and searches in natural scenes in which items are not clearly defined. To overcome these limitations the authors present a theoretical framework that accounts for the number of fixations in a scene based on the assumption of an adjustable functional visual field (FVF), across which parallel processing takes place. In considering eye movements as a fundamental part of search, however, a number of challenges arise that, once faced, promise important theoretical insights for studies interpreted in this new context and beyond. We will focus on two challenges here.
1. What's a fixation?
The authors' central aim is to understand search times based on the number of fixations during the search process. However, a fixation is not as clear-cut and discrete an entity as it might seem. Large primary saccades are frequently followed by smaller secondary saccades that often correct for errors in saccade landing position, but can also be observed after precise primary saccades (Ohl et al. Reference Ohl, Brandt and Kliegl2011). Both primary and secondary saccades meet the criteria for a saccadic eye movement, but it remains unclear whether the interval between primary and secondary saccades should be considered an independent fixation. Moreover, even during instructed fixation, small microsaccades are observed at a rate of 1–2 per second (Rolfs Reference Rolfs2009). Microsaccades have traditionally been considered fixational eye movements, suggesting that the interval between two microsaccades does not constitute an independent fixation. However, evidence accumulates that they are controlled by the same machinery as large saccades (Hafed et al. Reference Hafed, Goffart and Krauzlis2009; Rolfs et al. Reference Rolfs, Kliegl and Engbert2008) and fulfill the same purpose (Hafed Reference Hafed2011; Ko et al. Reference Ko, Poletti and Rucci2010), namely, bringing a stimulus onto the part of the fovea that affords the highest resolution. Fixations separated by microsaccades, therefore, may need to be included when computing visual search times. This acknowledgment has two interesting consequences. First, the proposed framework might help clarify whether the intervals bordering on a microsaccade should be considered separate fixations. By comparing empirically observed numbers of fixations contingent on their definition (as either including microsaccades or not), future research could evaluate what definition of a fixation more accurately predicts the observed search times. Second, the presence of microsaccades during fixations may help resolve the dilemma that H&O face when explaining how search can be successful even in the absence of (large) saccades. Observers are not aware of their own microsaccades, and the generation of microsaccades has been linked to shifts of covert attention (Engbert & Kliegl Reference Engbert and Kliegl2003; Hafed & Clark Reference Hafed and Clark2002; Yuval-Greenberg et al. Reference Yuval-Greenberg, Merriam and Heeger2014). The perpetual execution of microsaccades results in more than one fixation even when observers are explicitly instructed to fixate while performing the visual search task. This variable number of fixations could be informative for characterizing covert visual searches and provide an opportunity to conceptualize it in H&O's framework.
2. Constraints of transsaccadic vision
Active vision is characterized by severe processing limitations that present challenges and constraints for theories of visual search. With each saccade, the incoming light reflected by an object will fall onto a new part of the retina and is thus processed by largely different neural populations in every retinotopic area in the visual processing stream. As a consequence, the visual system needs to keep track of the locations of relevant items as well as of their identities (see Cavanagh et al. Reference Cavanagh, Hunt, Afraz and Rolfs2010 for a review), including potential targets and clear non-targets. There is strong psychophysical evidence that attended locations are updated across saccades (e.g., Jonikaitis et al. Reference Jonikaitis, Szinte, Rolfs and Cavanagh2013; Rolfs et al. Reference Rolfs, Jonikaitis, Deubel and Cavanagh2011), most likely relying on perisaccadic updating of visual priorities in visual attention-related brain areas (see Wurtz Reference Wurtz2008 for a review). Indeed physiological results suggest that this updating of visual priorities (and, hence, the distribution of attention) involves the entire visual field, including distractor locations (Mirpour & Bisley Reference Mirpour and Bisley2012). Whereas this evidence suggests that the system is keeping track of the locations of potential targets and distractors, the accumulation of spatially disperse stimulus feature information across fixations has severe capacity limits. Indeed only three to four items are remembered correctly across saccades (Irwin Reference Irwin1991), and visual memory is heavily biased towards the saccade target (Bays & Husain Reference Bays and Husain2008; Rolfs Reference Rolfs2015). Thus far, H&O's framework considers a restriction only in how many visited locations can be remembered but does not take into account the visual system's limited ability to keep track of stimulus information across saccades. To the extent that this stimulus information is relevant for the search task, the bottlenecks that saccades impose on visual perception and memory fundamentally constrain the relationship between the number of fixations and an observer's search efficiency. Although item-based models do not address saccade-related constraints at all, the framework put forth by H&O provides a fertile ground to incorporate these insights from the study of active vision into the domain of visual search.
To conclude, research on human eye movement has revealed innumerable determinants shaping the alternating sequence of saccades and fixations – including their fundamental link to visual perception and memory. The framework presented by H&O provides a basis for the inclusion of these insights in the formulation of fixation-based theories of visual search. We highlighted two aspects – the controversial definition of fixations and the constraints imposed by transsaccadic vision – that provide challenges and opportunities for theories of active visual search.
You have
Access
In their target article, Hulleman & Olivers (H&O) argue for a conceptual change in characterizing visual search efficiency. The classical view explains visual search times as a function of the number of stimuli in a display (i.e., set size). According to the critique by H&O, this perspective constrains the study of visual search to a scenario that requires clearly defined objects viewed during prolonged fixation. Moreover, they argue, the traditional approach falls short of incorporating results from a larger range of search conditions – including overt visual search and searches in natural scenes in which items are not clearly defined. To overcome these limitations the authors present a theoretical framework that accounts for the number of fixations in a scene based on the assumption of an adjustable functional visual field (FVF), across which parallel processing takes place. In considering eye movements as a fundamental part of search, however, a number of challenges arise that, once faced, promise important theoretical insights for studies interpreted in this new context and beyond. We will focus on two challenges here.
1. What's a fixation?
The authors' central aim is to understand search times based on the number of fixations during the search process. However, a fixation is not as clear-cut and discrete an entity as it might seem. Large primary saccades are frequently followed by smaller secondary saccades that often correct for errors in saccade landing position, but can also be observed after precise primary saccades (Ohl et al. Reference Ohl, Brandt and Kliegl2011). Both primary and secondary saccades meet the criteria for a saccadic eye movement, but it remains unclear whether the interval between primary and secondary saccades should be considered an independent fixation. Moreover, even during instructed fixation, small microsaccades are observed at a rate of 1–2 per second (Rolfs Reference Rolfs2009). Microsaccades have traditionally been considered fixational eye movements, suggesting that the interval between two microsaccades does not constitute an independent fixation. However, evidence accumulates that they are controlled by the same machinery as large saccades (Hafed et al. Reference Hafed, Goffart and Krauzlis2009; Rolfs et al. Reference Rolfs, Kliegl and Engbert2008) and fulfill the same purpose (Hafed Reference Hafed2011; Ko et al. Reference Ko, Poletti and Rucci2010), namely, bringing a stimulus onto the part of the fovea that affords the highest resolution. Fixations separated by microsaccades, therefore, may need to be included when computing visual search times. This acknowledgment has two interesting consequences. First, the proposed framework might help clarify whether the intervals bordering on a microsaccade should be considered separate fixations. By comparing empirically observed numbers of fixations contingent on their definition (as either including microsaccades or not), future research could evaluate what definition of a fixation more accurately predicts the observed search times. Second, the presence of microsaccades during fixations may help resolve the dilemma that H&O face when explaining how search can be successful even in the absence of (large) saccades. Observers are not aware of their own microsaccades, and the generation of microsaccades has been linked to shifts of covert attention (Engbert & Kliegl Reference Engbert and Kliegl2003; Hafed & Clark Reference Hafed and Clark2002; Yuval-Greenberg et al. Reference Yuval-Greenberg, Merriam and Heeger2014). The perpetual execution of microsaccades results in more than one fixation even when observers are explicitly instructed to fixate while performing the visual search task. This variable number of fixations could be informative for characterizing covert visual searches and provide an opportunity to conceptualize it in H&O's framework.
2. Constraints of transsaccadic vision
Active vision is characterized by severe processing limitations that present challenges and constraints for theories of visual search. With each saccade, the incoming light reflected by an object will fall onto a new part of the retina and is thus processed by largely different neural populations in every retinotopic area in the visual processing stream. As a consequence, the visual system needs to keep track of the locations of relevant items as well as of their identities (see Cavanagh et al. Reference Cavanagh, Hunt, Afraz and Rolfs2010 for a review), including potential targets and clear non-targets. There is strong psychophysical evidence that attended locations are updated across saccades (e.g., Jonikaitis et al. Reference Jonikaitis, Szinte, Rolfs and Cavanagh2013; Rolfs et al. Reference Rolfs, Jonikaitis, Deubel and Cavanagh2011), most likely relying on perisaccadic updating of visual priorities in visual attention-related brain areas (see Wurtz Reference Wurtz2008 for a review). Indeed physiological results suggest that this updating of visual priorities (and, hence, the distribution of attention) involves the entire visual field, including distractor locations (Mirpour & Bisley Reference Mirpour and Bisley2012). Whereas this evidence suggests that the system is keeping track of the locations of potential targets and distractors, the accumulation of spatially disperse stimulus feature information across fixations has severe capacity limits. Indeed only three to four items are remembered correctly across saccades (Irwin Reference Irwin1991), and visual memory is heavily biased towards the saccade target (Bays & Husain Reference Bays and Husain2008; Rolfs Reference Rolfs2015). Thus far, H&O's framework considers a restriction only in how many visited locations can be remembered but does not take into account the visual system's limited ability to keep track of stimulus information across saccades. To the extent that this stimulus information is relevant for the search task, the bottlenecks that saccades impose on visual perception and memory fundamentally constrain the relationship between the number of fixations and an observer's search efficiency. Although item-based models do not address saccade-related constraints at all, the framework put forth by H&O provides a fertile ground to incorporate these insights from the study of active vision into the domain of visual search.
To conclude, research on human eye movement has revealed innumerable determinants shaping the alternating sequence of saccades and fixations – including their fundamental link to visual perception and memory. The framework presented by H&O provides a basis for the inclusion of these insights in the formulation of fixation-based theories of visual search. We highlighted two aspects – the controversial definition of fixations and the constraints imposed by transsaccadic vision – that provide challenges and opportunities for theories of active visual search.