Classification videos reveal the visual information driving complex real-world speeded decisions

Humans can rapidly discriminate complex scenarios as they unfold in real time, for example during law enforcement or, more prosaically, driving and sport. Such decision-making improves with experience, as new sources of information are exploited. For example, sports experts are able to predict the outcome of their opponent’s next action (e.g. a tennis stroke) based on kinematic cues “read” from preparatory body movements. Here, we explore the use of psychophysical classification image techniques to reveal how participants interpret complex scenarios. We used sport as a test case, filming tennis players serving and hitting ground strokes, each with two possible directions. These videos were presented to novices and club-level amateurs, running from 0.8 seconds before to 0.2 seconds after racquet-ball contact. During practice, participants anticipated shot direction under a time limit targeting 90% accuracy. Participants then viewed videos through Gaussian windows ("bubbles") placed at random in the temporal, spatial or spatiotemporal domains. Comparing bubbles from correct and incorrect trials revealed how information from different regions contributed toward a correct response. Temporally, only later frames of the videos supported accurate responding (from ~0.05 seconds before ball contact to 0.1+ seconds afterwards). Spatially, information was accrued from the ball’s trajectory and from the opponent’s head. Spatiotemporal bubbles again highlighted ball trajectory information, but seemed susceptible to an attentional cuing artefact, which may caution against their wider use. Overall, bubbles proved effective in revealing regions of information accrual, and could thus be applied to help understand choice behavior in a range of ecologically valid situations.