Predicting Actions from Static Scenes

Tuan-Hung Vu¹, Catherine Olsson², Ivan Laptev¹, Aude Oliva², and Josef Sivic¹

¹WILLOW, ENS/INRIA/CNRS UMR 8548, Paris, France

²CSAIL, MIT, Cambridge, Massachusetts, USA

Abstract. Human actions naturally co-occur with scenes. In this work we aim to discover action-scene correlation for a large number of scene categories and to use such correlation for action prediction. Towards this goal, we collect a new SUN Action dataset with manual annotations of typical human actions for 397 scenes. We next discover action-scene associations and demonstrate that scene categories can be well identified from their associated actions. Using discovered associations, we address a new task of predicting human actions for images of static scenes. We evaluate prediction of 23 and 38 action classes for images of indoor and outdoor scenes respectively and show promising results. We also propose a new application of geo-localized action prediction and demonstrate ability of our method to automatically answer queries such as “Where is a good place for a picnic?” or “Can I cycle along this path?”.

Keywords: Action prediction, scene recognition, functional properties

LNCS 8693, p. 421 ff.

Full article in PDF | BibTeX

lncs@springer.com
© Springer International Publishing Switzerland 2014