Our goal is to establish the computational foundations for extracting, from visual information, intrinsic properties of real-world objects and scenes that exhibit their hard to verbalize but characteristic looks and feels, i.e., “Shitsukan.” By deriving computer vision methods that can recognize tangible and intangible shitsukan, we aim to gain insights into their perceptual mechanisms. For tangible shitsukan, we focus on estimating apparent attributes that encode physical properties of objects and scenes that are likely relevant to their shitsukan, including weight, size, softness, and condition, just from sight. For intangible shitsukan, we seek to uncover and quantify, from visual information, attributes of real-world environments that inform how to act in them. In particular, we consider objects, people, and the 3D space in which they are situated as key components of an environment, and derive methods that systematically estimate intrinsic semantic and contextual information that likely aids action planning and decision making of autonomous agents in them. By enabling perception, representation, and use of deep shitsukan with computer vision, we hope to aid in principled understanding and manipulation of human shitsukan perception.

Co-Investigator