English

 

Title of Project： Integrative studies of neural mechanisms and advanced information technologies for perception of material and surface qualities Term of Project FY2010-2014 Hidehiko Komatsu (National Institutes of Natural Sciences, National Institute for Physiological Sciences, Professor)

　【Purpose of the Research Project】
 We realize the richness of the real world through 'Shitsukan' perception, namely, perception of materials and surface qualities of natural and man-made objects. 'Shitsukan' also provides us with information which is vital in daily life. For instance, we use this information to identify objects and to judge their surface conditions such as friction to control grasping. 'Shitsukan' is also tightly related to preference / aversion or emotional reaction toward objects. Information for 'Shitsukan' perception is often given cross-modally by vision, touch and audition. However, little is known about what features contained in sensory stimuli are used for 'Shitsukan' perception. In vision, information related to 'Shitsukan' is embedded in a complicated high-dimensional manner in images generated through the interaction of surface reflectance, three-dimensional shape and illumination environment. We can effortlessly decipher such high-dimensional information, but how this can be achieved either theoretically or neurally is largely unknown. Similarly, how 'Shitsukan' is acquired or how it is used for various behaviors is not well understood. To understand these questions, cooperation among technology, psychophysics and neuroscience is very important. This project brings together expertise from these different fields to advance our knowledge on the mechanisms of human 'Shitsukan' perception and facilitate the progress of 'Shitsukan' technology science.

　【Content of the Research Project】
 This project consists of three groups: group A01 for engineering, B01 for psychophysics, and C01 for neuroscience. Group A01 develops technologies to systematically generate wide range of 'Shitsukan' stimuli and to faithfully reproduce 'Shitsukan' of objects employing the state-of-the-art theories and techniques of natural image processing, computer graphics (CG) and computational photography. This group also conducts research to develop novel algorithms to estimate factors such as object shape, surface reflectance and illumination from images employing advanced techniques of inverse rendering. Group B01 conducts research to study what features contained in sensory stimuli are used for 'Shitsukan', and how learning or refinement of such perception is achieved. This group in collaboration with other groups will explore the possibility of quantitative measurement and computer recognition of 'Shitsukan', and techniques to control it. Group C01 conducts research to understand neural mechanisms of 'Shitsukan' employing various techniques of neuroscience on the basis of the information on the properties of 'Shitsukan' gained by the collaboration with other groups. This group studies how the information related to 'Shitsukan' is represented in various brain areas, and how it is related to recognition and emotional reaction.

　【Expected Research Achievements】
 This project will bring us deeper understanding of the neural processing of 'Shitsukan' information and how it affects emotion and how it is used for behavior. Understanding of heuristics and neural mechanisms of 'Shitsukan' will promote development of novel algorithm of 'Shitsukan' estimation that supports efficient CG rendering and compression of infomation regarding object appearance. This will enable more systematic measurement, manipulation and reproduction of 'Shitsukan' information. 'Shitsukan' is related not only to arts and crafts, but also to a wide range of industries because values of many commodities are closely related to their 'Shitsukan'. We believe that our project will contribute to a wide range of activities in our society.

　【Key Words】
Shitsukan: A Japanese word to indicate perception of materials and surface qualities of objects as well as sensory impression of their fineness

 

A01-1  Computer Vision for Shitsukan Analysis

Head Imari Sato （National Institute of Informatics, Associate Professor）
Yoichi Sato（Institute of Industrial Science, The University of Tokyo, Professor）

 The goal of our project is modeling and predicting the complex appearance of real objects under natural illumination conditions. The appearance of an object is greatly influenced by its surface materials as well as the surrounding illumination, and it is essential for scene understanding to be able to predict its variation. We will develop image-based and inverse-rendering approaches to modeling objects’ shapes and reflectance solely from images taken under varying lighting conditions.

A01-2 Measurement, Synthesis and Reproduction for Shitsukan perception

Head Shinsaku Hiura （Hiroshima City University, Professor）
Daisuke Iwai（Osaka University, Associate Professor ）

  High fidelity and precise measurement and reproduction of object's appearance are widely demanded in various areas such as quality control, e-commerce or digital archive. Therefore we are developing reasonable technologies for recording and presenting physical aspects of Shitsukan. For this purpose, we utilize a emerging technology called computational photography, and record the appearance of objects from various lighting conditions and viewing directions. Photometric analysis techniques in computer vision are also exploited to parametrize optical factors such as diffuse and specularity components. For the presentation of Shitsukan esthetes, we apply superimposing techniques for display. Variable reflectance devices such as electronic paper or photochromic materials are combined with projectors to realize high-dynamic range, high-definition display.

B01-1 Human SHITSUKAN perception from visual, auditory and tactile information

Head Shin’ya Nishida （NTT Communication Science Laboratories, Senior Distinguished Scientist）
Keiji Uchikawa（Tokyo Institute of Technology, Professor） Isamu Motoyoshi（NTT Communication Science Laboratories, Distinguished Scientist） Waka Fujisaki（National Institute of Advanced Industrial Science and Technology, Research Scientist）

 The purpose of our project is to specify how the human brain uses sensory information for perception of qualities and materials of objects through vision, audition, and touch. Ongoing research topics include visual perception of gloss, translucency, metallicity and fluid viscosity; material-shape interactions; tactile perception of vibration frequencies; and audiovisual integration of material information. The research strategy is human psychophysics combined with computational stimulus analysis and brain imaging. For accurate color spectrum control, we are also developing a novel color display that can show monochromatic light images.

B01-2 Environmental and learning aspects in Shitsukan perception

Head Shigeki Nakauchi （Toyohashi University of Technology, Professor）
Michiteru Kitazaki（Toyohashi University of Technology, Associate Professor） Takehiro Nagai（Yamagata University , Associate Professor）

  We focus on ability of well-trained experts, who visually inspect surface quality of pearls, to investigate environmental and learning aspects in Shitsukan perception. First, we aim to investigate relationship between optical characteristics of pearl, such as Interference color, and expert's quality rating. Also, we conduct psychophysical experiments to reveal how inspection performance of experts depends on visual environment (e.g. light field) and object shape, etc.The investigation on pearl inspection experts will be extended and applied to other materials to reveal environmental and learning dependency in Shitsukan perception.

C01-1 Early visual mechanisms of shitsukan perception

Head Izumi Ohzawa （Osaka University, Professor）
Kota Sasaki（Osaka University, Assistant Professor） Hiroki Tanaka（Osaka University, Associate Professor）

 Specific aims of our research are as follows.
1. Development of analysis techniques for measuring high-order neural properties allowing separation of form and shitsukan parameters: Early visual neurons encode information in a highly generalized manner for later extraction of distinct information such as form, motion, color, etc. Embedded also among such a generalized representations is information related to surface and material properties. Using modern analysis techniques we have developed such as local spectral reverse correlation (LSRC), we will try to identify how shitsukan information is conveyed as neural signals in relatively early visual areas.

2. Monocular and binocular mechanisms of shitsukan perception: Aspects of shitsukan including specular and texture qualities of surfaces are known to be enhanced under binocular viewing compared with a monocular condition. We will explore and identify possible binocular contributions to shitsukan perception.

C01-2 Higher brain mechanisms of Shitsukan perception

Head Hidehiko Komatsu （National Institute for Physiological Sciences, Professor）
Noritaka Ichinohe（National Center of Neurology and Psychiatry, National Institute of Neuroscience, Director） Naokazu Goda（National Institute for Physiological Sciences, Assistant Professor）

 Our aim is to understand neural mechanisms of Shitsukan perception, namely, perception of materials and surface qualities of natural and man-made objects. Specifically, we are focusing on the neural representation of surface gloss and neural mechanisms of material identification. For this purpose, we are conducting neural recording experiments in monkeys and functional imaging experiments in both human subjects and monkeys to study which regions of the visual cortex are involved in the coding of the Shitsukan information, and how the high-dimensional information on Shitsukan is represented in these regions. We are also extending our research to study how the visual signals are integrated with signals from other sensory modalities in the cross-modal Shitsukan perception.

C01-3  

Head Manabu Honda