Click on title to see details.

A01 Shitsukan Mechanism

A01-1 Visual, auditory and tactile SHITUKAN recognition mechanisms based on signal modulations

Principal Investigator : Shin’ya Nishida(NTT Communication Science Labs, Nippon Telephone and Telegraph Corp)

西田 眞也

Principal Investigator

NTT Communication Science Labs, Nippon Telephone and Telegraph Corp

Shin’ya Nishida


Shigeto Furukawa

Co-Investigator

NTT Communication Science Labs, Nippon Telephone and Telegraph Corp

Shigeto Furukawa

Kyoko Suzuki

Co-Investigator

Yamagata University

Kyoko Suzuki

Keiji Yanai

Co-Investigator

University of Electro-Communications

Keiji Yanai

Outline

SHITSUKAN information of an object is embedded in the complex modulation patterns of sensory signals produced by interactions among object, environment and observer. Based on this idea, we attempt to identify the signal modulations responsible for a variety of visual, auditory and tactile SHITSUKAN, and the underlying recognition mechanisms. From the analysis of the physical properties of SHITSUKAN stimuli and the corresponding SHITSUKAN responses by human observers, we will reveal the functional and computational mechanisms of SHITSUKAN recognitions. With regard to research strategy, in addition of rigorous psychophysical experiments with limited number of observers in experimental rooms, we will develop a SHITSUKAN test battery to collect a large size of psychological response data through the internet. Adapting the same test battery to evaluate SHITSUKAN recognition performance of higher-brain dysfunction patients, we will gain insights into the neural mechanism of SHITSUKAN recognition.

Psychophysics of visual,auditory,and tactile SHITSUKAN recognition. SHITSUKAN test battery Higher brain dysfunction. Data collection via internet.

A01-2 Neural Basis of Shitsukan Perception and the Mechanisms of its Learning and Modulation

Principal Investigator : Hidehiko Komatsu(Brain Science Institute, Tamagawa University)

小松 英彦

Principal Investigator

Brain Science Institute, Tamagawa University

Hidehiko Komatsu


Isamu Motoyoshi

Co-Investigator

The University of Tokyo

Isamu Motoyoshi

Takeaki Shimokawa

Co-Investigator

Advanced Telecommunications Research Institute International

Takeaki Shimokawa

Naokazu Goda

Co-Investigator

National Institute for Physiological Sciences

Naokazu Goda

Outline

step-by-step processing. short-cut processing.

Shitsukan perception include functions such as material discrimination of objects and evaluation of surface qualities such as gloss and translucency. Our goal is to understand the multi-layered processing underlying Shitsukan perception, that consists of a step-by-step processing of Shitsukan information in the hierarchically organized visual system as well as a short-cut processing that takes advantage of relatively simple visual features that correlate with specific Shitsukan of objects. For this purpose, we will conduct psychophysical and behavioral experiments to study the properties of these two types of processings in humans and monkeys, and we will study brain activities related to the representation of Shitsukan information and how they will be modulated by the learning and experience.

A01-3 Neural Mechanism of Affective Response with Shitsukan Perception

Principal Investigator : Takafumi Minamimoto(National Institutes for Quantum and Radiological Science and Technology)

南本 敬史

Principal Investigator

National Institutes for Quantum and Radiological Science and Technology

Takafumi Minamimoto


Manabu Honda

Co-Investigator

National Center of Neurology and Psychiatry

Manabu Honda

Outline

Neural mechanism of attective response with Shitsukan.

Shitsukan perception often accompanies emotional/affective response as well as valuation and decision. Our goal is to understand the neural mechanism of affective response concomitant with Shitsukan perception, especially underlying neural circuit and molecular mechanism. We also aim to identify critical Shitsukan properties for specific affective response. We will develop integrated research on affective neuroscience using Shitsukan resources, including 3D objects and presentation methods. In addition to unique scientific outcomes, our research may provide industrially useful means for affective evaluation and improving Shitsukan quality.

A01-4 Modeling Complex Appearance of Real objects for Shitsukan Analysis

Principal Investigator : Imari Sato(National Institute of Informatics)

佐藤 いまり

Principal Investigator

National Institute of Informatics

Imari Sato


Ysuhiro Mukaigawa

Co-Investigator

Nara Institute of Science and Technology

Ysuhiro Mukaigawa

Yoichi Sato

Co-Investigator

The University of Tokyo

Yoichi Sato

Yinqiang Zheng

Co-Investigator

National Institute of Informatic

Yinqiang Zheng

Outline

The goal of our project is modeling photometric properties of real objects surfaces for analyzing Shitsukan perception. In computer vision, various techniques have been proposed for accurately modeling and rendering an object's appearance under complex lighting. In this project, we extend such analysis further to model more complicated photometric properties such as translucency, structured color, fluorescence and use them for Shitsukan analysis.

B01 Shitsukan Mining

B01-1 Deep Learning of Shitsukan Representational Space Using Images and Languages

Principal Investigator : Takauyki Okatani(Tohoku University)

岡谷 貴之

Principal Investigator

Tohoku University

Takauyki Okatani


Kota Yamaguchi

Co-Investigator

Tohoku University

Kota Yamaguchi

Outline

We aim to develop a computer system that recognizes surface qualities, or shitsukan, of an object from its image. Toward this end, we study methods for constructing a representational space of shitsukan information by performing deep learning that uses a large-scale dataset. We plan to mine representations for various shitsukan properties by performing a type of transfer learning using convolutional neural networks (CNNs). Our approach is to train CNNs for a "proxy task" of shitsukan recognition and then extract internal representations (activation patterns of internal layers for an input image) learned by the CNNs, by which we will construct a shitsukan representational space. This is based on our hypothesis that CNNs that have learned to solve the proxy task will automatically acquire sufficient factors for representing shitsukan information in their learned internal representation. Shitsukan is difficult to represent verbally, which makes it hard to transmit and share between human and human or between human and machines. The ultimate goal of the research project is to "quantify" such shitsukan and create a new research field by further extending the shitsukan representational space that will be developed in this project.

Extracting shitsukan properties from internal representation of CNNs.

B01-2 Data-mining of shitsukan representation using brain, image, and text data

Principal Investigator : Yukiyasu Kamitani(Advanced Telecommunications Research Institute International / Kyoto University)

神谷 之康

Principal Investigator

Advanced Telecommunications Research Institute International / Kyoto University

Yukiyasu Kamitani


Co-Investigator Tomoyasu Horikawa (Advanced Telecommunications Research Institute International)
Kei Majima (Advanced Telecommunications Research Institute International)
Tatsuya Harada (The University of Tokyo)

Outline

Shitsukan may have a complex information structure related to both physical and conceptual attributes. The conventional approach that focuses on a small number of physical attributes may fail to fully characterize shitsukan information. In this project, we propose a discovery-oriented approach to shitsukan that combines brain decoding and feature extraction from image and text databases.

Shitsukan feature extraction from image and text database. Brain decodeing of shitsukan features.

B01-3 Hierarchical Transformation of Shitsukan Information Representation in the Visual System

Principal Investigator : Izumi Ohzawa(Osaka University)

大澤 五住

Principal Investigator

Osaka University

Izumi Ohzawa


Hiroshi Tamura

Co-Investigator

Osaka University

Hiroshi Tamura

Kota Sasaki

Co-Investigator

Osaka University

Kota Sasaki

Co-Investigator Masato Okada (The University of Tokyo)

Outline

Model-free estimation of transformation in sensory representations from early to mid-level visual areas.

The primary objective of our research is to understand visual information processing related to "shitsukan" (perception of surface material and qualities) using data-driven approaches based on activities of single neurons in the brain. Electrophysiological and psychophysical methods are used. Since there are too many unknowns about details of processing at intermediate stages of the visual processing pathways, traditional methods based on a small number of hypotheses and models are applicable only in a limited manner. To overcome the limitation, data-driven approaches are needed. Specifically, new analytical methods and visual stimuli will be develped in order to reduce dependence of experiments on specific hypotheses and models. In addition to traditional stimuli, stimuli such as synthetic random stimuli based on elementary V1 signals are used. Neurons in V1 through mid-level visual areas in both the ventral and dorsal pathways will be studied.

B01-4 Shitsukan Information Based on Correspondences of Physics, Perception, and Affective Evaluations

Principal Investigator : Maki Sakamoto(University of Electro-Communications)

坂本 真樹

Principal Investigator

University of Electro-Communications

Maki Sakamoto


Shigeki Nakauchi

Co-Investigator

Toyohashi University of Technology

Shigeki Nakauchi

Outline

We construct Shitsukan database by exploring physical properties, material perceptions obtained by psychophysical experiments, affective evaluations described by language, which can be used as dataset for machine learning to build Shitsukan cognitive model or dataset for Shitsukan engineering. We also attempt to clarify Shitsukan cognitive mechanism how we perceive Shitsuan from materials and categorize it into onomatopoeia or linguitic sounds by analysing the relationship among physical properties, material perceptions, and subjective Shitsukan linguistic expressions. Furthermore, we construct a system which enables us to convert physical properties to onomatopoeia and vice versa.Finally, we want to contribute for developing new materials by generating onomatopoeia appropriate for physical or perceptual properties or discovering new Shitsukan using onomatopoeia.

Construction of Database Combining Physics,Perception,and Affective Evaluations for Shitsukan.

C01 Shitsukan Innovation

C01-1 Development of Record and Replay Technology for Haptic Feeling

Principal Investigator : Hiroyuki Kajimoto(The University of Electro-Communications)

梶本 裕之

Principal Investigator

The University of Electro-Communications

Hiroyuki Kajimoto


Shogo Okamoto

Co-Investigator

Nagoya University

Shogo Okamoto

Outline

Haptic feeling is one of important factors for the design of commercial products, and it recently attracts attention in the field of mobile or tangible electronic devices. However, current haptic technologies could present limited haptic feeling, mainly because of the hardware that achieved limited temporal/spatial resolution, and limited knowledge of temporal-spatial cross-interactions that contribute to haptic feeling.

The purposes of this research project are twofold. First, temporal-spatial characteristics that contribute to tactile feeling are determined by observation based study, and they are validated by mechanical stimulation. Second, touchpanel-based haptic display is developed, optimally combining several stimulation methods to present rich haptic feeling.

Find temporal-spatial characteristics that contribute to tactile feeling. Optimally design haptic touchpanel for rich haptic deeling.

C01-2 Modeling and Rendering of Appearances of Complex Objects by Computer Graphics Techniques

Principal Investigator : Yoshinori Dobashi(Hokkaido University)

土橋 宜典

Principal Investigator

Hokkaido University

Yoshinori Dobashi


Kei Iwasaki

Co-Investigator

Wakayama University

Kei Iwasaki

Makoto Okabe

Co-Investigator

University of Electro-Communications

Makoto Okabe

Takashi Ijiri

Co-Investigator

Ritsumeikan University

Takashi Ijiri

Hideki Todo

Co-Investigator

The University of Tokyo

Hideki Todo

Outline

Rendering complex appearances considering inner structures.

The goal of this project is to clarify the mechanism of material recognition and to find its engineering applications. We accelerate the research on the material recognition by using computer graphics techniques for virtually measuring some important factors that are impossible to be measured in the real world. Based on the recognition mechanism obtained by the scientific methods, we then develop novel appearance reproduction methods that are useful for practical applications. Unlike the previous methods for capturing and rendering object appearances, our approach treats a target object as a fully volumetric object by measuring its inner structures using computed tomography. By employing this approach, we aim at finding important factors for appearance reproduction that cannot be captured by the previous approach modeling the surface structures only. Furthermore, by making use of the fact that we acquire the volumetric structures, we extend our research to investigating the mechanism of material perception through sound and motion.

C01-3 Reproducing and Editing Material Properties on Real Object Surfaces by Programmable Illumination

Principal Investigator : Daisuke Iwai(Osaka University)

岩井 大輔

Principal Investigator

Osaka University

Daisuke Iwai


Shinsaku Hiura

Co-Investigator

Hiroshima City University

Shinsaku Hiura

Outline

Digital fabrication + projection mapping for SHITSUKAN reproduction and editing.

For understanding human recognition of material properties, we aim to realize a display which can computationally control and render various material appearances which are defined by complex and multidimensional reflectance properties or BTF (Bidirectional Texture Function). In this project, we develop a technology that can reproduce a desired BTF on a real object surface and edit it at a high flexibility. To achieve this goal, we particularly develop the following two technologies. First, we develop a calibration method of digital fabrication (D-Fab) equipment such as a milling machine and a multi-material 3D printer by applying computational photography techniques with programmable illumination. Using a calibrated D-Fab equipment, we can output a 3D object with a BTF which is close to a desired one. Second, we develop a spatially and angularly varying light projection system by which we edit the reflected light pattern from the surface of a D-Fab object. We apply computational display techniques to manipulate the light field in the space between the object and observers. Finally, we combine these two techniques to reproduce and edit material and surface qualities of real objects.

C01-4 Technologies for Analyzing, Controlling and Managing Information on Multimodal Material Perception in the Real World

Principal Investigator : Katsunori Okajima(Yokohama National University)

岡嶋 克典

Principal Investigator

Yokohama National University

Katsunori Okajima


Takahiko Horiuchi

Co-Investigator

Chiba University

Takahiko Horiuchi

Shoji Tominaga

Co-Investigator

Chiba University

Shoji Tominaga

Outline

We aim to establish "SHITSUKAN (Material Perception) engineering" as an academic system of SHITSUKAN management useful in industrial manufacturing sites, based on scientific understanding of material perception. "SHITSUKAN engineering" enables to control SHITSUKAN directly by operating physical parameters of stimuli by understanding information processing in our brain. For such a purpose, we will merge research results in the present research area, and apply the knowledge to solve various challenges in the real world. In addition, we will model and formulate SHITSUKAN mechanisms by analyzing SHITSUKAN information from many directions systematically. Finally, we will establish the comprehensive engineering system by which optical SHITSUKAN can be controlled and managed even in terms of wavelength dimension. From 2017, we will try to develop automatic measurement systems, analyzing systems, managing systems for optical SHITSUKAN information and to structure multimodal SHITSUKAN engineering by considering the five senses in cooperation with publicly-offered projects.

SHITSUKANs. Preprocessing. Measurement. Analyze/Control. Management.