Shih-Fu Chang, Columbia University
Departments of Computer Science and Electrical Engineering

Words and Pictures – Crowdsource Discovery beyond Image Semantics
Large annotated images from the Web and crowdsource, together with powerful machine learning tools, play a crucial role in rapid progress in semantic recognition of image data in recent years. However, as inferred in the saying “A Picture is Worth More Than 1,000 Words,” there is much richer information than just semantic labels associated with images from the Web resources and Crowdsource Fora. Such additional information covers the rich unexploited aspects, such as visual aesthetics, emotions, sentiments, user intention, and knowledge structure. Discovering such novel dimensions of image descriptions beyond semantics will have huge impact for exciting emerging applications such as personalized search and content recommendation. But it requires rigorous research in concept definition, task formulation, data crawling, and evaluation mechanisms. In this talk, I will address these issues by sharing our experiences in discovering beyond-semantic visual descriptions related to visual sentiment, video upload intent classification, cultural influence on visual sentiment, and finally a wiki-style video event ontology.



Shih-Fu Chang is the Senior Vice Dean and the Richard Dicker Professor of The Fu Foundation School of Engineering and Applied Science at Columbia University. His research is focused on multimedia information retrieval, signal processing, computer vision, and machine learning, with a goal to develop intelligent systems that can harness rich information from the vast amount of visual data such as those emerging on the Web, through pervasive sensing, and in gigantic content management systems. His vision is to unleash the power of massive visual content and enable users to be able to search, summarize, and create visual content in a way as easy as text. His work on content-based visual search in the early 90’s such as VisualSEEk and VideoQ helped set the foundation of this vibrant research area. Over the years, he has continued to create innovative solutions in image/video recognition, multimodal analysis, multimedia ontology, image authentication, and compact hashing for large-scale image databases. He has extended the unique search capabilities to mobile search, 3D object search, and brain machine interfaces. Impact of his work can be seen in more than 300 peer-reviewed publications, 25 issued patents, and technologies licensed to six companies. For his long-term pioneering contributions, he has been awarded the IEEE Signal Processing Society Technical Achievement Award, ACM Multimedia SIG Technical Achievement Award, the IEEE Kiyo Tomiyasu Award, and IBM Faculty Award. For his dedicated contributions to education, he received the Great Teacher Award from the Society of Columbia Graduates. He served as Chair of Columbia Electrical Engineering Department (2007-2010), the Editor-in-Chief of the IEEE Signal Processing Magazine (2006-8), and advisor for several international research institutions and companies. In his current capacity as Senior Executive Vice Dean of Columbia Engineering School, he plays a key role in the School’s strategic planning, special research initiatives, and faculty development. He is actively involved in planning and steering of initiatives such as Data Science, Nano Science, Precision Medicine, and Mind Brain and Behavior. He is a Fellow of the American Association for the Advancement of Science and IEEE.