Learning From Large-Scale Visual Data For Robots
Ozan Şener · eCommons (Cornell University) · 2016
Humans created a tremendous value by collecting and organizing all their knowledge in publicly accessible forms, as in Wikipedia and YouTube.The availability of such large knowledge bases not only changed the way we learn, it also changed how we design artificial intelligence algorithms.Recently, propelled by the available data and expressive models, many successful computer vision and natural language processing algorithms have emerged.However, we did not see a similar shift in robotics.Our robots are still having trouble recognizing basic objects, detecting humans, and even performing simple tasks like how to make an omelet.In this thesis, we study the type of knowledge robots need.Our initial analysis suggests that robots need a very unique type of knowledge base with many requirements like multi-modal data and physical grounding of concepts.We further design such a large-scale knowledge base and show how can it be used in many robotics tasks.Given this knowledge base, robots need to handle many challenges like scarcity of the supervision and the shift between different modalities and domains.We show that the common solution to all these problems is understanding the latent structure of the data.We also show that the key to discover and learn the latent structure is using large scale data.We propose machine learning algorithms, which can learn latent semantic structure with no supervision over multiple domains and modalities.Our algorithms show state-of-the-art performance in many robotics and computer vision benchmarks. BIOGRAPHICAL SKETCHOzan S ¸ener was born and raised in Sinop, a beautiful northern city in Turkey.His childhood memories mostly include writing various science-fiction stories full of robots and aliens.He learned programming during his high school in order to solve one nasty mathematical puzzle and his life changed when he realized he could build robots with that skill.He has built various robots since that day while competing in various robotics competitions.His next revelation was when he wanted to put a camera in one of his robots.He realized how challenging and fun the problem of robot perception/computer vision is and he lost his interest in mechanical and electrical engineering concepts.He obtained his Bachelor of Science degree from Middle East Technical University, Ankara, Turkey and joined a computer vision research lab in the same university to complete his masters.During his studies he collaborated closely with Nokia Research Center, Tampere, Finland developing mobile computer vision algorithms.His algorithms were deployed in production as part of multimedia engine in Nokia N9 series phones, and he still brags about it whenever he sees someone using a Nokia phone-unfortunately, it does not happen often anymore-.During his PhD studies at Cornell, he worked in the area of machine learning with applications to computer vision and robotics.He believes this is the perfect sweet spot for him, as he likes to work on theory, which can be applied on real systems and engineering systems, which has strong theoretical inspirations.In his free time, he likes to juggle, hike, and write.He re-activated the Cornell Juggling Club during his PhD.iii To the musicians of 70s and all the kids who listened them 1 1 In particular, two little kids from whom my parents grew iv ACKNOWLEDGEMENTS 1 -Tale of the White Rabbit [1] -Fox: "What are you working on?"-Rabbit: "My thesis."-Fox: "Hmmm.What's it about?"-Rabbit: "Oh, I'm writing about how rabbits eat foxes" -Fox: "That's ridiculous!Any fool knows that rabbits don't eat foxes" -Rabbit: "Sure they do, and I can prove it.Come with me."They both disappear into the rabbit's burrow.After a few minutes, the rabbit returns, alone, to his typewriter and resumes typing.Scene inside the rabbit's burrow: In one corner, there is a pile of fox bones.On the other side of the room, a huge lion is belching and picking his teeth.It doesn't matter what you choose for a thesis subject.It doesn't matter what you use for data.What does matter is who you have for a thesis advisor.Following the tale of the white rabbit, I was advised by Ashutosh Saxena and he was flanked by Silvio Savarese.I am grateful to both of them for their scientific contribution, as well as allowing me to pursue my own ideas and patiently supporting me during the process.I am also thankful to my thesis committee members Emin Gün Sirer and David Mimno for their useful comments and suggestions.2 -Down the Rabbit Hole @ Cornell First of all, I am grateful to Cornell Robot Learning Lab for creating a productive and fun environment both in Cornell and Stanford.I would like to thank Ashesh Jain, Dipendra Misra and Jae Sung for all the discussions and fun we had, Hema Koppula for all the guidance and help, and Ian Lenz and Chenxia Wu for their support and friendship.