CSE 591: Recognizing People, Objects, and Actions Instructor: Tamara Berg  (tlberg -at- cs.sunysb.edu) Office: 1411 Computer Science Lectures: Tues/Thurs 11:20-12:40pm Rm P112, Physics Office Hours: Tues/Thurs 1:00-2:00pm, and by appointment Course Webpage: http://tamaraberg.com/teaching/Spring_12/cse591

Introduction Recognition is one of the core pursuits of computer vision. In recognition one attempts to attach semantics to visual data such as images or video. Object recognition is an important subtopic where one builds models to recognize object categories or instances. Other subtopics include: activity recognition -- building descriptions of what people are doing from visual data, face recognition -- attaching identities to pictures or video of faces, and detection -- localizing all instances of a particular category in an image. This course will look at both historical and current methods for recognizing objects, people, actions, and scenes in images and video. Students will have a chance to define their own problems and work on solutions through a course project.

Topics Objects - single instance or category based People - faces, pedestrians, pose, and actions Scenes - whole image features, recognition in context, surfaces Recognition using vision + other modalities Human-centric recognition outputs

Tentative Schedule Date Topic Readings Presenter Assignments Jan 24 Intro & Overview of Course - Slides - Tamara Get access to matlab, do a tutorial. Jan 26 Computer Vision Review - Slides Sections 1.1-1.2 Tamara Choose a paper to present from the reading list, HW1 released Jan 31 Computer Vision Review (cont) see slides from Jan 26 - Tamara - Feb 2 Bag of Features Models - Slides Object Recognition from Local Scale-Invariant Features Tamara - Feb 7 Bag of Features Models - Slides Visual Categorization with Bags of Keypoints Tamara HW2 released Feb 9 BoF (cont) & Spatial Models - Slides Shape Matching and Object Recognition Using Low Distortion Correspondence Tamara, Kalyan - Feb 14 To Categorize or Not to Categorize - Slides Principles of Categorization, Beyond Categories: The Visual Memex Model for Reasoning About Object Relationships Tamara - Feb 16 Recognizing Attributes - Slides1 Attribute and Simile Classifiers for Face Verification, Relative Attributes Tamara, Chen - Feb 21 Faces Face Recognition using Eigenfaces, Robust Real-time Face Detection Chaitanya, Vinay HW3 released Feb 23 Faces (cont) - - - Feb 28 Intro to Pose & Actions - Slides - Tamara - March 1 Pedestrian Detection Histograms of Oriented Gradients for Human Detection, Pedestrian Detection in Crowded Scenes Hailin, Nihar - March 6 Project Proposals - - Prepare a 5 minute proposal presentation March 8 Pose Estimation in Images Recovering Human Body Configurations: Combining Segmentation and Recognition, Poselets: Body Part Detectors trained Using 3D Human Pose Annotations Keerthi, Rajan - March 13 Action Recognition Recognizing Action at a Distance, Learning Realistic Human Actions from Movies Swastika,Shobha - March 15 Intro to Scenes - Slides Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories Tamara - March 20 Scene Interpretation Automatic Photo Pop-Up, Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics Tamara,Rohit - March 22 Project Updates - - Prepare a 5 minute update presentation March 27 Project Help Day - - - March 29 Scenes & What We can Do in Them Recovering the Spatial Layout of Cluttered Rooms, From 3D Scene Geometry to Human Workspace Feifei, Fan - April 3 Spring Break - - - April 5 Spring Break - - - April 10 Words & Pictures - Tamara - April 12 Catch up Day - - - April 17 No Class - Traveling - - - April 19 Project Updates - - Prepare a 5 minute update presentation April 24 Generating Descriptions of Images Baby Talk: Understanding and Generating Simple Image Descriptions, Im2Text: Describing Images Using 1 Million Captioned Photographs Tamara - April 26 Predicting Aesthetic Quality High Level Describable Attributes for Predicting Aesthetics and Interestingness, Assessing the aesthetic quality of photographs using generic image descriptors Tamara - May 1 Final Project Presentations - Shobha & Rajan, Keerthi & Nihar, Chaitanya & Kalyan, Hanyu & Chen Prepare a 15 minute final presentation May 3 Final Project Presentations - FeiFei & Hailin, Swastika & Rohit & Vinay, Fan Prepare a 15 minute final presentation May 8 Final Project Write-up - - Project Write-up Due Grading There will be 3 simple homeworks during the first 1-2 months of the course to get students aquainted with computer vision and recognition. Over the final part of the course students will develop and present a project related to recognition. Students will also be responsible for leading one class paper discussion. A few short quizzes will also be given about assigned papers. Grading will consist of: Assignments (35%), Project (35%), Paper presentation (10%), Paper quizzes (10%), Participation (10%). No prior experience in computer vision is required to take this course. Homeworks should be done individually, but projects may be done individually or in pairs. Homeworks will be completed in matlab. Submit all homeworks, and presentations to: cse591@gmail.com Useful links Matlab Student Matlab licenses can be purchased from mathworks for $99 - Link. Matlab tutorial by Hany Farid and Eero Simoncelli - Link A more comprehensive Matlab tutorial by David Griffiths - Link Data Label Me - Link Tiny Images - Link Code for downloading Flickr images - Link Computing Features SIFT features - Link Scale Invariant Interest Points - Link Affine Covariant Regions - Link Shape Contexts - Link Gist - Link Other Useful Software Various Code from INRIA - Link Various Code from Oxford - Link Various useful machine learning tools - Link Reference Books Forsyth, David A., and Ponce, J. Computer Vision: A Modern Approach, Prentice Hall, 2003. Hartley, R. and Zisserman, A. Multiple View Geometry in Computer Vision, Academic Press, 2002. Stephen E Palmer, Vision Science: Photons to Phenomenology, MIT Press, 1999.