|
Research
My research interests lie at the boundary of computer vision and natural
language processing, specifically focused on understanding the connections
between these two related modalities. Today billions of images
with associated text are available on web pages, captioned photographs,
video with speech or closed captioning, and many others. In order to
organize, search, and exploit these enormous collections we work on developing
methods that combine information from both the visual and textual sources
effectively. Past and current projects include: automatically identifying
people in news photographs, classifying images from the web, selecting
aesthetically pleasing or interesting images, generating natural language
descriptions for images, visual social media analysis, and recognizing clothing
and style.
Bio
I received my B.S. in Mathematics and Computer Science from the University of Wisconsin, Madison in 2001. I then completed a PhD in Computer Science from the University of California, Berkeley in 2007 under the advisorship of Professor David Forsyth as a member of the Berkeley Computer Vision Group. Afterward, I spent 1 year as a research scientist at Yahoo! Research. From 2008-2013 I was an Assistant Professor in the Computer Science department at Stony Brook University and core member of the consortium for Digital Art, Culture, and Technology (cDACT). I joined the computer science department at the University of North Carolina Chapel Hill (UNC) in Fall 2013 and became an Associate Professor in 2015. I am the recipient of an NSF Career award, 2 google faculty awards, the 2013 Marr Prize, the 2016 UNC Hettleman Award.
Teaching
Spring 2016, Fall 2013, Spring 2012, Fall 2009 - CS 790-133 Recognizing People, Objects, and Actions
Fall 2015, Spring 2014 - CS 560 Artificial Intelligence
Spring 2015 - CS 790-133 Language and Vision
Fall 2012, Spring 2011 - CSE 595 Words and Pictures
Spring 2013/2012/2011/2010/2009 - CSE/ISE 364 Advanced Multimedia
Spring 2013, Fall 2011 - CSE 590 Computational Photography
Fall 2008 - CSE 690 Internet Vision
|
Publications
- TVQA+: Spatio-Temporal Grounding for Video Question Answering
Jie Lei,
Licheng Yu,
Tamara L. Berg
Mohit Bansal,
ArXiv, 2019
- Multi-Target Embodied Question Answering
Licheng Yu,
Xinlei Chen,
Georgia Gkioxari,
Mohit Bansal,
Tamara L. Berg
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Video
- TVQA: Localized, Compositional Video Question Answering
Jie Lei,
Licheng Yu,
Mohit Bansal,
Tamara L. Berg
Emperical Methods in Natural Language Processing (EMNLP), 2018.
- MAttNet: Modular Attention Network for Referring Expression Comprehension
Licheng Yu,
Zhe Lin,
Xiaohui Shen,
Jimei Yang,
Xin Lu,
Mohit Bansal,
Tamara L. Berg
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
- Visual to Sound: Generating Natural Sound for Videos in the Wild
Yipin Zhou,
Zhaowen Wang,
Chen Fang,
Trung Bui,
Tamara L. Berg
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Coverage in MIT Tech Review
- Combining Multiple Cues for Visual Madlibs Question Answering
Tatiana Tommasi
Arun Mallya
Bryan Plummer
Svetlana Lazebnik
Alexander C. Berg
Tamara L. Berg
International Journal on Computer Vision (IJCV), 2018.
- From Image to Language and Back Again
Anya Belz,
Tamara L. Berg,
Licheng Yu
Journal of Natural Language Engineering (JNLE), 2018.
- Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks
Yipin Zhou,
Yale Song,
Tamara L. Berg
Winter Conference on Applications of Computer Vision (WACV), 2018.
- Physics-Inspired Garment Recovery from a Single-View Image
Shan Yang,
Tanya Ambert,
Zherong Pan,
Ke Wang,
Licheng Yu,
Tamara L. Berg,
Ming C. Lin,
ACM Transactions on Graphics (TOG), 2018.
- Hierarchically-Attentive RNN for Album Summarization and Storytelling
Licheng Yu,
Mohit Bansal,
Tamara L. Berg,
Empirical Methods in Natural Language Processing (EMNLP), 2017.
- A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
Licheng Yu,
Hao Tan,
Mohit Bansal,
Tamara L. Berg,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
- When Was That Made?
Sirion Vittayakorn,
Alexander C. Berg,
Tamara L. Berg,
Winter Conference on Applications of Computer Vision (WACV), 2017.
- Modeling Context in Referring Expressions
Licheng Yu,
Patrick Poirson,
Shan Yang,
Alexander C. Berg,
Tamara L. Berg,
European Conference on Computer Vision (ECCV), 2016.
- Learning Temporal Transformations from Time-Lapse Videos
Yipin Zhou,
Tamara L. Berg,
European Conference on Computer Vision (ECCV), 2016.
- Solving Visual Madlibs with Multiple Cues
Tatiana Tommasi,
Arun Mallya,
Bryan Plummer,
Svetlana Lazebnik,
Alexander C. Berg,
Tamara L. Berg,
British Machine Vision Conference (BMVC), 2016.
- Auto-Illustrating Poems and Songs with Style
Katharina Schwarz,
Tamara L. Berg,
Asian Conference on Computer Vision (ACCV), 2016.
- When was that made?
Sirion Vittayakorn,
Alexander C. Berg,
Tamara L. Berg,
ArXiv, 2016.
- Detailed Garment Recovery from a Single-View Image
Shan Yang,
Tanya Ambert,
Zherong Pan,
Ke Wang,
Licheng Yu,
Tamara L. Berg,
Ming C. Lin,
ArXiv, 2016.
- Learning to Name Objects
Vicente Ordonez,
Wei Liu,
Jia Deng,
Yejin Choi,
Alexander C. Berg,
Tamara L. Berg,
Communications of the ACM (CACM) March, 2016. Technical perspective
- Combining Multiple Sources of Knowledge in Deep CNNs for Action Recognition
Eunbyung Park,
Xufeng Han,
Tamara L. Berg,
Alexander C. Berg,
Winter Conference on Applications of Computer Vision (WACV), 2016.
- Where to Buy It: Matching Street Clothing Photos in Online Shops
M. Hadi Kiapour,
Xufeng Han,
Svetlana Lazebnik,
Alexander C. Berg,
Tamara L. Berg,
International Conference on Computer Vision (ICCV), 2015.
Spotlight Video
- Visual Madlibs: Fill in the blank Description Generation and Question Answering
Licheng Yu,
Eunbyung Park,
Alexander C. Berg,
Tamara L. Berg,
International Conference on Computer Vision (ICCV), 2015.
Spotlight Video
- Temporal Perception and Prediction in Ego-Centric Video
Yipin Zhou,
Tamara L. Berg,
International Conference on Computer Vision (ICCV), 2015.
Spotlight Video
- Visual Madlibs: Fill in the blank Description Generation and Question Answering
Licheng Yu,
Eunbyung Park,
Alexander C. Berg,
Tamara L. Berg,
ArXiv, May 2015.
- Large Scale Retrieval and Generation of Image Descriptions
Vicente Ordonez,
Xufeng Han,
Polina Kuznetsova,
Girish Kulkarni,
Margaret Mitchell,
Kota Yamaguchi
Karl Stratos,
Amit Goyal,
Jesse Dodge,
Alyssa Mensch,
Hal Daumé III,
Alexander C. Berg,
Yejin Choi,
Tamara L. Berg,
International Journal of Computer Vision (IJCV), July 2015.
- Predicting Entry-Level Categories
Vicente Ordonez,
Wei Liu,
Jia Deng,
Yejin Choi,
Alexander C. Berg,
Tamara L. Berg,
To appear in International Journal of Computer Vision (IJCV) 2015.
- Runway to Realway: Visual Analysis of Fashion
Sirion Vittayakorn,
Kota Yamaguchi,
Alexander C. Berg,
Tamara L. Berg,
Winter Conference on Applications of Computer Vision (WACV) 2015.
- Refer-to-as Relations as Semantic Knowledge
Song Feng,
Sujith Ravi,
Ravi Kumar,
Polina Kuznetsova,
Wei Liu,
Alexander C. Berg,
Tamara L. Berg,
Yejin Choi,
AAAI Conference on Artificial Intelligence (AAAI) 2015.
- ReferItGame: Referring to Objects in Photographs of Natural Scenes
Sahar Kazemzadeh,
Vicente Ordonez,
Mark Matten,
Tamara L. Berg,
Empirical Methods in Natural Language Processing (EMNLP) 2014.
- TREETALK: Composition and Compression of Trees for Image Descriptions
Polina Kuznetsova,
Vicente Ordonez,
Tamara L. Berg,
Yejin Choi,
Transactions of the Association for Computational Linguistics (TACL) - to be presented at EMNLP 2014.
- Retrieving Similar Styles to Parse Clothing
Kota Yamaguchi,
Hadi Kiapour,
Luis E. Ortiz,
Tamara L. Berg,
Accepted to IEEE Transactions on Pattern Analysis (TPAMI) 2014.
- Hipster Wars: Discovering Elements of Fashion Styles
Hadi Kiapour,
Kota Yamaguchi,
Alexander C. Berg,
Tamara L. Berg,
European Conference on Computer Vision (ECCV) 2014.
- Learning High-level Judgments of Urban Perception
Vicente Ordonez,
Tamara L. Berg,
European Conference on Computer Vision (ECCV) 2014.
- Chic or Social: Visual Popularity Analysis in Online Fashion Networks
Kota Yamaguchi,
Tamara L. Berg,
Luis E. Ortiz,
ACM Multimedia (ACM MM) 2014.
- Materials Discovery: Fine-Grained Classification of X-ray Scattering Images
Hadi Kiapour,
Kevin Yager,
Alexander C. Berg,
Tamara L. Berg,
Winter Conference on Applications of Computer Vision (WACV) 2014.
- From Large Scale Image Categorization to Entry-Level Categories
Vicente Ordonez,
Jia Deng,
Yejin Choi,
Alexander C. Berg,
Tamara L. Berg,
International Conference on Computer Vision (ICCV) 2013 (oral).
Winner of the Marr prize
- Paper Doll Parsing: Retrieving Similar Styles to Parse Clothing
Kota Yamaguchi,
Hadi Kiapour,
Tamara L. Berg,
International Conference on Computer Vision (ICCV) 2013.
- Exploring the role of gaze behavior and object detection in scene understanding
Kiwon Yun,
Yifan Peng,
Dimitris Samaras,
Greg Zelinsky,
Tamara L Berg
Frontiers in Psychology, Perception Science, Dec 2013.
- Generalizing Image Captions for Image-Text Parallel Corpus
Polina Kuznetsova,
Vicente Ordonez,
Alexander C. Berg,
Tamara L. Berg,
Yejin Choi
Association for Computational Linguistics, (ACL) 2013.
- Studying Relationships Between Human Gaze, Description, and Computer Vision
Kiwon Yun,
Yifan Peng,
Greg Zelinsky,
Dimitris Samaras,
Tamara L Berg
Computer Vision and Pattern Recognition, (CVPR) 2013.
- Specifying the relationships between objects, gaze, and descriptions for scene understanding
Kiwon Yun,
Yifan Peng,
Hossein Adeli,
Tamara L Berg
Dimitris Samaras,
Greg Zelinsky,
Journal of Vision, Abstract presented at VSS, July 2013.
- BabyTalk: Understanding and Generating Simple Image Descriptions
Girish Kulkarni,
Visruth Premraj,
Vicente Ordonez,
Sagnik Dhar,
Siming Li,
Yejin Choi,
Alexander C. Berg,
Tamara L Berg
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Dec 2013.
- Parsing Clothing in Fashion Photographs
[pdf]
Kota Yamaguchi,
Hadi Kiapour,
Luis E. Ortiz,
Tamara L. Berg
Computer Vision and Pattern Recognition, (CVPR) 2012.
- Understanding and Predicting Importance in Images
[pdf]
Alexander C. Berg,
Tamara L. Berg,
Hal Daumé III,
Jesse Dodge,
Amit Goyal,
Xufeng Han,
Alyssa Mensch,
Margaret Mitchell,
Aneesh Sood,
Karl Stratos,
Kota Yamaguchi
Computer Vision and Pattern Recognition, (CVPR) 2012.
- Collective Generation of Natural Image Descriptions
[pdf]
Polina Kuznetsova,
Vicente Ordonez,
Alexander C. Berg,
Tamara L. Berg,
Yejin Choi
Association for Computational Linguistics (ACL) 2012.
- Detecting Visual Text
[pdf]
Jesse Dodge,
Amit Goyal,
Xufeng Han,
Alyssa Mensch,
Margaret Mitchell,
Karl Stratos,
Kota Yamaguchi,
Yejin Choi,
Hal Daumé III,
Alexander C. Berg,
Tamara L. Berg,
North American Chapter of the Association for Computational Linguistics (NAACL) 2012.
- Midge: Generating Image Descriptions From Computer Vision Detections
[pdf]
Margaret Mitchell,
Jesse Dodge,
Amit Goyal,
Kota Yamaguchi,
Karl Sratos,
Xufeng Han,
Alysssa Mensch,
Alexander C. Berg,
Tamara L. Berg,
Hal Daumé III
European Chapter of the Association for computational Linguistics, (EACL) 2012.
- Interactive Music: Human Motion Initiated Music Generation
Using Skeletal Tracking By Kinect
[pdf]
Tamara L. Berg,
Debaleena Chattopadhyay,
Margaret Schedel,
Timothy Vallier
SEAMUS, 2012.
- Two-person Interaction Detection Using Body-Pose Features and Multiple Instance Learning
[pdf]
Kiwon Yun,
Jean Honorio,
Debaleena Chattopadhyay,
Tamara L. Berg,
Dimitris Samaras
The 2nd International Workshop on Human Activity Understanding from 3D Data at Conference on Computer Vision and Pattern Recognition, (CVPR) 2012.
- JHU-CLSP Summer 2011 Workshop
Xufeng Han,
Alexander C. Berg,
Tamara L. Berg,
Hal Daumé III,
Jesse Dodge,
Amit Goyal,
Xufeng Han,
Alyssa Mensch,
Margaret Mitchell,
Karl Stratos,
Kota Yamaguchi
- Im2Text: Describing Images Using 1 Million Captioned Photographs
[pdf]
Vicente Ordonez,,
Girish Kulkarni,,
Tamara L. Berg
Neural Information Processing Systems (NIPS), 2011.
- Composing Simple Image Descriptions using Web-scale N-grams.
[pdf]
Siming Li,
Girish Kulkarni,
Tamara L. Berg,
Alexander C. Berg,
Yejin Choi
Computational Natural Language Learning (CoNLL), 2011.
- Iconizer: A Framework to Identify and Create Effective
Representations for Visual Information Encoding
[pdf]
Supriya Garg,
Tamara L. Berg,
Klaus Mueller
The 11th International Symposium on Smart Graphics (SG), 2011
- Baby Talk: Understanding and Generating Simple Image Descriptions
[pdf]
Girish Kulkarni,
Visruth Premraj,
Sagnik Dhar,
Siming Li,
Yejin Choi,
Alexander C. Berg,
Tamara L. Berg
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2011 (oral)
- High Level Describable Attributes for Predicting Aesthetics and Interestingness
[pdf]
Sagnik Dhar,
Vicente Ordonez,
Tamara L. Berg,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2011
- Who are you with and where are you going?
[pdf]
Kota Yamaguchi,
Alexander C. Berg,
Luis Ortiz
Tamara L. Berg,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2011
- Can Computers Master the Art of Communication? An Excursion with a Focus on Visual Analytics
Klaus Mueller,
Supriya Garg,
Julia Nam,
Tamara L. Berg,
Kevin McDonnell.
IEEE Computer Graphics and Applications, May/June 2011.
- Automatic Attribute Discovery and Characterization from Noisy Web Data
[pdf]
Tamara L. Berg,
Alexander C. Berg,
Jonathan Shih
The European Conference on Computer Vision (ECCV) 2010.
- iWalk, A Tool for Interacting with Geo-Located Data Through Movement and Gesture
[pdf]
Visruth Premraj,
Margaret Schedel,
Tamara L. Berg,
ACM Multimedia, Human Centered Multimedia Track (ACM MM) 2010.
- It's All About the Data
Tamara L. Berg, Alexander Sorokin, Gang Wang, David A. Forsyth, Derek Hoiem, Ali Farhadi, Ian Endres.
Proceedings of the IEEE, Special Issue on Internet Vision, August 2010, 98-8, 1434-1453.
- Finding Iconic Images
[pdf]
[ps]
Tamara L. Berg,
Alexander C. Berg,
The 2nd Internet Vision Workshop at Conference on Computer Vision and Pattern Recognition (CVPR) 2009.
- Words and Pictures: Categories, Modifiers, Depiction and Iconography
D.A. Forsyth, T.L. Berg, C. Alm, A. Farhadi, J. Hockenmaier, N. Loeff, G. Wang.
Object Categorization: Computer and Human Vision Perspectives. Cambridge University Press, 2009, in press. Sven Dickinson, Michael Tarr, Ales Leonardis, Bernt Schiele (eds)
- Names and Faces
[pdf]
[ps]
Tamara L. Berg,
Alexander C. Berg,
Jaety Edwards,
Michael Maire,
Ryan White,
Yee Whye Teh,
Erik Learned-Miller,
David A. Forsyth
In Submission
- Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments [pdf]
Gary B. Huang, Marwan Mattar, Tamara Berg, and Erik Learned-Miller.
The Workshop on Faces in Real-Life Images at European Conference on Computer Vision (ECCV) 2008.
- Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments [pdf]
Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller.
University of Massachusetts, Amherst, Technical Report 07-49, October, 2007
- Exploiting Words and Pictures
[pdf]
Tamara L. Berg
U.C. Berkeley Ph.D. Thesis, May. 2007
- Dataset Issues in Object Recognition
[pdf]
[ps]
J. Ponce, T. L. Berg, M. Everingham, D.A. Forsyth, M. Hebert, S. Lazebnik, M. Marszalek, C. Schmid, B.C. Russell, A. Torralba, C.K.I. Williams, J. Zhang and A. Zisserman,
Toward Category-Level Object Recognition,
Springer-Verlag Lecture Notes in Computer Science. J. Ponce, M. Hebert, C. Schmid and A. Zisserman (eds.), Feb 2007.
- Automatic Ranking of Iconic Images
[pdf]
[ps]
Tamara L. Berg,
David A. Forsyth
U.C. Berkeley Technical Report, Jan. 2007
- Names and Faces
[pdf]
[ps]
Tamara L. Berg,
Alexander C. Berg,
Jaety Edwards,
Michael Maire,
Ryan White,
Yee Whye Teh,
Erik Learned-Miller,
David A. Forsyth
U.C. Berkeley Technical Report, Jan. 2007
- Animals on the Web
[pdf]
[ps]
Tamara L. Berg,
David A. Forsyth
Computer Vision and Pattern Recognition (CVPR) 2006
-
Shape Matching and Object Recognition using Low Distortion Correspondence
[pdf]
[ps]
[ppt]
Alexander C. Berg,
Tamara L. Berg,
Jitendra Malik
Computer Vision and Pattern Recognition (CVPR), 2005
- Shape Matching and Object Recognition using Low Distortion Correspondence
[pdf]
[ps]
Alexander C. Berg,
Tamara L. Berg,
Jitendra Malik
U.C. Berkeley Technical Report, Dec. 2004
- Who's in the Picture?
[pdf]
[ps]
Tamara L. Berg,
Alexander C. Berg,
Jaety Edwards,
David A. Forsyth
Neural Information Processing Systems (NIPS), 2004
- Names and Faces in the News
[pdf]
[ps]
Tamara L. Berg,
Alexander C. Berg,
Jaety Edwards,
Michael Maire,
Ryan White,
Yee Whye Teh,
Erik Learned-Miller,
David A. Forsyth
Computer Vision and Pattern Recognition (CVPR), 2004
|
Past Funding
BIGDATA: Award #1633295 - BIGDATA: F: Collaborative Research: From Visual Data to Visual Understanding, Project Website
NSF IIS Core: Award #1562098 - RI: Medium: Collaborative Research: Text-to-Image Reference Resolution for Image Understanding and Manipulation
NSF CI-New: Collaborative Research: Award #1405822 - Federated Data Set Infrastructure for Recognition Problems in Computer Vision
NSF II-New: Award #1405847 - Seeing the Future: Ubiquitous Computing in EyeGlasses
Microsoft, Google, Adobe
NSF Faculty Early Career Development (CAREER) Program: Award #1444234 - Toward a General Framework for Words & Pictures. (previously Award #1054133) Project Page
NSF IIS Core: Award #1445409 - RI: Medium: Integrating Humans and Computers for Image and Video Understanding (previously Award #1161876)
CI-P:Collaborative Research Award #1417991 - Visual Entailment data set and challenge for the language and vision communities (previously Award #1205354)
Seeing Social: Exploiting Computer Vision in Online Communities. Google Faculty Research Award
SBU/BNL Seed Grant: "The Data Sensorium: Multi-Modal Explorations of Scientific Data". Personel - Dan Weymouth, Kevin Yager, Tamara Berg, Margaret Schedel, Klaus Mueller, Dimitris Samaras, Tony Phillips, Rita Goldstein, Nelly Alia-Klein, Zabet Patterson.
NSF MRI-R2 grant: "Development of an Immersive Giga-pixel Display". Contributor as Senior Personel.
Stony Brook FAHSS grant: "Encountering Data". Daniel Weymouth, Tamara Berg, Zabet Patterson, Margaret Schedel, John Lutterbie.
Stony Brook FAHSS grant: "Hybrid Geographies". Zabet Patterson, Christa Erickson, Margaret Schedel, Tamara Berg, Raiford Guins, Andrew Uroskie.
Former Students & Postdocs
Yipin Zhou (PhD, UNC)
Licheng Yu (PhD, UNC)
Jie Lei (PhD, UNC)
Sirion Vittayakorn (PhD, UNC) 2016 - Assistant Professor Thailand
Tatiana Tommasi (Postdoc, UNC 2015-2016)
Hadi Kiapour (PhD, UNC) 2015 - Research Scientist at eBay Research
Zhilan Zhou (Undergrad, UNC)
Justo (Miko) Marquez (Undergrad, UNC)
Jesse Hu (Undergrad, Duke)
Fred Landis (Undergrad, UNC)
Joe Puccio (Undergrad, UNC)
Vicente Ordonez (PhD, UNC) 2015 - Assistant Professor University of Virginia.
Rob Lewis (MS, UNC) 2015
Brian Christante (MS, UNC) 2015
Hannah Kerner (MS, UNC) 2015
Kota Yamaguchi (PhD, SBU) 2014 - Assistant Professor Tohoku University
Sahar Kazemzadeh (Undergrad, UNC) 2014
Aaron Brown (Undergrad, UNC) 2014
Josh Potter (Undergrad, UNC) 2014
Sean Freemerman (Undergrad, UNC) 2014
Roman Myers (Undergrad, UNC) 2014
Sebo Kim (Undergrad, SBU) 2014
Vinson Luo (High School, SBU Simons program) - summer 2013
Mark Matten (High School, SBU Simons program) - summer 2013
Priyanka Khante (Undergrad, SBU) - summer 2013
Chen Ling (MS, SBU) - 2013
Hanyu Liu (MS, SBU) - 2013
Chaitanya Kommini (MS Indepdendent Study, SBU) - 2012
Deepak Venkatachalam (MS Independent Study, SBU) - 2011
Farheen Noorie (MS Independent Study, SBU) - 2011
Girish Kulkarni (MS, SBU) - 2011 Epic Systems
Debaleena Chattopadhy (MS, SBU) - 2011 Indiana School of Informatics PhD
Sagnik Dhar (MS, SBU) 2010 - Volkswagen
Visruth Premraj (MS, SBU) 2010 Epic Systems
Erin Palmer (MS, SBU) 2009 - Factset
Jose Villa (MS, SBU) 2010
Piyush Kumat, (MS Indendent Study, SBU) Fall 2009
Alex Berg, my husband.
Arnold Miller, my dad.
|