Deep Learning Seminar 2016

In recent years, deep neural networks have been used to solve complex machine-learning problems and have achieved significant state-of-the-art results in many areas. The whole field of deep learning has been developing rapidly, with new methods and techniques emerging steadily.

The goal of the seminar is to follow the newest advancements in the deep learning field. The course takes form of a reading group – each lecture a paper is presented by one of the students. The paper is announced in advance, hence all participants can read it beforehand and can take part in the discussion of the paper.

Related Courses

  • Deep Learning – Course introducing deep neural networks, from the basics to the latest advances, focusing both on theory as well as on practical aspects.

Summer Semester

In summer semester 2016, the Deep Learning Seminar takes place on Tuesday at 12:20 in S1. We will first meet on Tuesday Feb 28.

If you want to receive announcements about chosen papers, sign up to our mailing list.

Program

Date Who Title Link
28 Feb 2017 Mirek Olšák C. Kaliszyk, F. Chollet, C. Szegedy: HolStep: A Machine Learning Dataset for Higher-order Logic Theorem Proving https://openreview.net/pdf?id=ryuxYmvel
    TreeRNN-based implementation of Mirek Olšák surpassing accuracy of above paper from 83% to 88% https://github.com/mirefek/HolStep-Tree
07 Mar 2017 Dušan Variš Jason Lee, Kyunghyun Cho, Thomas Hofmann: Fully Character-Level Neural Machine Translation without Explicit Segmentation https://arxiv.org/abs/1610.03017
14 Mar 2017 Karel Král Geoffrey Hinton, Oriol Vinyals, Jeff Dean: Distilling the Knowledge in a Neural Network https://arxiv.org/abs/1503.02531
    Lei Jimmy Ba, Rich Caruana: Do Deep Nets Really Need to be Deep? https://arxiv.org/abs/1312.6184
21 Mar 2017 Milan Straka Moshe Looks, Marcello Herreshoff, DeLesley Hutchins, Peter Norvig: Deep Learning with Dynamic Computation Graphs https://arxiv.org/abs/1702.02181
    Lingpeng Kong, Chris Alberti, Daniel Andor, Ivan Bogatyy, David Weiss: DRAGNN: A Transition-Based Framework for Dynamically Connected Neural Networks https://arxiv.org/abs/1703.04474
28 Mar 2017 Lukáš Jendele Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick: Mask R-CNN https://arxiv.org/abs/1703.06870
    Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, Yichen Wei: Fully Convolutional Instance-aware Semantic Segmentation https://arxiv.org/abs/1611.07709
04 Apr 2017 Ondrej Škopek Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, Samy Bengio: Neural Combinatorial Optimization with Reinforcement Learning https://arxiv.org/abs/1611.09940
    Oriol Vinyals, Meire Fortunato, Navdeep Jaitly: Pointer Networks https://arxiv.org/abs/1506.03134
11 Apr 2017 Jan Hajič jr. Diederik P Kingma, Max Welling: Auto-Encoding Variational Bayes https://arxiv.org/abs/1312.6114
    Francisco J. R. Ruiz, Michalis K. Titsias, David M. Blei: The Generalized Reparameterization Gradient https://papers.nips.cc/paper/6328-the-generalized-reparameterization-gradient.pdf
18 Apr 2017 Jindřich Libovický Holger Schwenk, Ke Tran, Orhan Firat, Matthijs Douze: Learning Joint Multilingual Sentence Representations with Neural Machine Translation https://arxiv.org/abs/1704.04154
25 Apr 2017 Milan Straka Mevlana Gemici et al.: Generative Temporal Models with Memory https://arxiv.org/abs/1702.04649
    Alex Graves et al.: Hybrid computing using a neural network with dynamic external memory https://www.gwern.net/docs/2016-graves.pdf
02 May 2017 David Mareček Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, Wang Ling: Learning to Compose Words into Sentences with Reinforcement Learning https://arxiv.org/abs/1611.09100
09 May 2017 Rudolf Rosa Michael Sejr Schlichtkrull, Anders Søgaard: Cross-Lingual Dependency Parsing with Late Decoding for Truly Low-Resource Languages https://www.aclweb.org/anthology/E/E17/E17-1021.pdf
16 May 2017 Jindřich Helcl Noam Shazeer et al.: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer https://arxiv.org/abs/1701.06538
23 May 2017 Peter Zborovský Luca Bertinetto et al.: Fully-Convolutional Siamese Networks for Object Tracking https://arxiv.org/abs/1606.09549

Papers for Inspiration

You can choose any paper you find interesting, but if you would like some inspiration, you can look at the following list.

Collections of Deep Learning Papers

Word Embeddings

  • Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, Adam Kalai: Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. https://arxiv.org/abs/1607.06520

Parsing

Neural Machine Translation

Language Correction

  • Ziang Xie, Anand Avati, Naveen Arivazhagan, Dan Jurafsky, Andrew Y. Ng: Neural Language Correction with Character-Based Attention. https://arxiv.org/abs/1603.09727

Language Modelling

Reinforcement Learning

  • Frank S. He, Yang Liu, Alexander G. Schwing, Jian Peng: Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening. https://arxiv.org/abs/1611.01606
  • Natasha Jaques, Shixiang Gu, Richard E. Turner, Douglas Eck: Tuning Recurrent Neural Networks with Reinforcement Learning. https://arxiv.org/abs/1611.02796
  • Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, Raia Hadsell: Learning to Navigate in Complex Environments. https://arxiv.org/abs/1611.03673
  • Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, Wang Ling: Learning to Compose Words into Sentences with Reinforcement Learning. https://arxiv.org/abs/1611.09100
  • Chelsea Finn, Tianhe Yu, Justin Fu, Pieter Abbeel, Sergey Levine: Generalizing Skills with Semi-Supervised Reinforcement Learning. https://arxiv.org/abs/1612.00429

Program Generation

Adversarial Networks

Network Architectures

  • Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, Nando de Freitas: Learning to learn by gradient descent by gradient descent. https://arxiv.org/abs/1606.04474
  • Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, Jürgen Schmidhuber: Recurrent Highway Networks. https://arxiv.org/abs/1607.03474
  • Lingpeng Kong, Chris Alberti, Daniel Andor, Ivan Bogatyy, David Weiss: DRAGNN: A Transition-Based Framework for Dynamically Connected Neural Networks. https://openreview.net/pdf?id=BycCx8qex
  • Bowen Baker, Otkrist Gupta, Nikhil Naik, Ramesh Raskar: Designing Neural Network Architectures using Reinforcement Learning. https://arxiv.org/abs/1611.02167

Structured Prediction

  • Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, Michael Collins: Globally Normalized Transition-Based Neural Networks. https://arxiv.org/abs/1603.06042

Image Labeling

  • Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan: Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge. https://arxiv.org/abs/1609.06647

Image Recognition

Image Enhancement

  • Justin Johnson, Alexandre Alahi, Li Fei-Fei: Perceptual Losses for Real-Time Style Transfer and Super-Resolution. https://arxiv.org/abs/1603.08155
  • Richard Zhang, Phillip Isola, Alexei A. Efros: Colorful Image Colorization. https://arxiv.org/abs/1603.08511
  • Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi: Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. https://arxiv.org/abs/1609.04802
  • Ryan Dahl, Mohammad Norouzi, Jonathon Shlens: Pixel Recursive Super Resolution. https://arxiv.org/pdf/1702.00783.pdf

Speech Synthesis

  • Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu: WaveNet: A Generative Model for Raw Audio. https://arxiv.org/abs/1609.03499