Deep Learning Starter Guide

Understanding core ideas

• Vanishing gradient problem
  • Decent formal explanation: https://ayearofai.com/rohan-4-the-vanishing-gradient-problem-ec68f76ffb9b
• Information bottleneck: Original paper: Deep Learning and the Information Bottleneck Principle
• Entire set of well written class notes: http://cs231n.github.io/

Understanding core solutions

• Convolutional networks:
  • Explanation with lots of examples and visualizations: https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/
  • Clear explanation of convolutional structure: http://colah.github.io/posts/2014-07-Conv-Nets-Modular/
• Long Short Term Memory (LSTM)
  • Intuitive explanation of structure: http://colah.github.io/posts/2015-08-Understanding-LSTMs/
  • Fun example, good starting point to implement LSTMs yourself: http://karpathy.github.io/2015/05/21/rnn-effectiveness/
• Dropout
  • the original paper is pretty good, Dropout: A Simple Way to Prevent Neural Networks from Overfitting
• Alternate cost functions (softmax, cross-entropy, etc).
  • http://neuralnetworksanddeeplearning.com/chap3.html
• More details on optimization methods
  • http://ruder.io/optimizing-gradient-descent/
• Alternate activation functions (Adam, Adagrad, RMSprop, etc.)
  • https://en.wikipedia.org/wiki/Activation_function

My links

• This cheatsheet: https://benblack769.github.io/neural_posts/getting-started-cheatsheet/
• Slides: link

Other fun sites

• Visualization: http://playground.tensorflow.org

Getting started implementing

My recommendation is to have a starting goal of having a 97% test accuracy rate with the MNIST dataset: http://yann.lecun.com/exdb/mnist/

• I got started here: http://neuralnetworksanddeeplearning.com
• Tensorflow’s MNIST tutorial: https://www.tensorflow.org/get_started/mnist/beginners

Once you get 97%, then try to do even better using some of the above techniques.

More?

Please feel free to comment if you think there are other resources or ideas I should put on here.