UFLDL Recommended Readings
From Ufldl
| Line 3: | Line 3: | ||
The basics: | The basics: | ||
* [[http://cs294a.stanford.edu CS294A]] neural network/sparse autoencoder tutorial. (Most of this is now in the [[UFLDL Tutorial]], but the exercise is still on the CS294A website.) | * [[http://cs294a.stanford.edu CS294A]] neural network/sparse autoencoder tutorial. (Most of this is now in the [[UFLDL Tutorial]], but the exercise is still on the CS294A website.) | ||
| - | * Natural Image Statistics book, Hyvarinen et al. This is long, so just skim or skip the chapters that you already know. Important chapters: 5 (PCA and whitening; you'll probably already know the PCA stuff), 6 (sparse coding), 7 (ICA), 10 (ISA), 11 (TICA), 16 (temporal models). | + | * Natural Image Statistics book, Hyvarinen et al. This is long, so just skim or skip the chapters that you already know. |
| - | * Olshausen and Field Sparse Coding | + | ** Important chapters: 5 (PCA and whitening; you'll probably already know the PCA stuff), 6 (sparse coding), 7 (ICA), 10 (ISA), 11 (TICA), 16 (temporal models). |
| + | * [http://redwood.psych.cornell.edu/papers/olshausen_field_nature_1996.pdf] Olshausen and Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images Nature 1996. (Sparse Coding) | ||
* [http://www.cs.stanford.edu/~ang/papers/icml07-selftaughtlearning.pdf] Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer and Andrew Y. Ng. Self-taught learning: Transfer learning from unlabeled data. ICML 2007 | * [http://www.cs.stanford.edu/~ang/papers/icml07-selftaughtlearning.pdf] Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer and Andrew Y. Ng. Self-taught learning: Transfer learning from unlabeled data. ICML 2007 | ||
| Line 17: | Line 18: | ||
* Larochelle, Erhan, Courville, Bergstra, Bengio, ICML 2007. (Someone read this and let us know if this is worth keeping,.) | * Larochelle, Erhan, Courville, Bergstra, Bengio, ICML 2007. (Someone read this and let us know if this is worth keeping,.) | ||
* [http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf] Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why Does Unsupervised Pre-training Help Deep Learning? JMLR 2010 | * [http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf] Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why Does Unsupervised Pre-training Help Deep Learning? JMLR 2010 | ||
| - | * | + | * [http://cs.stanford.edu/~ang/papers/nips09-MeasuringInvariancesDeepNetworks.pdf] Ian J. Goodfellow, Quoc V. Le, Andrew M. Saxe, Honglak Lee and Andrew Y. Ng.Measuring invariances in deep networks. NIPS 2009. |
RBMs: | RBMs: | ||
| Line 25: | Line 26: | ||
* Computer Vision | * Computer Vision | ||
** [http://www.ifp.illinois.edu/~jyang29/ScSPM.htm] Jianchao Yang, Kai Yu, Yihong Gong, Thomas Huang. Linear Spatial Pyramid Matching using Sparse Coding for Image Classification, CVPR 2009 | ** [http://www.ifp.illinois.edu/~jyang29/ScSPM.htm] Jianchao Yang, Kai Yu, Yihong Gong, Thomas Huang. Linear Spatial Pyramid Matching using Sparse Coding for Image Classification, CVPR 2009 | ||
| - | ** Small codes and large image databases for recognition. | + | ** [http://people.csail.mit.edu/torralba/publications/cvpr2008.pdf] A. Torralba, R. Fergus and Y. Weiss. Small codes and large image databases for recognition. CVPR 2008. |
* Audio Recognition | * Audio Recognition | ||
| - | ** [http://www.cs.stanford.edu/people/ang/papers/nips09-AudioConvolutionalDBN.pdf] Unsupervised feature learning for audio classification using convolutional deep belief networks, Honglak Lee, Yan Largman, Peter Pham and Andrew Y. Ng. In NIPS | + | ** [http://www.cs.stanford.edu/people/ang/papers/nips09-AudioConvolutionalDBN.pdf] Unsupervised feature learning for audio classification using convolutional deep belief networks, Honglak Lee, Yan Largman, Peter Pham and Andrew Y. Ng. In NIPS 2009. |
| - | + | ||
| Line 35: | Line 35: | ||
* [http://ronan.collobert.com/pub/matos/2008_nlp_icml.pdf] R. Collobert and J. Weston. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. ICML 2008. | * [http://ronan.collobert.com/pub/matos/2008_nlp_icml.pdf] R. Collobert and J. Weston. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. ICML 2008. | ||
* [http://www.cs.toronto.edu/~hinton/absps/threenew.pdf] Mnih, A. and Hinton, G. E. Three New Graphical Models for Statistical Language Modelling. ICML 2007 | * [http://www.cs.toronto.edu/~hinton/absps/threenew.pdf] Mnih, A. and Hinton, G. E. Three New Graphical Models for Statistical Language Modelling. ICML 2007 | ||
| + | |||
Advanced stuff: | Advanced stuff: | ||
| Line 44: | Line 45: | ||
Mean-Covariance models | Mean-Covariance models | ||
| - | * 3- | + | * [http://www.cs.toronto.edu/~ranzato/publications/ranzato_aistats2010.pdf] M. Ranzato, A. Krizhevsky, G. Hinton. Factored 3-Way Restricted Boltzmann Machines for Modeling Natural Images. In AISTATS 2010. |
| - | * | + | * [http://www.cs.toronto.edu/~ranzato/publications/ranzato_cvpr2010.pdf] M. Ranzato, G. Hinton, Modeling Pixel Means and Covariances Using Factorized Third-Order Boltzmann Machines. CVPR 2010 |
| + | ** (someone and tell us if you need to read the 3-way RBM paper before the mcRBM one) | ||
* [http://www.cs.toronto.edu/~hinton/absps/mcphone.pdf] Dahl, G., Ranzato, M., Mohamed, A. and Hinton, G. E. Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine. NIPS 2010. | * [http://www.cs.toronto.edu/~hinton/absps/mcphone.pdf] Dahl, G., Ranzato, M., Mohamed, A. and Hinton, G. E. Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine. NIPS 2010. | ||
| - | * | + | * [http://www.nature.com/nature/journal/v457/n7225/pdf/nature07481.pdf] Y. Karklin and M. S. Lewicki, Emergence of complex cell properties by learning to generalize in natural scenes, Nature, 2008. |
| + | ** (someone tell us if this should be here. Interesting algorithm + nice visualizations, though maybe slightly hard to understand.) | ||
Overview | Overview | ||
* [http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf] Yoshua Bengio. Learning Deep Architectures for AI. FTML 2009. (Broad landscape description of the field, but technical details there are hard to follow so ignore that. This is also easier to read after you've gone over some of literature of the field.) | * [http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf] Yoshua Bengio. Learning Deep Architectures for AI. FTML 2009. (Broad landscape description of the field, but technical details there are hard to follow so ignore that. This is also easier to read after you've gone over some of literature of the field.) | ||
| - | |||
Practical guides: | Practical guides: | ||
| - | * A practical guide (read if you're trying to implement and RBM; but otherwise skip since this is not really a tutorial). [http:// | + | * [http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf] Geoff Hinton. A practical guide to training restricted Boltzmann machines. UTML TR 2010–003. |
| - | + | ** A practical guide (read if you're trying to implement and RBM; but otherwise skip since this is not really a tutorial). | |
| + | * [http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf] Y. LeCun, L. Bottou, G. Orr and K. Muller. Efficient Backprop. Neural Networks: Tricks of the trade, Springer, 1998 | ||
| + | ** Read if you're trying to run backprop; but otherwise skip since very low-level engineering/hackery tricks and not that satisfying to read. | ||
