Exercise:PCA and Whitening
From Ufldl
(→Step 5: ZCA whitening) |
|||
| Line 22: | Line 22: | ||
==== Step 1a: Implement PCA ==== | ==== Step 1a: Implement PCA ==== | ||
| - | In this step, you will implement PCA to obtain <math>x_{\rm rot}</math>, the matrix in which the data is "rotated" to the basis comprising the principal components (i.e. the | + | In this step, you will implement PCA to obtain <math>x_{\rm rot}</math>, the matrix in which the data is "rotated" to the basis comprising the principal components (i.e. the eigenvectors of <math>\Sigma</math>). Note that in this part of the exercise, you should ''not'' whiten the data. |
==== Step 1b: Check covariance ==== | ==== Step 1b: Check covariance ==== | ||
| Line 38: | Line 38: | ||
Now that you have found <math>k</math>, compute <math>\tilde{x}</math>, the reduced-dimension representation of the data. This gives you a representation of each image patch as a <math>k</math> dimensional vector instead of a 144 dimensional vector. If you are training a sparse autoencoder or other algorithm on this reduced-dimensional data, it will run faster than if you were training on the original 144 dimensional data. | Now that you have found <math>k</math>, compute <math>\tilde{x}</math>, the reduced-dimension representation of the data. This gives you a representation of each image patch as a <math>k</math> dimensional vector instead of a 144 dimensional vector. If you are training a sparse autoencoder or other algorithm on this reduced-dimensional data, it will run faster than if you were training on the original 144 dimensional data. | ||
| - | To see the effect of dimension reduction, go back from <math>\tilde{x}</math> to produce the matrix <math>\hat{x}</math>, the dimension-reduced data but expressed in the original 144 dimensional space of image patches. Visualise <math>\hat{x}</math> and compare it to the raw data, <math>x</math>. You will observe that there is little loss due to throwing away the principal components that correspond to dimensions with low variation. For comparison, you may also wish to generate and visualise <math>\hat{x}</math> for when only | + | To see the effect of dimension reduction, go back from <math>\tilde{x}</math> to produce the matrix <math>\hat{x}</math>, the dimension-reduced data but expressed in the original 144 dimensional space of image patches. Visualise <math>\hat{x}</math> and compare it to the raw data, <math>x</math>. You will observe that there is little loss due to throwing away the principal components that correspond to dimensions with low variation. For comparison, you may also wish to generate and visualise <math>\hat{x}</math> for when only 90% of the variance is retained. |
<table> | <table> | ||
| Line 67: | Line 67: | ||
That would be the case if you were doing whitening alone with no regularization. However, in this case you are whitening with regularization, to avoid numerical/etc. problems associated with small eigenvalues. As a result of this, some of the diagonal entries of the covariance of your <math>x_{\rm PCAwhite}</math> will be smaller than 1. | That would be the case if you were doing whitening alone with no regularization. However, in this case you are whitening with regularization, to avoid numerical/etc. problems associated with small eigenvalues. As a result of this, some of the diagonal entries of the covariance of your <math>x_{\rm PCAwhite}</math> will be smaller than 1. | ||
| - | To verify that your implementation of PCA whitening with and without regularization is correct, you can check these properties. Implement code to compute the covariance matrix and verify this property. (To check the result of PCA without whitening, simply set epsilon to 0, or close to 0, say <tt>1e-10</tt>). As earlier, you can visualise the covariance matrix with <tt>imagesc</tt>. When visualised as an image, for PCA whitening without regularization you should see a red line across the diagonal (corresponding to the one entries) against a blue background (corresponding to the zero entries); for PCA whitening with | + | To verify that your implementation of PCA whitening with and without regularization is correct, you can check these properties. Implement code to compute the covariance matrix and verify this property. (To check the result of PCA without whitening, simply set epsilon to 0, or close to 0, say <tt>1e-10</tt>). As earlier, you can visualise the covariance matrix with <tt>imagesc</tt>. When visualised as an image, for PCA whitening without regularization you should see a red line across the diagonal (corresponding to the one entries) against a blue background (corresponding to the zero entries); for PCA whitening with regularization you should see a red line that slowly turns blue across the diagonal (corresponding to the 1 entries slowly becoming smaller). |
<table> | <table> | ||
<tr> | <tr> | ||
| - | <td>[[File:pca_whitened_covar.png|360px|alt=Covariance for PCA whitening with | + | <td>[[File:pca_whitened_covar.png|360px|alt=Covariance for PCA whitening with regularization|Covariance for PCA whitening with regularization]]</td> |
| - | <td>[[File:pca_whitened_unregularised_covar.png|360px|alt=Covariance for PCA whitening with | + | <td>[[File:pca_whitened_unregularised_covar.png|360px|alt=Covariance for PCA whitening with regularization|Covariance for PCA whitening without regularization]]</td> |
</tr> | </tr> | ||
<tr> | <tr> | ||
| - | <td><center>Covariance for PCA whitening with | + | <td><center>Covariance for PCA whitening with regularization</center></td> |
| - | <td><center>Covariance for PCA whitening without | + | <td><center>Covariance for PCA whitening without regularization</center></td> |
</tr> | </tr> | ||
</table> | </table> | ||
| Line 82: | Line 82: | ||
=== Step 5: ZCA whitening === | === Step 5: ZCA whitening === | ||
| - | Now implement ZCA whitening to produce the matrix <math>x_{ZCAWhite}</math>. Visualize <math>x_{ZCAWhite}</math> and compare it to the raw data, <math>x</math>. You should observe that whitening results in, among other things, enhanced edges. Try repeating this with <tt>epsilon</tt> set to 1, 0.1, and 0.01, and see what you obtain. | + | Now implement ZCA whitening to produce the matrix <math>x_{ZCAWhite}</math>. Visualize <math>x_{ZCAWhite}</math> and compare it to the raw data, <math>x</math>. You should observe that whitening results in, among other things, enhanced edges. Try repeating this with <tt>epsilon</tt> set to 1, 0.1, and 0.01, and see what you obtain. The example shown below (left image) was obtained with <tt>epsilon</tt> = 0.1. |
<table> | <table> | ||
| Line 99: | Line 99: | ||
[[Category:Exercises]] | [[Category:Exercises]] | ||
| + | |||
| + | |||
| + | {{PCA}} | ||
