Exercise:PCA and Whitening
From Ufldl
(→Step 3: PCA with dimension reduction) |
|||
| Line 22: | Line 22: | ||
==== Step 1a: Implement PCA ==== | ==== Step 1a: Implement PCA ==== | ||
| - | In this step, you will implement PCA to obtain <math>x_{\rm rot}</math>, the matrix in which the data is "rotated" to the basis comprising the principal components (i.e. the | + | In this step, you will implement PCA to obtain <math>x_{\rm rot}</math>, the matrix in which the data is "rotated" to the basis comprising the principal components (i.e. the eigenvectors of <math>\Sigma</math>). Note that in this part of the exercise, you should ''not'' whiten the data. |
==== Step 1b: Check covariance ==== | ==== Step 1b: Check covariance ==== | ||
| Line 67: | Line 67: | ||
That would be the case if you were doing whitening alone with no regularization. However, in this case you are whitening with regularization, to avoid numerical/etc. problems associated with small eigenvalues. As a result of this, some of the diagonal entries of the covariance of your <math>x_{\rm PCAwhite}</math> will be smaller than 1. | That would be the case if you were doing whitening alone with no regularization. However, in this case you are whitening with regularization, to avoid numerical/etc. problems associated with small eigenvalues. As a result of this, some of the diagonal entries of the covariance of your <math>x_{\rm PCAwhite}</math> will be smaller than 1. | ||
| - | To verify that your implementation of PCA whitening with and without regularization is correct, you can check these properties. Implement code to compute the covariance matrix and verify this property. (To check the result of PCA without whitening, simply set epsilon to 0, or close to 0, say <tt>1e-10</tt>). As earlier, you can visualise the covariance matrix with <tt>imagesc</tt>. When visualised as an image, for PCA whitening without regularization you should see a red line across the diagonal (corresponding to the one entries) against a blue background (corresponding to the zero entries); for PCA whitening with | + | To verify that your implementation of PCA whitening with and without regularization is correct, you can check these properties. Implement code to compute the covariance matrix and verify this property. (To check the result of PCA without whitening, simply set epsilon to 0, or close to 0, say <tt>1e-10</tt>). As earlier, you can visualise the covariance matrix with <tt>imagesc</tt>. When visualised as an image, for PCA whitening without regularization you should see a red line across the diagonal (corresponding to the one entries) against a blue background (corresponding to the zero entries); for PCA whitening with regularization you should see a red line that slowly turns blue across the diagonal (corresponding to the 1 entries slowly becoming smaller). |
<table> | <table> | ||
<tr> | <tr> | ||
| - | <td>[[File:pca_whitened_covar.png|360px|alt=Covariance for PCA whitening with | + | <td>[[File:pca_whitened_covar.png|360px|alt=Covariance for PCA whitening with regularization|Covariance for PCA whitening with regularization]]</td> |
| - | <td>[[File:pca_whitened_unregularised_covar.png|360px|alt=Covariance for PCA whitening with | + | <td>[[File:pca_whitened_unregularised_covar.png|360px|alt=Covariance for PCA whitening with regularization|Covariance for PCA whitening without regularization]]</td> |
</tr> | </tr> | ||
<tr> | <tr> | ||
| - | <td><center>Covariance for PCA whitening with | + | <td><center>Covariance for PCA whitening with regularization</center></td> |
| - | <td><center>Covariance for PCA whitening without | + | <td><center>Covariance for PCA whitening without regularization</center></td> |
</tr> | </tr> | ||
</table> | </table> | ||
| Line 82: | Line 82: | ||
=== Step 5: ZCA whitening === | === Step 5: ZCA whitening === | ||
| - | Now implement ZCA whitening to produce the matrix <math>x_{ZCAWhite}</math>. Visualize <math>x_{ZCAWhite}</math> and compare it to the raw data, <math>x</math>. You should observe that whitening results in, among other things, enhanced edges. Try repeating this with <tt>epsilon</tt> set to 1, 0.1, and 0.01, and see what you obtain. | + | Now implement ZCA whitening to produce the matrix <math>x_{ZCAWhite}</math>. Visualize <math>x_{ZCAWhite}</math> and compare it to the raw data, <math>x</math>. You should observe that whitening results in, among other things, enhanced edges. Try repeating this with <tt>epsilon</tt> set to 1, 0.1, and 0.01, and see what you obtain. The example shown below (left image) was obtained with <tt>epsilon</tt> = 0.1. |
<table> | <table> | ||
| Line 99: | Line 99: | ||
[[Category:Exercises]] | [[Category:Exercises]] | ||
| + | |||
| + | |||
| + | {{PCA}} | ||
