Whitening
From Ufldl
(→2D example) |
(→2D example) |
||
| Line 21: | Line 21: | ||
[[File:PCA-rotated.png | 600px]] | [[File:PCA-rotated.png | 600px]] | ||
| - | The covariance matrix of this data is given by | + | The covariance matrix of this data is given by: |
| - | + | ||
| - | + | <math>\begin{align} | |
| - | + | ||
| - | + | ||
| - | + | ||
| - | + | ||
\begin{bmatrix} | \begin{bmatrix} | ||
7.29 & 0 \\ | 7.29 & 0 \\ | ||
| Line 33: | Line 29: | ||
\end{bmatrix}. | \end{bmatrix}. | ||
\end{align}</math> | \end{align}</math> | ||
| + | |||
| + | (Note: Technically, many of the | ||
| + | statements in this section about the "covariance" will be true only if the data | ||
| + | has zero mean. In the rest of this section, we will take this assumption as | ||
| + | implicit in our statements. However, even if the data's mean isn't exactly zero, | ||
| + | the intuitions we're presenting here still hold true, and so this isn't something | ||
| + | that you should worry about.) | ||
| + | |||
It is no accident that the diagonal values are <math>\textstyle \lambda_1</math> and <math>\textstyle \lambda_2</math>. | It is no accident that the diagonal values are <math>\textstyle \lambda_1</math> and <math>\textstyle \lambda_2</math>. | ||
Further, | Further, | ||
| Line 46: | Line 50: | ||
\end{align}</math> | \end{align}</math> | ||
Plotting <math>\textstyle x_{{\rm PCAwhite}}</math>, we get: | Plotting <math>\textstyle x_{{\rm PCAwhite}}</math>, we get: | ||
| - | + | ||
| - | + | [[File:PCA-whitened.png | 600px]] | |
| - | + | ||
| - | + | ||
This data now has covariance equal to the identity matrix <math>\textstyle I</math>. We say that | This data now has covariance equal to the identity matrix <math>\textstyle I</math>. We say that | ||
<math>\textstyle x_{{\rm PCAwhite}}</math> is our '''PCA whitened''' version of the data: The | <math>\textstyle x_{{\rm PCAwhite}}</math> is our '''PCA whitened''' version of the data: The | ||
| Line 55: | Line 58: | ||
unit variance. | unit variance. | ||
| - | |||
| - | |||
'''Whitening combined with dimensionality reduction.''' | '''Whitening combined with dimensionality reduction.''' | ||
If you want to have data that is whitened and which is lower dimensional than | If you want to have data that is whitened and which is lower dimensional than | ||
| Line 62: | Line 63: | ||
<math>\textstyle x_{{\rm PCAwhite}}</math>. When we combine PCA whitening with regularization | <math>\textstyle x_{{\rm PCAwhite}}</math>. When we combine PCA whitening with regularization | ||
(described later), the last few components of <math>\textstyle x_{{\rm PCAwhite}}</math> will be | (described later), the last few components of <math>\textstyle x_{{\rm PCAwhite}}</math> will be | ||
| - | nearly zero anyway, and thus can safely be dropped. | + | nearly zero anyway, and thus can safely be dropped. |
| - | + | ||
| - | + | ||
| - | + | ||
== ZCE Whitening == | == ZCE Whitening == | ||
