Whitening
From Ufldl
(→Introduction) |
|||
Line 40: | Line 40: | ||
Further, | Further, | ||
the off-diagonal entries are zero; thus, | the off-diagonal entries are zero; thus, | ||
- | <math>\textstyle x_{{\rm rot},1}</math> and <math>\textstyle x_{{\rm rot},2}</math> are uncorrelated, satisfying one of | + | <math>\textstyle x_{{\rm rot},1}</math> and <math>\textstyle x_{{\rm rot},2}</math> are uncorrelated, satisfying one of our desiderata |
- | our desiderata for whitened data. | + | for whitened data (that the features be less correlated). |
To make each of our input features have unit variance, we can simply rescale | To make each of our input features have unit variance, we can simply rescale | ||
Line 91: | Line 91: | ||
When implementing PCA whitening or ZCA whitening in practice, sometimes some | When implementing PCA whitening or ZCA whitening in practice, sometimes some | ||
of the eigenvalues <math>\textstyle \lambda_i</math> will be numerically close to 0, and thus the scaling | of the eigenvalues <math>\textstyle \lambda_i</math> will be numerically close to 0, and thus the scaling | ||
- | step | + | step where we divide by <math>\sqrt{\lambda_i}</math> would involve dividing by a value close to zero; this |
- | the data to blow up (take on large values) or otherwise be numerically unstable. In practice, we | + | may cause the data to blow up (take on large values) or otherwise be numerically unstable. In practice, we |
- | implement | + | therefore implement this scaling step using |
a small amount of regularization, and add a small constant <math>\textstyle \epsilon</math> | a small amount of regularization, and add a small constant <math>\textstyle \epsilon</math> | ||
to the eigenvalues before taking their square root and inverse: | to the eigenvalues before taking their square root and inverse: | ||
Line 100: | Line 100: | ||
\end{align}</math> | \end{align}</math> | ||
When <math>\textstyle x</math> takes values around <math>\textstyle [-1,1]</math>, a value of <math>\textstyle \epsilon \approx 10^{-5}</math> | When <math>\textstyle x</math> takes values around <math>\textstyle [-1,1]</math>, a value of <math>\textstyle \epsilon \approx 10^{-5}</math> | ||
- | might be typical. | + | might be typical. |
For the case of images, adding <math>\textstyle \epsilon</math> here also has the effect of slightly smoothing (or low-pass | For the case of images, adding <math>\textstyle \epsilon</math> here also has the effect of slightly smoothing (or low-pass | ||
Line 118: | Line 118: | ||
performed by ZCA. This results in a less redundant representation of the input | performed by ZCA. This results in a less redundant representation of the input | ||
image, which is then transmitted to your brain. | image, which is then transmitted to your brain. | ||
+ | |||
+ | |||
+ | |||
+ | {{PCA}} | ||
+ | |||
+ | |||
+ | {{Languages|白化|中文}} |