Question

What are different methods by which you can compute the PCA? Does every method will yield same result or answer at the end?

Answer 1

1) Using covariance

Let X be a d-dimensional random vector expressed as a column vector. Without loss of generality, assume X has zero mean.

We want to find a d × d orthonormal transformation matrix P so that PX has a diagonal covariance matrix (i.e. PX is a random vector with all its distinct components pairwise uncorrelated).

A quick computation assuming were unitary yields:

Hence holds if and only if were diagonalizable by .

This is very constructive, as cov(X) is guaranteed to be a non-negative definite matrix and thus is guaranteed to be diagonalizable by some unitary matrix.

2) Iterative computation method

The pseudocode is given below:

3) The NIPALS method

Non-linear iterative partial least squares (NIPALS) is a variant the classical power iteration with matrix deflation by subtraction implemented for computing the first few components in a principal component or partial least squares analysis. For very-high-dimensional datasets, such as those generated in the *omics sciences (e.g., genomics, metabolomics) it is usually only necessary to compute the first few PCs. The non-linear iterative partial least squares (NIPALS) algorithm updates iterative approximations to the leading scores and loadings t₁ and r₁^T by the power iteration multiplying on every iteration by X on the left and on the right, i.e. calculation of the covariance matrix is avoided, just as in the matrix-free implementation of the power iterations to X^TX, based on the function evaluating the product X^T(X r) = ((X r)^TX)^T

⁴⁾ Online/sequential estimation

In an "online" or "streaming" situation with data arriving piece by piece rather than being stored in a single batch, it is useful to make an estimate of the PCA projection that can be updated sequentially. This can be done efficiently, but requires different algorithms.