Question

10.2

Suppose we a data set where each data point represents a single student's scores on a math test, a physics test, a reading comprehension test, and a vocabulary test.

We find the first two principal components, which capture 90% of the variability in the data, and interpret their loadings. We conclude that the first principal component represents overall academic ability, and the second represents a contrast between quantitative ability and verbal ability.

What loadings would be consistent with that interpretation? Choose all that apply.

a) (0.5, 0.5, 0.5, 0.5) and (0.71, 0.71, 0, 0)

b) (0.5, 0.5, 0.5, 0.5) and (0, 0, -0.71, -0.71)

c) (0.5, 0.5, 0.5, 0.5) and (0.5, 0.5, -0.5, -0.5)

d) (0.5, 0.5, 0.5, 0.5) and (-0.5, -0.5, 0.5, 0.5)

e) (0.71, 0.71, 0, 0) and (0, 0, 0.71, 0.71)

f) (0.71, 0, -0.71, 0) and (0, 0.71, 0, -0.71)

Answer 1

Loadings (which should not be confused with eigenvectors) have the following properties:

1. Their sums of squares within each component are the eigenvalues (components' variances).
2. Loadings are coefficients in linear combination predicting a variable by the (standardized) components.

You extracted 2 first PCs out of 4. Matrix of loadings A

and the eigenvalues:

A (loadings)
         PC1           PC2
X1   .5000000000   .5000000000 
X2   .5000000000   .5000000000 
X3   .5000000000  -.5000000000 
X4   .5000000000  -.5000000000
Eigenvalues:
    1.0000000000  1.0000000000

In this instance, both eigenvalues are equal. It is a rare case in real world, it says that PC1 and PC2 are of equal explanatory "strength".

Suppose you also computed component values, Nx2 matrix C

, and you z-standardized (mean=0, st. dev.=1) them within each column. Then (as point 2 above says), X^=CA′. But, because you left only 2 PCs out of 4 (you lack 2 more columns in A) the restored data values X^

are not exact, - there is an error (if eigenvalues 3, 4 are not zero).

OK. What are the coefficients to predict components by variables? Clearly, if A

were full 4x4, these would be B=(A−1)′. With non-square loading matrix, we may compute them as B=A⋅diag(eigenvalues)−1=(A+)′

, where diag(eigenvalues) is the square diagonal matrix with the eigenvalues on its diagonal, and + superscript denotes pseudoinverse. In your case:

diag(eigenvalues):
1 0
0 1

B (coefficients to predict components by original variables):
    PC1           PC2
X1 .5000000000   .5000000000 
X2 .5000000000   .5000000000 
X3 .5000000000  -.5000000000 
X4 .5000000000  -.5000000000

So, if X

is Nx4 matrix of original centered variables (or standardized variables, if you are doing PCA based on correlations rather than covariances), then C=XB; C

are standardized principal component scores. Which in your example is:

PC1 = 0.5*X1 + 0.5*X2 + 0.5*X3 + 0.5*X4 ~ (X1+X2+X3+X4)/4

"the first component is proportional to the average score"

PC2 = 0.5*X1 + 0.5*X2 - 0.5*X3 - 0.5*X4 = (0.5*X1 + 0.5*X2) - (0.5*X3 + 0.5*X4)

"the second component measures the difference between the first pair of scores and the second pair of scores"

In this example it appeared that B=A

, but in general case they are different.

Note: The above formula for the coefficients to compute component scores, B=A⋅diag(eigenvalues)−1

, is equivalent to B=R−1A, with R being the covariance (or correlation) matrix of variables. The latter formula comes directly from linear regression theory. The two formulas are equivalent within PCA context only. In factor analysis, they are not and to compute factor scores (which are always approximate in FA) one should rely on the second formula.