In: Statistics and Probability
Matlab Code
Write a procedure to calculate the log discriminant function for a given multi-variate Gaussian distribution and prior probability
solution:
Normal distribution and discriminant functions
Linear discriminant functions have a variety of pleasant analytical properties. They can be optimal if the underlying distributions are cooperative, such as Gaussians having equal covariance, as might be obtained through an intelligent choice of feature detectors. Even when they are not optimal, we might be willing to sacrifice some performance in order to gain the advantage of their simplicity. Linear discriminant functions are relatively easy to compute and in the absence of information suggesting otherwise, linear classifiers are attractive candidates for initial, trial classifiers.
Discriminant analysis is a classification method. It assumes that different classes generate data based on different Gaussian distributions.
To train (create) a classifier, the fitting function estimates the parameters of a Gaussian distribution for each class
To predict the classes of new data, the trained classifier finds the class with the smallest misclassification cost
in a problem with feature vector y and state of nature variable w, we can represent the discriminant function as:
gi(x)=-1/2 (x−μ_{i})t Σ_{i-1 }(x−μ_{i}) −d/2 ln 2π−1/2 ln |Σi| + lnP(wi)
case 1: Σ_{i} = σ^{2}I
This is the simplest case and it occurs when the features are statistically independent and each feature has the same variance, σ^{2}. Here, the covariance matrix is diagonal since its simply σ^{2} times the identity matrix I. This means that each sample falls into equal sized clusters that are centered about their respective mean vectors. The computation of the determinant and the inverse |Σ_{i}| = σ^{2d} and Σ_{i}^{-1} = (1/σ^{2})I. Because both |Σ_{i}| and the (d/2) ln 2π term in the equation above are independent of i, we can ignore them and thus we obtain this simplified discriminant function:
that is
g_{i} (x)=−||x−μ_{i}||^{2} /2σ^{2} + ln P(wi)
||x−μ_{i}|| 2 =(x−μ_{i})t (x−μ_{i})
If the prior probabilities are not equal, then the discriminant function shows that the squared distance ||x - μ||^{2} must be normalized by the variance σ^{2} and offset by adding ln P(w_{i}); therefore if x is equally near two different mean vectors, the optimal decision will favor the priori more likely. Expansion of the quadratic form (x - μ_{i})^{t}(x - μ_{i}) yields :
g_{i}(x)=−1/2 σ ^{2} [x^{t}x − 2μ_{i}^{t}x+ μ_{i}^{t}μ_{i}] + ln P(wi)
which looks like a quadratic function of x. However, the quadratic term x^{t}x is the same for all i, meaning it can be ignored since it just an additive constant, thereby we obtain the equivalent discriminant function:
where
g_{i}(x)=w_{i}^{t}+w_{i}0
wi=1σ^{2}μ_{i}
_{and }
w_{i}0= −1/2σ^{2}μ_{i}^{t}iμ_{i} ln P(wi)
w_{i0} is the threshold or bias for the ith category.
A classifier that uses linear discriminants is called a linear machine. For a linear machine, the decision surfaces for a linear machine are just pieces of hyperplanes defined by the linear equations g_{i}(x) = g_{j}(x) for the two categories with the highest posterior probabilities. In this situation, the equation can be written as
where
w^{t} ( x−x0 ) = 0
w=μ_{i}−μ_{j}
x0=1/2(μ_{i}+μ_{j})−σ^{2}/ ||μ_{i}−μ_{j}|| 2 lnP (wi)/P(wj) (μi−μj)
The Multivariate Gaussian Distribution
A vector-valued random variable X=[X1···Xn]T is said to have a multivariate normal (or Gaussian) distribution with mean μ∈R^{n} and covariance matrix Σ∈S^{n}++ 1
if its probability density function 2 is given by
p(x;μ,Σ) =1/(2^{n/2})|Σ|^{1/2} exp (-1/2(x-)^{T}Σ^{-1}x-)
We write this asX∼N(μ,Σ). In these notes, we describe multivariate Gaussians and some of their basic properties.
The model for discriminant analysis is
Each class (Y) generates data (X) using a multivariate normal distribution. In other words, the model assumes X has a Gaussian mixture distribution (gmdistribution).
For linear discriminant analysis, the model has the same covariance matrix for each class; only the means vary.
For quadratic discriminant analysis, both means and covariances of each class vary.
For linear discriminant analysis, it computes the sample mean of each class. Then it computes the sample covariance by first subtracting the sample mean of each class from the observations of that class, and taking the empirical covariance matrix of the result.
For quadratic discriminant analysis, it computes the sample mean of each class. Then it computes the sample covariances by first subtracting the sample mean of each class from the observations of that class, and taking the empirical covariance matrix of each class.
The fit method does not use prior probabilities or costs for fitting.
please give me thumb up