Question

Normal distribution and discriminant functions

Matlab
- Write a procedure to calculate the log discriminant function for a given
multi-variate Gaussian distribution and prior probability.

Answer 1

solution:

Normal distribution and discriminant functions

Linear discriminant functions have a variety of pleasant analytical properties. They can be optimal if the underlying distributions are cooperative, such as Gaussians having equal covariance, as might be obtained through an intelligent choice of feature detectors. Even when they are not optimal, we might be willing to sacrifice some performance in order to gain the advantage of their simplicity. Linear discriminant functions are relatively easy to compute and in the absence of information suggesting otherwise, linear classifiers are attractive candidates for initial, trial classifiers.

Discriminant analysis is a classification method. It assumes that different classes generate data based on different Gaussian distributions.

To train (create) a classifier, the fitting function estimates the parameters of a Gaussian distribution for each class

To predict the classes of new data, the trained classifier finds the class with the smallest misclassification cost

in a problem with feature vector y and state of nature variable w, we can represent the discriminant function as:

gi(x)=-1/2 (x−μ_i)t Σ_i^-1  (x−μ_i) −d/2 ln 2π−1/2 ln |Σi| + lnP(wi)

case 1: Σ_i = σ²I

This is the simplest case and it occurs when the features are statistically independent and each feature has the same variance, σ². Here, the covariance matrix is diagonal since its simply σ² times the identity matrix I. This means that each sample falls into equal sized clusters that are centered about their respective mean vectors. The computation of the determinant and the inverse |Σ_i| = σ^2d and Σ_i^-1 = (1/σ²)I. Because both |Σ_i| and the (d/2) ln 2π term in the equation above are independent of i, we can ignore them and thus we obtain this simplified discriminant function:

that is

g_i (x)=−||x−μ_i||² /2σ² + ln P(wi)

||x−μ_i|| 2 =(x−μ_i)t (x−μ_i)

If the prior probabilities are not equal, then the discriminant function shows that the squared distance ||x - μ||² must be normalized by the variance σ² and offset by adding ln P(w_i); therefore if x is equally near two different mean vectors, the optimal decision will favor the priori more likely. Expansion of the quadratic form (x - μ_i)^t(x - μ_i) yields :

g_i(x)=−1/2 σ ² [x^tx − 2μ_i^tx+ μ_i^tμ_i] + ln P(wi)

which looks like a quadratic function of x. However, the quadratic term x^tx is the same for all i, meaning it can be ignored since it just an additive constant, thereby we obtain the equivalent discriminant function:

where

g_i(x)=w_i^t+w_i0

wi=1σ²μ_i

_and  

w_i0= −1/2σ²μ_i^tiμ_i ln P(wi)

w_i0 is the threshold or bias for the ith category.

       A classifier that uses linear discriminants is called a linear machine. For a linear machine, the decision surfaces for a linear machine are just pieces of hyperplanes defined by the linear equations g_i(x) = g_j(x) for the two categories with the highest posterior probabilities. In this situation, the equation can be written as

where

w^t ( x−x0 ) = 0

w=μ_i−μ_j

x0=1/2(μ_i+μ_j)−σ²/ ||μ_i−μ_j|| 2 lnP (wi)/P(wj) (μi−μj)

The Multivariate Gaussian Distribution

A vector-valued random variable X=[X1···Xn]T is said to have a multivariate normal (or Gaussian) distribution with mean μ∈Rⁿ and covariance matrix Σ∈Sⁿ++ 1

if its probability density function 2 is given by

p(x;μ,Σ) =1/(2^n/2)|Σ|^1/2 exp (-1/2(x-)^TΣ^-1x-)

We write this asX∼N(μ,Σ). In these notes, we describe multivariate Gaussians and some of their basic properties.

The model for discriminant analysis is

Each class (Y) generates data (X) using a multivariate normal distribution. In other words, the model assumes X has a Gaussian mixture distribution (gmdistribution).

• For linear discriminant analysis, the model has the same covariance matrix for each class; only the means vary.

• For quadratic discriminant analysis, both means and covariances of each class vary.

• For linear discriminant analysis, it computes the sample mean of each class. Then it computes the sample covariance by first subtracting the sample mean of each class from the observations of that class, and taking the empirical covariance matrix of the result.

• For quadratic discriminant analysis, it computes the sample mean of each class. Then it computes the sample covariances by first subtracting the sample mean of each class from the observations of that class, and taking the empirical covariance matrix of each class.

• The fit method does not use prior probabilities or costs for fitting.

please give me thumb up