For a school project, I have to design a QDA classifier for 28x28 pixel images of letters in sign language. I have been given 27455 images for training, which have to be flattened to a 784 pixel vector in order to continue with the calculation. The main problems I have been facing are: overflows in the exponent when using non log likelihood discrimination, and problems of 0 or negative covariance matrix determinants, which wreck havoc on the log calculation.
There are 24 letters as 2 of the 26 English letters are not images rather motion videos/gif, so there are 24 classes. The steps I am supposed to use (I am free for advice in this part as well):
- Flattening the image to a row vector of size 784. The dimension of each input is therefore 784.
- Classifying all images by their respective labels into groups - I have used a dictionary for this.
- For every class and its relative images, calculate the Pi (initial probability) and the covariance matrix after calculating the axis-0 average using the following formula:
$$ \Sigma_c = (\frac{(x_{c})^T * x_{c}}{ycount}) - (\mu_c)^T * \mu_c $$ where $ycount$ is the number of samples which initially belonged to that specific class. - For some $xtest$, use the following formula to get the probability of the input image belonging to that class and multiply it by the initial probability Pi: $$ p_c = (2\pi)^{-D/2} * |\Sigma_c|^{-0.5} * e^{-0.5 (x - \mu_c)^T * \Sigma_c^{-1} * (x - \mu_c)} \\ P_c = p_c * \pi_c $$
- The Bayesian probabilities of every class will be $P_c$ divided by the sum of all $P_c$: $$ BP_c = \frac{P_c}{\sum{P_c}} $$
- The classification should be the class $c$ with the highest $BP_c$.
My lecturer also wants the Log Likelihood, while the preprocessing and flattening of the data remains the same, as well as the calculation of the covariance matrix - which is to be done as follows:
- Calculate the determinant of the covariance matrix by: $$|\Sigma_c| = \prod{\delta_{c,d}}$$ where $\delta_{c,d}$ are the eigenvalues of the covariance matrix of class $c$.
- Use the following formula to calculate: $$ P_c = -0.5 |\Sigma_c| -0.5 (x_{c} - \mu)^T * (\Sigma_c)^{-1} * (x_{c} - \mu) + ln(\pi_c) $$
- The Bayesian probability part remains the same.
According to my lecturer, the classification of the images should be the same for both discriminants: non log likelihood and log likelihood, which I completely agree with as the natural logarithm is a monotonous ascending function.
But! Both ways lead me to some mathematical error in my numpy code, which I will attach below. Apparently the determinant goes astronomically high or the exponent overflows in the non logarithmic approach. In the logarithmic approach, the determinant is sometimes zero or negative and the np.log throws a runtime error. No matter which error, the classification stops and throws a constant classification, 3.
My code:
def classifyMatricifiedData(flattened_stack, yl1):
ctr = Counter(yl1)
assert len(ctr.keys()) == C
cont = {}
for i in ctr.keys():
cont[i] = flattened_stack[yl1 == i]
return cont, ctr
def NormalizedCovarianceMatrix(classX, yc):
assert classX.shape[1] == D
avg = np.average(classX, axis=0)
part1 = (classX.T @ classX) / yc
part2 = (np.reshape(avg,(D,1)) @ np.reshape(avg,(1,D)))
return part1 - part2, avg
def genClassDistribution(data2, labelctr2):
assert len(labelctr2) == C
covMats, avgs = {}, {}
NCM = NormalizedCovarianceMatrix
for i in labelctr2:
covMats[i], avgs[i] = NCM(data2[i], labelctr2[i])
return covMats, avgs
def calcDet(matrix1):
eigenvalues = np.linalg.eigvals(matrix1)
#print(eigenvalues.shape)
product1 = 1
for i in eigenvalues:
product1 *= i
return product1
def MultivariateGaussian(xt1, covMat1, mu1):
assert covMat1.shape == (D,D)
assert mu1.shape == (D,)
#determinanta = np.linalg.det(covMat1)
determinanta = calcDet(covMat1)
invMat = np.linalg.inv(covMat1)
part1 = np.power(2*np.pi, -D/2) * np.power(determinanta, -0.5)
part2 = -0.5 * (xt1 - mu1) @ invMat @ (xt1 - mu1).T
ans = part1 * np.exp(part2)
return ans
def LogMultivariateGaussian(xt1, covMat1, mu1):
assert covMat1.shape == (D,D)
assert mu1.shape == (D,)
#determinanta = np.linalg.det(covMat1)
determinanta = calcDet2(covMat1)
invMat = np.linalg.pinv(covMat1)
#part1 = -D//2*1.837877066
part1 = -0.5*np.log(determinanta)
part2 = -0.5 * (xt1 - mu1).T @ invMat @ (xt1 - mu1)
ans = part1 + part2
return ans
def OptimizedBayesianClassifier(xt2, covMats, muVecs, labelctr3):
cont = {}
denom = 0
for i in labelctr3:
cont[i] = MultivariateGaussian(xt2, covMats[i], muVecs[i])
#cont[i] = LogMultivariateGaussian(xt2, covMats[i], muVecs[i])
cont[i] += np.log(labelctr3[i] / L) # Ln added
denom += cont[i]
for i in labelctr3:
cont[i] /= denom
keys, values = list(cont.keys()), list(cont.values())
print(cont)
return keys[np.argmax(values)]
The function classifyMatricifiedData is supposed to take in flattened images and the respective labels, and return a dict with the classified images where the keys are the classes and the values are lists of vectors/matrices.
The function NormalizedCovarianceMatrix is supposed to take in data of some class and how many images are there in the class and return a covariance matrix.
The function calcDet calculates the determinant of the matrix as per my lecturer instructed and MultivariateGaussian calculates the probability as per the average and the covariance matrix. The last function calculates the Bayesian Probability.
- Can I use the absolute value of the determinant if it is troubling the Log Likelihood function?
- Can anybody pls tell me how to proceed? I tried using np.pinv to invert the matrix even when it is not invertible, but to no avail!
Subset of errors raised:
- RuntimeWarning: invalid value encountered in power part1 = np.power(2*np.pi, -D/2) * np.power(determinanta, -0.5)
- RuntimeWarning: overflow encountered in exp ans = part1 * np.exp(part2)
- The probabilities (almost of all of them) are 'Nan'.
