1
$\begingroup$

I'm trying to understand a code snippet from my lecture on Machine Learning (see the code below).

It extracts the mean and standard deviation of the features and uses them to 'normalize' (standardize) the features. The data X contains all-ones in the first column. The variable t is the output value on which we train the weights. First the mean of the output is obtained so that ridge regression can be applied without a bias term (intercept), then the all-ones column is removed and the data is normalized. The weights are obtained by the formula:

enter image description here

What I don't understand is that the weights obtained by using the normalized data are once again divided by the standard deviation of the features W[1:, 0] = Wn[:, 0] * Sinv. Can someone help me out? Thanks!

PS: I know normalizing is something else and I actually mean standardizing, but my professor uses normalizing instead for some reason..

Here's the code:

def NormalisedRidgeRegression(X, t, p_lambda=0.0):
    K = X.shape[1] - 1

    M = np.mean(X[:, 1:], axis=0)
    S = np.std(X[:, 1:], axis=0)
    Sinv = 1 / S

    targetmean = np.mean(t)
    tn = t - targetmean

    Xn = np.dot(X[:, 1:] - M, np.diag(Sinv))

    Wn = np.dot(np.dot(np.linalg.pinv(np.dot(Xn.T, Xn) + p_lambda * np.eye(K)), Xn.T), tn)
    W = np.zeros((K + 1, 1))
    W[1:, 0] = Wn[:, 0] * Sinv
    W[0, 0] = targetmean
    y_hat = np.dot(X, W)
    return W, y_hat
$\endgroup$
1
$\begingroup$

I found my answer myself.. I'm not sure if I should delete the question, so I'll just share:

Say for a given data point before normalising we have a*x + b*z = y Normalising makes this a * x/std(x) + b * z/std(z) = y This means that for standard deviations greater than 1 the features are shrunk and therefore the weights are blown up. In the code the y_hat values are obtained using the original data, which means the coefficients have to be shrunk to reflect the relationship between the output and the original feature values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.