0
$\begingroup$

According to Q8,9 of HW5, Caltech's Learning from data course, we have to generate 100 test points of the form (x1,x2) and get their outputs 1/0 depending on which side of a random line they lie on. image classification We need to use this training data for stochastic gradient descent, using the cross-entropy error and calculate the error E_out on testing data produced in a similar fashion.

I used the following python code for the same:

def sgd(inputs, outputs, weights = np.array([0,0,0]), eeta = 0.1):
    weights.shape = (3,1)
    jj = 0
    datasets = len(outputs)

    while(True):
        temp = weights
        iterator = np.random.permutation(datasets)
        for i in iterator:
            x = inputs[i]
            y = outputs[i]
            x.shape = (3,1)
            w_x = np.dot(weights.transpose(), x)
            delta = -(y*x)/(1 + exp(y*w_x))
            weights = weights - eeta*delta

        diff_abs = np.linalg.norm(temp-weights)
        sume=0
        for i in range(datasets):
            y = outputs[i]
            x = inputs[i]
            x.shape = (3,1)
            w_x = np.dot(weights.transpose(), x)
            sume += log(1 + e**(-y*w_x))
        ein = sume/datasets

        if diff_abs<0.005:
            print(jj,'\t',ein)
            # print(ein)
            return ein
            break
        jj+=1
    return weights

def calc_eout(weights, datasets):
    weights.shape = (3,1)
    inputs = generate_inputs(datasets)
    outputs = get_outputs(inputs)
    sume = 0
    for i in range(datasets):
        y = outputs[i]
        x = inputs[i]
        x.shape = (3,1)
        w_x = np.dot(weights.transpose(), x)
        sume += log(1 + e**(-y*w_x))
    return sume/datasets

eout = []
ein = []
for i in range(100):
    print(i,'\n')
    eeta = 0.01
    datasets = 100

    weights = np.array([0,0,0])

    inputs = generate_inputs(datasets)
    outputs = get_outputs(inputs)
    weights = sgd(inputs, outputs, weights, eeta)
    e_out = calc_eout(weights, datasets)
    eout.append(e_out)

print("eout = ", sum(eout)/len(eout))

Using this code, the e_in becomes almost constant to around 35-38% which really shouldn't be the case as the data is properly linearly separable. Secondly, the answer to this question is that the E_out is around 0.1, and reaches there in around 350 epochs. While my error becomes almost constant to 0.38 in 100 epochs. Can someone tell me what mistake I am doing? It's probably something very trivial.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.