1
$\begingroup$

I am trying to program incremental stochastic gradient descent (ISGD) algorithm in logistic regression. Initially, I coded respective logistic regression' loss function and its gradient, also got some idea to proceed rest of workflow. But, I have no idea how to apply sequential operation in incremental stochastic gradient descent algorithm which can be used in the respective logistic regression. How can I implement the sequential operation in incremental SGD? Any way to make this happen in Python? How can I do that? Any idea?

Initial implementation

import numpy as np
import scipy as sp
import sklearn as sl
from scipy import special as ss
from  sklearn import datasets

## load input dataset
X_train, y_train=datasets.load_svmlight_file('/path/to/train_dataset')
X_test,y_test=datasets.load_svmlight_file('/path/to/train_dataset.txt', 
                                          n_features=X_train.shape[1])

# logistic loss function
def lossFunc(x_i,y_i,w):
    w.resize((w.shape[0],1))
    y_i.resize((y_i.shape[0],1))

    lossFnc=ss.log1p(1+np.nan_to_num(ss.expm1(-y_i* np.dot(x_i,w,))))
    rslt=np.float(lossFnc)
    return rslt

#gradient function

def gradFnc(X,y,w):
    w.resize((w.shape[0],1))
    y.resize((y.shape[0],1))

    gradF1=-y*np.nan_to_num(ss.expm1(-y))
    gradF2=gradF1/(1+np.nan_to_num(ss.expm1(-y*np.dot(X,w))))
    gradF3=gradF2.resize(gradF2.shape[0],)
    return gradF3

class ISGD:
    def _init_(self, learnRate=0.0001, num_iter=100, verbose=False):
        self.w=None
        self.learnRate=learnRate
        self.verbose=verbose
        self.num_iter=num_iter


    def fitt(self, X,y):
        n,d=X.shape
        self.w=np.zeros(shape=(d,))

        for i in range(self.num_iter):
            print ("\n:", "Iteration:", i)

            grd=gradFnc(self.w, X,y)
            grd.resize((grd.shape[0],1))
            self.w=self.w-grd
            print "Loss:", lossFunc(self.w,X,y)

        return self

def predict(x_i, w):
        y_hat=w[0]
        for idx in range(len(x_i)-1):
            y_hat+=w[i+1]*x_i[idx]
        return 1.0/(1.0+np.nan_to_num(ss.expm1(-y_hat)))

def update_weights(x_i, y_i,w):
    lr=0.8
    yhat=predict(x_i, w)
    error=y_i-yhat
    return w+lr*(y_i-yhat)*x_i

How to proceed rest of workflow?

Here is blog about HogWild! for parallel machine learning. The particular interpretation of incremental SGD can be found here: hogwild! algorithm for logistic regression.

Now I have no idea how to apply sequential operation in incremental SGD which can be used in the respective logistic regression. How can I make this happen? Is there any efficient workaround to implement sequential incremental SGD algorithm for logistic regression? What is the efficient programming pipeline to accomplish the task that I stated above? Any more thoughts?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.