I have a difficulty where to start the implementation of incremental stochastic gradient descent algorithm and its respective implementation in logistic regression. I don't quite understand this algorithm and there are few sources to explain it with crystal-clear interpretation and possible demo code. I am quite new with ML algorithm and I have no idea which would be efficient workaround to solve this problem.
In particular, the problem I am studying is to implement hogWild! algorithm for logistic regression, which asks me to program incremental SGD algorithm with a sequential order. Can anyone give me a general idea or possible pipeline to make this happen in python?
logistic loss function and gradient
Here is my implementation:
import numpy as np import scipy as sp import sklearn as sl from scipy import special as ss from sklearn import datasets X_train, y_train=datasets.load_svmlight_file('/path/to/train_dataset') X_test,y_test=datasets.load_svmlight_file('/path/to/train_dataset.txt', n_features=X_train.shape) class ISGD: def lossFunc(X,y,w): w.resize((w.shape,1)) y.resize((y.shape,1)) lossFnc=ss.log1p(1+np.nan_to_num(ss.expm1(-y* np.dot(X,w,)))) rslt=np.float(lossFnc) return rslt def gradFnc(X,y,w): w.resize((w.shape,1)) y.resize((y.shape,1)) gradF1=-y*np.nan_to_num(ss.expm1(-y)) gradF2=gradF1/(1+np.nan_to_num(ss.expm1(-y*np.dot(X,w)))) gradF3=gradF2.resize(gradF2.shape,) return gradF3 def _init_(self, learnRate=0.0001, num_iter=100, verbose=False): self.w=None self.learnRate=learnRate self.verbose=verbose self.num_iter=num_iter def fitt(self, X,y): n,d=X.shape self.w=np.zeros(shape=(d,)) for i in range(self.num_iter): print ("\n:", "Iteration:", i) grd=gradFnc(self.w, X,y) grd.resize((grd.shape,1)) self.w=self.w-grd print "Loss:", lossFunc(self.w,X,y) return self
Seems my above implementation has some problems. Can anyone help me how to correct that? Plus, I don't have a solid idea how to implement incremental SGD sequentially. How can I make this happen? Any idea?