# How to plot cost versus number of iterations in scikit learn?

One of the recommendations in the Coursera Machine Learning course when working with gradient descent based algorithms is:

Debugging gradient descent. Make a plot with number of iterations on the x-axis. Now plot the cost function, J(θ) over the number of iterations of gradient descent. If J(θ) ever increases, then you probably need to decrease α.

Do gradient descent based models in scikit-learn provide a mechanism for retrieving the cost vs the number of iterations?

Based on the answer here, use the following code:

old_stdout = sys.stdout
sys.stdout = mystdout = StringIO()
clf = SGDClassifier(**kwargs, verbose=1)
clf.fit(X_tr, y_tr)
sys.stdout = old_stdout
loss_history = mystdout.getvalue()
loss_list = []
for line in loss_history.split('\n'):
if(len(line.split("loss: ")) == 1):
continue
loss_list.append(float(line.split("loss: ")[-1]))
plt.figure()
plt.plot(np.arange(len(loss_list)), loss_list)
plt.savefig("warmstart_plots/pure_SGD:"+str(kwargs)+".png")
plt.xlabel("Time in epochs")
plt.ylabel("Loss")
plt.close()


Also take a look at here

• Clever workaround! Mar 1 '18 at 22:27

### Note The Code Is Self Explanatory(It's Hard-coded)..

Here's the function which we are considering

def f(a,b):
return a**2 + b**2

fig = plt.figure(figsize=(10, 6))
ax = fig.gca(projection='3d')
plt.hold(True)
a = np.arange(-2, 2, 0.25)
b = np.arange(-2, 2, 0.25)
a, b = np.meshgrid(a, b)
c = f(a,b)
surf = ax.plot_surface(a, b, c, rstride=1, cstride=1, alpha=0.3,
linewidth=0, antialiased=False,cmap='rainbow')
ax.set_zlim(-0.01, 8.01) Here's a 3D view of Gradient Descent Reaching The Optimum(in case interested, it will not always work, Have a look at the end plot..)

def gradient_descent(theta0, iters, alpha):
history = [theta0] # to store all thetas
theta = theta0     # initial values for thetas
# main loop by iterations:
for i in range(iters):
gradient = [2.0*x for x in theta]
# update parameters:
theta = [a - alpha*b for a,b in zip(theta, gradient)]
history.append(theta)
return history

history = gradient_descent(theta0 = [-1.8, 1.6], iters = 30, alpha = 0.03)

fig = plt.figure(figsize=(10, 6))
ax = fig.gca(projection='3d')
plt.hold(True)
a = np.arange(-2, 2, 0.25)
b = np.arange(-2, 2, 0.25)
a, b = np.meshgrid(a, b)
c = f(a,b)
surf = ax.plot_surface(a, b, c, rstride=1, cstride=1, alpha=0.3,
linewidth=0, antialiased=False)
ax.set_zlim(-0.01, 8.01)

a = np.array([x for x in history])
b = np.array([x for x in history])
c = f(a,b)
ax.scatter(a, b, c, color="r");

plt.show()


And Here's the output we will get When Gradient Descent will fail (unfortunately)..