# Huge cost not converging well with TensorFlow logistic regression

I try to use Logistic Regression for a dataset which contains 15 numeric features and 4238 rows of examples. The calculated cost started at 415.91, and converged when the cost was reduced to 220.119 only. I think there must be something wrong, but as I am not sure what to do, I would like to share the code with you, it would be helpful for me to know what is not alright in the code and might cause the issue. I would appreciate your advises and experiences a lot!

import tensorflow as tf
import pandas as pd
from sklearn.model_selection import train_test_split

dataset = pd.DataFrame.from_csv('framingham_heart_disease.csv', index_col = None)
print(dataset.shape)
dataX, dataY = dataset.iloc[:,:-1], dataset.iloc[:,-1:]
dataX = dataX.values/50
dataY = dataY.values

trainX, testX, trainY, testY = train_test_split(dataX, dataY, test_size=0.20, random_state=42)

numTrainData = trainX.shape[0]
numFeatures = trainX.shape[1]
numLabels = trainY.shape[1]

X = tf.placeholder(tf.float32, [numTrainData,numFeatures])
yExpected = tf.placeholder(tf.float32, [numTrainData, numLabels])

tf.set_random_seed(1)
weights = tf.Variable(tf.random_normal([numFeatures,numLabels],
mean=0,
stddev=0.01,
name="weights"))
bias = tf.Variable(tf.random_normal([1,numLabels],
mean=0,
stddev=0.01,
name="bias"))

apply_weights_OP = tf.matmul(X, weights, name="apply_weights")
weights_after_nan = tf.where(tf.is_nan(apply_weights_OP), tf.ones_like(apply_weights_OP) * 0, apply_weights_OP);

learningRate = tf.train.exponential_decay(learning_rate=0.0001,
global_step= 1,
decay_steps=trainX.shape[0],
decay_rate= 0.95,
staircase=True)
cost_OP = tf.nn.l2_loss(activation_OP-yExpected, name="squared_error_cost")

sess = tf.Session()
init_OP = tf.global_variables_initializer()
sess.run(init_OP)

numEpochs = 3000
cost = 0.0
diff = 1
epoch_values = []
accuracy_values = []
cost_values = []

for i in range(numEpochs):
if i > 1 and diff < .0001:
print("change in cost %g; convergence."%diff)
break
else:
step = sess.run(training_OP, feed_dict={X: trainX, yExpected: trainY})
# Report occasional stats
if i % 100 == 0:
epoch_values.append(i)
# Generate accuracy stats on test data
newCost = sess.run(cost_OP, feed_dict={X: trainX, yExpected: trainY})
# Add cost to live graphing variable
cost_values.append(newCost)
# Re-assign values for variables
diff = abs(newCost - cost)
cost = newCost

#generate print statements
print("step %d, cost %g, change in cost %g"%(i, newCost, diff))


I'd expect a better convergence with lower cost, but instead I get this: step 0, cost 415.91, change in cost 415.91 step 100, cost 229.459, change in cost 186.45 step 200, cost 221.717, change in cost 7.74254 step 300, cost 220.504, change in cost 1.2124 step 400, cost 220.225, change in cost 0.279007 step 500, cost 220.15, change in cost 0.0752258 step 600, cost 220.127, change in cost 0.022522 step 700, cost 220.121, change in cost 0.00689697 step 800, cost 220.119, change in cost 0.00166321 step 900, cost 220.119, change in cost 6.10352e-05 change in cost 6.10352e-05; convergence.

I would appreciate your advises a lot, have a good day people :)

• what loss do you expect and why? do you have any baseline? – oW_ Jul 6 '19 at 2:08
• sadly i dont have any baseline, but i expect the loss being smaller for sure, as the 15 features are normalized (in a simple way by dividing by 50 for the first approach, because the values were in different ranges). The result of activation_OP-yExpected is also in the expected range (numbers like 0.01212), so I don't quite get the operation l2_loss. TensorFlow docu says that it is in the file python/ops/gen_nn_ops.py, but I can't find it actually.. – Enyang Wang Jul 6 '19 at 6:53