0
$\begingroup$

I trained a logistic regression model with tensorflow but the accuracy of the model was so poor (accuracy = 0.68). The model was trained using simulated dataset and the result should be very good. is there something wrong with the code ?

 #simulated dataSet
sim_data <- function(n=2000){
  library(dummies)
  age <- round(abs(rnorm(n,mean = 60, sd = 20)))
  lac <- round(abs(rnorm(n,3,1)),1)
  wbc <- round(abs(rnorm(n,10,3)),1)
  sex <- factor(rbinom(n,size = 1,prob = 0.6),
                   labels = c("Female","Male"));
  type <- as.factor(sample(c("Med","Emerg","Surg"),
                           size = n,replace = T,
                           prob = c(0.4,0.4,0.2)))
  linPred <- cbind(1,age,lac,wbc,dummy(sex)[,-1],
                   dummy(type)[,-1]) %*%
    c(-30,0.2,4,1,-2,3,-3)
  pi <- 1/(1+exp(-linPred))
  mort <- factor(rbinom(n,size = 1, prob = pi),
                 labels = c("Alive","Died"))
  dat <- data.frame(age=age,lac=lac,wbc=wbc,
                    sex=sex,type=type,
                    mort = mort)
  return(dat)
}
set.seed(123)
dat <- sim_data()
dat_test <- sim_data(n=1000)
#logistic regression in conventional method
mod <- glm(mort~.,data = dat,family = "binomial")
library(tableone)
ShowRegTable(mod)
#diagnotic accuracy
pred <- predict.glm(mod,newdata = dat_test,
                    type = "response")
library(pROC)
roc(response = dat_test$mort,predictor = pred,ci=T)
predBi <- pred >= 0.5
crossTab <- table(predBi,dat_test$mort)
(crossTab[1]+crossTab[4])/sum(crossTab)
#choose different cutoff for the accuracy
DTaccuracy <- data.frame()
for (cutoff in seq(0,1,by = 0.01)) {
  predBi <- pred >= cutoff;
  crossTab <- table(predBi,dat_test$mort)
  accuracy = (crossTab[1]+crossTab[4])/sum(crossTab)
  DTaccuracy <- rbind(DTaccuracy,c(accuracy,cutoff))
}
names(DTaccuracy) <- c('Accuracy','Cutoff')
qplot(x=Cutoff, y = Accuracy, data = DTaccuracy)


#tensorflow method
library(caret)
y = with(dat, model.matrix(~ mort + 0))
x = model.matrix(~.,dat[,!names(dat)%in%"mort"])
trainIndex = createDataPartition(1:nrow(x), 
                                 p=0.7, list=FALSE,times=1)

x_train = x[trainIndex,]
x_test = x[-trainIndex,]
y_train = y[trainIndex,]
y_test = y[-trainIndex,]

# Hyper-parameters
epochs = 30             # Total number of training epochs
batch_size = 30        # Training batch size
display_freq = 10      # Frequency of displaying the training results
learning_rate = 0.1   # The optimization initial learning rate
#Then we will define the placeholders for features and labels:
library(tensorflow)
X <- tf$placeholder(tf$float32, shape(NULL, ncol(x)),
                    name = "X")
Y = tf$placeholder(tf$float32, shape(NULL, 2L), name = "Y")

#we will define the parameters. We will randomly initialize the weights with mean “0” and a standard deviation of “1.” We will initialize bias to “0.”

W = tf$Variable(tf$random_normal(shape(ncol(x),2L),
                                 stddev = 1.0), 
                name = "weghts")
b = tf$Variable(tf$zeros(shape(2L)), name = "bias")

#Then we will compute the logit.

logits = tf$add(tf$matmul(X, W), b)
pred = tf$nn$sigmoid(logits)
#The next step is to define the loss function. We will use sigmoid cross entropy with logits as a loss function.

entropy = tf$nn$sigmoid_cross_entropy_with_logits(labels = Y, 
                                                  logits = logits)
loss = tf$reduce_mean(entropy,name = "loss")
#The last step of the model composition is to define the training op. We will use a gradient descent with a learning rate 0.1 to minimize cost.

optimizer = tf$train$GradientDescentOptimizer(learning_rate = learning_rate)$minimize(loss)
init_op = tf$global_variables_initializer()
#Now that we have trained the model, let’s evaluate it:

correct_prediction <- tf$equal(tf$argmax(logits, 1L), tf$argmax(Y, 1L),
                               name = "correct_pred")
accuracy <- tf$reduce_mean(tf$cast(correct_prediction, tf$float32),
                           name = "accuracy")
#Having structured the graph, let’s execute it:

with(tf$Session() %as% sess, {
  sess$run(init_op)
  for (i in 1:5000) {
    sess$run(optimizer,
             feed_dict = dict(X=x_train, Y=y_train))
  }
  sess$run(accuracy, 
           feed_dict=dict(X = x_test, Y = y_test))
})

The accuracy obtained by glm() method is quite good (accuracy=0.95), which is as expected; however, the accuracy was only 0.68 with TensorFlow method. How can I solve the problem?

$\endgroup$
0
$\begingroup$

Your learning rate is too high. In TensorFlow models I usually set a learning rate between 0.001 and 0.00005 to achieve acceptable results.

$\endgroup$
  • $\begingroup$ Thank you for your comments; I have just tried the learning rate of 0.00005 but the accuracy did not improve at all. $\endgroup$ – Z. Zhang Sep 10 at 8:21
  • $\begingroup$ I think it's too slow for a simple model such as logistic regression. Please remember that lower learning rate means improved model's performance, but also much longer training times (and a lot of additional training epochs). If you set it too slow, it might take too much time to converge. Try with 0.001, and then run the model multiple times changing its value. $\endgroup$ – Leevo Sep 10 at 8:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.