I am interested in predicting if a doctor would prescribe a specific drug and have chosen Logistic Regression as a starting point.

I have a few questions:

  1. Is feature selection the first step to take to choose relevant variables?
  2. Is Logistic Regression only for binary output? For each doctor, can I get probability of drug prescription (e.g. doctor 1 = 0.87, doctor 2 = 0.56)?
  3. How can my model be deployed into production? Is it a huge task?
  • $\begingroup$ You would add doctor as an input, so you predict probability of prescription given doctor 1,2,.. $\endgroup$
    – seanv507
    Jul 18, 2019 at 7:23

2 Answers 2


I would not start with (manual) feature selection. Use Lasso instead to "automatically" shrink/select features (this is Logit with shrinking of features basically). Logit (or Logit with Lasso as here) is for binary cases, but you can also do "Multinominal Logit" (option multi_class='multinomial' in sklearn), which is for more than two classes. Usually you use sklearn in Python for such things. Also see the examples in the sklearn docs.

Make sure you have a test and trainings set. Also make sure that you do not use data from your test set for training. Train on the train set only and use the test set to see how your model performs on data NOT seen during training.

It is not clear what you mean when you say "move to production". This depends on your problem. You simply need to make predictions here, but the implementation is of course contingent on the environment.

It is okay to play around with data. However, if you really want to go for serious data science, you should have a look at the methods behind all this magic. I recommend "Introduction to Statistical Learning". It is a really good book with many code examples and it is not very technical.

Note that there is no silver bullet. Lasso or Logit may be okay, but other methods may be better. This really depends on the problem/data.

Here is a little sample code for Lasso:

# Split test/train
from sklearn.model_selection import train_test_split
xtrain, xtest, ytrain, ytest = train_test_split(X, y, test_size=0.20, random_state=7)

from sklearn.metrics import roc_auc_score
from sklearn.metrics import confusion_matrix
from sklearn.linear_model import Lasso
from sklearn.linear_model import LassoCV

# Perform lasso CV to get the best parameter alpha for regulation
lasso = Lasso(max_iter=10000)
lassocv = LassoCV(alphas=None, cv=10, max_iter=10000)
lassocv.fit(xtrain, ytrain.values.ravel())

# Fit lasso using the best alpha
lasso.fit(xtrain, ytrain)

# Look at results (coefficients)
la1 = pd.Series(abs(lasso.coef_), name="lasso")
la2 = pd.Series(X.columns, name="names")
dflasso = pd.concat([la2,la1], axis=1)
dflasso = dflasso.sort_values(by=['lasso'], ascending=False)

# Look at AUC
print("AUC Lasso: %.3f" %roc_auc_score(ytest.values, lasso.predict(xtest)))

# Predict probs 
lasspreds0 = lasso.predict(xtest)
# Classes
lasspreds = np.round(lasspreds0)

# Confusion matrix
tnlog, fplog, fnlog, tplog = confusion_matrix(ytest, lasspreds).ravel() #y_true, y_pred
print("True negative: %s, False positive: %s, False negative: %s, True positive %s" %(tnlog, fplog, fnlog, tplog))
print("Share false %.2f" %(((fplog+fnlog)/(fplog+fnlog+tplog+tnlog))))

# Look at probs
print("Min. prob. of belonging to class 0: %.3f" %lasspreds0.min())
print("Max. prob. of belonging to class 0: %.3f" %lasspreds0.max())


Please note that the sklearn Lasso as described above does not do a logistic regression, which means predictions can be smaller zero or larger one. To use Lasso with Logit (ensuring predictions are zero or one), one can use LogisticRegression:

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
log = LogisticRegression(penalty='l1', solver='liblinear')
log.fit(X, y)
  • $\begingroup$ quick question, for states column, which is a varchar, I use labelencoder from sklearn, isn't it the best practice? $\endgroup$
    – Dwight K
    Jul 15, 2019 at 16:02
  • $\begingroup$ Not sure what you mean exactly. But I guess you want to include a factor (from string) in your model. I would go for one hot (or dummy) encoding in this case scikit-learn.org/stable/modules/generated/… or if you use Pandas pandas.pydata.org/pandas-docs/stable/reference/api/… $\endgroup$
    – Peter
    Jul 15, 2019 at 16:05
  • $\begingroup$ Also @Peter , I am not sure if you answered by probability question, I would like my final prediction in 0.somethings not 0 and 1 as Logistic regression would give as an output. I hope I am clear. Thank you. $\endgroup$
    – Dwight K
    Jul 15, 2019 at 16:06
  • $\begingroup$ See the example # Predict probs lasspreds0 = lasso.predict(xtest) $\endgroup$
    – Peter
    Jul 15, 2019 at 16:24

To add on @Peter's answer, you can use the method: classifier.predict_proba(X_test) to get the probability of X_test belonging to each class.

This is called a soft prediction and would most likely need something called probability calibration to get usable probabilities. Hard prediction is what the classifier.predict() method does. It takes the class with the highest probability and assigns its label to your X_test.

PS : If you are sticking with Logistic Regression you won't need probability calibration since LR automatically optimizes logloss probabilities. However just in case you opted for another classifier, you will need to calibrate it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.