# Regression Algorithms in Production

I am interested in predicting if a doctor would prescribe a specific drug and have chosen Logistic Regression as a starting point.

I have a few questions:

1. Is feature selection the first step to take to choose relevant variables?
2. Is Logistic Regression only for binary output? For each doctor, can I get probability of drug prescription (e.g. doctor 1 = 0.87, doctor 2 = 0.56)?
3. How can my model be deployed into production? Is it a huge task?
• You would add doctor as an input, so you predict probability of prescription given doctor 1,2,.. Jul 18, 2019 at 7:23

I would not start with (manual) feature selection. Use Lasso instead to "automatically" shrink/select features (this is Logit with shrinking of features basically). Logit (or Logit with Lasso as here) is for binary cases, but you can also do "Multinominal Logit" (option multi_class='multinomial' in sklearn), which is for more than two classes. Usually you use sklearn in Python for such things. Also see the examples in the sklearn docs.

Make sure you have a test and trainings set. Also make sure that you do not use data from your test set for training. Train on the train set only and use the test set to see how your model performs on data NOT seen during training.

It is not clear what you mean when you say "move to production". This depends on your problem. You simply need to make predictions here, but the implementation is of course contingent on the environment.

It is okay to play around with data. However, if you really want to go for serious data science, you should have a look at the methods behind all this magic. I recommend "Introduction to Statistical Learning". It is a really good book with many code examples and it is not very technical.

Note that there is no silver bullet. Lasso or Logit may be okay, but other methods may be better. This really depends on the problem/data.

Here is a little sample code for Lasso:

# Split test/train
from sklearn.model_selection import train_test_split
xtrain, xtest, ytrain, ytest = train_test_split(X, y, test_size=0.20, random_state=7)

from sklearn.metrics import roc_auc_score
from sklearn.metrics import confusion_matrix
from sklearn.linear_model import Lasso
from sklearn.linear_model import LassoCV

lasso = Lasso(max_iter=10000)
lassocv = LassoCV(alphas=None, cv=10, max_iter=10000)
lassocv.fit(xtrain, ytrain.values.ravel())

# Fit lasso using the best alpha
lasso.set_params(alpha=lassocv.alpha_)
lasso.fit(xtrain, ytrain)

# Look at results (coefficients)
la1 = pd.Series(abs(lasso.coef_), name="lasso")
la2 = pd.Series(X.columns, name="names")
dflasso = pd.concat([la2,la1], axis=1)
dflasso = dflasso.sort_values(by=['lasso'], ascending=False)
print(dflasso)

# Look at AUC
print("AUC Lasso: %.3f" %roc_auc_score(ytest.values, lasso.predict(xtest)))

# Predict probs
lasspreds0 = lasso.predict(xtest)
# Classes
lasspreds = np.round(lasspreds0)

# Confusion matrix
tnlog, fplog, fnlog, tplog = confusion_matrix(ytest, lasspreds).ravel() #y_true, y_pred
print("True negative: %s, False positive: %s, False negative: %s, True positive %s" %(tnlog, fplog, fnlog, tplog))
print("Share false %.2f" %(((fplog+fnlog)/(fplog+fnlog+tplog+tnlog))))

# Look at probs
print("Min. prob. of belonging to class 0: %.3f" %lasspreds0.min())
print("Max. prob. of belonging to class 0: %.3f" %lasspreds0.max())


EDIT:

Please note that the sklearn Lasso as described above does not do a logistic regression, which means predictions can be smaller zero or larger one. To use Lasso with Logit (ensuring predictions are zero or one), one can use LogisticRegression:

from sklearn.linear_model import LogisticRegression

To add on @Peter's answer, you can use the method: classifier.predict_proba(X_test) to get the probability of X_test belonging to each class.
This is called a soft prediction and would most likely need something called probability calibration to get usable probabilities. Hard prediction is what the classifier.predict() method does. It takes the class with the highest probability and assigns its label to your X_test.