0
$\begingroup$

I am working on a logistic regression case with an imbalance in the target variable. To fix this I am using SMOTE (Synthetic Minority Oversampling Technique), but each time I run my regression model, I get different numbers in my confusion matrix. I have set random_state parameters while invoking SMOTE as well as Logistic Regression still to no avail. Even my features are the same in each iteration. I was able to get the best value for recall as 0.81 and AUC value as 0.916 once but they are not coming anymore. On some occasions, the value of False Positives and False Negatives shoots up very much indicating that the classifier is very bad.

Please guide what I am doing wrong here, below is the code snippet.

# Feature Selection
features = [ 'FEMALE','MALE','SINGLE','UNDER_WEIGHT','OBESE','PROFESSION_ANYS',
            'PROFESSION_PROF_UNKNOWN']

# Set X and Y Variables
X5 = dataframe[features]

# Target variable
Y5 = dataframe['PURCHASE']

# Splitting using SMOTE
from imblearn.over_sampling import SMOTE
os = SMOTE(random_state = 4)

X5_train, X5_test, Y5_train, Y5_test = train_test_split(X5,Y5, test_size=0.20)
os_data_X5,os_data_Y5 = os.fit_sample(X5_train, Y5_train)
columns = X5_train.columns

os_data_X5 = pd.DataFrame(data = os_data_X5, columns = columns )
os_data_Y5 = pd.DataFrame(data = os_data_Y5, columns = ['PURCHASE'])

# Instantiate Logistic Regression model (using the default parameters)
logreg_5 = LogisticRegression(random_state = 4, penalty='l1', class_weight = 'balanced')

# Fit the model with train data
logreg_5.fit(os_data_X5,os_data_Y5)

# Make predictions on test data set
Y5_pred = logreg_5.predict(X5_test)

# Make Confusion Matrix to compare results against actual values
cnf_matrix = metrics.confusion_matrix(Y5_test, Y5_pred)
cnf_matrix
$\endgroup$
1
$\begingroup$

I do not have enough reputation to make a comment.

For each run, you use different data for training and testing. To obtain stable results you should also fix the partition of the data into train/test set by providing random_state for the train_test_split:

X5_train, X5_test, Y5_train, Y5_test = train_test_split(X5,Y5, test_size=0.20, random_state=4)

A completely different question, however, is why with a small/large? change in training data you have drastically different classification performance.

$\endgroup$
  • $\begingroup$ thanks for the suggestion, after adding random_state value during test/train split, my result is not changing with different iterations. But now I am concerned that it is fixed to an average classification performance which is less than the best figures that I obtained in one of the runs, how can I solve this issue now. $\endgroup$ – tanmay Nov 19 at 15:28
  • $\begingroup$ This should remain an answer; it is the answer to OP. @tanmay, your higher scores were probably from lucky test set splits; they were probably not indicative of future performance. $\endgroup$ – Ben Reiniger Nov 19 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.