I'm quite new to Python coding so maybe this is an obvious thing.

I am dealing with an imbalanced data set. I have encoded the categorical features and transformed them into numpy.array. After this, I used the SKlearn train test split and then utilized the SMOTE. Now I want to know how I can split the data again so I can use my resampled data for my models. So basically, any code recommendations if you have any would be super appreciated!

I have read that you should not use the train test split sklearn function again after resampling and it is the only method I know for splitting data so I was hoping to get some clues on what to do.

See my code below:

import pandas as pd
import numpy as np
from sklearn.preprocessing import OneHotEncoder, LabelEncoder, label_binarize, StandardScaler, MinMaxScaler
df = pd.read_csv("file.csv", header=None)


df.columns = [
    "Status of checking account",
    "Duration in months",
    "Credit history",
    "Credit amount",
    "Savings account/bond",
    "Present employment since",
    "Installment rate in percentage of disposable income",
    "Personal status and sex",
    "Other debtors",
    "Present residence since",
    "Age in years",
    "Other installment plans",
    "Number of existing credits",
    "Number of people providing maintenance for",
    "Foreign worker",
    "Credit risk",


for col in df.select_dtypes(include="int64"):    
    if len(df[col].unique()) <= 5:
        df[col] = df[col].astype("category")

for col in df.select_dtypes(include="object"):    
    df[col] = df[col].astype("category")


from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(drop="first")
x_transform = enc.fit_transform(df.select_dtypes(include="category")).toarray()


df_enc = pd.DataFrame(x_transform, columns=enc.get_feature_names_out())

df_sklearn = pd.concat(
    [df.select_dtypes(include="int64"), df_enc], axis=1, ignore_index=False

x_2 = df_sklearn.iloc[:,:-1].to_numpy()
y_2 = df_sklearn.iloc[:,-1].to_numpy()


from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(x_2, y_2, test_size=0.2, random_state=1)

from imblearn.over_sampling import SMOTE

sm = SMOTE(sampling_strategy='auto')
x_train_bal, y_train_bal = sm.fit_resample(x_train, y_train)

print("Before/After clean")
unique, counts = np.unique(y_train, return_counts=True)
print(dict(zip(unique, counts)))
unique, counts = np.unique(y_train_bal, return_counts=True)
print(dict(zip(unique, counts)))

The dataset that I am working with: https://archive.ics.uci.edu/ml/datasets/statlog+(german+credit+data)

ALSO, I have tried applying a LIME model but also always run into the problem that my data is numpy array. When looking at tutorials where they use similar datasets to mine they never get that so I assume it has something to do with how I encode my variables.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.