I'm quite new to Python coding so maybe this is an obvious thing.
I am dealing with an imbalanced data set. I have encoded the categorical features and transformed them into numpy.array. After this, I used the SKlearn train test split and then utilized the SMOTE. Now I want to know how I can split the data again so I can use my resampled data for my models. So basically, any code recommendations if you have any would be super appreciated!
I have read that you should not use the train test split sklearn function again after resampling and it is the only method I know for splitting data so I was hoping to get some clues on what to do.
See my code below:
import pandas as pd import numpy as np from sklearn.preprocessing import OneHotEncoder, LabelEncoder, label_binarize, StandardScaler, MinMaxScaler df = pd.read_csv("file.csv", header=None) df.shape df.columns = [ "Status of checking account", "Duration in months", "Credit history", "Purpose", "Credit amount", "Savings account/bond", "Present employment since", "Installment rate in percentage of disposable income", "Personal status and sex", "Other debtors", "Present residence since", "Property", "Age in years", "Other installment plans", "Housing", "Number of existing credits", "Job", "Number of people providing maintenance for", "Telephone", "Foreign worker", "Credit risk", ] df.info() for col in df.select_dtypes(include="int64"): if len(df[col].unique()) <= 5: df[col] = df[col].astype("category") for col in df.select_dtypes(include="object"): df[col] = df[col].astype("category") df.dtypes from sklearn.preprocessing import OneHotEncoder enc = OneHotEncoder(drop="first") x_transform = enc.fit_transform(df.select_dtypes(include="category")).toarray() print(x_transform) print("----------------\n") print(enc.categories_) print("----------------\n") print(enc.get_feature_names_out()) df_enc = pd.DataFrame(x_transform, columns=enc.get_feature_names_out()) df_enc.head() df_sklearn = pd.concat( [df.select_dtypes(include="int64"), df_enc], axis=1, ignore_index=False ) df_sklearn.head() x_2 = df_sklearn.iloc[:,:-1].to_numpy() y_2 = df_sklearn.iloc[:,-1].to_numpy() df_sklearn.iloc[:,-1].value_counts()/df_sklearn.iloc[:,-1].count() from sklearn.model_selection import train_test_split x_train, x_test, y_train, y_test = train_test_split(x_2, y_2, test_size=0.2, random_state=1) from imblearn.over_sampling import SMOTE sm = SMOTE(sampling_strategy='auto') x_train_bal, y_train_bal = sm.fit_resample(x_train, y_train) print("Before/After clean") unique, counts = np.unique(y_train, return_counts=True) print(dict(zip(unique, counts))) unique, counts = np.unique(y_train_bal, return_counts=True) print(dict(zip(unique, counts)))
The dataset that I am working with: https://archive.ics.uci.edu/ml/datasets/statlog+(german+credit+data)
ALSO, I have tried applying a LIME model but also always run into the problem that my data is numpy array. When looking at tutorials where they use similar datasets to mine they never get that so I assume it has something to do with how I encode my variables.