0
$\begingroup$

I'm new to this field and trying to learn by working with a fraud dataset. Initially, I used the dataset as is, but now I'm trying unsupervised learning without the labels. I've tried clustering algorithms like K-Means, DBSCAN, and Isolation Forests to create my own labels, but it's not working well.

I think one problem might be the large number of features in the dataset. Do you have any tips on how to simplify the features for unlabeled data?

Here is my code until now:

train_df = pd.read_csv("freezed")

rep_mapping = {
    'SM': 'Samsung',
    'SAMSUNG': 'Samsung',
    'GT-': 'Samsung',
    'MOTO': 'Motorola',
    'LG': 'LG',
    'RV:': 'RV',
    'HUAWEI': 'Huawei',
    'ALE-': 'Huawei',
    '-L': 'Huawei',
    'BLADE': 'ZTE',
    'LINUX': 'Linux',
    'XT': 'Sony',
    'HTC': 'HTC',
    'ASUS': 'Asus',
    'LENOVO': 'Lenovo',
    'WINDOWS': 'Windows',
    'PIXEL': 'Pixel',
    'HISENSE': 'Hisense'
}

train_df['DeviceInfo'] = train_df['DeviceInfo'].str.split(' ', expand=True)[0]
for pattern, replacement in rep_mapping.items():
    train_df.loc[train_df['DeviceInfo'].str.upper().str.contains(pattern, na=False), 'DeviceInfo'] = replacement

device_counts = train_df['DeviceInfo'].value_counts()
other_devices = device_counts[device_counts < 200].index.tolist()

train_df['DeviceInfo'].replace(other_devices, 'Others', inplace=True)

train_df['Transaction_day_of_week'] = np.floor((train_df['TransactionDT'] / (3600 * 24) - 1) % 7)
train_df['Transaction_hour'] = np.floor(train_df['TransactionDT'] / 3600) % 24
train_df['os'] = train_df['id_30'].str.split(' ', expand=True)[0]
train_df['version_os'] = train_df['id_30'].str.split(' ', expand=True)[1]
train_df['browser'] = train_df['id_31'].str.split(' ', expand=True)[0]
train_df['version_browser'] = train_df['id_31'].str.split(' ', expand=True)[1]
train_df['screen_width'] = train_df['id_33'].str.split('x', expand=True)[0]
train_df['screen_height'] = train_df['id_33'].str.split('x', expand=True)[1]

fill_values = {col: "Unknown" if train_df[col].dtype == "object" else train_df[col].mean() for col in train_df.columns}
train_df = train_df.fillna(value=fill_values)

train_df['P_em_origin'] = train_df['P_emaildomain'].map(lambda x: str(x).split('.')[0])
train_df['R_em_origin'] = train_df['R_emaildomain'].map(lambda x: str(x).split('.')[0])
train_df['P_em_suffix'] = train_df['P_emaildomain'].map(lambda x: str(x).split('.')[-1])
train_df['R_em_suffix'] = train_df['R_emaildomain'].map(lambda x: str(x).split('.')[-1])
train_df['P_em_freq'] = train_df.groupby(["P_em_origin"])["P_em_origin"].transform('count')
train_df['R_em_freq'] = train_df.groupby(["R_em_origin"])["R_em_origin"].transform('count')

train_df = train_df.drop(["TransactionID", "TransactionDT", "P_emaildomain",
                          "R_emaildomain", "id_30", "id_31", "id_33"],
                         axis=1)
labels = train_df.pop("isFraud")

le = LabelEncoder()
for col in train_df.columns:
    if train_df[col].dtype == "object":
        train_df[col] = train_df[col].fillna("Unknown")
        le.fit(list(train_df[col].astype(str).values))
        train_df[col] = le.transform(list(train_df[col].astype(str).values))
    else:
        train_df[col] = train_df[col].fillna(train_df[col].mean())

scaler = MinMaxScaler()
train_df = pd.DataFrame(scaler.fit_transform(train_df), columns=train_df.columns)
$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.