Skip to main content
Data Science

Return to Question

edited title
Link
Mas
Mas
  • 55
  • 4

As a beginner, I need help with this Beginner basic clustering model and one-hot encoding?

Source Link
Mas
Mas
  • 55
  • 4

As a beginner, I need help with this clustering model and one-hot encoding?

I have a dataframe of natural disaster incidents in Afghanistan from 2016 - 2023.

Column names:

  • REGION (Northern, Eastern etc)
  • PROV_CODE (province)
  • PROV_NAME
  • DIST_CODE (district)
  • DIST_NAME
  • INC_DATE (incident date)
  • INC_TYPE (5 types: Avalanche, Earthquake, Landslide, Heavy Snowfall, Flood)
  • Persons_killed
  • Persons_injured
  • Individuals_affected
  • Families_affected
  • Houses_damaged
  • Houses_destroyed
  • SEASON

I want to perform a basic clustering experiment to see what patterns exist in the data. I am a beginner and am really struggling. I have finished cleaning the data and exploratory data analysis. Here is what I do next:

  1. PROV_CODE and PROV_NAME are redundant. As are DIST_CODE and DIST_NAME. These are the Province/District codes and names. So I will drop PROV_CODE and DIST_CODE. I will also drop INC_DATE, because I don't know how to work with dates (I created SEASON to replace it).
df = pd.read_csv('cleaned_disaster_data.csv')
df.drop(['PROV_CODE', 'DIST_CODE', 'INC_DATE'], axis=1)
  1. Now, from what I understand, there are several 'object' categorical features that need to be converted to numerical types, in order to be used in clustering (REGION, PROV_NAME, DIST_NAME, INC_TYPE, and SEASON). And I must use one hot encoding instead of label encoding, because these variables have no inherent ranking.
df_encoded = pd.get_dummies(df, columns=['REGION', 'PROV_NAME', 'DIST_NAME', 'INC_TYPE', 'Season'])

This seems to introduce the issue of having a highly dimensional dataset, as I now have the following 408 columns:

  • Persons_killed
  • Persons_injured
  • Individuals_affected
  • Families_affected
  • Houses_damaged
  • Houses_destroyed
  • REGION_Capital
  • REGION........and so on for a total of 8 REGION columns
  • PROV_NAME_Badakhshan
  • PROVE_NAME........and so on for a total of 34 PROV_NAME columns
  • DIST_NAME_Abkamari
  • DIST_NAME........ and so on for a total of 351 DIST_NAME columns
  • INC_TYPE_Avalanche
  • INC_TYPE........ and so on for a total of 5 INC_TYPE columns
  • SEASON_Autumn
  • SEASON........ and so on for a total of 4 SEASON columns
  • Cluster
  1. Next I start the clustering algorithm (I did use the elbow method to try and find the best number of clusters but I exclude it from here to keep it simple).
df = df_encoded

#Define our features
X = df 

#Scale the data
scaler = MinMaxScaler()
scaled_X = pd.DataFrame(scaler.fit_transform(X), columns = X.columns)

#Clustering
kmeans = KMeans(n_clusters = 5)
df['cluster'] = kmeans.fit_predict(scaled_X)
  1. Now I have absolutely no idea how to analyze this data. I want to visualize results. But how? I can make a plots of numerical types like Persons_killed vs Persons_injured. But I want to see if there are patterns for example geographical patterns that may show certain regions/provinces with certain incident types, but I don't know how to use the encoded columns in plotting. Maybe I am not asking the right scientific question to begin with. Does clustering even make sense with this dataset? What do you think of my experiment so far? I will take ANY suggestions or advice to improve this (I don't mind dropping the districts and just using provinces, that would help get rid of a lot of dimensions). And how do I analyze the resulting data?