Beginner basic clustering model and one-hot encoding?

Question

I have a dataframe of natural disaster incidents in Afghanistan from 2016 - 2023.

Column names:

• REGION (Northern, Eastern etc)
• PROV_CODE (province)
• PROV_NAME
• DIST_CODE (district)
• DIST_NAME
• INC_DATE (incident date)
• INC_TYPE (5 types: Avalanche, Earthquake, Landslide, Heavy Snowfall, Flood)
• Persons_killed
• Persons_injured
• Individuals_affected
• Families_affected
• Houses_damaged
• Houses_destroyed
• SEASON

I want to perform a basic clustering experiment to see what patterns exist in the data. I am a beginner and am really struggling. I have finished cleaning the data and exploratory data analysis. Here is what I do next:

1. PROV_CODE and PROV_NAME are redundant. As are DIST_CODE and DIST_NAME. These are the Province/District codes and names. So I will drop PROV_CODE and DIST_CODE. I will also drop INC_DATE, because I don't know how to work with dates (I created SEASON to replace it).

df = pd.read_csv('cleaned_disaster_data.csv')
df.drop(['PROV_CODE', 'DIST_CODE', 'INC_DATE'], axis=1)

2. Now, from what I understand, there are several 'object' categorical features that need to be converted to numerical types, in order to be used in clustering (REGION, PROV_NAME, DIST_NAME, INC_TYPE, and SEASON). And I must use one hot encoding instead of label encoding, because these variables have no inherent ranking.

df_encoded = pd.get_dummies(df, columns=['REGION', 'PROV_NAME', 'DIST_NAME', 'INC_TYPE', 'Season'])

This seems to introduce the issue of having a highly dimensional dataset, as I now have the following 408 columns:

• Persons_killed
• Persons_injured
• Individuals_affected
• Families_affected
• Houses_damaged
• Houses_destroyed
• REGION_Capital
• REGION........and so on for a total of 8 REGION columns
• PROV_NAME_Badakhshan
• PROVE_NAME........and so on for a total of 34 PROV_NAME columns
• DIST_NAME_Abkamari
• DIST_NAME........ and so on for a total of 351 DIST_NAME columns
• INC_TYPE_Avalanche
• INC_TYPE........ and so on for a total of 5 INC_TYPE columns
• SEASON_Autumn
• SEASON........ and so on for a total of 4 SEASON columns
• Cluster

3. Next I start the clustering algorithm (I did use the elbow method to try and find the best number of clusters but I exclude it from here to keep it simple).

df = df_encoded

#Define our features
X = df 

#Scale the data
scaler = MinMaxScaler()
scaled_X = pd.DataFrame(scaler.fit_transform(X), columns = X.columns)

#Clustering
kmeans = KMeans(n_clusters = 5)
df['cluster'] = kmeans.fit_predict(scaled_X)

4. Now I have absolutely no idea how to analyze this data. I want to visualize results. But how? I can make a plots of numerical types like Persons_killed vs Persons_injured. But I want to see if there are patterns for example geographical patterns that may show certain regions/provinces with certain incident types, but I don't know how to use the encoded columns in plotting. Maybe I am not asking the right scientific question to begin with. Does clustering even make sense with this dataset? What do you think of my experiment so far? I will take ANY suggestions or advice to improve this (I don't mind dropping the districts and just using provinces, that would help get rid of a lot of dimensions). And how do I analyze the resulting data?

Answer 1

Once you are done fitting the model, you can label each of your records based on the cluster.

df['cluster_labels'] = kmeans.labels_

For ease of analysis, you can convert all the one-hot encoding back to the original format data i.e. create a df column to convert back 8 region columns to the single region column in data frame. Once you are done converting all the columns back to their original format. You can filter out each cluster one by one.

df[df['cluster_labels']==0]

Say you first pickup this cluster and then you can plot the distribution of each column to see if there are any patterns say this cluster has most of the inc_type as Avalanche, number of houses destroyed were more and most state of the rows in the cluster is Badakhshan. You can say Badakhshan is a mountain surface that is why there are more houses destroyed due to Avalanche.

This way you can filter out next cluster and plot distribution of each column to find patterns

Other way to analyze the data is to calculate the mean/median/mode of the entire dataset and then calculating the mean/median/mode of the individual clusters and then comparing the values amongst all the cluster to draw inferences. Example say cluster 0 have average value of the number of houses destroyed as 90 whereas cluster 1 has average value of the number of houses destroyed as 5.