I have a dataframe of natural disaster incidents in Afghanistan from 2016 - 2023.
- REGION (Northern, Eastern etc)
- PROV_CODE (province)
- DIST_CODE (district)
- INC_DATE (incident date)
- INC_TYPE (5 types: Avalanche, Earthquake, Landslide, Heavy Snowfall, Flood)
I want to perform a basic clustering experiment to see what patterns exist in the data. I am a beginner and am really struggling. I have finished cleaning the data and exploratory data analysis. Here is what I do next:
- PROV_CODE and PROV_NAME are redundant. As are DIST_CODE and DIST_NAME. These are the Province/District codes and names. So I will drop PROV_CODE and DIST_CODE. I will also drop INC_DATE, because I don't know how to work with dates (I created SEASON to replace it).
df = pd.read_csv('cleaned_disaster_data.csv') df.drop(['PROV_CODE', 'DIST_CODE', 'INC_DATE'], axis=1)
- Now, from what I understand, there are several 'object' categorical features that need to be converted to numerical types, in order to be used in clustering (REGION, PROV_NAME, DIST_NAME, INC_TYPE, and SEASON). And I must use one hot encoding instead of label encoding, because these variables have no inherent ranking.
df_encoded = pd.get_dummies(df, columns=['REGION', 'PROV_NAME', 'DIST_NAME', 'INC_TYPE', 'Season'])
This seems to introduce the issue of having a highly dimensional dataset, as I now have the following 408 columns:
- REGION........and so on for a total of 8 REGION columns
- PROVE_NAME........and so on for a total of 34 PROV_NAME columns
- DIST_NAME........ and so on for a total of 351 DIST_NAME columns
- INC_TYPE........ and so on for a total of 5 INC_TYPE columns
- SEASON........ and so on for a total of 4 SEASON columns
- Next I start the clustering algorithm (I did use the elbow method to try and find the best number of clusters but I exclude it from here to keep it simple).
df = df_encoded #Define our features X = df #Scale the data scaler = MinMaxScaler() scaled_X = pd.DataFrame(scaler.fit_transform(X), columns = X.columns) #Clustering kmeans = KMeans(n_clusters = 5) df['cluster'] = kmeans.fit_predict(scaled_X)
- Now I have absolutely no idea how to analyze this data. I want to visualize results. But how? I can make a plots of numerical types like Persons_killed vs Persons_injured. But I want to see if there are patterns for example geographical patterns that may show certain regions/provinces with certain incident types, but I don't know how to use the encoded columns in plotting. Maybe I am not asking the right scientific question to begin with. Does clustering even make sense with this dataset? What do you think of my experiment so far? I will take ANY suggestions or advice to improve this (I don't mind dropping the districts and just using provinces, that would help get rid of a lot of dimensions). And how do I analyze the resulting data?