Beginner clustering project, what are the input features and how do I analyze the data?

Question

I am a beginner to data science. I have this dataset on natural disaster events in Afghanistan from 2016 - 2017. Columns:

• REGION (ex. North, North West, etc)
• PROVINCE_NAME (kind of like US 50 states)
• DISTRICT_NAME (kind of like US counties)
• INCIDENT_DATE (5 types: Flood, Earthquake, Land slide, Avalanche, and Heavy Snowfall)
• INCIDENT_TYPE
• Persons_killed
• Persons_injured
• Individuals_affected
• Families_affected
• Houses_damaged
• Houses_destroyed

I need to do any basic ML model on this dataset. I thought of predicting the disaster type given the other features using classification, or predicting the number of persons killed using regression. But I think some of these ideas are silly because they aren't useful in real life. For example, if I'm predicting Persons_killed, would I realistically have access to Persons_injured? (I don't know, if you have a good scientific question I can answer using regression etc, please let me know.)

A more meaningful experiment to try might be clustering. Since clustering is unsupervised, I'm just looking for any patterns. Does this mean I put all 13 columns into my model? I am a bit stuck on how to design this model but here is my thought process:

• I have checked my dataset for missing values, typos, etc. I have done some EDA.
• Do I need to encode categorical vars like REGION, PROVINCE_NAME, DISTRICT_NAME, and INCIDENT_TYPE? And what do I do with INCIDENT_DATE? Should I make a new feature called "Season", since I am not sure how to work with dates, or if I should just leave it out?
• Another issue is there are 2 natural disasters that are outliers (Earthquakes), they had a very large number of Persons_killed. This was factual, so would I leave these in the dataset or remove them? Because they cause the plot to zoom out very far and then you can't see the other data.
• I would then scale the data using StandardScaler (am I using all 13 columns in my model?)
• I don't fully understand dimensionality reduction with PCA, so I may leave this out for now and repeat this whole experiment with applying PCA at this step.
• Then I would create the model object and fit the model to the scaled data, and predict the cluster assignments. For the number of clusters, I could try a random number to start, but in class I learned about silhouette analysis and using a range of k to loop through and find the best k value.
• Now I'm confused on how to analyze the clusters. I would be analyzing the characteristics of each cluster. Perhaps use mean values of features. I am not sure how I would look at this on a map to see geographic patterns.

I apologize, as a beginner this is my first project, I would appreciate ANY advice, even if you cannot answer the whole series of questions.

Answer 1

Let me provide some guidance on the various aspects you've mentioned:

1. Encoding Categorical Variables

Yes, you should encode categorical variables like REGION, PROVINCE_NAME, DISTRICT_NAME, and INCIDENT_TYPE. Use techniques like one-hot encoding or label encoding, depending on the nature of the variables. For INCIDENT_DATE, creating a new feature like SEASON is a good idea. You can extract this information from the date and then encode it.

2. Handling Outliers

Whether to keep or remove outliers depends on the goals of your analysis. If earthquakes are significant events in your context, and you want the model to capture their impact, keep them. However, if they are skewing the results and making it hard to visualize other patterns, you may consider creating separate models with and without these outliers for comparison.

3. Scaling Data

Yes, use StandardScaler to scale your data. Include all relevant columns (features) in your model, as they contribute to the clustering process.

4. Dimensionality Reduction with PCA

While PCA can be beneficial, it's not always necessary, especially if your dataset isn't too large. You can start without PCA and explore it later if needed. It helps in reducing the number of features while preserving the most important information.

5. Determining the Number of Clusters

Silhouette analysis is a good approach. You can also try the elbow method, which involves plotting the explained variance as a function of the number of clusters and selecting the "elbow" point where the improvement slows down.

6. Analyzing Clusters

Once you have your clusters, analyze them by examining the mean values of features within each cluster. To visualize geographic patterns, consider plotting the clusters on a map using geospatial visualization tools. This can help you understand if certain regions or provinces are more prone to specific types of natural disasters.