Finding Correlations in two Datasets

Question

I have an assignment where I am trying to find correlations between Lightning Strikes and Telecommunication damage. The two datasets consist of many columns (especially the human-recorded Telecommunication damage one), but let's assume it was something like this :

Telecommunication_Damage_df.columns = (timestamp, geolocation(lat, lon), type_of_damage) and
Lightning_Strikes.columns = (timestamp, geolocation(lat, lon))

I have done some EDA and cleaned the data, also assigned each lightning strike and each telecommunication damage row to a particular location(cities/areas), but now I am confused as to how I should proceed. Every other Data Science / Machine Learning project I have been involved in was much more direct and usually had one training/testing dataset whereas with this one I am stuck as to how I should proceed, is there a model that could help? Is there a methodology I am unfamiliar with?

I tried following this crime/geolocation tutorial (https://www.kdnuggets.com/2020/02/introduction-geographical-time-series-crime-r-sql-tableau.html) but it's not exactly the same, because there is one dataset being used, crimes, if a crime occurred then that's it, whereas here if a lightning strike occurred that doesn't necessarily mean a telecommunication problem was found and vice versa.

I know it's a bit vague, but I've been stuck for quite a while, and I was hoping that someone could guide me to any direction, because at the moment I am idle.

Answer 1

About the type of problem: apparently here the goal is not to do any kind of supervised learning, it seems to be more a kind of descriptive task.

The first thing I would try to do is to align the two datasets, i.e. merge them based on events in one occuring at the same time and place as in the other one. In order to do that you need to be able to compute for any two events whether they appear in the same area around the same time. I guess you'll need a window, for example defining two events as related if they occur within 1h of each other and are distant by less than 10km. Be careful about noise in the data: the time of a telecom incident might be the time it's reported, not the time it actually happens.

Once you have linked potential related events, you can merge the two datasets into one. You should probably preserve the events which have no match in the other dataset for statistical purposes, so use an outer join.

At this stage you have an exploitable dataset. I would start by doing plots such as overall probability distribution of related events, probability of related event by location. Probability of related events across time.