I have a data science challenge in which two datasets are provided, the first one contains weather data (temperature, wind speed, and precipitation) for a number of days, and the other contains flight information (duration and number of passengers aboard) for the same days. The flight dataset can contain multiple flights on the same day.
First of all, we are asked to merge the two datasets. As a result weather data values can be duplicates since there can be multiple flights on the same day. Moreover, half of the number_of_passengers values are missing, in future questions, we will have to fill them using a predictive model.
I have a couple of questions concerning outlier detection:
Since many weather data values will be duplicated in the big dataset, should I detect weather outliers in the weather dataset first or in the big dataset where a value can be present multiple times (using IQR for outlier detection). (Knowing that the outlier question comes after the join question).
Should I detect outliers univariately for data points regardless of whether number_of_passengers is missing or not, or should I first divide my dataset into points where number_of_passengers is given and others where it is not and then perform outlier detection for the two datasets and merge them afterwards? Or a third option which is not removing outliers for points where number_of_passengers is missing even though they are clearly outliers (not rare case outliers but error impossible values outliers)