0
$\begingroup$

I have a data science challenge in which two datasets are provided, the first one contains weather data (temperature, wind speed, and precipitation) for a number of days, and the other contains flight information (duration and number of passengers aboard) for the same days. The flight dataset can contain multiple flights on the same day.

First of all, we are asked to merge the two datasets. As a result weather data values can be duplicates since there can be multiple flights on the same day. Moreover, half of the number_of_passengers values are missing, in future questions, we will have to fill them using a predictive model.

I have a couple of questions concerning outlier detection:

  1. Since many weather data values will be duplicated in the big dataset, should I detect weather outliers in the weather dataset first or in the big dataset where a value can be present multiple times (using IQR for outlier detection). (Knowing that the outlier question comes after the join question).

  2. Should I detect outliers univariately for data points regardless of whether number_of_passengers is missing or not, or should I first divide my dataset into points where number_of_passengers is given and others where it is not and then perform outlier detection for the two datasets and merge them afterwards? Or a third option which is not removing outliers for points where number_of_passengers is missing even though they are clearly outliers (not rare case outliers but error impossible values outliers)

Improve this question
$\endgroup$
1
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Commented Jan 16, 2022 at 13:25

1 Answer 1

Reset to default
1
$\begingroup$

As you are doing joins, I would suggest you to remove outliers and anything on original weather data itself. As once you join number fo flights on each day may have an impact on it.

As in future number of passenger would be a target variable, i would suggest you to seprate your data into training (Rows where Number of missing is present) vs number of missing is not present. This will make sure that there is no data leakage between train and test set.

Also, as Number of Missing may be highly correlated to extreme weather condition i would suggest to have this in your data in some form maybe as flag or some clipping

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.