2
$\begingroup$

Assume that there is a very large dataset of hundreds of sites which contains only the PM2.5 level, the site ID, and the Latitude and Longitude as features. The independent feature to be predicted is PM2.5. The dependent features are site ID and Latitude/Longitude.

Conversely, a second dataset pertains to only three of the above sites (A, B, C). It has the same features as the first. However, in addition to the location and PM2.5 features, there are also perhaps 50 environmental features collected daily for each site, such as daily high air temperature, daily max wind speed, and so on.

The goal is to determine which dataset can produce the better prediction for sites A, B, C on a given day in the future.

Question 1: Suppose that classifiers build using first dataset perform much better than classifiers built using the second dataset. Are classifiers built on the first dataset valid as they are basing some of their predictive value simply on the site ID? Is site ID (and to a similar extend Lat/Lon) a valid feature to include?

Question 2: Suppose that the task is now to predict PM2.5 at a fourth unseen location. There is no training data for the fourth location, only a test event. Both datasets do not include this location. The Lat/Lon of the fourth location is included in test. Site ID of this fourth location is N/A during test. The second dataset includes many environmental features for this fourth location during test. Suppose that based solely on location, the classifiers from the first dataset (many examples of Lat/Lon, site ID, previous day PM2.5 levels) perform much better at predicting PM2.5 at this new location than the models built using few locations. Is this a valid result?

$\endgroup$

1 Answer 1

2
$\begingroup$

Are classifiers built on the first dataset valid as they are basing some of their predictive value simply on the site ID? Is site ID (and to a similar extend Lat/Lon) a valid feature to include?

It's fine to include site ID as long as the site ID will also be available at inference time. If you want to nickpick, it is probably a bit redundant to include both site ID and lat/long. Lat/long contains the same amount of information (from the model's PoV) but in a more difficult-to-model format. You will probably get equivalently good results with site ID only.

Suppose that based solely on location, the classifiers from the first dataset (many examples of Lat/Lon, site ID, previous day PM2.5 levels) perform much better at predicting PM2.5 at this new location than the models built using few locations. Is this a valid result?

Assuming there is no data leakage, it's a very strange result. Considering that the second dataset contains more information, you would expect that a model trained on the second dataset would perform at least as well as the first. If the first-dataset model is performing better, then either (1) the additional dependent features in the second dataset are uncorrelated with the independent feature or (2) your model overfit when training on the second dataset.

I think it is unlikely that the additional features are completely uncorrelated with the target. I would recommend trying some type of regularization method and retraining on the second dataset

$\endgroup$
3
  • 1
    $\begingroup$ Strange, sure. But the first dataset contains much more training data to go off of. Let's say 1000 sites. The second dataset is only using 3 sites. Let's assume train contains 3 years of daily values. So adding an unknown site, the 1000 sites' worth of data appears to make up for including zero environmental variables. $\endgroup$
    – There
    Jun 13, 2022 at 18:30
  • 1
    $\begingroup$ yeah in that case it sounds reasonable to me $\endgroup$
    – zachdj
    Jun 13, 2022 at 20:24
  • 1
    $\begingroup$ I missed the part where you mentioned the second dataset had only three sites. sorry about that $\endgroup$
    – zachdj
    Jun 13, 2022 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.