1
$\begingroup$

I have a huge dataset of 10 million observations but most of the variables are missing for 40% records. There are couple of variables available for the whole dataset such as sic code(Industry category) and addresses of the companies.

What may go wrong if I build a model on other 60% records(Non-missing values)? Can you suggest any measures/actions to take before spending time on building the model?

  • Couple of things I planned to do to ensure 60% records represent the population:
  1. Distribution of companies by industry category/state for 60% records(Non-missing values) vs 100% records

  2. Target variable distribution of 60% records(non-missing) vs 100% records

$\endgroup$
3
  • $\begingroup$ what are your variables ? $\endgroup$ Commented Oct 15, 2020 at 20:04
  • $\begingroup$ Building a classification model to target companies for mail campaigns. Variables missing for 40 % of the companies are some categorical (Type of business, owns or leases the property) and some continuous (employee count, revenue, etc.,) $\endgroup$
    – Max64
    Commented Oct 15, 2020 at 20:19
  • $\begingroup$ Do you know the ground truth for the groups with missing values? If yes, you can weight or oversample according to the stratification in the non-missing cases. So you can possibly mimic the true distribution by post-stratification. However, this is contingent on what you actually want to do model-wise. Your question is too vague overall. Also the question "what may go wrong" is highly contingent on your overall aim $\endgroup$
    – Peter
    Commented Oct 16, 2020 at 15:57

2 Answers 2

2
$\begingroup$

Generally speaking, you should investigate the process by which your values are missing and try to deal with it. I assume you checked that :

  • There is no meaningfull way to fill those missing values. Sometimes, typically with companies data that often represent amount of money, missing values means 0$. For exemple values missing by block may correspond to a tax form they didn't fill because they don't pay it and most features could be set to 0.
  • Similarly you don't have enough data to fill the blanks. Sometimes you can do missing value imputation based on what information you have. Specifically with companies data, you might be able to calculate financial ratios based on other metrics. Basically, you might want to check that your missing values do not correspond to some calculation that hasn't been made (then either correct that calculation or do it yourself in your pre-processing). There are more advanced imputations techniques, but I don't think they would be usefull when values are missing by blocks.

If you checked that, you can proceed to the next step : modeling. How you build your model depends on what is its purpose, what you want to do with it. Here, you should ask yourself some practical questions about those 40% records with missing values.

The first one is : do you actually need to output something for those instances with a lot of missing values ?

In some cases, notably when the output is public (or can be contested individually) it might just be best to output nothing or something along the line of 'we do not have enough data' for those instances. Practically that means removing them from your data set and building your models on your other instances.

On the contrary, when you look for performance (and not explainability to someone) you might want to use that pattern as an info. The best way to do so would be to introduce some feature that count the proportion of missing value for that instance. It might help a general model (see below) deal with that info.

If you need to output something, because you need some sort of an average, or because management asked for it, then consider the second question : should you make multiple models ?

One approach is to use just one model. Some models, mainly trees, can handle missing values. More advanced methods like XGBoost (eXtreme Gradient Boosted trees) are even considered state of the art for tabular data like companies data. Generally speaking this works (and might works very well with XGBoost). However, having done that myself, you'll end up with something quite hard to explain and overall you'll obfuscate that a significant chunk of your population is missing data. More specifically with data missing in blocks like yours, you'll probably end up with a very early split in your tree and actually have two models inside of one, with possible bizare interactions.

The alternative is to devise two models: one, very performant (say XGBoost), on your data with few missing values and another, way simpler on your other data. This will help you deal with instances with lot of missing values quite efficiently. For companies data with lots of missing data, you might just want to build a simple table of average of the outcome by your main descriptors (size, industry, geographical zone). This will leave you with most of your time to deal with the more complex model and its explanation (which is probably why you are paid for and is way more fun).

$\endgroup$
1
$\begingroup$

I agree that the main potential issue is bias due to a particular group of instances being over-represented in the missing values. For example it's possible that the type of company is missing in cases where it's unknown or ambiguous, and this might correspond to a particular company profile (e.g. smaller, more recent, ...). To me the measures you propose to ensure that the remaining data is representative look good (of course this depends on the specifics of the data).

Another important point to think about is what the model is going to be used for: if it's going to be used to predict something from unseen instances, chances are that some of these future instances will also have missing values. In this case the model might be less useful if it cannot deal with them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.