I'm given a data set to create a model that would predict whether a certain supply chain would be able to deliver the goods without delay or not. I'm doing this in Python.

The data set has 93000 data points has at least 100 features as given. What's worse, most of these features are categorical data and each feature has around 200 unique categories.

I want to develop the best model I can but I find the size of the data set is too large for this.

I tried different methods of feature reduction. But even for feature reduction methods, it takes infeasible amount of time in my laptop. So I thought the best thing I could do is to first select a sub sample of the data points and find the best model and best parameters and then train the whole data set with this model.

Is selecting a sample from the data set the best thing I could do in this situation?

If not, your kind suggestions are welcome.

If it is, what's the best method of selecting these samples?

  • $\begingroup$ This is interesting to me. I have a similar problem. Did you ever find an answer on your own? $\endgroup$
    – mccurcio
    Oct 1, 2020 at 2:05
  • 1
    $\begingroup$ @oaxacamatt it turned out I had errors in my code slowing the whole process down. My bad I should have marked Fnguyen s answer below as the correct answer. Sorry I have nothing more to offer. $\endgroup$
    – PPGoodMan
    Oct 2, 2020 at 2:12

2 Answers 2


9300 cases is not a lot, this doesn't even fall within the realm of big data or anything were you would have to apply special techniques.

Also given a more restricted feature set there are almost no computational issues you should encounter unless you work on an actual brick or Atari. As an example benchmark I am able to fit a model for >4M cases with up to 20-30 features (some with ~40 levels) on my 16gb Laptop in <10mins.

So let's focus on the real issue, the complexity of the data i.e. the amount of available features/feature levels and most importantly efficient code.

Assuming you don't have the resources to simply throw some cloud processing power on this problem and call it a day we have to restrict the complexity.

Note that this does not mean "sampling the data" which would indicate using only a subset of cases, as said in the beginning this isn't your problem.

Instead let's workshop ways to cut down computational requirements:

1. Optimize your code

This sounds simple but research and make sure that you are using the best packages for the job, adjusted parameters correctly and cut fat from your code. This is hard work but it is necessary and worth it.

E.g. have you checked that all code is parallelized as much as you can and runs on all CPUs?

2. Optimize your data structure

Factor levels normally aren't a problem computationally speaking but even so >100 levels might be too hard to understand anyway. Try to transform your data by e.g. grouping certain factor levels together. Also try to pivot parts of your data e.g. by wide to long data.

3. Use your domain knowledge

You surely understand your business and this problem. Don't just throw all the data at it! Think for a minute and remove unnecessary or duplicate columns.

4. Work your way upwards

ML modeling has a simply yardstick, if the validated prediction quality is good, your model is good. If you manage to do that with two variables, nobody cares.

So start with a bare-bones data set that includes ALL cases (still keep split-sample validation in mind here) but only the 2-3 simplest, most relevant predictors (based on domain knowledge) as well as the depended variable.

Fit a model and check it's quality, is it good enough? Yeah your done! If not include the next round of variables...

5. Modulize your EDA and have patience

I cannot stress enough that your data set really isn't that big or complex, so optimize your code first. BUT if basic EDA needed to find the best features still takes too long, do the following:

  1. Structure and modulize your code by e.g. doing a PCA for only chunks of the columns (again use all case but only a subset of variables).

  2. Let it run one-by-one and just have patience if the code runs for an hour, sometimes that is how it goes.

  3. Look at the results, compare and decide how to merge results.

6. Talk to your boss

Yes I said I assume you do not have the resources to throw more power on it but if this is an important business problem there should be resources for cloud computation. Fitting a model on AWS is dirt cheap and will only take seconds, so realistically we are talking about <100$ here to train multiple models.

If you don't have a boss to pay for this stuff or he doesn't want to, try free resources. There are a lot of cloud based services that offer enough free GPU or CPU time that you should be able to fit a model.

  • $\begingroup$ Very methodical approach. I likes. $\endgroup$
    – mccurcio
    Oct 1, 2020 at 2:10

sampling data might be a possible approach. Given a sample you might check correlation, or even fit a model without waiting so much time and using any feature ranking method. On the other hand you may try if any dimensionality reduction technique you like (pca,ca,mfa, etc...) work to reduce the complexity.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.