2
$\begingroup$

I have a column called item_colour which describes the colour of products in my dataset.

There are 85 colours and some of the colours are unique or they represent a small part of the data ( I have 100,000 observations). For example, there is only one "almond" colour or only two "sky blue". I want to recombine rare values and put them together into a group called "other colour".

How can I find the threshold to recombine them? For example, combine the colours together which have values less 50 or so.

P.S.I am working with R

$\endgroup$
5
  • 1
    $\begingroup$ Welcome to the Site! Have you tried using for loops in r, making a new column with frequency of unique values and based on that you can combine them together? $\endgroup$
    – Toros91
    Nov 18, 2017 at 17:18
  • $\begingroup$ Is it helping find the threshold of necessity for being "rare"? Because I know how to recombine values. $\endgroup$
    – Anes
    Nov 18, 2017 at 17:24
  • $\begingroup$ Yes it can be done, if you can write a function to find the colour whose value is less than 2 % of the total. If yes then you can segregate those records and place it in the Rare colour segment. $\endgroup$
    – Toros91
    Nov 18, 2017 at 17:27
  • $\begingroup$ So threshold is 5% of total data. Got it! And do you think recombining them will affect my prediction model in a bad way? Because when I recombine them only 6 colours left and approximately 80 colours are recombined into the rare colour segment. $\endgroup$
    – Anes
    Nov 18, 2017 at 17:34
  • $\begingroup$ I think even 5% is also more. $\endgroup$
    – Toros91
    Nov 18, 2017 at 17:58

1 Answer 1

1
$\begingroup$

This is going to be a situation where there will be no fixed rule. One important factor is how meaningful colour differences are to the other parts of your problem. If colour has low correlation/impact in a supervised learning/prediction scenario for example, and the dataset is noisy, then you will want to merge more colours (at a higher fraction of total number) to reduce sampling bias effects that might otherwise assign importance to the colour and increase error rates in test and production.

The safest approach is to treat the colour combination threshold as a hyper-parameter to the model building process, and test to see what differences it makes. If there is little or no impact to model effectiveness, then a higher threshold could be useful purely to reduce number of parameters in the model - decreasing resources used to train and run it.

If that seems time-consuming, then picking something by feel (e.g. your idea of picking count less than 50 in the dataset) is not usually too bad, at least to start with. You can go back and re-evaluate your choice if you have problems with the model.

One other possibility for feature engineering is to use the rareness of the specific colour as an additional feature. So in addition to categories for the popular colours and an "other colour" category, add a real value "colour frequency" = the observed ratio of that colour in the training set. Whether or not this is useful will depend on the problem, but it may help address some of the lost information when merging categories with a wider range of rareness values, assuming that unusual colours indicate anything at all (they may not).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.