0
$\begingroup$

I have a categorical variable with 4 levels ('8 c', '6 c','NAN','Others') and I want to convert it to numerical form. an Obvious way is to simply remove the 'c' part from the first two categories and replace NAN with 0. However, I was wondering about the 'Others' level?

What could be the best way to transform this level? Please note that the variable represents the number of cylinders for a given car.

Improve this question
$\endgroup$
4
  • $\begingroup$ Firstly, what is the meaning of (others) in this column? $\endgroup$
    – user119783
    Jul 16, 2021 at 9:19
  • $\begingroup$ it wasn't specified in the dataset description but I guess it means any other value than 8 c or 6 c $\endgroup$
    – user120794
    Jul 16, 2021 at 9:47
  • $\begingroup$ Could you please tell me what are the possible max number of cylinders you may have other than 6 and 8? Can this column have 0 number of cylinder, does that make sense? I think I need to understand more the dataset to give you a better solution. If the dataset is publicly available, you can share it with us. $\endgroup$
    – user119783
    Jul 16, 2021 at 10:27
  • $\begingroup$ I'm trying to analyse this dataset: kaggle.com/austinreese/craigslist-carstrucks-data $\endgroup$
    – user120794
    Jul 16, 2021 at 11:39

3 Answers 3

Reset to default
2
$\begingroup$

I spent some time exploring this dataset:

There are some findings I want to share it with you:

  • Number of samples is 426880 samples.
  • Number of categories in the cylinder column are: 3,4,5,6,7,8,10,12,others,Blank cell. You could take a look at the cylinder list at the beginning. here
  • There are no 7, 9, and 11 cylinders. Then, 'others' could contain more likely either 1 or 2 cylinders. click here
  • Number of samples contain the 'others' value in the cylinder column is 1298 samples which is equivalent to 0.3% of the total number of samples.

You can handle missing value and 'others' as following:

  1. Others: due to the very small amount of samples containing 'others', you could remove all these samples. Alternatively, you could replace them with 1 or 2 cylinders where 2 is the most popular.

  2. Blank cells: from my naive knowledge of cars, if the cars are from the same manufacturer and the same model, the same type of fuel, they have probably the same number of cylinders. Then, you can replace the blank values with the number of cylinders from the other cars which have the number of cylinders. Please, see the example below:

enter image description here

Another example:

enter image description here

Another example:

enter image description here

  • If you find the car model is missing. Then, I recommend removing these samples (they are 2673).
Improve this answer
$\endgroup$
1
$\begingroup$

I would suggest in this case 2 steps as part of your data preparation:

  • substitute 'NAN' for 'Others', since both labels are giving you no info and can be considered as unknown values
  • once you have finally 3 labels ('8 c', '6 c','Others'), apply one hot encoding, since you only have 3 possible categories (which prevents your dataset from being too sparse) and at the same time you do not assume that the unknown values are 0 cylinders.
Improve this answer
$\endgroup$
3
  • $\begingroup$ But he wants to convert them to ordinal data(6<8) not one-hot code. $\endgroup$
    – user119783
    Jul 16, 2021 at 10:42
  • $\begingroup$ I thought at the beginning like you with dealing with nan and others together. But then I thought that nan values can contain 6, 8 or any value but others contains any value but not 6 and 8 because this is the meaning of others. $\endgroup$
    – user119783
    Jul 16, 2021 at 10:46
  • $\begingroup$ Ok to your second point, nice appreciation. About the first point, I think she/he should not force ordinal encoding, since does not know whether the 'other' values are higher or lower than 6, 8... $\endgroup$
    – German C M
    Jul 16, 2021 at 16:13
0
$\begingroup$

It really depends what your variable refers to, and which kind of model you want to use.

A few things you can do :

  • OneHotEncoding : will create binary variables for each possibility for your variable : in your case, it'll create 4 variables '8 c', '6 c','NAN','Others', that take 1 or 0. This way, each possible variable output is now a binary variable, independant from others. Example : 'Var'='8 c' becomes '8 c' = 1, '6 c' = 0, 'NAN' = 0, 'Others' = 0

  • Manual coding with order : You can yourself transform '8 c' as 1, '6 c' as 2, 'NAN' as 3 and 'Others' as 4 for example, but this will mean for your model than '8 c' and '6 c' are closer than '8 c' and 'Others' for example. That's a method you can use for ordinal values (when V1 > V2 > ... )

  • Target Encoding : Often used when you have too much possible values, and can't afford to OneHot (which creates as much variables as potential values). Target Encoding will code your values as numbers, refering to their potential link to the target variable (y in your modelisation). I don't recommend it in your case since you only have 4 potential values.

Improve this answer
$\endgroup$
6
  • $\begingroup$ You can't replace nan with 0 because nan means no information. There are some techniques to deal with this. $\endgroup$
    – user119783
    Jul 16, 2021 at 10:52
  • $\begingroup$ can you please share with me the name of one of these techniques? $\endgroup$
    – user120794
    Jul 16, 2021 at 11:40
  • $\begingroup$ @Hamzawi : it doens't replace NaN to 0, it creates as much BINARY variables as potential values, and when Value is NaN, it assigns '1' to NaN value and 0 to others. $\endgroup$
    – Adept
    Jul 16, 2021 at 12:00
  • $\begingroup$ @maryamBlushi OneHotEncoding and TargetEncoding are the names of the techniques $\endgroup$
    – Adept
    Jul 16, 2021 at 12:00
  • $\begingroup$ @MaryamBlushi. If you mean techniques to replace Nan values in your dataset. There is a lot you could take a look at here: analyticsindiamag.com/… $\endgroup$
    – user119783
    Jul 16, 2021 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.