Questions tagged [categorical-data]
Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.
398
questions
200
votes
13
answers
302k
views
K-Means clustering for mixed numeric and categorical data
My data set contains a number of numeric attributes and one categorical.
Say, NumericAttr1, NumericAttr2, ..., NumericAttrN, CategoricalAttr,
where ...
- 5,454
172
votes
4
answers
122k
views
When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?
I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building ...
- 1,843
28
votes
3
answers
13k
views
How to combine categorical and continuous input features for neural network training
Suppose we have two kinds of input features, categorical and continuous. The categorical data may be represented as one-hot code A, while the continuous data is just a vector B in N-dimension space. ...
- 515
26
votes
2
answers
30k
views
Why do we need to discard one dummy variable?
I have learned that, for creating a regression model, we have to take care of categorical variables by converting them into dummy variables. As an example, if, in our data set, there is a variable ...
15
votes
3
answers
32k
views
How to convert categorical data to numerical data in Pyspark
I am using Ipython notebook to work with pyspark applications. I have a CSV file with lots of categorical columns to determine whether the income falls under or over the 50k range. I would like to ...
- 1,045
15
votes
2
answers
6k
views
Why does frequency encoding work?
Frequency encoding is a widely used technique in Kaggle competitions, and many times proves to be a very reasonable way of dealing with categorical features with high cardinality. I really don't ...
- 6,031
14
votes
10
answers
4k
views
How can I appropriately handle cleaning of gender data?
I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform an analysis of it.
In this dataset, the gender ...
- 263
14
votes
3
answers
9k
views
How can I dynamically distinguish between categorical data and numerical data?
I know someone who is working on a project that involves ingesting files of data without regard to the columns or data types. The task is to take a file with any number of columns and various data ...
- 243
13
votes
3
answers
10k
views
How can I do classification with categorical data which is not fixed?
I have a classification problem with both categorical and numerical data. The problem I'm facing is that my categorical data is not fixed, that means that the new candidate whose label I want to ...
- 281
12
votes
3
answers
29k
views
Mass convert categorical columns in Pandas (not one-hot encoding)
I have pandas dataframe with tons of categorical columns, which I am planning to use in decision tree with scikit-learn. I need to convert them to numerical values (not one hot vectors). I can do it ...
- 345
12
votes
2
answers
7k
views
Catboost Categorical Features Handling Options (CTR settings)?
I am working with a dataset with large number of categorical features (>80%) predicting a continuous target variable (i.e. Regression). I have been reading quite a bit about ways to handle categorical ...
- 4,207
12
votes
1
answer
4k
views
Feature importance with high-cardinality categorical features for regression (numerical depdendent variable)
I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many ...
- 331
11
votes
1
answer
7k
views
Confusion about Entity Embeddings of Categorical Variables - Working Example!
Problem Statement: I have problem making the Entity Embedding of Categorical Variable works for a simple dataset. I have followed the original github, or paper, or other blogposts[1,2,or this 3], or ...
- 4,207
11
votes
4
answers
28k
views
Clustering for mixed numeric and nominal discrete data
My data includes survey responses that are binary (numeric) and nominal / categorical. All responses are discrete and at individual level.
Data is of shape (n=7219, p=105).
Couple things:
I am ...
- 310
10
votes
4
answers
26k
views
How can Time Series Analysis be done with Categorical Variables
Most of the time series analysis tutorials/textbooks I've read about, be they for univariate or multivariate time series data, usually deal with continuous numerical variables.
I currently have a ...
- 101
9
votes
4
answers
14k
views
How to combine PCA and MCA on mixed data?
Suppose I have mixed data and (python) code which is capable of doing PCA (principal component analysis) on continuous predictors and MCA (multiple correspondence analysis) on nominal predictors. Is ...
9
votes
2
answers
2k
views
Using NLP to automate the categorization of user description
I have a huge file of customer complaints about the products my company owns and I would like to do a data analysis on those descriptions and tag a category to each of them.
For example: I need to ...
- 1,045
9
votes
2
answers
24k
views
Always drop the first column after performing One Hot Encoding?
Since one of the columns can be generated completely from the others, and hence retaining this extra column does not add any new information for the modelling process, would it be good practice to ...
- 403
8
votes
5
answers
6k
views
How do I encode the categorical columns if there are more than 15 unique values?
I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have ...
- 81
8
votes
2
answers
20k
views
Why don't tree ensembles require one-hot-encoding?
I know that models such as random forest and boosted trees don't require one-hot encoding for predictor levels, but I don't really get why. If the tree is making a split in the feature space, then isn'...
- 83
8
votes
4
answers
10k
views
One Hot encoding for large number of values
How do we use one hot encoding if the number of values which a categorical variable can take is large ?
In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary ...
- 367
8
votes
1
answer
10k
views
Keras categorical_crossentropy loss (and accuracy)
When training a neural network with keras for the categorical_crossentropy loss, how exactly is the loss defined? I expect it to be the average over all samples of
$$\textstyle\text{loss}(p^\text{true}...
- 194
7
votes
3
answers
10k
views
How to deal with categorical feature of very high cardinality?
I would like to train a binary classifier on feature vectors.
One of the features is categorical feature with string, it is the zip codes of a country.
Typically, there is thousands of zip codes, and ...
- 594
7
votes
1
answer
752
views
How to deal with missing data for only some categories
Or in other words, data for category A is irrelevant for category B. So it is not present, how can imputing missing data distort/effect learning models broadly.
I can't find any logic how to deal ...
- 453
7
votes
3
answers
3k
views
Classifier and Technique to use for large number of categories
I am designing a scikit learn classifier for a sequence labelling task which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year. I have ...
- 191
6
votes
1
answer
4k
views
How to handle columns with categorical data and many unique values
I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.
I also have another column with 145 nunique values that I could also use ...
- 175
6
votes
1
answer
2k
views
Why after adding categorical data the Linear Regression fails?
Based on a training set we applied a simple Linear Regression on some attributes that all were numeric.
Now we have more attributes in terms of categories and of course we applied one-hot-encoding to ...
6
votes
2
answers
4k
views
Dummy coding a column in R with multiple levels
I have a dependent variable measuring the net revenue. One of the major predictor affecting this is "product" i.e. the product sold to the customer. My randomly sampled dataset contains 1.4 million ...
- 155
6
votes
2
answers
4k
views
One hot encoding large dataset
Initially, I have a dataset where for each row there is user_id and product_ids he bought.
In that dataset there are 478191 products bought by different users.
In order to discover frequent items ...
- 191
6
votes
2
answers
2k
views
Machine Learning - Where is the difference between one-class, binary-class and multinominal-class classification?
Where is the difference between one-class, binary-class and multinominal-class classification?
If I like to classify text in lets say four classes and also want the system to be able to tell me that ...
- 163
6
votes
1
answer
281
views
Mapping of categorical features into binary indicator features
I am currently reading an introductory machine learning book by Daumé (ch. 03, p. 30) and when discussing the mapping of categorical features with "n" possible values into "n" binary indicator ...
6
votes
3
answers
319
views
Anomaly detection using clustering of highly correlated Categorical data
My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else it's Anomaly. Likewise, there are thousands ...
5
votes
3
answers
1k
views
Quasi-categorical variables - any ideas?
Let's say I'm trying to predict a person's electricity consumption, using the time of day as a predictor (hours 00-23), and further assume I have a hefty but finite amount of historical measurements.
...
- 206
5
votes
5
answers
10k
views
Large no of categorical variables with large no of categories
I'm working on a binary classification problem where the dataset is slightly imbalanced (30% class 0 | 70% class 1).
Most of my features are categorical with large number of categories. For example: ...
- 51
5
votes
1
answer
550
views
Extracting encoded features after CatBoost
I have a dataset containing numerical as well as categorical variables.
After I've fit my dataset to a CatBoostClassifier, I want to extract the entire feature set, with the categorical variables ...
- 229
5
votes
2
answers
713
views
How to continue incremental learning when a categorical variable has been assigned additional category labels?
Please help answer this question or point me to any resource.
There is a model in an environment where training happens with new data and the data is discarded after training is completed.
This keeps ...
5
votes
1
answer
15k
views
Categorical and ordinal feature data representation in regression analysis? [closed]
I am trying to fully understand difference between categorical and ordinal data when doing regression analysis. For now, what is clear:
Categorical feature and data example:
Color: red, white, black ...
- 177
5
votes
2
answers
5k
views
Feature Selection with one-hot-encoded categorical data
I have a dataset with 400+ columns. Almost 90% of these are categorical data with One-Hot-Encoding (OHE). I'm using the dataset for a classification problem.
My professors asked me to perform feature ...
- 175
5
votes
2
answers
2k
views
Nested features with one to many relationships
I have a dataset that has (among others) a categorical variable with many levels and further attributes associated with each level.
For example, consider predicting machine failure based on its last ...
oW_♦
- 6,262
5
votes
1
answer
81
views
Relation mining of multivariant categorical timeseries without excluding the temporal nature
To all:
I have been wracking my brain at this for a while and thought maybe someone here would know of a package or algorithm to handle the following:
I have nominal multivariant timeseries that ...
- 51
5
votes
1
answer
6k
views
Feature selection for data with both continuous and categorical features?
I am working on a classification problem with 4 ordinal classes to predict, labelling/predicting samples as either a number from 1-4. My training dataset has 284 features by ~40,000 samples and I am ...
- 155
5
votes
1
answer
950
views
Array of categorical variables vs one-hot encoding
I have some JSON data to be transformed to a machine-learning friendly format. Every object in my data, which will eventually become an instance in my dataset, has the exact same fields (in this case, ...
- 151
5
votes
0
answers
2k
views
Dealing with categorical variables in Isolation Forest
Isolation Forest is widely used when dealing with outlier/anomaly detection when we have no labels. The theory behind is that making random split at random points and counting how many splits you do ...
- 6,112
5
votes
2
answers
195
views
Selecting ML algorithm for music composition
I'm a composer and programmer. I'd like to use ML for composing music. There's already research on the general problem of composing by machine in known musical styles. I'm more interested in the ...
- 173
4
votes
6
answers
18k
views
Different number of features in train vs test
I'm doing the titanic exercise on kaggle and there is a categorical Cabin attribute that has a lot of different strings: C41, C11, B20 etc. (about 100).
To be able to train my model I'm converting it ...
- 198
4
votes
1
answer
2k
views
Is there an asymmetric version of nominal correlation?
I use Cramer's V to calculate correlation of features in a dataset made of only nominal features.
Let's consider the following dataset:
...
- 689
4
votes
4
answers
2k
views
How to visualise multidimensional categorical data with additional time dimension
I am trying to visualize a dataset that registers the career of people in an organisation and their backgrounds before starting in that particular organisation. I 'd like to show how individuals are ...
- 141
4
votes
3
answers
2k
views
What's the best way to use binned data in a tree-based model?
I have some numeric data that has come 'binned', but the bins are not of equal sizes in terms of scale or quantile
For example, an age variable that is [0-16), [16-21), [21-30), [30-45), [45-65), [65,...
- 141
4
votes
1
answer
735
views
Decision Tree only splits to the left
I can’t really understand, why my decision tree only splits to the left.
I originally have 2 categorical features (further named feature 0 and 1), which I concat to one feature since feature 1 is ...
- 41
4
votes
1
answer
3k
views
Strategies to encode categorical variables with many categories
I was going over the Kaggle competitions IEEE,Categorical Feature Encoding Challenge and one of the ways in which categorical variables have been handled is by replacing the variables by the ...
user86099