Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
167
votes
13answers
220k views

K-Means clustering for mixed numeric and categorical data

My data set contains a number of numeric attributes and one categorical. Say, NumericAttr1, NumericAttr2, ..., NumericAttrN, CategoricalAttr, where ...
142
votes
4answers
83k views

When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?

I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building ...
19
votes
2answers
17k views

Why do we need to discard one dummy variable?

I have learned that, for creating a regression model, we have to take care of categorical variables by converting them into dummy variables. As an example, if, in our data set, there is a variable ...
18
votes
3answers
6k views

How to combine categorical and continuous input features for neural network training

Suppose we have two kinds of input features, categorical and continuous. The categorical data may be represented as one-hot code A, while the continuous data is just a vector B in N-dimension space. ...
14
votes
10answers
3k views

How can I appropriately handle cleaning of gender data?

I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform an analysis of it. In this dataset, the gender ...
12
votes
3answers
24k views

Mass convert categorical columns in Pandas (not one-hot encoding)

I have pandas dataframe with tons of categorical columns, which I am planning to use in decision tree with scikit-learn. I need to convert them to numerical values (not one hot vectors). I can do it ...
12
votes
3answers
8k views

How can I do classification with categorical data which is not fixed?

I have a classification problem with both categorical and numerical data. The problem I'm facing is that my categorical data is not fixed, that means that the new candidate whose label I want to ...
12
votes
2answers
21k views

How to convert categorical data to numerical data in Pyspark

I am using Ipython notebook to work with pyspark applications. I have a CSV file with lots of categorical columns to determine whether the income falls under or over the 50k range. I would like to ...
12
votes
3answers
7k views

How can I dynamically distinguish between categorical data and numerical data?

I know someone who is working on a project that involves ingesting files of data without regard to the columns or data types. The task is to take a file with any number of columns and various data ...
12
votes
2answers
6k views

Catboost Categorical Features Handling Options (CTR settings)?

I am working with a dataset with large number of categorical features (>80%) predicting a continuous target variable (i.e. Regression). I have been reading quite a bit about ways to handle categorical ...
12
votes
1answer
4k views

Feature importance with high-cardinality categorical features for regression (numerical depdendent variable)

I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many ...
11
votes
4answers
24k views

Clustering for mixed numeric and nominal discrete data

My data includes survey responses that are binary (numeric) and nominal / categorical. All responses are discrete and at individual level. Data is of shape (n=7219, p=105). Couple things: I am ...
11
votes
1answer
707 views

Why does frequency encoding work?

Frequency encoding is a widely used technique in Kaggle competitions, and many times proves to be a very reasonable way of dealing with categorical features with high cardinality. I really don't ...
10
votes
1answer
6k views

Confusion about Entity Embeddings of Categorical Variables - Working Example!

Problem Statement: I have problem making the Entity Embedding of Categorical Variable works for a simple dataset. I have followed the original github, or paper, or other blogposts[1,2,or this 3], or ...
9
votes
4answers
11k views

How to combine PCA and MCA on mixed data?

Suppose I have mixed data and (python) code which is capable of doing PCA (principal component analysis) on continuous predictors and MCA (multiple correspondence analysis) on nominal predictors. Is ...
9
votes
2answers
2k views

Using NLP to automate the categorization of user description

I have a huge file of customer complaints about the products my company owns and I would like to do a data analysis on those descriptions and tag a category to each of them. For example: I need to ...
8
votes
3answers
6k views

One Hot encoding for large number of values

How do we use one hot encoding if the number of values which a categorical variable can take is large ? In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary ...
8
votes
1answer
9k views

Keras categorical_crossentropy loss (and accuracy)

When training a neural network with keras for the categorical_crossentropy loss, how exactly is the loss defined? I expect it to be the average over all samples of $$\textstyle\text{loss}(p^\text{true}...
6
votes
3answers
8k views

How to deal with categorical feature of very high cardinality?

I would like to train a binary classifier on feature vectors. One of the features is categorical feature with string, it is the zip codes of a country. Typically, there is thousands of zip codes, and ...
6
votes
5answers
103 views

How do I encode the categorical columns if there are more than 15 unique values?

I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have ...
6
votes
2answers
12k views

Why don't tree ensembles require one-hot-encoding?

I know that models such as random forest and boosted trees don't require one-hot encoding for predictor levels, but I don't really get why. If the tree is making a split in the feature space, then isn'...
6
votes
1answer
2k views

Why after adding categorical data the Linear Regression fails?

Based on a training set we applied a simple Linear Regression on some attributes that all were numeric. Now we have more attributes in terms of categories and of course we applied one-hot-encoding to ...
6
votes
1answer
236 views

How to deal with missing data for only some categories

Or in other words, data for category A is irrelevant for category B. So it is not present, how can imputing missing data distort/effect learning models broadly. I can't find any logic how to deal ...
6
votes
3answers
7k views

How can Time Series Analysis be done with Categorical Variables

Most of the time series analysis tutorials/textbooks I've read about, be they for univariate or multivariate time series data, usually deal with continuous numerical variables. I currently have a ...
6
votes
2answers
4k views

Dummy coding a column in R with multiple levels

I have a dependent variable measuring the net revenue. One of the major predictor affecting this is "product" i.e. the product sold to the customer. My randomly sampled dataset contains 1.4 million ...
6
votes
3answers
2k views

Classifier and Technique to use for large number of categories

I am designing a scikit learn classifier for a sequence labelling task which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year. I have ...
6
votes
1answer
15k views

Always drop the first column after performing One Hot Encoding?

Since one of the columns can be generated completely from the others, and hence retaining this extra column does not add any new information for the modelling process, would it be good practice to ...
6
votes
2answers
1k views

Machine Learning - Where is the difference between one-class, binary-class and multinominal-class classification?

Where is the difference between one-class, binary-class and multinominal-class classification? If I like to classify text in lets say four classes and also want the system to be able to tell me that ...
6
votes
1answer
105 views

Mapping of categorical features into binary indicator features

I am currently reading an introductory machine learning book by Daumé (ch. 03, p. 30) and when discussing the mapping of categorical features with "n" possible values into "n" binary indicator ...
5
votes
3answers
789 views

Quasi-categorical variables - any ideas?

Let's say I'm trying to predict a person's electricity consumption, using the time of day as a predictor (hours 00-23), and further assume I have a hefty but finite amount of historical measurements. ...
5
votes
1answer
2k views

How to handle columns with categorical data and many unique values

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world. I also have another column with 145 nunique values that I could also use ...
5
votes
2answers
394 views

How to continue incremental learning when a categorical variable has been assigned additional category labels?

Please help answer this question or point me to any resource. There is a model in an environment where training happens with new data and the data is discarded after training is completed. This keeps ...
5
votes
1answer
2k views

Logic behind SMOTE-NC?

In the SMOTE paper here, the authors present the logic for creating synthetic examples when some of the features are nominal and some are continuous (section 6.1, SMOTE-NC). This example is provided: ...
5
votes
2answers
3k views

Feature Selection with one-hot-encoded categorical data

I have a dataset with 400+ columns. Almost 90% of these are categorical data with One-Hot-Encoding (OHE). I'm using the dataset for a classification problem. My professors asked me to perform feature ...
5
votes
1answer
74 views

Relation mining of multivariant categorical timeseries without excluding the temporal nature

To all: I have been wracking my brain at this for a while and thought maybe someone here would know of a package or algorithm to handle the following: I have nominal multivariant timeseries that ...
5
votes
2answers
2k views

Nested features with one to many relationships

I have a dataset that has (among others) a categorical variable with many levels and further attributes associated with each level. For example, consider predicting machine failure based on its last ...
5
votes
1answer
823 views

Array of categorical variables vs one-hot encoding

I have some JSON data to be transformed to a machine-learning friendly format. Every object in my data, which will eventually become an instance in my dataset, has the exact same fields (in this case, ...
4
votes
1answer
1k views

Is there an asymmetric version of nominal correlation?

I use Cramer's V to calculate correlation of features in a dataset made of only nominal features. Let's consider the following dataset: ...
4
votes
4answers
1k views

How to visualise multidimensional categorical data with additional time dimension

I am trying to visualize a dataset that registers the career of people in an organisation and their backgrounds before starting in that particular organisation. I 'd like to show how individuals are ...
4
votes
3answers
2k views

What's the best way to use binned data in a tree-based model?

I have some numeric data that has come 'binned', but the bins are not of equal sizes in terms of scale or quantile For example, an age variable that is [0-16), [16-21), [21-30), [30-45), [45-65), [65,...
4
votes
2answers
3k views

One hot encoding large dataset

Initially, I have a dataset where for each row there is user_id and product_ids he bought. In that dataset there are 478191 products bought by different users. In order to discover frequent items ...
4
votes
4answers
3k views

Scikit Learn Missing Data - Categorical values

I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier) ...
4
votes
1answer
13k views

Categorical and ordinal feature data representation in regression analysis? [closed]

I am trying to fully understand difference between categorical and ordinal data when doing regression analysis. For now, what is clear: Categorical feature and data example: Color: red, white, black ...
4
votes
3answers
2k views

Dealing with a dataset with a mix of continuous and categorical variables

How do the choice of machine learning algorithm and preprocessing change when some of the independent variables are categorical while others are continuous? Can such data be directly applied to the ...
4
votes
2answers
3k views

Categorical data for sklearns Isolation Forrest

I'm trying to do anomaly detection with Isolation Forests (IF) in sklearn. Except for the fact that it is a great method of anomaly detection, I also want to use it because about half of my features ...
4
votes
2answers
130 views

Why Decision Tree Classifier is not working with categorical value?

I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx. I am trying to model a Decision Tree Classifier as part of an ...
4
votes
3answers
81 views

Bayesian combination of multi-dimensional experts?

I have what seems to me to be a slightly complex version of a decision tree problem, that can't figure out how to model, and I'm trying to avoid the "just dump it into an NN" solution. I have a ...
4
votes
3answers
3k views

Categorical Variables - Classification

I have a categorical variable, country which takes on values like India, US, Pakistan etc. I am currently using a linear SLM for a classification task. So my country value varies from 1-20. How ...
4
votes
3answers
9k views

Large categorical dataset for regression

I need to collect several large datasets (thousands of samples, dozens of features) for regression with only categorical inputs. I already look for such datasets in the UCI repository, but I did not ...
4
votes
1answer
1k views

Feature selection for data with both continuous and categorical features?

I am working on a classification problem with 4 ordinal classes to predict, labelling/predicting samples as either a number from 1-4. My training dataset has 284 features by ~40,000 samples and I am ...

1
2 3 4 5 6