Questions tagged [categorical-data]
Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.
391
questions
1
vote
1
answer
95
views
Anomaly detection on sparse categorical data
I have a big dataset with a column "clientid" and a categorical column "choice".
I want to find out what are the clients that have strange combinations of choices (less frequent ...
- 691
2
votes
1
answer
673
views
Why do my target labels need to begin at 0 for sparse categorical cross entropy to work?
I'm following a guide here to implement image segmentation in Keras.
One thing I'm confused about are these lines:
...
- 6,862
0
votes
1
answer
228
views
Problem with binning
I am trying to change continuous data points to categorical by using binning. I know two techniques, i) equal width bins ii) bins with equal number of elements.
My questions are:
Which type of ...
- 91
2
votes
1
answer
369
views
Categories with the same mean in target encoding
While doing target encoding it can happen that two categories have the same target mean.
This is bad because there will be no difference in the new feature in it and we will lose some information.
...
- 1
1
vote
1
answer
2k
views
Calculating correlation for categorical variables
I am struggling to find out a suitable way to calculate correlation coefficient for categorical variables. Pearson's coefficient is not supported for categorical ...
- 564
2
votes
1
answer
209
views
Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality
When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less ...
- 24.5k
0
votes
1
answer
156
views
Dealing with unseen data/categories in machine learning models for stream data
I want to build a machine learning model (xgb and lgbm) that has to handle streaming data on a weekly basis. The models are trained on a bi-weekly basis. The data includes order information and I want ...
- 1,859
1
vote
1
answer
77
views
Find correlation between grades from two raters [duplicate]
The question is whether we can find a correlation between two sets of grades (categorical data).
Let’s say we have a dog competition and there are 1000 dogs participating.
There are two rounds of ...
- 93
1
vote
0
answers
348
views
Partial Dependence Plot and categorical variables
While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables:
For each of the categories, we get ...
- 1,595
1
vote
0
answers
12
views
multi-observed features [closed]
I am working on a ML model where individual features may have a highly variable number of observed values.
The model will predict a continuous variable so I am planning to use a Regressor.
More ...
- 1,859
0
votes
1
answer
26
views
Determining which categorical data is beneficial in predictive modelling
I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
CommunityBot
- 1
0
votes
1
answer
20
views
Does produced data fall under the "data collection" category?
I am not sure whether data governance and policy is covered by this community. I'm giving it a try and let me know if I need to adjust.
While surveying all the data of my current employer, and ...
- 1,595
2
votes
0
answers
39
views
ML recommendation system with items organized in a tree
I would like to develop a recommendation system (probably hybrid, user-based and feature-based) for items which are organized in a tree (there are categories, divided in sub-categories, divided in sub-...
- 1,595
1
vote
1
answer
4k
views
Scikit Learn OneHotEncoded Features causing error in classifier
I’m trying to prepare data for input to a Decision Tree and Multinomial Naïve Bayes Classifier.
This is what my data looks like (pandas dataframe):
...
- 1,012
4
votes
4
answers
4k
views
Scikit Learn Missing Data - Categorical values
I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier)
...
- 1,012
3
votes
1
answer
107
views
Is there a name for a scale which mixes ordinal and nominal?
The textbooks I have differentiate between nominal, ordinal, interval and ratio scales. The ordinal scale is quite popular in the wild, used for basically all subjective data, and also for dividing ...
- 19.5k
0
votes
0
answers
65
views
Calculating Uncertainty for categorical predictions
I am wondering what is the best way to calculate the uncertainty for my categorical predictions. I have created a model that predicts what rating a movie is getting based on certain keywords and the ...
- 1
2
votes
1
answer
95
views
What is the best alternative for Fisher's Exact test for contigency tables that are NOT 2x2?
I am a newbie to data mining. I am trying to find associations between two categorical variables. Since more than 20% of my expected frequencies are less than 5, I wanted to use Fisher exact test but ...
- 161
1
vote
1
answer
31
views
How to deal with variable number of permutation invariant features?
I want to learn from data where each record has a variable number of features that have no inherent order to them. Take as an example the task to predict whether a repair is worth it of some item. ...
CommunityBot
- 1
0
votes
1
answer
34
views
For very simple linear regression can we quantify the prediction accuracy hit between using one hot encoding and simple numerical mapping?
Suppose I had a simple linear regression model that had the following input or X variable:
...
- 842
1
vote
0
answers
13
views
Very infrequent values in embedding layers
I have a categorical input that is very imbalanced. 90% of the values are either A or B and frequencies for C, D, E, F, etc are as little as 1.
I am using an embedding layer for this input and the ...
0
votes
1
answer
70
views
How to Present All Categories in All Samples
I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with ...
- 5,404
14
votes
10
answers
4k
views
How can I appropriately handle cleaning of gender data?
I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform an analysis of it.
In this dataset, the gender ...
- 5,776
1
vote
1
answer
684
views
unsupervised anomaly detection on sparse data
Given that I have a very sparse data matrix with continuous features, like this dataframe for example
...
CommunityBot
- 1
1
vote
1
answer
49
views
How to handle One Hot Encoded columns with changing categories in supervised ML Problem?
Scenario:
I have the following game data about participants, game and the winner in the following format:
...
- 5,776
0
votes
1
answer
152
views
What is the best way to encode an arbitrary collection of strings into int categorical variables?
I have a bunch of categorical labels which I want to transform into int categorical features for an ML algorithm.
The problem is I don't have a prior list of the categories, so that I can't just ...
CommunityBot
- 1
0
votes
1
answer
35
views
How would I approach training a model and encoding this categorical data
So I have the following data:
I have one series where each word has a value that describes the average review score that would get.
For example, if the word "excellent" showed up in reviews ...
- 488
3
votes
1
answer
420
views
"Binary Encoding" in "Decision Tree" / "Random Forest" Algorithms
Is it OK to use Binary Encoding in a dataset containing categorical columns with very high cardinalities?
Some facts about my dataset:
My dataset has ~170,000 rows
One of the categoric variables has ...
- 19.5k
0
votes
1
answer
2k
views
SMOTE-NC does not help to oversample my mixed continuous/categorical dataset
When I use SMOTE-NC to oversample three classes of a 4-class classification problem, the Prec, Recall, and F1 metrics for minority classes are still VERY low (~3%). I have 32 categorical and 30 ...
CommunityBot
- 1
1
vote
0
answers
33
views
How does the R implementation of RandomForest split nodes on categorical data?
The R implementation of RandomForest can take in categorical features as factors and train and predict on these features without encoding. Normally, I use the python implementation from scikit-learn ...
- 165
1
vote
2
answers
615
views
Is there a RandomForest implementation that handles categorical data without encoding in python?
I am working on a binary classification project with both continuous and categorical features. I know that the R implementation of RandomForest can handle categorical data passed in as factor type ...
- 174
5
votes
2
answers
5k
views
Feature Selection with one-hot-encoded categorical data
I have a dataset with 400+ columns. Almost 90% of these are categorical data with One-Hot-Encoding (OHE). I'm using the dataset for a classification problem.
My professors asked me to perform feature ...
- 31
0
votes
1
answer
258
views
How to perform SMOTE-N when there is no majority vote?
In the SMOTE paper, the authors present the logic of creating synthetic examples when all features are nominal (section 6.2, SMOTE-N):
To generate new minority class feature vectors, we can create ...
- 209
4
votes
1
answer
2k
views
Logic behind SMOTE-NC?
In the SMOTE paper here, the authors present the logic for creating synthetic examples when some of the features are nominal and some are continuous (section 6.1, SMOTE-NC).
This example is provided:
...
- 209
0
votes
1
answer
80
views
Categorical data - how to handle [closed]
Few questions on categorical data. Need suggestions / pointers:
How can we check for correlation between categorical features and target or between the features themselves?
How about correlation ...
CommunityBot
- 1
1
vote
0
answers
94
views
Endogenous and exogenous ordinal variables in R studio Lavaan
I am new to using R studio, so apologies for the basic question. I have run a number of Confirmatory Factor Analyses using the Lavaan package. Each questionnaire item is on a 4 point Likert scale 0 = ...
- 2,492
5
votes
1
answer
397
views
Extracting encoded features after CatBoost
I have a dataset containing numerical as well as categorical variables.
After I've fit my dataset to a CatBoostClassifier, I want to extract the entire feature set, with the categorical variables ...
- 884
0
votes
0
answers
82
views
looking for a correlation function for categorical + numerical datasets in matlab
I am looking for a function in Matlab that calculates the correlation coefficient of a data set that includes categorical + numerical data. I am quite surprised this isn't as straightforward as I ...
2
votes
1
answer
303
views
splitting mechanism with one hot encoded variables (tree based/boosting)
I am using xgboost and have a categorical unordered feature with 25 levels. So when i apply one hot encoding i have 25 columns. This introduces alot of sparsity. Even more unusual, my feature ...
- 10.8k
2
votes
1
answer
230
views
Dealing with categorical variables in regression problems which method to use?
Usually if I have regression problem and my initial dataset contains categorical variables like :
column 1:
Math
Science
Science
English
I would convert this ...
- 854
0
votes
1
answer
77
views
how to handle different categorical values in train and test dataset?
I have a dataset in which if i do
train_df["era"].value_counts()
then it will show 120 different type of categorical values
and then if i do
...
- 262
3
votes
1
answer
382
views
Average of importance gain for a categorical variable
Suppose I have a set of M categorical variables, some of them with a different number of categories (for instance, var1 has five categories, var2 has three, etc).
I train an XGBoost model on a numeric ...
CommunityBot
- 1
9
votes
2
answers
2k
views
Using NLP to automate the categorization of user description
I have a huge file of customer complaints about the products my company owns and I would like to do a data analysis on those descriptions and tag a category to each of them.
For example: I need to ...
- 997
26
votes
3
answers
11k
views
How to combine categorical and continuous input features for neural network training
Suppose we have two kinds of input features, categorical and continuous. The categorical data may be represented as one-hot code A, while the continuous data is just a vector B in N-dimension space. ...
- 997
3
votes
1
answer
131
views
Is there an encoder which can automatically detect the intrinsic order of an ordinal variable and assign values accordingly?
Given data with an ordinal variable, says "house quality" with values ex (excellent), gd (good), ...
- 5,921
1
vote
2
answers
257
views
How to find correlation between categorical data and continuous data
I'm working on imputing null values in the Titanic dataset.
The 'Embarked' column has some. I do NOT want to just set them all to the most common value, ...
- 1,869
11
votes
4
answers
27k
views
Clustering for mixed numeric and nominal discrete data
My data includes survey responses that are binary (numeric) and nominal / categorical. All responses are discrete and at individual level.
Data is of shape (n=7219, p=105).
Couple things:
I am ...
- 310
2
votes
2
answers
2k
views
Why is count encoding effective in improving accuracy? [duplicate]
Can someone please explain why/how Count encoding of categorical features improve accuracy in classification when compared to simply label encoding them ?
I found one explanation in kaggle " ...
- 2,246
0
votes
1
answer
2k
views
Categorical Variable and Target Variable
Though a similar question is answered here , but I wanted to take a different approach. Assuming that I have a binary target variable 1/0 and a categorical variable Gender M/F. From this, I can have a ...
CommunityBot
- 1
0
votes
1
answer
86
views
Is this attribute numeric or categorical (ordinal)? Help!
So I have this dataset I need to perform several techniques on as part of a data mining/machine learning project of some sort in PYTHON. There are a couple of features however, that have me very ...
- 6,525