Questions tagged [categorical-data]
Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.
391
questions
2
votes
2
answers
711
views
Appropriate loss function for multi-hot output vectors
I have some data in which model inputs and outputs (which are the same size) belong to multiple classes concurrently. A single input or output is a vector of zeros somewhere between one and four ...
- 1,490
2
votes
1
answer
67
views
is there an adequate number of levels of categorical variables?
I have a project that I'm working on. The dataset contains many categorical variables and some of them have too many levels (+100). My question is : is there any advice to know the "adequate"...
- 24.5k
2
votes
0
answers
70
views
Regression dataset with categorical features
I have thought of a regression technique that I want to try on several datasets. I would like these datasets to have the following properties:
Be a tabular dataset (no images).
Have at least 20k rows,...
- 5,921
5
votes
0
answers
2k
views
Dealing with categorical variables in Isolation Forest
Isolation Forest is widely used when dealing with outlier/anomaly detection when we have no labels. The theory behind is that making random split at random points and counting how many splits you do ...
- 5,776
1
vote
1
answer
222
views
Does converting continuous variable to discrete(categorical) variable increases accuracy of a tree based model?
I've read other questions regarding if a continuous feature should be converted to categorical or not. But I'm interested in case of tree based classifiers such as Decision Tree, Random Forest, ...
- 10.8k
0
votes
2
answers
144
views
Categorical and non-categorical data in the same column
I have a unique dataset that has many columns and most columns contain both categorical and non-categorical data. For example, let's say that one column is attribute_1 and for observations that have ...
- 5,897
1
vote
1
answer
915
views
WHY or WHEN to convert numeric data to a categorical data?
This is an open ended WHY TO or WHEN TO question rather than a question on HOW TO encode numeric to categorical data.
I am currently working on Telco Customer Churn dataset from kaggle.
This is ...
- 19.5k
0
votes
0
answers
18
views
Model performance metrics
I have a dataset with multiple numeric input values and a categorical output. How can I measure model performance with different algorithms. As the results are categorical, we can not obtain r squared ...
- 37
0
votes
1
answer
25
views
How to pre-process the name String of a customer?
I implement logistic regression to predict if a customer is a business or a non-business customer with the help of TensorFlow in Python. I have several feature candidates like name, street, zip, ...
- 241
1
vote
2
answers
291
views
Dummy encoding the categorical variables using the changed version of OneHotEncoder [duplicate]
This is my code, I was trying to dummy encode the first column of X using OneHotEncoder but it was showing error and the documentation page of OneHotEncoder says that it has been changed and I wasn't ...
CommunityBot
- 1
1
vote
1
answer
879
views
sklearn serialize label encoder for multiple categorical columns
I have a model with several categorical features that need to be converted to numeric format. I am using a combination of LabelEncoder and OneHotEncoder to achieve this.
Once in production, I need to ...
- 10.8k
4
votes
1
answer
2k
views
Strategies to encode categorical variables with many categories
I was going over the Kaggle competitions IEEE,Categorical Feature Encoding Challenge and one of the ways in which categorical variables have been handled is by replacing the variables by the ...
CommunityBot
- 1
0
votes
1
answer
37
views
Data analysis PCA
I have a question about the functioning of PCA. I have a dataset with only 2 categorical attributes out of 9. Is it good to calculate pca between those two? Does it help me understanding anything ...
- 784
1
vote
2
answers
118
views
Clustering on categorical attributes
I have a dataset with only 2 categorical attributes out of 9. How can I get a clustering analysis on it?
I am using R. Do you have any advices about instructions, how to do it, topics, ...?
here's my ...
- 688
4
votes
3
answers
3k
views
Dealing with a dataset with a mix of continuous and categorical variables
How do the choice of machine learning algorithm and preprocessing change when some of the independent variables are categorical while others are continuous? Can such data be directly applied to the ...
CommunityBot
- 1
1
vote
1
answer
113
views
Handling Numerical Categorical Column in ML models in Python
When I was exploring the titanic dataset to estimate the probability of a person of surviving using the Logistic Model, I realized there are two ways of handling numerical categorical variables :
Use ...
- 146
1
vote
1
answer
175
views
LSTM Time-series classification - derived feature
I have a time-series dataset and I want to derive a new feature based on a date column which I believe might improve my predictive model. The feature is if it's weekday or weekend.
I am not sure how ...
- 371
3
votes
1
answer
209
views
binning high cardinality categorical features
one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently ...
CommunityBot
- 1
1
vote
2
answers
4k
views
what should i do if my target variable is categorical when using decision tree? (many categorical variables)
all,
i'm trying to classify a set of features to belong to a particular company (my dependent variable). my independent variables are a mixture of continuous and categorical features.
my data-set ...
- 5,404
1
vote
0
answers
84
views
Using a Subset of Categories in a Categorical Column
I have a XGBoost model and I'm going to retrain it by adding new features. There is a column in my data and it's about professions of the customers. It has 60 categories. I suppose there is no need to ...
- 422
1
vote
0
answers
294
views
Implementing Scikit Learn's FeatureHasher for High Cardinality Categorical Data
Background: I am working on a binary classification of health insurance claims. The data I am working with has approximately 1 million rows and a mix of numeric features and categorical features (all ...
0
votes
1
answer
681
views
tensorflow categorical data with vocabulary list - Expected binary or Unicode string, got [0,1,2,...]
I'm brand new to machine learning (having just completed the google machine learning crash course) and thought it would be good to try my hand at a Kaggle competition as a good starter to some real ...
- 7,207
1
vote
1
answer
330
views
A Loss of 55.2164 with a sparse_categorical_crossentropy in a sequential neural network?
I'm following Aurélion Géron's book on Machine Learning.
The following code tries to evaluate a neural network with a sparse categorical cross entropy loss function, on the Fashion Mnist data set. ...
- 6,862
1
vote
0
answers
13
views
Compare the variances of two categorical distributions in a repeated measure design
I ran two model-building procedures with different parameters on the same sample and obtained the selection of my optimized hyperparameter for each outer fold (each of the analyses had 100 outer folds ...
- 141
0
votes
2
answers
1k
views
How to handle different input sizes of an NN when One-Hot-Encoding a categorical input?
let's assume an input dataset that is a mix of categorical values and real values. When preprocessing this data into an appropriate NN input, OHE is recommended because it doesn't assume any order of ...
CommunityBot
- 1
-1
votes
2
answers
28
views
Categorical feature as output and perform a classification
I have a database in which the output feature Y is categorical, for example (oversimplification)
...
- 1,355
25
votes
2
answers
27k
views
Why do we need to discard one dummy variable?
I have learned that, for creating a regression model, we have to take care of categorical variables by converting them into dummy variables. As an example, if, in our data set, there is a variable ...
- 10.8k
1
vote
1
answer
111
views
Can we optimize regression problems that have categorical variables by encoding them if on the other hand we are inserting multicollinearity? [duplicate]
Can we optimize regression problems that have categorical variables by encoding them if, on the other hand, we are inserting multicollinearity?
- 19.5k
1
vote
0
answers
27
views
Variable selection involving mixture of numerical, high cardinal,low cardinal features
Consider a dummy dataframe:
A B C D …. Z
1 2 as we 2
2 4 qq rr 5
4 5 tz rc 9
This dataframe has 25 independent variables and one target variable ,the ...
- 19
3
votes
2
answers
303
views
How can I perform categorical encoding when the dataset is too large for memory?
I generally do preprocessing before fitting estimators using Scikit-Learn. My latest project is using significantly more data than I have used in the past, and whilst I know I can use online learning ...
- 1,714
1
vote
0
answers
85
views
why keras gives me desired results for my Entity Embedding but not pytorch?
I tried to build Entity Embeddings of categorical data from a dataset.
I took a dataset - "Bike share”.This dataset shows number of bike share/rent/sales in every ...
- 1,154
4
votes
1
answer
2k
views
Naive Bayes for Categorical Features (Non Binary)
How do i use Naive Bayes Classifier (Using sklearn) for a Dataset
considering that my feature set is categorical, ie more than 2 categories per feature are present.
I've looked everywhere, some ...
- 116
12
votes
2
answers
7k
views
Catboost Categorical Features Handling Options (CTR settings)?
I am working with a dataset with large number of categorical features (>80%) predicting a continuous target variable (i.e. Regression). I have been reading quite a bit about ways to handle categorical ...
CommunityBot
- 1
0
votes
2
answers
129
views
How to handle different categorical embedding sizes in hold out data set
I have a pytorch tabular dataset with zip code as a categorical embedding. I'm getting great results on the test set. When I go to run my hold out sample through, it errors out because I have more ...
- 23
2
votes
3
answers
577
views
Categorical variables with multiple entries transformed to entity embedding
I have structured data with lots (tens of thousads) of categories organized into columns. The goal is to enter the data into gradient boosting machine algorithm for a specific prediction.
Some ...
- 23
2
votes
2
answers
226
views
Classification when variables are in ranges
I want to classify my data and some of my variables are ranges.
I classify location so for example, school, the hours that people are at school are from 7:00 to 14:00, some of my variables are ...
- 5,260
2
votes
1
answer
140
views
Different encoders applied to a dataset
I have a dataset which have both categorical features with high cardinality (>8000) and low cardinality (4 or 5).
Would that be ok to encode the high cardinality ones with one encoder (target encoder,...
- 5,776
1
vote
3
answers
3k
views
What should I use if I have millions of possible values for a feature in a sklearn predictive model?
I am trying to create a large model. One of the features is categorical, and it has almost 100 million entries.
I have looked at sklearn LabelEncoder, but I am concerned that it will still create an ...
- 1,941
3
votes
0
answers
22
views
Why RANDOM noise images always predicted as BIRD?
Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%.
However when feeding into 5000 random noise images (Gaussian noise with the ...
- 31
4
votes
1
answer
2k
views
Is there an asymmetric version of nominal correlation?
I use Cramer's V to calculate correlation of features in a dataset made of only nominal features.
Let's consider the following dataset:
...
4
votes
2
answers
436
views
Why Decision Tree Classifier is not working with categorical value?
I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx.
I am trying to model a Decision Tree Classifier as part of an ...
- 5,404
0
votes
1
answer
559
views
What is the best way to encode features when clustering data? [duplicate]
I have a dataset with numerical and categorical features. I am trying to run a k-means algorithm to find clusters of data.
What is the best way to encode categorical features?
I have been doing one ...
- 1,773
2
votes
2
answers
10k
views
Checking Correlation of Categorical variables in SPSS
I am building a predictive model for a classification problem using SPSS. Of the Independent variables, I have both Continuous and Categorical variables. SPSS gives only correlation between continuous ...
CommunityBot
- 1
0
votes
2
answers
205
views
Encoding Categorical Data Without Increasing the Dimension
I've been exploring methods for encoding categorical data. I was hoping to find a good method that does not increase the dimension of the dataset, similar to the one used on this dataset about drug ...
- 2,981
1
vote
3
answers
289
views
Correlation between categorical variables based on the target distribution
Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way ...
CommunityBot
- 1
0
votes
1
answer
509
views
Help making a custom categorical loss function in Keras
I am a bit new to machine learning, and I'm trying to get the basics working towards a bigger project using a very simple encoder-decoder model. It looks like this:
...
- 1,093
1
vote
3
answers
3k
views
Transformation of categorical variables
I have a data with continous variables and categorical variables. I am using Random Forest and have made my continues variables Gaussian by transformation and have standardized it. Should categorical ...
-1
votes
1
answer
465
views
How to find and calculate correlation in a data set which has category and continuous variables? [duplicate]
I am working on an Insurance domain use case to predict if an existing customer will buy a second insurance policy or not. I have a few personal details saved under different categories like Marital ...
- 5,260
0
votes
1
answer
150
views
Clustering categorical variable values based on continuous target values [closed]
Let's say I have $n$ data points with just one categorical feature $x$ and a continuous target variable $y$. I want to divide the possible values of $x$ into subsets such that the value of $y$ doesn't ...
- 5,260
3
votes
2
answers
305
views
Dealing with categorical variables
I have a panel data set. My dependent variable is total costs, and almost all of my independent variables are categorical variables. For instance, age is "old","new". Now i have some questions.
...
- 609