Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
1
vote
0answers
12 views

Compare the variances of two categorical distributions in a repeated measure design

I ran two model-building procedures with different parameters on the same sample and obtained the selection of my optimized hyperparameter for each outer fold (each of the analyses had 100 outer folds ...
0
votes
2answers
847 views

How to handle different input sizes of an NN when One-Hot-Encoding a categorical input?

let's assume an input dataset that is a mix of categorical values and real values. When preprocessing this data into an appropriate NN input, OHE is recommended because it doesn't assume any order of ...
-1
votes
2answers
19 views

Categorical feature as output and perform a classification

I have a database in which the output feature Y is categorical, for example (oversimplification) ...
21
votes
2answers
21k views

Why do we need to discard one dummy variable?

I have learned that, for creating a regression model, we have to take care of categorical variables by converting them into dummy variables. As an example, if, in our data set, there is a variable ...
1
vote
1answer
41 views

Can we optimize regression problems that have categorical variables by encoding them if on the other hand we are inserting multicollinearity? [duplicate]

Can we optimize regression problems that have categorical variables by encoding them if, on the other hand, we are inserting multicollinearity?
1
vote
0answers
22 views

Variable selection involving mixture of numerical, high cardinal,low cardinal features

Consider a dummy dataframe: A B C D …. Z 1 2 as we 2 2 4 qq rr 5 4 5 tz rc 9 This dataframe has 25 independent variables and one target variable ,the ...
3
votes
2answers
87 views

How can I perform categorical encoding when the dataset is too large for memory?

I generally do preprocessing before fitting estimators using Scikit-Learn. My latest project is using significantly more data than I have used in the past, and whilst I know I can use online learning ...
1
vote
0answers
68 views

why keras gives me desired results for my Entity Embedding but not pytorch?

I tried to build Entity Embeddings of categorical data from a dataset. I took a dataset - "Bike share”.This dataset shows number of bike share/rent/sales in every ...
2
votes
1answer
1k views

Naive Bayes for Categorical Features (Non Binary)

How do i use Naive Bayes Classifier (Using sklearn) for a Dataset considering that my feature set is categorical, ie more than 2 categories per feature are present. I've looked everywhere, some ...
12
votes
2answers
6k views

Catboost Categorical Features Handling Options (CTR settings)?

I am working with a dataset with large number of categorical features (>80%) predicting a continuous target variable (i.e. Regression). I have been reading quite a bit about ways to handle categorical ...
0
votes
2answers
56 views

How to handle different categorical embedding sizes in hold out data set

I have a pytorch tabular dataset with zip code as a categorical embedding. I'm getting great results on the test set. When I go to run my hold out sample through, it errors out because I have more ...
1
vote
3answers
331 views

Categorical variables with multiple entries transformed to entity embedding

I have structured data with lots (tens of thousads) of categories organized into columns. The goal is to enter the data into gradient boosting machine algorithm for a specific prediction. Some ...
2
votes
2answers
49 views

Classification when variables are in ranges

I want to classify my data and some of my variables are ranges. I classify location so for example, school, the hours that people are at school are from 7:00 to 14:00, some of my variables are ...
2
votes
1answer
55 views

Different encoders applied to a dataset

I have a dataset which have both categorical features with high cardinality (>8000) and low cardinality (4 or 5). Would that be ok to encode the high cardinality ones with one encoder (target encoder,...
1
vote
3answers
3k views

What should I use if I have millions of possible values for a feature in a sklearn predictive model?

I am trying to create a large model. One of the features is categorical, and it has almost 100 million entries. I have looked at sklearn LabelEncoder, but I am concerned that it will still create an ...
3
votes
0answers
20 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
4
votes
1answer
2k views

Is there an asymmetric version of nominal correlation?

I use Cramer's V to calculate correlation of features in a dataset made of only nominal features. Let's consider the following dataset: ...
4
votes
2answers
187 views

Why Decision Tree Classifier is not working with categorical value?

I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx. I am trying to model a Decision Tree Classifier as part of an ...
0
votes
1answer
223 views

What is the best way to encode features when clustering data? [duplicate]

I have a dataset with numerical and categorical features. I am trying to run a k-means algorithm to find clusters of data. What is the best way to encode categorical features? I have been doing one ...
2
votes
2answers
8k views

Checking Correlation of Categorical variables in SPSS

I am building a predictive model for a classification problem using SPSS. Of the Independent variables, I have both Continuous and Categorical variables. SPSS gives only correlation between continuous ...
0
votes
2answers
97 views

Encoding Categorical Data Without Increasing the Dimension

I've been exploring methods for encoding categorical data. I was hoping to find a good method that does not increase the dimension of the dataset, similar to the one used on this dataset about drug ...
1
vote
3answers
240 views

Correlation between categorical variables based on the target distribution

Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way ...
12
votes
1answer
1k views

Why does frequency encoding work?

Frequency encoding is a widely used technique in Kaggle competitions, and many times proves to be a very reasonable way of dealing with categorical features with high cardinality. I really don't ...
0
votes
1answer
385 views

Help making a custom categorical loss function in Keras

I am a bit new to machine learning, and I'm trying to get the basics working towards a bigger project using a very simple encoder-decoder model. It looks like this: ...
1
vote
3answers
3k views

Transformation of categorical variables

I have a data with continous variables and categorical variables. I am using Random Forest and have made my continues variables Gaussian by transformation and have standardized it. Should categorical ...
-1
votes
1answer
348 views

How to find and calculate correlation in a data set which has category and continuous variables? [duplicate]

I am working on an Insurance domain use case to predict if an existing customer will buy a second insurance policy or not. I have a few personal details saved under different categories like Marital ...
0
votes
1answer
117 views

Clustering categorical variable values based on continuous target values [closed]

Let's say I have $n$ data points with just one categorical feature $x$ and a continuous target variable $y$. I want to divide the possible values of $x$ into subsets such that the value of $y$ doesn't ...
3
votes
2answers
141 views

Dealing with categorical variables

I have a panel data set. My dependent variable is total costs, and almost all of my independent variables are categorical variables. For instance, age is "old","new". Now i have some questions. ...
2
votes
0answers
278 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
1
vote
1answer
21 views

Turning Histogram values into Numerical format ( Excel-xslx, Pandas-DataFrame, etc.)

I am trying to do a correlation study about personality traits as described in Hofstede's :https://www.hofstede-insights.com/product/compare-countries/ . I would like to have the values described in ...
-2
votes
1answer
338 views

Apply a clustering algorithm on categorical data with features of multiple values [duplicate]

Let us I have a people data like gender, age, marital status, education, employment, hobbies. I want to make clusters of those people, having some similarity/common among them (for example they have ...
2
votes
2answers
1k views

Should we use one hot encoder class in data having 2 as maximum numeric representation of categorical feature in each column?

I am testing the Play Golf data set using Decision tree classifier: I am splitting the data into Outlook, Temp, Humidity and windy as features, and ...
2
votes
2answers
218 views

Categorical features preprocessing for clustering

Can anyone tell suggest the best practice for clustering data with mixtured features (both with categorical and continuous). I am struggling with a problem; I realized that for all metrics algorithms ...
3
votes
2answers
7k views

How to implement feature selection for categorical variables (especially with many categories)?

I've been trying to get some ideas of how I could treat categorical variables when doing feature selection. Mainly I've been running Random Forest feature importance on Python for which preprocessing ...
6
votes
3answers
9k views

How to deal with categorical feature of very high cardinality?

I would like to train a binary classifier on feature vectors. One of the features is categorical feature with string, it is the zip codes of a country. Typically, there is thousands of zip codes, and ...
1
vote
0answers
49 views

Bayesian classification of “JSON” data

"Machine Learning over JSON" describes some issues surrounding the classification of JSON documents. Namely, Categorical Features Data is Hierarchical Missingness is Chunky The first two have fairly ...
3
votes
2answers
47 views

how to handle values that only appear once in a column?

Counting the values of a column using pandas I got the following result: ...
1
vote
1answer
46 views

What are the best practises to decide whether a variable is categorical?

What are some of the systematic ways to categorise variables into categorical or numeric? I believe using only intuition in such scenarios can many-a-times lead to major irreversible errors. What are ...
7
votes
1answer
414 views

How to deal with missing data for only some categories

Or in other words, data for category A is irrelevant for category B. So it is not present, how can imputing missing data distort/effect learning models broadly. I can't find any logic how to deal ...
3
votes
5answers
5k views

Large no of categorical variables with large no of categories

I'm working on a binary classification problem where the dataset is slightly imbalanced (30% class 0 | 70% class 1). Most of my features are categorical with large number of categories. For example: ...
1
vote
0answers
15 views

Poisson Model (w/ multiple levels X)

Question Is Poisson model the best method for predicting counts among multiple levels within nominal variable? Details Imagine data of 7000 observations, where output= Obs.Count {numeric,0,1,2..8} ...
0
votes
0answers
32 views

The Difference between One Hot Encoding and LabelEncoder? [duplicate]

I am working on a ML problem to predict house prices and Zip Code is one feature which will be useful. I am also trying to use ...
1
vote
1answer
115 views

Linear regression model with (categorical) predictor variables

I used LM model with (categorical) predictor variables on my data in r like this (I have count variable as dependent/target variable): ...
1
vote
1answer
466 views

How to handle large number of categories in a dataset?

I have one dataset of "Books" which contains 8 columns initially and out of which 3 of them contains text values which can be categorized. The 3 columns contains "Language-code", "Author Name" and "...
1
vote
2answers
180 views

Scikit alternative for categorical data modeling?

So, sklearn doesn't support categorical data in its models. Is there a known alternative for categorical data modeling (such as random forests, etc.) for Python?
1
vote
0answers
120 views

Measuring the similarity between a numeric data matrix and one or more categorical variables?

Given a numeric data matrix $A$ of size $n \times p$, which each row represents an observation along $p$ variables, and a second categorical data matrix $M$ of size $n \times z$, where each row ...
2
votes
0answers
52 views

Why don't Target/LeaveOneOut Encoders work well for Regression Tasks?

In this review of categorical encoding, it states early on that For regression tasks, Target and LeaveOneOut probably won’t work well and later repeats that Target/LeaveOneOut (Owen Zhang's ...
6
votes
1answer
131 views

Mapping of categorical features into binary indicator features

I am currently reading an introductory machine learning book by Daumé (ch. 03, p. 30) and when discussing the mapping of categorical features with "n" possible values into "n" binary indicator ...
3
votes
3answers
4k views

Purpose of converting continuous data to categorical data

I was reading through a notebook tutorial working with the Titanic dataset, linked here, and noticed that they highly favored ordinal data to continuous data. For example, they converted both the Age ...
1
vote
1answer
1k views

High train and val results. Bad test and predict results

For my thesis project I've been trying to make a CNN for some challenging data. There's four classes with the following amount of images respectively [410, 410, 269, 206] = 1,295 total. Now I know ...

1 2 3
4
5
7