Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
0
votes
0answers
19 views

Variable selection involving mixture of numerical, high cardinal,low cardinal features

Consider a dummy dataframe: A B C D …. Z 1 2 as we 2 2 4 qq rr 5 4 5 tz rc 9 This dataframe has 25 independent variables and one target variable ,the ...
2
votes
0answers
13 views

ML methods for prediction, using categorical variables and time

Most of the time series analysis tutorials/textbooks I found time series data, usually deal with continuous numerical variables. I am currently trying to solve a problem that deals with multivariate ...
2
votes
1answer
25 views

Extracting encoded features after CatBoost

I have a dataset containing numerical as well as categorical variables. After I've fit my dataset to a CatBoostClassifier, I want to extract the entire feature set, with the categorical variables ...
1
vote
0answers
19 views

why keras gives me desired results for my Entity Embedding but not pytorch?

I tried to build Entity Embeddings of categorical data from a dataset. I took a dataset - "Bike share”.This dataset shows number of bike share/rent/sales in every ...
0
votes
0answers
20 views

Should the type of Boolean categorical features be numerical or categorical after encoding?

There are categorical features which have two different value in my dataframe next to numerical features. I've converted these categorical values to 0 or 1. I will apply correalation elimination on ...
1
vote
3answers
27 views

Categorical variables with multiple entries transformed to entity embedding

I have structured data with lots (tens of thousads) of categories organized into columns. The goal is to enter the data into gradient boosting machine algorithm for a specific prediction. Some ...
2
votes
1answer
26 views

How to retrain a K-Modes model based on daily data?

I have read that retraining a model depends highly on what you are trying to achieve. I am conscious that maybe I need to retrain my model daily and after a certain time I have to train the model ...
2
votes
2answers
24 views

Classification when variables are in ranges

I want to classify my data and some of my variables are ranges. I classify location so for example, school, the hours that people are at school are from 7:00 to 14:00, some of my variables are ...
2
votes
2answers
41 views

How can I perform categorical encoding when the dataset is too large for memory?

I generally do preprocessing before fitting estimators using Scikit-Learn. My latest project is using significantly more data than I have used in the past, and whilst I know I can use online learning ...
2
votes
1answer
20 views

Different encoders applied to a dataset

I have a dataset which have both categorical features with high cardinality (>8000) and low cardinality (4 or 5). Would that be ok to encode the high cardinality ones with one encoder (target encoder,...
3
votes
0answers
16 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
2
votes
1answer
40 views

Logistic Regression Model for categorical features with multiple values in each category

I am working on an insurance use case to build a logistic regression classifier to predict if a policy will lapse or not. The dataset has more than 20 categorical features for a policy. Each ...
4
votes
2answers
68 views

Why Decision Tree Classifier is not working with categorical value?

I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx. I am trying to model a Decision Tree Classifier as part of an ...
0
votes
1answer
26 views

What is the best way to encode features when clustering data? [duplicate]

I have a dataset with numerical and categorical features. I am trying to run a k-means algorithm to find clusters of data. What is the best way to encode categorical features? I have been doing one ...
0
votes
2answers
83 views

Dummy encoding the categorical variables using the changed version of OneHotEncoder

This is my code, I was trying to dummy encode the first column of X using OneHotEncoder but it was showing error and the documentation page of OneHotEncoder says that it has been changed and I wasn't ...
2
votes
1answer
40 views

Strategies to encode categorical variables with many categories

I was going over the Kaggle competitions IEEE,Categorical Feature Encoding Challenge and one of the ways in which categorical variables have been handled is by replacing the variables by the ...
2
votes
5answers
61 views

How do you predict a continuous variable when all your independent variables are categorical

I am new to data science and ML. Recently I have been given a sales dataset which contains weekly sales of a fashion brand. It has information about product like category(t shirt, polo shirt, cotton ...
0
votes
2answers
21 views

How to handle different categorical embedding sizes in hold out data set

I have a pytorch tabular dataset with zip code as a categorical embedding. I'm getting great results on the test set. When I go to run my hold out sample through, it errors out because I have more ...
1
vote
3answers
69 views

Correlation between categorical variables based on the target distribution

Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way ...
9
votes
1answer
183 views

Why does frequency encoding work?

Frequency encoding is a widely used technique in Kaggle competitions, and many times proves to be a very reasonable way of dealing with categorical features with high cardinality. I really don't ...
0
votes
2answers
49 views

Encoding Categorical Data Without Increasing the Dimension

I've been exploring methods for encoding categorical data. I was hoping to find a good method that does not increase the dimension of the dataset, similar to the one used on this dataset about drug ...
0
votes
0answers
18 views

Categorial Encoding with different cardinality

I have some user data for his data activities. Some examples of the columns are : Activity : ex. youtbube , viber, whatsapp etc etc. Cardinality > 1000 Region : Area identifier Cardinality > 10000 ...
0
votes
1answer
47 views

Help making a custom categorical loss function in Keras

I am a bit new to machine learning, and I'm trying to get the basics working towards a bigger project using a very simple encoder-decoder model. It looks like this: ...
0
votes
1answer
36 views

Clustering categorical variable values based on continuous target values [closed]

Let's say I have $n$ data points with just one categorical feature $x$ and a continuous target variable $y$. I want to divide the possible values of $x$ into subsets such that the value of $y$ doesn't ...
2
votes
2answers
51 views

Dealing with categorical variables

I have a panel data set. My dependent variable is total costs, and almost all of my independent variables are categorical variables. For instance, age is "old","new". Now i have some questions. ...
-1
votes
1answer
150 views

How to find and calculate correlation in a data set which has category and continuous variables? [duplicate]

I am working on an Insurance domain use case to predict if an existing customer will buy a second insurance policy or not. I have a few personal details saved under different categories like Marital ...
0
votes
0answers
19 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
2
votes
1answer
44 views

Average of importance gain for a categorical variable

Suppose I have a set of M categorical variables, some of them with a different number of categories (for instance, var1 has five categories, var2 has three, etc). I train an XGBoost model on a numeric ...
0
votes
0answers
14 views

Categorical loss functions with similar properties to Kullback-Leibler loss function

When using the Kullback-Leibler divergence as loss function for predicting the probabilities of a categorical (multinomial) distribution, one of the properties is that the difference between $a$ and $...
0
votes
0answers
144 views

Frequency/Count encoding

How do I perform frequency/count encoding for a train and test set? The implementations of this encoding I've seen simply frequency encode the categorical variables on a particular dataset (no ...
2
votes
2answers
38 views

How to leverage description data in multi-class classification (dimensionality reduction)

I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as ...
1
vote
1answer
20 views

Turning Histogram values into Numerical format ( Excel-xslx, Pandas-DataFrame, etc.)

I am trying to do a correlation study about personality traits as described in Hofstede's :https://www.hofstede-insights.com/product/compare-countries/ . I would like to have the values described in ...
0
votes
0answers
5 views

Finding relevant pain points in feedbacks(open text)

I have employee feedback and need to find the appropriate pain points out of their feedback. Need help with the approach and analysis. I have provided a couple of examples below. Note: The feedbacks ...
0
votes
0answers
14 views

How to encode factor predictors in prediction models

The response variable as well as all predictor variables in my dataset are factors. I want to build a model for predicting the response variable. As I understand I have to first encode my predictor ...
0
votes
0answers
6 views

Include or exclude original features after encoding

I have some categorical features and they are encoded by different types of encoding (one-hot, label, target, etc). My question is whether you usually include the original categorical features with ...
-1
votes
1answer
55 views

Apply a clustering algorithm on categorical data with features of multiple values [duplicate]

Let us I have a people data like gender, age, marital status, education, employment, hobbies. I want to make clusters of those people, having some similarity/common among them (for example they have ...
0
votes
0answers
13 views

Recursive feature elimination on train data or complete dataset and dummy encoding

I am using RFE with logistic regression. I will also be doing cross validation with RFE (RFECV in sklearn) to get the optimum number of features. I am not sure whether to use RFECV on just train ...
2
votes
2answers
146 views

Should we use one hot encoder class in data having 2 as maximum numeric representation of categorical feature in each column?

I am testing the Play Golf data set using Decision tree classifier: I am splitting the data into Outlook, Temp, Humidity and windy as features, and ...
2
votes
2answers
44 views

Categorical features preprocessing for clustering

Can anyone tell suggest the best practice for clustering data with mixtured features (both with categorical and continuous). I am struggling with a problem; I realized that for all metrics algorithms ...
1
vote
1answer
44 views

Encode missing data and unseen data

Let's assume that I have a classification problem and all my features are categorical data. I have missing data (and I do not want to do any imputation). Also, I know that I will have some unseen ...
1
vote
0answers
9 views

Bayesian classification of “JSON” data

"Machine Learning over JSON" describes some issues surrounding the classification of JSON documents. Namely, Categorical Features Data is Hierarchical Missingness is Chunky The first two have fairly ...
1
vote
1answer
194 views

Naive Bayes for Categorical Features (Non Binary)

How do i use Naive Bayes Classifier (Using sklearn) for a Dataset considering that my feature set is categorical, ie more than 2 categories per feature are present. I've looked everywhere, some ...
3
votes
2answers
44 views

how to handle values that only appear once in a column?

Counting the values of a column using pandas I got the following result: ...
1
vote
1answer
28 views

What are the best practises to decide whether a variable is categorical?

What are some of the systematic ways to categorise variables into categorical or numeric? I believe using only intuition in such scenarios can many-a-times lead to major irreversible errors. What are ...
1
vote
1answer
29 views

Modeling the Price Movement- What analysis should be used

I am trying to model the price of a hotel as the check-in date arrives. I have a data set which looks like- For e.g- if I am looking at the booking date of Dec 31st, I would want to analyze the ...
0
votes
0answers
47 views

Categorical Multivariate Time Series

I have a small dataset of products of which the price varies along time. Each product is represented by categorical features mostly ( type, matter, use, location ...) and one or two scalar features ( ...
2
votes
0answers
57 views

binning high cardinality categorical features

one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently ...
0
votes
1answer
340 views

SMOTE-NC does not help to oversample my mixed continuous/categorical dataset

When I use SMOTE-NC to oversample three classes of a 4-class classification problem, the Prec, Recall, and F1 metrics for minority classes are still VERY low (~3%). I have 32 categorical and 30 ...
0
votes
0answers
68 views

Hotel Booking Analytics: Perform an analysis in order to understand the movement of the price as the day approaches the check-in date

I am working on a hotel booking dataset. I have transactional level booking data, where each row corresponds to a booking. Please refer to below snippet of the data: I am trying to find out the ...
1
vote
0answers
12 views

Poisson Model (w/ multiple levels X)

Question Is Poisson model the best method for predicting counts among multiple levels within nominal variable? Details Imagine data of 7000 observations, where output= Obs.Count {numeric,0,1,2..8} ...

1 2 3 4 5