Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

106 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6 votes
3 answers
322 views

Anomaly detection using clustering of highly correlated Categorical data

My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else it's Anomaly. Likewise, there are thousands ...
viral kapadia's user avatar
5 votes
0 answers
2k views

Dealing with categorical variables in Isolation Forest

Isolation Forest is widely used when dealing with outlier/anomaly detection when we have no labels. The theory behind is that making random split at random points and counting how many splits you do ...
Carlos Mougan's user avatar
3 votes
0 answers
23 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
dmrak's user avatar
  • 31
2 votes
0 answers
530 views

Library for Phi correlation coefficient in python?

I want to calculate correlation b/w categorical features in my data. I reviewed the literature and found phi coefficient can be used for this purpose. I found one library called phik in python enter ...
As13's user avatar
  • 75
2 votes
1 answer
500 views

Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?

I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
2much2code's user avatar
2 votes
0 answers
46 views

(R) can I convert a categorical variable into a numeric equivalent in linear regression to predict a continuous variable?

Specifically, I have an item code as one of the independent variables that can have several hundred possible values results in underfitting when predicting the projected availability of that item. I'd ...
Stephen's user avatar
  • 21
2 votes
0 answers
242 views

How to get the maximum likelihood estimate of the categorical distribution parameters using Lagrange optimization?

Let's say our data is discrete-valued and belongs to one of $K$ classes. The underlying probability distribution is assumed to be a categorical/multinoulli distribution given as $p(\textbf{x}) = \...
Shashank Kumar's user avatar
2 votes
0 answers
86 views

Regression dataset with categorical features

I have thought of a regression technique that I want to try on several datasets. I would like these datasets to have the following properties: Be a tabular dataset (no images). Have at least 20k rows,...
David Masip's user avatar
  • 6,051
2 votes
2 answers
1k views

Clustering mixed data types - numeric, categorical, arrays, and text

I have a dataset with 4 types of data columns: ...
Malki's user avatar
  • 121
2 votes
0 answers
40 views

ML recommendation system with items organized in a tree

I would like to develop a recommendation system (probably hybrid, user-based and feature-based) for items which are organized in a tree (there are categories, divided in sub-categories, divided in sub-...
Corlin's user avatar
  • 27
2 votes
1 answer
81 views

What to do if a specific label of a category appears only a few times?

Let's say I am trying to predict whether a car will be auctioned or not (not what I'm actually trying to do, but it represents it pretty well) using tabular data. I have the year the car was made, its ...
user avatar
2 votes
1 answer
185 views

How to retrain a K-Modes model based on daily data?

I have read that retraining a model depends highly on what you are trying to achieve. I am conscious that maybe I need to retrain my model daily and after a certain time I have to train the model ...
davidaap's user avatar
  • 121
2 votes
1 answer
283 views

Logistic Regression Model for categorical features with multiple values in each category

I am working on an insurance use case to build a logistic regression classifier to predict if a policy will lapse or not. The dataset has more than 20 categorical features for a policy. Each ...
tanmay's user avatar
  • 173
2 votes
0 answers
677 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
daanvdn's user avatar
  • 21
2 votes
2 answers
138 views

How to leverage description data in multi-class classification (dimensionality reduction)

I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as ...
Nick Bohl's user avatar
2 votes
0 answers
72 views

Why don't Target/LeaveOneOut Encoders work well for Regression Tasks?

In this review of categorical encoding, it states early on that For regression tasks, Target and LeaveOneOut probably won’t work well and later repeats that Target/LeaveOneOut (Owen Zhang's ...
Alexander Wang's user avatar
2 votes
1 answer
190 views

How to deal with a potencially multiple categorical variable

I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring....
Diogo Santos's user avatar
2 votes
1 answer
4k views

How to plot a heatmap-like plot for categorical features?

I would greatly appreciate let me know how to plot a heatmap-like plot for categorical features? In fact, based on this post, the association between categorical ...
ebrahimi's user avatar
  • 1,307
2 votes
1 answer
30 views

Individual differences in response time (RT) experiment - searching for the right test

Given a distribution of response times to different categorical variables, what's the best way to test for individual differences? or more specifically: There are 100 people pressing buttons in 10 ...
Sharonio's user avatar
2 votes
0 answers
1k views

How to choose the optimal k in k-protoypes?

To analyze a dataset from banking I have both numerical and categorical values. I transform them to analyze with k-prototypes. The original dataset: The modified dataset: E.g.: Job (for 1 to 12 '...
Sergi F.'s user avatar
2 votes
0 answers
103 views

Type of Test to Determine Correlation in R

I have a dataset of approximately 48,000 rows each one a click of a an article, some of these clicks were also comments. For each article I have the country and subject of the article and name of ...
Jonathan Dine's user avatar
2 votes
0 answers
70 views

Tag categorizer

I'm using Google prediction API and I need to perform tag categorization for free text. So when I receive a text like "Cant update application" is tagged with #Apps #General #CantUpdate. My ...
gogasca's user avatar
  • 749
1 vote
1 answer
240 views

Encode categorical data for unsupervised learning

What is the best encoder for categorical data in unsupervised learning? I am using unsupervised learning on mixed data (such as K-means). Before running my unsupervised algorithm, I am using dimension ...
Julien PETOT's user avatar
1 vote
1 answer
526 views

How to predict on data that is label encoded as end user will input a categorical data?

My dataset contains about 29 features with 3 class labels as result. Among these 29 features around 24 features are categorical i cannot transform each category into numbers as there are many more ...
Muhammad Minhas's user avatar
1 vote
0 answers
46 views

Intuition behind catboost encoding techniques

Can anyone please help me in understanding the effect of various bucketing techniques used in CatBoost Algorithm for categorical features? Like there is border, buckets, binarized target mean, counter ...
Mimansa Maheshwari's user avatar
1 vote
0 answers
23 views

What methods I could use to analyze the contingency table?

I am data science beginner, and I have a question about methods that I could use to analyze the following data. It is a simple case, I am trying to check the influence of cohabitation before marriage ...
Karpi's user avatar
  • 11
1 vote
0 answers
226 views

Encode each comma separated value in Pandas

I have a dataset ...
spd's user avatar
  • 119
1 vote
0 answers
18 views

Model a classification problem with multiple categorical varialbes as input features only. Diff Model performance

I'm having an input data with 100k rows, 8 input features, I'm trying to predict y (binary 1/0). But all the X are categorical variables(strictly nominal variables, not ordinal). Some with 8 levels, ...
Martin's user avatar
  • 11
1 vote
0 answers
134 views

Handling date and time fields for classification task

I'm working on a classification task(The dataset is 400,000 rows and 30 columns) and one of my features was date-time. I've extracted the month, day of the week, and hour from the dataset (year is a ...
insomniac's user avatar
1 vote
1 answer
27 views

How to test whether data is clustered wrt. subcategories?

I have a dataset of about 2000 entries, containing two numerical values, one categorical and one sub-categorical label for each entry. The data is from chemistry lab data, but for the purpose of this ...
user avatar
1 vote
0 answers
46 views

Why is a column detected as text and not as categorical when opening excel file?

I have the following values in an excel sheet Aguascalientes Baja California Baja California Sur Campeche Chiapas Chihuahua Ciudad de México Coahuila Colima Durango Estado de México Guanajuato ...
Gabriel Barrera's user avatar
1 vote
0 answers
89 views

When doing feature selection, are features like "year", "month" considered as ordinal features or should I convert them to strings?

I am working on a hotel reservation dataset that has both categorical and numerical (continuous and discrete) features (26 columns, 30244 rows). Target is also categorical and it says if the user &...
leahnanno's user avatar
1 vote
1 answer
98 views

An Unsupervised learning method suitable for large categorical data sets

I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
HoonP's user avatar
  • 11
1 vote
1 answer
43 views

How to build multiple variable regression having a mix of numerical & categorical features?

There is a need to estimate Annual Average Daily Traffic Volume (AADT). We have bunch of data about vehicles' speeds during several years. It is noticed that AADT depends on the average number of such ...
Артём Ощепков's user avatar
1 vote
1 answer
42 views

How to get correlation between the categories of two categorical variable?

I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua&...
bonaqua's user avatar
  • 11
1 vote
0 answers
30 views

Feature Selection Statistical Test for Nominal Response Vs Continuous Predictors? (in R)

I cannot find much information on this, none so far useful. I have a sparse data set with 17K+ columns of continuous gene expressions, an example of a typical column: (3.15, 0, 7.1294, 0, 0, 0, 2300.2,...
mnsml's user avatar
  • 11
1 vote
0 answers
15 views

Very infrequent values in embedding layers

I have a categorical input that is very imbalanced. 90% of the values are either A or B and frequencies for C, D, E, F, etc are as little as 1. I am using an embedding layer for this input and the ...
Hiro Nakagame's user avatar
1 vote
1 answer
35 views

How to deal with variable number of permutation invariant features?

I want to learn from data where each record has a variable number of features that have no inherent order to them. Take as an example the task to predict whether a repair is worth it of some item. ...
TomTom's user avatar
  • 111
1 vote
0 answers
48 views

How does the R implementation of RandomForest split nodes on categorical data?

The R implementation of RandomForest can take in categorical features as factors and train and predict on these features without encoding. Normally, I use the python implementation from scikit-learn ...
David Stein's user avatar
1 vote
0 answers
138 views

Endogenous and exogenous ordinal variables in R studio Lavaan

I am new to using R studio, so apologies for the basic question. I have run a number of Confirmatory Factor Analyses using the Lavaan package. Each questionnaire item is on a 4 point Likert scale 0 = ...
Jayne's user avatar
  • 11
1 vote
2 answers
256 views

Model for predicting duration based on categorical data

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
Kadin's user avatar
  • 111
1 vote
0 answers
398 views

Partial Dependence Plot and categorical variables

While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables: For each of the categories, we get ...
Carlos Mougan's user avatar
1 vote
2 answers
1k views

What are some good methods to forecast future revenue on categorical and value based data?

I have monthly snapshots (3 years) of all the contract data. It includes following information: Contract status [Categorical]: Proposed, tracked, submitted, won, lost, etc Contract stages [...
RWS's user avatar
  • 11
1 vote
0 answers
118 views

Using a Subset of Categories in a Categorical Column

I have a XGBoost model and I'm going to retrain it by adding new features. There is a column in my data and it's about professions of the customers. It has 60 categories. I suppose there is no need to ...
tkarahan's user avatar
  • 422
1 vote
0 answers
336 views

Implementing Scikit Learn's FeatureHasher for High Cardinality Categorical Data

Background: I am working on a binary classification of health insurance claims. The data I am working with has approximately 1 million rows and a mix of numeric features and categorical features (all ...
Jacob Niederer's user avatar
1 vote
0 answers
13 views

Compare the variances of two categorical distributions in a repeated measure design

I ran two model-building procedures with different parameters on the same sample and obtained the selection of my optimized hyperparameter for each outer fold (each of the analyses had 100 outer folds ...
Johannes Wiesner's user avatar
1 vote
2 answers
943 views

Aggregating multiple encoded categorical values

I am trying find commonly used techniques when dealing with high cardinality multi-valued categorical variables. I am currently using a dataset with a feature CATEGORY which has a cardinality of ~20,...
Vishwa Kalyanaraman's user avatar
1 vote
0 answers
28 views

Variable selection involving mixture of numerical, high cardinal,low cardinal features

Consider a dummy dataframe: A B C D …. Z 1 2 as we 2 2 4 qq rr 5 4 5 tz rc 9 This dataframe has 25 independent variables and one target variable ,the ...
Chinti's user avatar
  • 19
1 vote
0 answers
89 views

why keras gives me desired results for my Entity Embedding but not pytorch?

I tried to build Entity Embeddings of categorical data from a dataset. I took a dataset - "Bike share”.This dataset shows number of bike share/rent/sales in every ...
Akshay Tiwari's user avatar
1 vote
0 answers
25 views

Poisson Model (w/ multiple levels X)

Question Is Poisson model the best method for predicting counts among multiple levels within nominal variable? Details Imagine data of 7000 observations, where output= Obs.Count {numeric,0,1,2..8} ...
OctoCatKnows's user avatar