Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
2
votes
1answer
63 views

Clustering with Only Categorical Features

I am trying to do clustering with a bunch (24) of categorical features. I have done some research and found a lot of people recommending something such as K-Modes. I tried running K-Modes on my data ...
12
votes
3answers
24k views

How to convert categorical data to numerical data in Pyspark

I am using Ipython notebook to work with pyspark applications. I have a CSV file with lots of categorical columns to determine whether the income falls under or over the 50k range. I would like to ...
1
vote
1answer
47 views

one hot encoding target variable in tree and non tree (knn) methods

I am learning about label encoders, one hot encoding etc applied to datasets for classification via KNN and XGBoost type trees. However, I am a bit confused as to whether the target variable should be ...
2
votes
1answer
27 views

Individual differences in response time (RT) experiment - searching for the right test

Given a distribution of response times to different categorical variables, what's the best way to test for individual differences? or more specifically: There are 100 people pressing buttons in 10 ...
2
votes
1answer
130 views

How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline

Let us say we have a dataset with a feature such as Surname. arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith'] And I want to encode this categorical info ...
1
vote
2answers
33 views

Linear Regression with Category variables

I'm currently learning and exploring machine learning and understand the basics of linear regression based on two numerical variables, but now I wish to go a little further and need some guidance ...
1
vote
1answer
103 views

Alternatives for MultiLabelBinarizer

There a lot of information on how to handle categorical variables when preprocessing data for ML classification. However, I cannot find any feedback on how to handle categorical variables, where each ...
1
vote
0answers
22 views

How to get correlation between the categories of two categorical variable?

I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua&...
1
vote
1answer
116 views

Which classification model to use on large, high-dimensional dataset?

I face a classification task: with several features a target features is to be predicted. I'm working with python. My dataset includes 60 features from which I picked 16 which I think could be ...
6
votes
2answers
18k views

Always drop the first column after performing One Hot Encoding?

Since one of the columns can be generated completely from the others, and hence retaining this extra column does not add any new information for the modelling process, would it be good practice to ...
1
vote
2answers
31 views

Different number of features in train vs test when using Label Encoding

This is not a duplicate of Different number of features in train vs test There are some categorical columns in my data, and the cardinality for each of them is large, so I chose to use LabelEncoding ...
0
votes
0answers
28 views

Equitable selection of users through ranking

I am looking to take a dataset largely derived of user input in categorical form, this sign up sheet asks for many data points such as age group, race, sign up date, as well as a few others. My goal ...
0
votes
0answers
16 views

Which are the features selection techniques depending on the combination on cat num columns in independent and dependent features?

I am very confused: For what I understood I should: Multicollinearity check with Pearson corr and possibly consider to drop multicolliner features Then? I am very confused feature selection should be ...
1
vote
1answer
60 views

Anomaly detection on sparse categorical data

I have a big dataset with a column "clientid" and a categorical column "choice". I want to find out what are the clients that have strange combinations of choices (less frequent ...
2
votes
1answer
84 views

Why do my target labels need to begin at 0 for sparse categorical cross entropy to work?

I'm following a guide here to implement image segmentation in Keras. One thing I'm confused about are these lines: ...
0
votes
0answers
13 views

How to convert the categorical features into one-hot encoded dense vectors before applying the min-max scaler of sklearn?

like the question says I am trying to figure out how to convert one-hot encoded vectors into dense vectors, because my data-set contains both continuous and categorical features, I used the ...
0
votes
1answer
54 views

Problem with binning

I am trying to change continuous data points to categorical by using binning. I know two techniques, i) equal width bins ii) bins with equal number of elements. My questions are: Which type of ...
2
votes
1answer
184 views

Categories with the same mean in target encoding

While doing target encoding it can happen that two categories have the same target mean. This is bad because there will be no difference in the new feature in it and we will lose some information. ...
1
vote
1answer
64 views

Calculating correlation for categorical variables

I am struggling to find out a suitable way to calculate correlation coefficient for categorical variables. Pearson's coefficient is not supported for categorical ...
2
votes
1answer
35 views

Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality

When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less ...
0
votes
1answer
32 views

Dealing with unseen data/categories in machine learning models for stream data

I want to build a machine learning model (xgb and lgbm) that has to handle streaming data on a weekly basis. The models are trained on a bi-weekly basis. The data includes order information and I want ...
1
vote
1answer
60 views

Find correlation between grades from two raters [duplicate]

The question is whether we can find a correlation between two sets of grades (categorical data). Let’s say we have a dog competition and there are 1000 dogs participating. There are two rounds of ...
1
vote
0answers
263 views

Partial Dependence Plot and categorical variables

While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables: For each of the categories, we get ...
4
votes
1answer
3k views

Feature selection for data with both continuous and categorical features?

I am working on a classification problem with 4 ordinal classes to predict, labelling/predicting samples as either a number from 1-4. My training dataset has 284 features by ~40,000 samples and I am ...
1
vote
0answers
11 views

multi-observed features [closed]

I am working on a ML model where individual features may have a highly variable number of observed values. The model will predict a continuous variable so I am planning to use a Regressor. More ...
6
votes
5answers
424 views

How do I encode the categorical columns if there are more than 15 unique values?

I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have ...
0
votes
1answer
22 views

Determining which categorical data is beneficial in predictive modelling

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
1
vote
1answer
62 views

Modeling the Price Movement- What analysis should be used

I am trying to model the price of a hotel as the check-in date arrives. I have a data set which looks like- For e.g- if I am looking at the booking date of Dec 31st, I would want to analyze the ...
0
votes
1answer
17 views

Does produced data fall under the “data collection” category?

I am not sure whether data governance and policy is covered by this community. I'm giving it a try and let me know if I need to adjust. While surveying all the data of my current employer, and ...
2
votes
0answers
34 views

ML recommendation system with items organized in a tree

I would like to develop a recommendation system (probably hybrid, user-based and feature-based) for items which are organized in a tree (there are categories, divided in sub-categories, divided in sub-...
1
vote
1answer
4k views

Scikit Learn OneHotEncoded Features causing error in classifier

I’m trying to prepare data for input to a Decision Tree and Multinomial Naïve Bayes Classifier. This is what my data looks like (pandas dataframe): ...
4
votes
4answers
3k views

Scikit Learn Missing Data - Categorical values

I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier) ...
2
votes
0answers
27 views

How to get the maximum likelihood estimate of the categorical distribution parameters using Lagrange optimization?

Let's say our data is discrete-valued and belongs to one of $K$ classes. The underlying probability distribution is assumed to be a categorical/multinoulli distribution given as $p(\textbf{x}) = \...
3
votes
1answer
99 views

Is there a name for a scale which mixes ordinal and nominal?

The textbooks I have differentiate between nominal, ordinal, interval and ratio scales. The ordinal scale is quite popular in the wild, used for basically all subjective data, and also for dividing ...
2
votes
2answers
292 views

Feature Engineering

I have a data frame with around 37,000 rows and 54 columns. Out of these 54 columns, two columns namely 'user_id' and 'mail_id' are provided in a very creepy format as shown below: ...
0
votes
0answers
19 views

Calculating Uncertainty for categorical predictions

I am wondering what is the best way to calculate the uncertainty for my categorical predictions. I have created a model that predicts what rating a movie is getting based on certain keywords and the ...
0
votes
0answers
47 views

What's the best method to merge N categorical features into one and keep it as categorical

I'm training a Transformer model and it requires one input sentence and N optional labels, not classes cause it's a multi-label and multi-class problem so the unique classes turned into labels. I have ...
2
votes
1answer
27 views

What is the best alternative for Fisher's Exact test for contigency tables that are NOT 2x2?

I am a newbie to data mining. I am trying to find associations between two categorical variables. Since more than 20% of my expected frequencies are less than 5, I wanted to use Fisher exact test but ...
1
vote
1answer
20 views

How to deal with variable number of permutation invariant features?

I want to learn from data where each record has a variable number of features that have no inherent order to them. Take as an example the task to predict whether a repair is worth it of some item. ...
0
votes
1answer
24 views
1
vote
0answers
8 views

Very infrequent values in embedding layers

I have a categorical input that is very imbalanced. 90% of the values are either A or B and frequencies for C, D, E, F, etc are as little as 1. I am using an embedding layer for this input and the ...
0
votes
1answer
26 views

How to Present All Categories in All Samples

I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with ...
14
votes
10answers
4k views

How can I appropriately handle cleaning of gender data?

I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform an analysis of it. In this dataset, the gender ...
1
vote
1answer
238 views

unsupervised anomaly detection on sparse data

Given that I have a very sparse data matrix with continuous features, like this dataframe for example ...
1
vote
1answer
41 views

How to handle One Hot Encoded columns with changing categories in supervised ML Problem?

Scenario: I have the following game data about participants, game and the winner in the following format: ...
0
votes
1answer
38 views

What is the best way to encode an arbitrary collection of strings into int categorical variables?

I have a bunch of categorical labels which I want to transform into int categorical features for an ML algorithm. The problem is I don't have a prior list of the categories, so that I can't just ...
0
votes
1answer
30 views

How would I approach training a model and encoding this categorical data

So I have the following data: I have one series where each word has a value that describes the average review score that would get. For example, if the word "excellent" showed up in reviews ...
3
votes
1answer
252 views

“Binary Encoding” in “Decision Tree” / “Random Forest” Algorithms

Is it OK to use Binary Encoding in a dataset containing categorical columns with very high cardinalities? Some facts about my dataset: My dataset has ~170,000 rows One of the categoric variables has ...
0
votes
1answer
1k views

SMOTE-NC does not help to oversample my mixed continuous/categorical dataset

When I use SMOTE-NC to oversample three classes of a 4-class classification problem, the Prec, Recall, and F1 metrics for minority classes are still VERY low (~3%). I have 32 categorical and 30 ...
1
vote
0answers
27 views

How does the R implementation of RandomForest split nodes on categorical data?

The R implementation of RandomForest can take in categorical features as factors and train and predict on these features without encoding. Normally, I use the python implementation from scikit-learn ...

1
2
3 4 5
7