Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
1
vote
2answers
26 views

Linear Regression with Category variables

I'm currently learning and exploring machine learning and understand the basics of linear regression based on two numerical variables, but now I wish to go a little further and need some guidance ...
1
vote
0answers
22 views

How to get correlation between the categories of two categorical variable?

I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua&...
1
vote
2answers
23 views

Different number of features in train vs test when using Label Encoding

This is not a duplicate of Different number of features in train vs test There are some categorical columns in my data, and the cardinality for each of them is large, so I chose to use LabelEncoding ...
0
votes
0answers
26 views

Equitable selection of users through ranking

I am looking to take a dataset largely derived of user input in categorical form, this sign up sheet asks for many data points such as age group, race, sign up date, as well as a few others. My goal ...
0
votes
0answers
15 views

Which are the features selection techniques depending on the combination on cat num columns in independent and dependent features?

I am very confused: For what I understood I should: Multicollinearity check with Pearson corr and possibly consider to drop multicolliner features Then? I am very confused feature selection should be ...
2
votes
1answer
47 views

Why do my target labels need to begin at 0 for sparse categorical cross entropy to work?

I'm following a guide here to implement image segmentation in Keras. One thing I'm confused about are these lines: ...
0
votes
0answers
11 views

How to convert the categorical features into one-hot encoded dense vectors before applying the min-max scaler of sklearn?

like the question says I am trying to figure out how to convert one-hot encoded vectors into dense vectors, because my data-set contains both continuous and categorical features, I used the ...
0
votes
1answer
18 views

Problem with binning

I am trying to change continuous data points to categorical by using binning. I know two techniques, i) equal width bins ii) bins with equal number of elements. My questions are: Which type of ...
1
vote
1answer
45 views

Calculating correlation for categorical variables

I am struggling to find out a suitable way to calculate correlation coefficient for categorical variables. Pearson's coefficient is not supported for categorical ...
2
votes
1answer
27 views

Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality

When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less ...
1
vote
1answer
47 views

Anomaly detection on sparse categorical data

I have a big dataset with a column "clientid" and a categorical column "choice". I want to find out what are the clients that have strange combinations of choices (less frequent ...
1
vote
0answers
11 views

Feature Selection Statistical Test for Nominal Response Vs Continuous Predictors? (in R)

I cannot find much information on this, none so-far useful. I have a sparse data set with 17K+ columns of continuous gene expressions, an example of a typical column: (3.15, 0, 7.1294, 0, 0, 0, 2300.2,...
0
votes
1answer
23 views

Dealing with unseen data/categories in machine learning models for stream data

I want to build a machine learning model (xgb and lgbm) that has to handle streaming data on a weekly basis. The models are trained on a bi-weekly basis. The data includes order information and I want ...
1
vote
1answer
53 views

Find correlation between grades from two raters [duplicate]

The question is whether we can find a correlation between two sets of grades (categorical data). Let’s say we have a dog competition and there are 1000 dogs participating. There are two rounds of ...
1
vote
0answers
10 views

multi-observed features [closed]

I am working on a ML model where individual features may have a highly variable number of observed values. The model will predict a continuous variable so I am planning to use a Regressor. More ...
2
votes
0answers
20 views

How to get the maximum likelihood estimate of the categorical distribution parameters using Lagrange optimization?

Let's say our data is discrete-valued and belongs to one of $K$ classes. The underlying probability distribution is assumed to be a categorical/multinoulli distribution given as $p(\textbf{x}) = \...
6
votes
5answers
206 views

How do I encode the categorical columns if there are more than 15 unique values?

I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have ...
0
votes
0answers
15 views

Calculating Uncertainty for categorical predictions

I am wondering what is the best way to calculate the uncertainty for my categorical predictions. I have created a model that predicts what rating a movie is getting based on certain keywords and the ...
0
votes
0answers
26 views

What's the best method to merge N categorical features into one and keep it as categorical

I'm training a Transformer model and it requires one input sentence and N optional labels, not classes cause it's a multi-label and multi-class problem so the unique classes turned into labels. I have ...
2
votes
1answer
17 views

What is the best alternative for Fisher's Exact test for contigency tables that are NOT 2x2?

I am a newbie to data mining. I am trying to find associations between two categorical variables. Since more than 20% of my expected frequencies are less than 5, I wanted to use Fisher exact test but ...
0
votes
1answer
20 views

For very simple linear regression can we quantify the prediction accuracy hit between using one hot encoding and simple numerical mapping?

Suppose I had a simple linear regression model that had the following input or X variable: ...
1
vote
0answers
8 views

Very infrequent values in embedding layers

I have a categorical input that is very imbalanced. 90% of the values are either A or B and frequencies for C, D, E, F, etc are as little as 1. I am using an embedding layer for this input and the ...
0
votes
1answer
26 views

How to Present All Categories in All Samples

I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with ...
1
vote
1answer
39 views

How to handle One Hot Encoded columns with changing categories in supervised ML Problem?

Scenario: I have the following game data about participants, game and the winner in the following format: ...
0
votes
1answer
27 views

How would I approach training a model and encoding this categorical data

So I have the following data: I have one series where each word has a value that describes the average review score that would get. For example, if the word "excellent" showed up in reviews ...
1
vote
1answer
19 views

How to deal with variable number of permutation invariant features?

I want to learn from data where each record has a variable number of features that have no inherent order to them. Take as an example the task to predict whether a repair is worth it of some item. ...
1
vote
1answer
42 views

DBSCAN on textual and numerical columns

I have a dataset which has two columns: title price sentence1 12 sentence2 13 I have used doc2vec to convert the ...
1
vote
1answer
131 views

unsupervised anomaly detection on sparse data

Given that I have a very sparse data matrix with continuous features, like this dataframe for example ...
1
vote
0answers
27 views

How does the R implementation of RandomForest split nodes on categorical data?

The R implementation of RandomForest can take in categorical features as factors and train and predict on these features without encoding. Normally, I use the python implementation from scikit-learn ...
1
vote
2answers
30 views

Is there a RandomForest implementation that handles categorical data without encoding in python?

I am working on a binary classification project with both continuous and categorical features. I know that the R implementation of RandomForest can handle categorical data passed in as factor type ...
3
votes
1answer
218 views

How to Set the Same Categorical Codes to Train and Test data? Python-Pandas

NOTE: If someone else it's wondering about this topic, I understand you're getting deeper in the Data Analysis world, so I did this question before to learn that: You encode categorical values as ...
0
votes
1answer
29 views

How to encode high cardinality categorical data?

I have a dataset of 1600 rows and 28 columns. Only one column is partially complete with 1300 records. The rest is NaN. I did a value count of this columns and it has 84 different categories that are ...
0
votes
0answers
16 views

How can the forecast accuracy of three models for a categorical time series be compared?

This is a general question. Can anybody please point me in the direction of how I can compare the forecasts of a 3 level categorical time series by three competing models? I would like to compare the ...
1
vote
0answers
28 views

Endogenous and exogenous ordinal variables in R studio Lavaan

I am new to using R studio, so apologies for the basic question. I have run a number of Confirmatory Factor Analyses using the Lavaan package. Each questionnaire item is on a 4 point Likert scale 0 = ...
0
votes
0answers
21 views

Collapse categorical variable to reduce levels using a decision tree

I am using zip codes as an independent variable as part of a binary classification problem. Naturally, this feature has many different levels (around 2,000), so I was wondering if there is a ...
2
votes
1answer
37 views

Dealing with categorical variables in regression problems which method to use?

Usually if I have regression problem and my initial dataset contains categorical variables like : column 1: Math Science Science English I would convert this ...
0
votes
2answers
36 views

How would I classify this variable?

I am learning about the difference between categorical, ordinal and numerical variables. From what I understand: Categorical variables have 2+ categories without any intrinsic order. Ordinal ...
0
votes
1answer
28 views

how to handle different categorical values in train and test dataset?

I have a dataset in which if i do train_df["era"].value_counts() then it will show 120 different type of categorical values and then if i do ...
1
vote
1answer
123 views

splitting mechanism with one hot encoded variables (tree based/boosting)

I am using xgboost and have a categorical unordered feature with 25 levels. So when i apply one hot encoding i have 25 columns. This introduces alot of sparsity. Even more unusual, my feature ...
0
votes
1answer
21 views

Determining which categorical data is beneficial in predictive modelling

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
0
votes
1answer
29 views

Model for predicting duration based on categorical data

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
1
vote
0answers
225 views

Partial Dependence Plot and categorical variables

While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables: For each of the categories, we get ...
3
votes
1answer
30 views

Is there an encoder which can automatically detect the intrinsic order of an ordinal variable and assign values accordingly?

Given data with an ordinal variable, says "house quality" with values ex (excellent), gd (good), ...
1
vote
2answers
96 views

How to find correlation between categorical data and continuous data

I'm working on imputing null values in the Titanic dataset. The 'Embarked' column has some. I do NOT want to just set them all to the most common value, ...
1
vote
2answers
409 views

Why is count encoding effective in improving accuracy? [duplicate]

Can someone please explain why/how Count encoding of categorical features improve accuracy in classification when compared to simply label encoding them ? I found one explanation in kaggle " ...
1
vote
2answers
166 views

What are some good methods to forecast future revenue on categorical and value based data?

I have monthly snapshots (3 years) of all the contract data. It includes following information: Contract status [Categorical]: Proposed, tracked, submitted, won, lost, etc Contract stages [...
0
votes
1answer
66 views

Is this attribute numeric or categorical (ordinal)? Help!

So I have this dataset I need to perform several techniques on as part of a data mining/machine learning project of some sort in PYTHON. There are a couple of features however, that have me very ...
2
votes
1answer
30 views

is there an adequate number of levels of categorical variables?

I have a project that I'm working on. The dataset contains many categorical variables and some of them have too many levels (+100). My question is : is there any advice to know the "adequate"...
2
votes
0answers
46 views

Regression dataset with categorical features

I have thought of a regression technique that I want to try on several datasets. I would like these datasets to have the following properties: Be a tabular dataset (no images). Have at least 20k rows,...
2
votes
1answer
205 views

Clustering mixed data types - numeric, categorical, arrays, and text

I have a dataset with 4 types of data columns: ...

1
2 3 4 5
7