Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
1 vote
1 answer
7 views

Encode categorical data for unsupervised learning

What is the best encoder for categorical data in unsupervised learning? I am using unsupervised learning on mixed data (such as K-means). Before running my unsupervised algorithm, I am using dimension ...
0 votes
0 answers
4 views

Clustering mixed type variables with Orange

I wonder if with Orange it is possible to cluster mixed type data, so a dataset with numeric as well as discrete (categorical) data (ordered / unordered). Can you show an example of how that could be ...
0 votes
1 answer
15 views

Is there a procedure for determining if a classification problem is ill-defined?

Consider a group of objects denoted $O = \{o_0, o_1, \cdots\}$ where each object is associated with a feature vector $F = \{f_0, f_1, \cdots\, f_{N-1}\}$. For this case, assume the features are ...
  • 101
2 votes
1 answer
24 views

How to use categorical data in forecasting with Prophet?

I'm trying to create a model to predict the number of players on a video game at a certain time and was trying to figure out how to integrate categorical data into my forecasting problem. So far, my ...
0 votes
1 answer
21 views

Does Scikit-Learn's OneHotEncoder make all Columns Categorical?

I've been using Scikit-Learn's OneHotEncoder to turn categorical data into binary columns, however, it seems that fitting ...
  • 227
1 vote
1 answer
18 views

N-ary decision tree with categorical features

I want to build an n-ary decision tree with categorical features. I am using ordinary ID3 algorithm to build a tree. Lets take the next dataset as a training dataset for building a decision tree: ...
  • 111
0 votes
0 answers
21 views

What Would Be a Good Measure of Feature Importance in Regression?

Doing simple supervised regression where the label is a floating point number (guaranteed positive) and the features are a mix of continuous floating point values and some categorical features. What ...
  • 303
0 votes
0 answers
11 views

Dealing with categorical columns with unbalanced value count

I'm doing some data processing and wondering what is the best practice for dealing with categorical columns that has a value counts plot looking like the below (these are one-hot-encoded at a later ...
0 votes
0 answers
10 views

How do I use one hot encoding with 240 nominal variables and each with equal occurrence?

The method I saw that's generally used to deal with large # of nominal variables is to keep the most frequent variables and introduce a new "other" category. But that's not possible with my ...
1 vote
1 answer
18 views

How do well informed labels for ordinal encoding improve model performance?

From Kaggle's intermediate machine learning tutorial, it was stated that for each column, we randomly assign each unique value to a different integer. This is a common approach that is simpler than ...
0 votes
0 answers
9 views

How to classify and organize very complex data for easy future reference?

I've studied binomial nomenclature in college, but forgot all about it! I use Joplin to organize my life and business, but it's getting out of hand complicated and I can't find stuff. Struggling on ...
1 vote
1 answer
23 views

How to predict on data that is label encoded as end user will input a categorical data?

My dataset contains about 29 features with 3 class labels as result. Among these 29 features around 24 features are categorical i cannot transform each category into numbers as there are many more ...
1 vote
1 answer
39 views

Encoding for Linear Regression

I have a CSV file with salary information and other columns. I am trying to transform some of these columns into proper values, for a ...
1 vote
1 answer
38 views

working principle of Support Vector Machine

I have a dataset consisting of numerical features and categorical features. I want train the training set using SVM. SVM is a quadratic optimization algorithm. I would like to know the how SVM works ...
  • 359
0 votes
0 answers
23 views

How to cluster data where each sample has a different amount of different features?

I decompiled one million binaries and stored a representation of every function in a database. Each binary has a list of functions. Each function has a number of callees (how often its called by ...
0 votes
1 answer
38 views

SkLearn Categorical Naive Bayes Vs Mathematical theory of Naive Bayes

The Naive Bayes classification based on the following formula $P(C_i|X) = {P(X|C_i)P(C_i) \over P(X)} ... i)$ $P(X|C_i)$ is the posterior probability of $X$ conditioned on $C_i$, $P(X)$ prior ...
  • 359
1 vote
1 answer
18 views

If I use Weight of Evidence to transform categorical variables, do I still need to inform their indexes to Catboost

I'm using Weight of Evidence (WOE) to encode my categorical features. Do I still need to inform Catboost that they are categorical features by using cat_features parameter?
1 vote
0 answers
24 views

Intuition behind catboost encoding techniques

Can anyone please help me in understanding the effect of various bucketing techniques used in CatBoost Algorithm for categorical features? Like there is border, buckets, binarized target mean, counter ...
0 votes
0 answers
14 views

OHE vs dummy variables

Could someone explain to me the difference between creating categorical “embeddings” with StringIndexer and OneHotEncoder, vs just creating dummy variables for each category? Aside from it being more ...
0 votes
0 answers
28 views

Do you use categorical data types?

Personally I've never used the categorical data type in pandas, and leave eveything as objects. I've seen it has the capability to be saved as parquet files, saves data etc... What are the pros and ...
  • 435
0 votes
1 answer
47 views

Is it possible to implement logistic regression (or any other ML method) to impute null values in a categorical feature with multiple values?

I'm doing a Data Science project, and I'm on the stage of cleaning categorical features. I've been researching, and it seems that imputing the mean or median can change the distribution. Therefore, a ...
0 votes
0 answers
7 views

Behavioral Segmentation Clustering Mixed Large Data - with or without categorical variables

This question may be closed due to being too broad but I feel as though this is the best place to ask my question. At the moment, I am dealing with some customer data and am looking on segmenting them ...
0 votes
1 answer
22 views

Method of accuracy improve for binary classification imbalanced dataset

I have an imbalance data set where the imbalance ratio No: Yes is 8:1. If I run classifiers on the groundtruth dataset I got recall and F2 measure for Naive bayes, Logistic regression, random forest. ...
  • 359
0 votes
0 answers
38 views

how to handle categorical data that has two or more columns with unique values?

...
2 votes
1 answer
70 views

Clustering ordered categorical data

Suppose I have a list of, say, 100 countries, as well as their respective historical sovereign credit ratings as such ...
  • 123
1 vote
0 answers
18 views

What methods I could use to analyze the contingency table?

I am data science beginner, and I have a question about methods that I could use to analyze the following data. It is a simple case, I am trying to check the influence of cohabitation before marriage ...
  • 11
0 votes
0 answers
14 views

How to handle tags/lists with CatBoost?

I have database like this: Id, type, category1, category2, tags 1, ‘cosmetics’, 123, 456, [446, 354] 2, ‘electronics’, 234, 213, [55, 978, 12] … And I want to predict some value with ...
0 votes
0 answers
21 views

Constructing pandas DF with model trained categorical variable

I've trained a lightGBM model on a dataset X where X has a categorical (in the pandas sense) variable. This model trains fine and when I predict using it all looks good - I can even change the value ...
0 votes
0 answers
219 views

Library for Phi correlation coefficient in python?

I want to calculate correlation b/w categorical features in my data. I reviewed the literature and found phi coefficient can be used for this purpose. I found one library called phik in python enter ...
  • 37
0 votes
0 answers
13 views

improve the project of binary classification with categorical features

Here is a interview task of training a binary classification model. I will not show the detail of training data as the confidential issue. The variable $X$ has ten features and all the features are ...
0 votes
0 answers
50 views

How to calculate distance for symmetric binary and nomianl variables?

In the existing function dist(), the only method for nominal variable is 'binary', and it's for asymmetric binary. However, I ...
0 votes
0 answers
13 views

How do GANs learn category distributions

I'm currently getting more into the topic of GANs and Generating Models. I've understood how the Generator and Discriminator work together in optimization to generate synthetic samples. Now I'm ...
  • 1
1 vote
0 answers
141 views

Encode each comma separated value in Pandas

I have a dataset ...
  • 119
0 votes
1 answer
67 views

Categorical data preprocessing for training a algorithm

I have a training dataset where values of "Output" col is dependent on three columns (which are categorical [No ordering]). ...
  • 119
1 vote
0 answers
16 views

Model a classification problem with multiple categorical varialbes as input features only. Diff Model performance

I'm having an input data with 100k rows, 8 input features, I'm trying to predict y (binary 1/0). But all the X are categorical variables(strictly nominal variables, not ordinal). Some with 8 levels, ...
  • 11
0 votes
0 answers
6 views

What kind of hypothesis testing in Python can be used to validate that 4 job titles are significantly different based on their skillset?

I have 4 job titles, for each of which I scraped hundreds of job descriptions and classified them by if they contain words related to a predefined list of skills. For each job description, I now have ...
0 votes
0 answers
7 views

Whether effect of one categorical variable on a continuous variable depends on levels of another categorical variable

In the dataset I need to analyse, I need to look at whether the effect of people's profession (3 categories) on their scores on a test (I have already tested for this effect and found one) differs ...
2 votes
1 answer
29 views

How do I get the mean values that are greater than .5 for my model?

I am trying to build a classification model. One of the variables called specialty has 200 values. Based on a previous post I saw, I decided I wanted to include the values that have the highest mean. ...
0 votes
0 answers
668 views

Value Error: Shapes (None,128,128) and (None, 4) are incompatible

I am trying to perform CNN on my dataset. I came across the below error ValueError: Shapes (None, 128, 128) and (None, 4) are incompatible The shape of my xTrain ...
0 votes
1 answer
276 views

XGBOOST with target column has categorical data and features also has categorical data

I have a huge dataset with the categorical columns in features and also my target variable is categorical. All the values are not ordinal so I think it is best to use one hot encoding. But I have one ...
1 vote
0 answers
54 views

Handling date and time fields for classification task

I'm working on a classification task(The dataset is 400,000 rows and 30 columns) and one of my features was date-time. I've extracted the month, day of the week, and hour from the dataset (year is a ...
1 vote
1 answer
726 views

How to tell CatBoost which feature is categorical?

I am excited to learn that CatBoost can handle categorical features by itself. One of my features, Department ID, is categorical. However, it looks like numeric, since the values are like 1001, 1002, ....
2 votes
0 answers
190 views

Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?

I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
0 votes
2 answers
131 views

Custom Encoding for Categorical Features - sklearn

Just wanted to check if there are any obvious flaws with a custom encoding idea I have - for categorical features used with RandomForestClassifer or any tree-based classifier. As all of you would know ...
  • 101
0 votes
1 answer
83 views

Collapsing categorical data into more than 3 categories

I have a bunch of categorical, part of speech data that I want to collapse into fewer categories. np.where() won't do because I want to have 6 categories at the end: noun, verb, adjective, adverb, ...
0 votes
0 answers
519 views

How to group multiple categories of a categorical variable before feeding the data to a machine learning algorithm?

I have a labelled dataset to which I wish to fit a classification model (say, a Decision Tree). One of the categorical variables (say STATE) in the data has a lot ...
0 votes
1 answer
30 views

What's the minimum percentage of categories should be present in the categorical variable for to ignore the variable entirely

For example, if i have a feature "colour_codes" that has close to 5000 distinct color codes inside it. And the number of samples/rows is 10 million. Then should I ignore the feature "...
0 votes
1 answer
25 views

Over-sampling when predicting a contionuous variable

Lets say i am predicting house selling prices (continuous) and therefore have multiple independent variables (numerical and categorical). Is it common practice to balance the dataset when the ...
  • 9
0 votes
1 answer
88 views

Separating numerical and categorical features in a binary classification problem

I have a dataset with employee data with around 9500 rows, and have to predict if the target is 0 or 1. Some of my features are the department of an employee, gender, salary, review_score(numerical),...
  • 101
0 votes
0 answers
26 views

Anomaly Detection with totally Categorical features

I am working on an anomaly detection project that aims to discover which merchant is the point of compromise. The data contains no numerical value and it looks like this; date account merchant fraud ...

1
2 3 4 5
8