Questions tagged [categorical-data]
Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.
398
questions
2
votes
1
answer
81
views
What to do if a specific label of a category appears only a few times?
Let's say I am trying to predict whether a car will be auctioned or not (not what I'm actually trying to do, but it represents it pretty well) using tabular data. I have the year the car was made, its ...
CommunityBot
- 1
1
vote
3
answers
164
views
Anomaly detection in nominal big data
I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
CommunityBot
- 1
2
votes
1
answer
188
views
How to deal with a potencially multiple categorical variable
I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring....
CommunityBot
- 1
3
votes
1
answer
422
views
Encode missing data and unseen data
Let's assume that I have a classification problem and all my features are categorical data.
I have missing data (and I do not want to do any imputation).
Also, I know that I will have some unseen ...
CommunityBot
- 1
1
vote
1
answer
29
views
How to deal with categorical disalignment in test and train in binary classification problems
I have a train and test datasets (600k observations) that have different categories for the same categorical variable.
For example train has the categorical variable Letters having unique categories ...
- 338
10
votes
4
answers
26k
views
How can Time Series Analysis be done with Categorical Variables
Most of the time series analysis tutorials/textbooks I've read about, be they for univariate or multivariate time series data, usually deal with continuous numerical variables.
I currently have a ...
- 101
2
votes
1
answer
172
views
How to retrain a K-Modes model based on daily data?
I have read that retraining a model depends highly on what you are trying to achieve. I am conscious that maybe I need to retrain my model daily and after a certain time I have to train the model ...
CommunityBot
- 1
0
votes
0
answers
14
views
Feature selection on datasets with both categorical and numerical features
I'm proposing a novel methodology for feature selection in the context of tabular datasets that contain both numerical and categorical features. In order to prove the efficacy of my methodology, I ...
0
votes
1
answer
130
views
Categorical data preprocessing for training a algorithm
I have a training dataset where values of "Output" col is dependent on three columns (which are categorical [No ordering]).
...
CommunityBot
- 1
14
votes
10
answers
4k
views
How can I appropriately handle cleaning of gender data?
I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform an analysis of it.
In this dataset, the gender ...
- 6,112
0
votes
1
answer
226
views
How to efficiently reduce dimensions of one-hot encoded categorical values?
I'm currently working on a project where I'm using an LSTM to learn and predict sequences of categorical data.
My dataset consists of variable-length sequences of items $s_i = [x_{i_0}, x_{i_1}, ..., ...
CommunityBot
- 1
2
votes
1
answer
261
views
Logistic Regression Model for categorical features with multiple values in each category
I am working on an insurance use case to build a logistic regression classifier to predict if a policy will lapse or not. The dataset has more than 20 categorical features for a policy. Each ...
CommunityBot
- 1
1
vote
1
answer
42
views
How to get correlation between the categories of two categorical variable?
I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua&...
- 351
0
votes
2
answers
90
views
How would I classify this variable?
I am learning about the difference between categorical, ordinal and numerical variables.
From what I understand:
Categorical variables have 2+ categories without any intrinsic order.
Ordinal ...
CommunityBot
- 1
4
votes
1
answer
735
views
Decision Tree only splits to the left
I can’t really understand, why my decision tree only splits to the left.
I originally have 2 categorical features (further named feature 0 and 1), which I concat to one feature since feature 1 is ...
- 1,352
1
vote
2
answers
247
views
Model for predicting duration based on categorical data
I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
CommunityBot
- 1
0
votes
0
answers
11
views
classification using simple relationships between time series data
I am looking to predict which courses are taught by which university professors at my school. More specifically, for each semester and professor I want to know the probability breakdown of which ...
- 101
1
vote
2
answers
904
views
Aggregating multiple encoded categorical values
I am trying find commonly used techniques when dealing with high cardinality multi-valued categorical variables.
I am currently using a dataset with a feature CATEGORY which has a cardinality of ~20,...
CommunityBot
- 1
0
votes
1
answer
82
views
What's the minimum percentage of categories should be present in the categorical variable for to ignore the variable entirely
For example, if i have a feature "colour_codes" that has close to 5000 distinct color codes inside it. And the number of samples/rows is 10 million. Then should I ignore the feature "...
CommunityBot
- 1
1
vote
1
answer
223
views
Is there any way to collect categorical features quickly in Julia DataFrames?
I'm using Julia 0.6.3 with Dataframes.jl
I was wondering if there was any way to get categorial features easily in Julia?
For large datasets it can be impossible to enter everything by hand.
My ...
CommunityBot
- 1
6
votes
3
answers
319
views
Anomaly detection using clustering of highly correlated Categorical data
My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else it's Anomaly. Likewise, there are thousands ...
CommunityBot
- 1
4
votes
2
answers
5k
views
Imputation of missing values and dealing with categorical values
I have a dataset (10 million rows, 55 columns) with many missing values. I need to predict those values somehow using other non-missing values, i.e. replace them with something that is not NaN. Mean ...
- 364
1
vote
1
answer
462
views
How to predict on data that is label encoded as end user will input a categorical data?
My dataset contains about 29 features with 3 class labels as result. Among these 29 features around 24 features are categorical i cannot transform each category into numbers as there are many more ...
CommunityBot
- 1
1
vote
1
answer
107
views
Queries regarding feature importance for categorical features
Queries regarding feature importance for categorical features:
Context:
I have almost 185 categorical features and these categorical features have either 2 or 3 or 8 or 1 or sometimes 4 categories, ...
CommunityBot
- 1
1
vote
1
answer
884
views
faster alternatives to sparse.model.matrix?
I have a large dataset that is entirely categorical. I'm trying to train with it using xgboost, so I must first convert this categorical data to numerical. So far I've been using sparse.model.matrix() ...
CommunityBot
- 1
1
vote
1
answer
96
views
An Unsupervised learning method suitable for large categorical data sets
I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
CommunityBot
- 1
0
votes
0
answers
29
views
Combining Textual, Categorical and Numerical data for Semantic Search using SentenceTransformers model
I'm building a food semantic search model and I want to use a pre-trained SentenceTransformers model with cosine similarity. I'm using Epicurious dataset for the corpus which consists of textual (&...
- 1
1
vote
2
answers
1k
views
What are some good methods to forecast future revenue on categorical and value based data?
I have monthly snapshots (3 years) of all the contract data. It includes following information:
Contract status [Categorical]: Proposed, tracked, submitted, won, lost, etc
Contract stages [...
CommunityBot
- 1
0
votes
1
answer
185
views
Collapsing categorical data into more than 3 categories
I have a bunch of categorical, part of speech data that I want to collapse into fewer categories. np.where() won't do because I want to have 6 categories at the end: noun, verb, adjective, adverb, ...
CommunityBot
- 1
1
vote
1
answer
188
views
Encode categorical data for unsupervised learning
What is the best encoder for categorical data in unsupervised learning?
I am using unsupervised learning on mixed data (such as K-means).
Before running my unsupervised algorithm, I am using dimension ...
CommunityBot
- 1
0
votes
0
answers
28
views
Training Biased/Uneven Categorical Data with CatBoost, Unbalanced/Unseen Categories Handling
Summary:
I am training a discount eligibility model where the dataset represents historical data for products where people availed discounts based on simple features like product category, discount ...
- 101
1
vote
2
answers
3k
views
Dealing with multiple distinct-value categorical variables
So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values.
For instance, one column have more ...
CommunityBot
- 1
2
votes
1
answer
382
views
Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?
I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
CommunityBot
- 1
0
votes
2
answers
19
views
How to encode Income Type Ordinal Data into numbers?
I am doing a mini project on Credit card Approval Prediction. The Dataset I have taken is from Kaggle: https://www.kaggle.com/datasets/rikdifos/credit-card-approval-prediction
Problem: I want to ...
- 23
1
vote
1
answer
238
views
Alternatives for MultiLabelBinarizer
There a lot of information on how to handle categorical variables when preprocessing data for ML classification. However, I cannot find any feedback on how to handle categorical variables, where each ...
CommunityBot
- 1
2
votes
2
answers
990
views
Clustering mixed data types - numeric, categorical, arrays, and text
I have a dataset with 4 types of data columns:
...
CommunityBot
- 1
2
votes
2
answers
137
views
How to leverage description data in multi-class classification (dimensionality reduction)
I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as ...
CommunityBot
- 1
0
votes
2
answers
341
views
Custom Encoding for Categorical Features - sklearn
Just wanted to check if there are any obvious flaws with a custom encoding idea I have - for categorical features used with RandomForestClassifer or any tree-based classifier.
As all of you would know ...
CommunityBot
- 1
2
votes
1
answer
4k
views
How to plot a heatmap-like plot for categorical features?
I would greatly appreciate let me know how to plot a heatmap-like plot for categorical features?
In fact, based on this post, the association between categorical ...
CommunityBot
- 1
0
votes
0
answers
7
views
What are some standard methods for studying co-ocurrence patterns?
We have a series of academic articles annotated with several tags eg "environmental issues", "legal issues", etc.
I wish to understand whether some of these topics are frequently ...
- 161
1
vote
1
answer
126
views
Dummy variables for unseen data in R
I got the following problem: When I trained my model I created my dummy variables(before train-test split) in the following way:
...
CommunityBot
- 1
0
votes
1
answer
100
views
Encoding concept for categorical data - pick one for all the columns or different for different kinds in the same df
[Beginner here] If dataset contains - both ordinal, nonordinal (few categories) & nonordinal (multiple categories > 30). Is one supposed to pick one to encapsulate of all such situations or ...
CommunityBot
- 1
0
votes
0
answers
17
views
Encode multiple label categorical variables with consideration of the frequency and standardization
I'm currently working on a dataset containing, among other features, multi-labeled categorical data per person for the last 3 years. I'm not sure how to handle this kind of categorical data where ...
- 1
0
votes
0
answers
15
views
Unsupervised rule extraction of categorical data
I have a dataset of network traffic with three features that I would like to extract rules from in order to apply firewall/flow control rules i.e. the permitted flows.
I am able to classify a ...
1
vote
2
answers
116
views
How to encode high cardinality categorical data?
I have a dataset of 1600 rows and 28 columns. Only one column is partially complete with 1300 records. The rest is NaN. I did a value count of this columns and it has 84 different categories that are ...
CommunityBot
- 1
1
vote
1
answer
124
views
Mixing categorical data and time-series data for regression purpose
I came across a problem and I have been looking on internet how to solve it without finding a solution that fits my need. I am trying to predict investors behavior. To be precise, I would like to ...
- 1,352
0
votes
2
answers
505
views
XGBOOST with target column has categorical data and features also has categorical data
I have a huge dataset with the categorical columns in features and also my target variable is categorical.
All the values are not ordinal so I think it is best to use one hot encoding.
But I have one ...
CommunityBot
- 1
0
votes
1
answer
33
views
Converting categorical to the percentage
How do I convert the categorical value to the percentage?|
I have this asset wellness data:
Poor: 3
Warning: 27
Good: 120
How do I convert it to the percentage ...
- 947
0
votes
1
answer
52
views
When using Chi-Squrare test in feature selection makes sense?
What are the prerequisites that need to be fulfilled before conducting a chi-square test (Bivariate analysis)? For instance, before having a correlation matrix, we should first ensure linearity. What ...
- 947
0
votes
1
answer
45
views
Feature Selection - determining the significance of imbalanced categorical data column
I have a dataset with a categorical column that contains three categories. One of the categories represents 98% of the data, while the remaining 2% are distributed between the other two categories, ...
