Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
0
votes
2answers
36 views

How would I classify this variable?

I am learning about the difference between categorical, ordinal and numerical variables. From what I understand: Categorical variables have 2+ categories without any intrinsic order. Ordinal ...
1
vote
1answer
166 views

Is there any way to collect categorical features quickly in Julia DataFrames?

I'm using Julia 0.6.3 with Dataframes.jl I was wondering if there was any way to get categorial features easily in Julia? For large datasets it can be impossible to enter everything by hand. My ...
1
vote
0answers
10 views

multi-observed features [closed]

I am working on a ML model where individual features may have a highly variable number of observed values. The model will predict a continuous variable so I am planning to use a Regressor. More ...
6
votes
5answers
103 views

How do I encode the categorical columns if there are more than 15 unique values?

I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have ...
0
votes
1answer
21 views

Determining which categorical data is beneficial in predictive modelling

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
1
vote
1answer
138 views

Encode missing data and unseen data

Let's assume that I have a classification problem and all my features are categorical data. I have missing data (and I do not want to do any imputation). Also, I know that I will have some unseen ...
1
vote
1answer
46 views

Modeling the Price Movement- What analysis should be used

I am trying to model the price of a hotel as the check-in date arrives. I have a data set which looks like- For e.g- if I am looking at the booking date of Dec 31st, I would want to analyze the ...
0
votes
1answer
17 views

Does produced data fall under the “data collection” category?

I am not sure whether data governance and policy is covered by this community. I'm giving it a try and let me know if I need to adjust. While surveying all the data of my current employer, and ...
2
votes
0answers
30 views

ML recommendation system with items organized in a tree

I would like to develop a recommendation system (probably hybrid, user-based and feature-based) for items which are organized in a tree (there are categories, divided in sub-categories, divided in sub-...
0
votes
1answer
67 views

Queries regarding feature importance for categorical features

Queries regarding feature importance for categorical features: Context: I have almost 185 categorical features and these categorical features have either 2 or 3 or 8 or 1 or sometimes 4 categories, ...
0
votes
1answer
23 views

Model for predicting duration based on categorical data

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
0
votes
1answer
574 views

faster alternatives to sparse.model.matrix?

I have a large dataset that is entirely categorical. I'm trying to train with it using xgboost, so I must first convert this categorical data to numerical. So far I've been using sparse.model.matrix() ...
0
votes
1answer
23 views

Entropy loss from collapsing/merging two categories

Suppose I am counting occurrences in a sequence. For a classical example, let's say I'm counting how many of each kind of car comes down a highway. After keeping tally for a while, I see there are ...
1
vote
1answer
88 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
142
votes
4answers
83k views

When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?

I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building ...
1
vote
1answer
4k views

Scikit Learn OneHotEncoded Features causing error in classifier

I’m trying to prepare data for input to a Decision Tree and Multinomial Naïve Bayes Classifier. This is what my data looks like (pandas dataframe): ...
0
votes
1answer
614 views

Should the type of Boolean categorical features be numerical or categorical after encoding?

There are categorical features which have two different value in my dataframe next to numerical features. I've converted these categorical values to 0 or 1. I will apply correalation elimination on ...
4
votes
4answers
3k views

Scikit Learn Missing Data - Categorical values

I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier) ...
2
votes
0answers
11 views

How to get the maximum likelihood estimate of the categorical distribution parameters using Lagrange optimization?

Let's say our data is discrete-valued and belongs to one of $K$ classes. The underlying probability distribution is assumed to be a categorical/multinoulli distribution given as $p(\textbf{x}) = \...
1
vote
2answers
105 views

What are some good methods to forecast future revenue on categorical and value based data?

I have monthly snapshots (3 years) of all the contract data. It includes following information: Contract status [Categorical]: Proposed, tracked, submitted, won, lost, etc Contract stages [...
3
votes
1answer
96 views

Is there a name for a scale which mixes ordinal and nominal?

The textbooks I have differentiate between nominal, ordinal, interval and ratio scales. The ordinal scale is quite popular in the wild, used for basically all subjective data, and also for dividing ...
2
votes
2answers
2k views

Dealing with multiple distinct-value categorical variables

So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values. For instance, one column have more ...
0
votes
1answer
171 views

Aggregating multiple encoded categorical values

I am trying find commonly used techniques when dealing with high cardinality multi-valued categorical variables. I am currently using a dataset with a feature CATEGORY which has a cardinality of ~20,...
2
votes
2answers
290 views

Feature Engineering

I have a data frame with around 37,000 rows and 54 columns. Out of these 54 columns, two columns namely 'user_id' and 'mail_id' are provided in a very creepy format as shown below: ...
3
votes
1answer
235 views

Anomaly detection using clustering of highly correlated Categorical data

My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else its Anomaly. Likewise there are thousands of ...
0
votes
0answers
13 views

Calculating Uncertainty for categorical predictions

I am wondering what is the best way to calculate the uncertainty for my categorical predictions. I have created a model that predicts what rating a movie is getting based on certain keywords and the ...
0
votes
0answers
18 views

What's the best method to merge N categorical features into one and keep it as categorical

I'm training a Transformer model and it requires one input sentence and N optional labels, not classes cause it's a multi-label and multi-class problem so the unique classes turned into labels. I have ...
2
votes
1answer
13 views

What is the best alternative for Fisher's Exact test for contigency tables that are NOT 2x2?

I am a newbie to data mining. I am trying to find associations between two categorical variables. Since more than 20% of my expected frequencies are less than 5, I wanted to use Fisher exact test but ...
1
vote
1answer
19 views

How to deal with variable number of permutation invariant features?

I want to learn from data where each record has a variable number of features that have no inherent order to them. Take as an example the task to predict whether a repair is worth it of some item. ...
2
votes
1answer
72 views

How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline

Let us say we have a dataset with a feature such as Surname. arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith'] And I want to encode this categorical info ...
2
votes
1answer
27 views

Individual differences in response time (RT) experiment - searching for the right test

Given a distribution of response times to different categorical variables, what's the best way to test for individual differences? or more specifically: There are 100 people pressing buttons in 10 ...
1
vote
1answer
48 views

Clustering with Only Categorical Features

I am trying to do clustering with a bunch (24) of categorical features. I have done some research and found a lot of people recommending something such as K-Modes. I tried running K-Modes on my data ...
0
votes
1answer
19 views
1
vote
0answers
7 views

Very infrequent values in embedding layers

I have a categorical input that is very imbalanced. 90% of the values are either A or B and frequencies for C, D, E, F, etc are as little as 1. I am using an embedding layer for this input and the ...
0
votes
1answer
54 views

What to do if a specific label of a category appears only a few times?

Let's say I am trying to predict whether a car will be auctioned or not (not what I'm actually trying to do, but it represents it pretty well) using tabular data. I have the year the car was made, its ...
0
votes
1answer
25 views

How to Present All Categories in All Samples

I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with ...
1
vote
1answer
86 views

Which classification model to use on large, high-dimensional dataset?

I face a classification task: with several features a target features is to be predicted. I'm working with python. My dataset includes 60 features from which I picked 16 which I think could be ...
2
votes
2answers
70 views

How to leverage description data in multi-class classification (dimensionality reduction)

I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as ...
14
votes
10answers
3k views

How can I appropriately handle cleaning of gender data?

I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform an analysis of it. In this dataset, the gender ...
1
vote
1answer
51 views

unsupervised anomaly detection on sparse data

Given that I have a very sparse data matrix with continuous features, like this dataframe for example ...
1
vote
1answer
37 views

How to handle One Hot Encoded columns with changing categories in supervised ML Problem?

Scenario: I have the following game data about participants, game and the winner in the following format: ...
0
votes
1answer
28 views

What is the best way to encode an arbitrary collection of strings into int categorical variables?

I have a bunch of categorical labels which I want to transform into int categorical features for an ML algorithm. The problem is I don't have a prior list of the categories, so that I can't just ...
0
votes
1answer
26 views

How would I approach training a model and encoding this categorical data

So I have the following data: I have one series where each word has a value that describes the average review score that would get. For example, if the word "excellent" showed up in reviews ...
2
votes
1answer
186 views

“Binary Encoding” in “Decision Tree” / “Random Forest” Algorithms

Is it OK to use Binary Encoding in a dataset containing categorical columns with very high cardinalities? Some facts about my dataset: My dataset has ~170,000 rows One of the categoric variables has ...
1
vote
0answers
26 views

DBSCAN on textual and numerical columns

I have a dataset which has two columns: title price sentence1 12 sentence2 13 I have used doc2vec to convert the ...
1
vote
1answer
1k views

SMOTE-NC does not help to oversample my mixed continuous/categorical dataset

When I use SMOTE-NC to oversample three classes of a 4-class classification problem, the Prec, Recall, and F1 metrics for minority classes are still VERY low (~3%). I have 32 categorical and 30 ...
2
votes
1answer
22 views

How to Set the Same Categorical Codes to Train and Test data? Python-Pandas

NOTE: If someone else it's wondering about this topic, I understand you're getting deeper in the Data Analysis world, so I did this question before to learn that: You encode categorical values as ...
1
vote
0answers
25 views

How does the R implementation of RandomForest split nodes on categorical data?

The R implementation of RandomForest can take in categorical features as factors and train and predict on these features without encoding. Normally, I use the python implementation from scikit-learn ...
0
votes
2answers
25 views

Is there a RandomForest implementation that handles categorical data without encoding in python?

I am working on a binary classification project with both continuous and categorical features. I know that the R implementation of RandomForest can handle categorical data passed in as factor type ...

1
2 3 4 5 6