Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

Categorial Encoding with different cardinality

I have some user data for his data activities. Some examples of the columns are : Activity : ex. youtbube , viber, whatsapp etc etc. Cardinality > 1000 Region : Area identifier Cardinality > 10000 ...
0
votes
1answer
14 views

Help making a custom categorical loss function in Keras

I am a bit new to machine learning, and I'm trying to get the basics working towards a bigger project using a very simple encoder-decoder model. It looks like this: ...
0
votes
1answer
28 views

Clustering categorical variable values based on continuous target values [on hold]

Let's say I have $n$ data points with just one categorical feature $x$ and a continuous target variable $y$. I want to divide the possible values of $x$ into subsets such that the value of $y$ doesn't ...
2
votes
2answers
36 views

Dealing with categorical variables

I have a panel data set. My dependent variable is total costs, and almost all of my independent variables are categorical variables. For instance, age is "old","new". Now i have some questions. ...
-1
votes
1answer
39 views

How to find and calculate correlation in a data set which has category and continuous variables? [duplicate]

I am working on an Insurance domain use case to predict if an existing customer will buy a second insurance policy or not. I have a few personal details saved under different categories like Marital ...
0
votes
0answers
12 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
1
vote
1answer
30 views

Average of importance gain for a categorical variable

Suppose I have a set of M categorical variables, some of them with a different number of categories (for instance, var1 has five categories, var2 has three, etc). I train an XGBoost model on a numeric ...
0
votes
0answers
9 views

Categorical loss functions with similar properties to Kullback-Leibler loss function

When using the Kullback-Leibler divergence as loss function for predicting the probabilities of a categorical (multinomial) distribution, one of the properties is that the difference between $a$ and $...
0
votes
0answers
36 views

Frequency/Count encoding

How do I perform frequency/count encoding for a train and test set? The implementations of this encoding I've seen simply frequency encode the categorical variables on a particular dataset (no ...
2
votes
2answers
31 views

How to leverage description data in multi-class classification (dimensionality reduction)

I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as ...
1
vote
1answer
20 views

Turning Histogram values into Numerical format ( Excel-xslx, Pandas-DataFrame, etc.)

I am trying to do a correlation study about personality traits as described in Hofstede's :https://www.hofstede-insights.com/product/compare-countries/ . I would like to have the values described in ...
-1
votes
0answers
12 views

How To Do Cluster Analysis with a Categorical Index Column?

I have This DF : Amount_A Pos Code 0012 1251 10 0211 154 5 0321 35465 6 The Code Column is a category but i need it to do my ...
0
votes
0answers
4 views

Finding relevant pain points in feedbacks(open text)

I have employee feedback and need to find the appropriate pain points out of their feedback. Need help with the approach and analysis. I have provided a couple of examples below. Note: The feedbacks ...
0
votes
0answers
8 views

How to encode factor predictors in prediction models

The response variable as well as all predictor variables in my dataset are factors. I want to build a model for predicting the response variable. As I understand I have to first encode my predictor ...
0
votes
0answers
6 views

Include or exclude original features after encoding

I have some categorical features and they are encoded by different types of encoding (one-hot, label, target, etc). My question is whether you usually include the original categorical features with ...
-1
votes
1answer
41 views

Apply a clustering algorithm on categorical data with features of multiple values [duplicate]

Let us I have a people data like gender, age, marital status, education, employment, hobbies. I want to make clusters of those people, having some similarity/common among them (for example they have ...
0
votes
0answers
8 views

Recursive feature elimination on train data or complete dataset and dummy encoding

I am using RFE with logistic regression. I will also be doing cross validation with RFE (RFECV in sklearn) to get the optimum number of features. I am not sure whether to use RFECV on just train ...
2
votes
2answers
34 views

Should we use one hot encoder class in data having 2 as maximum numeric representation of categorical feature in each column?

I am testing the Play Golf data set using Decision tree classifier: I am splitting the data into Outlook, Temp, Humidity and windy as features, and ...
2
votes
2answers
32 views

Categorical features preprocessing for clustering

Can anyone tell suggest the best practice for clustering data with mixtured features (both with categorical and continuous). I am struggling with a problem; I realized that for all metrics algorithms ...
2
votes
0answers
18 views

Encode missing data and unseen data

Let's assume that I have a classification problem and all my features are categorical data. I have missing data (and I do not want to do any imputation). Also, I know that I will have some unseen ...
1
vote
0answers
7 views

Bayesian classification of “JSON” data

"Machine Learning over JSON" describes some issues surrounding the classification of JSON documents. Namely, Categorical Features Data is Hierarchical Missingness is Chunky The first two have fairly ...
1
vote
0answers
95 views

Naive Bayes for Categorical Features (Non Binary)

How do i use Naive Bayes Classifier (Using sklearn) for a Dataset considering that my feature set is categorical, ie more than 2 categories per feature are present. I've looked everywhere, some ...
3
votes
2answers
44 views

how to handle values that only appear once in a column?

Counting the values of a column using pandas I got the following result: ...
1
vote
1answer
26 views

What are the best practises to decide whether a variable is categorical?

What are some of the systematic ways to categorise variables into categorical or numeric? I believe using only intuition in such scenarios can many-a-times lead to major irreversible errors. What are ...
1
vote
1answer
29 views

Modeling the Price Movement- What analysis should be used

I am trying to model the price of a hotel as the check-in date arrives. I have a data set which looks like- For e.g- if I am looking at the booking date of Dec 31st, I would want to analyze the ...
0
votes
0answers
24 views

Categorical Multivariate Time Series

I have a small dataset of products of which the price varies along time. Each product is represented by categorical features mostly ( type, matter, use, location ...) and one or two scalar features ( ...
2
votes
0answers
49 views

binning high cardinality categorical features

one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently ...
0
votes
1answer
178 views

SMOTE-NC does not help to oversample my mixed continuous/categorical dataset

When I use SMOTE-NC to oversample three classes of a 4-class classification problem, the Prec, Recall, and F1 metrics for minority classes are still VERY low (~3%). I have 32 categorical and 30 ...
0
votes
0answers
56 views

Hotel Booking Analytics: Perform an analysis in order to understand the movement of the price as the day approaches the check-in date

I am working on a hotel booking dataset. I have transactional level booking data, where each row corresponds to a booking. Please refer to below snippet of the data: I am trying to find out the ...
1
vote
0answers
11 views

Poisson Model (w/ multiple levels X)

Question Is Poisson model the best method for predicting counts among multiple levels within nominal variable? Details Imagine data of 7000 observations, where output= Obs.Count {numeric,0,1,2..8} ...
0
votes
0answers
25 views

The Difference between One Hot Encoding and LabelEncoder? [duplicate]

I am working on a ML problem to predict house prices and Zip Code is one feature which will be useful. I am also trying to use ...
2
votes
1answer
27 views

How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline

Let us say we have a dataset with a feature such as Surname. arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith'] And I want to encode this categorical info ...
0
votes
0answers
9 views

What statistical test would I use to test that both data sources are matching

Suppose I am only working with two datasets, Data Source 1 with columns Customer ID and ...
1
vote
0answers
23 views

Alternatives for MultiLabelBinarizer

There a lot of information on how to handle categorical variables when preprocessing data for ML classification. However, I cannot find any feedback on how to handle categorical variables, where each ...
0
votes
1answer
59 views

How to handle large number of categories in a dataset?

I have one dataset of "Books" which contains 8 columns initially and out of which 3 of them contains text values which can be categorized. The 3 columns contains "Language-code", "Author Name" and "...
1
vote
0answers
19 views

Measuring the similarity between a numeric data matrix and one or more categorical variables?

Given a numeric data matrix $A$ of size $n \times p$, which each row represents an observation along $p$ variables, and a second categorical data matrix $M$ of size $n \times z$, where each row ...
0
votes
0answers
19 views

Class Size Imbalance for LDA or any other Content based analysis

I am running some content analysis studies on my dataset which has two different classes, and each class has a respective list of the document I am analyzing. I compare the LDA topic model inference ...
1
vote
1answer
29 views

Which classification model to use on large, high-dimensional dataset?

I face a classification task: with several features a target features is to be predicted. I'm working with python. My dataset includes 60 features from which I picked 16 which I think could be ...
3
votes
0answers
29 views

Why don't Target/LeaveOneOut Encoders work well for Regression Tasks?

In this review of categorical encoding, it states early on that For regression tasks, Target and LeaveOneOut probably won’t work well and later repeats that Target/LeaveOneOut (Owen Zhang's ...
0
votes
0answers
16 views

Identifying patterns/motifs in categorical time series

There is a data set I currently possess of the following form: $D = [(s_0, t_0), (s_1, t_1), ..., (s_N, t_N)]$, where $s_i \in \mathcal{S}, 0 \leq t_i < T$. Here $\mathcal{S}$ is a finite alphabet ...
2
votes
1answer
43 views

Mapping of categorical features into binary indicator features

I am currently reading an introductory machine learning book by Daumé (ch. 03, p. 30) and when discussing the mapping of categorical features with "n" possible values into "n" binary indicator ...
2
votes
3answers
173 views

Purpose of converting continuous data to categorical data

I was reading through a notebook tutorial working with the Titanic dataset, linked here, and noticed that they highly favored ordinal data to continuous data. For example, they converted both the Age ...
0
votes
2answers
589 views

How can Time Series Analysis be done with Categorical Variables

Most of the time series analysis tutorials/textbooks I've read about, be they for univariate or multivariate time series data, usually deal with continuous numerical variables. I currently have a ...
1
vote
0answers
16 views

Reducing the dimensions of data who's predominant categorical feature, its layer, has depths that overlaps with other samples layer values

I am working with a data set of soil types with multiple layers of varying depths and sizes with multiple features. There are $1-9$ layers each with differing dimensions, for example, a soil type ...
1
vote
1answer
101 views

High train and val results. Bad test and predict results

For my thesis project I've been trying to make a CNN for some challenging data. There's four classes with the following amount of images respectively [410, 410, 269, 206] = 1,295 total. Now I know ...
1
vote
0answers
55 views

Embedding Layer on unseen data

Let's say we have a categorical variables with 5 different categories (levels). I train and get a good model based on this dataset using embedding layer with, say, 3 embedding size and with some ...
2
votes
5answers
220 views

Large no of categorical variables with large no of categories

I'm working on a binary classification problem where the dataset is slightly imbalanced (30% class 0 | 70% class 1). Most of my features are categorical with large number of categories. For example: ...
0
votes
1answer
117 views

Categorical Variable and Target Variable

Though a similar question is answered here , but I wanted to take a different approach. Assuming that I have a binary target variable 1/0 and a categorical variable Gender M/F. From this, I can have a ...
3
votes
1answer
913 views

Feature Selection with one-hot-encoded categorical data

I have a dataset with 400+ columns. Almost 90% of these are categorical data with One-Hot-Encoding (OHE). I'm using the dataset for a classification problem. My professors asked me to perform feature ...
0
votes
0answers
35 views

Unsupervised learning/ clustering for data with multiple categorical variables

Dataset: I have been trying unsupervised clustering algorithms (K-modes & SOM) to cluster the students based on their grades in 3 exams. Should I one-hot encode the data (even though grades are ...