Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
1
vote
0answers
10 views

Alternatives for MultiLabelBinarizer

There a lot of information on how to handle categorical variables when preprocessing data for ML classification. However, I cannot find any feedback on how to handle categorical variables, where each ...
0
votes
1answer
18 views

How to handle large number of categories in a dataset?

I have one dataset of "Books" which contains 8 columns initially and out of which 3 of them contains text values which can be categorized. The 3 columns contains "Language-code", "Author Name" and "...
1
vote
0answers
7 views

Measuring the similarity between a numeric data matrix and one or more categorical variables?

Given a numeric data matrix $A$ of size $n \times p$, which each row represents an observation along $p$ variables, and a second categorical data matrix $M$ of size $n \times z$, where each row ...
0
votes
0answers
13 views

Class Size Imbalance for LDA or any other Content based analysis

I am running some content analysis studies on my dataset which has two different classes, and each class has a respective list of the document I am analyzing. I compare the LDA topic model inference ...
1
vote
1answer
20 views

Which classification model to use on large, high-dimensional dataset?

I face a classification task: with several features a target features is to be predicted. I'm working with python. My dataset includes 60 features from which I picked 16 which I think could be ...
3
votes
0answers
19 views

Why don't Target/LeaveOneOut Encoders work well for Regression Tasks?

In this review of categorical encoding, it states early on that For regression tasks, Target and LeaveOneOut probably won’t work well and later repeats that Target/LeaveOneOut (Owen Zhang's ...
0
votes
0answers
9 views

Identifying patterns/motifs in categorical time series

There is a data set I currently possess of the following form: $D = [(s_0, t_0), (s_1, t_1), ..., (s_N, t_N)]$, where $s_i \in \mathcal{S}, 0 \leq t_i < T$. Here $\mathcal{S}$ is a finite alphabet ...
2
votes
1answer
36 views

Mapping of categorical features into binary indicator features

I am currently reading an introductory machine learning book by Daumé (ch. 03, p. 30) and when discussing the mapping of categorical features with "n" possible values into "n" binary indicator ...
2
votes
3answers
37 views

Purpose of converting continuous data to categorical data

I was reading through a notebook tutorial working with the Titanic dataset, linked here, and noticed that they highly favored ordinal data to continuous data. For example, they converted both the Age ...
0
votes
2answers
26 views

How can Time Series Analysis be done with Categorical Variables

Most of the time series analysis tutorials/textbooks I've read about, be they for univariate or multivariate time series data, usually deal with continuous numerical variables. I currently have a ...
1
vote
0answers
14 views

Reducing the dimensions of data who's predominant categorical feature, its layer, has depths that overlaps with other samples layer values

I am working with a data set of soil types with multiple layers of varying depths and sizes with multiple features. There are $1-9$ layers each with differing dimensions, for example, a soil type ...
1
vote
1answer
32 views

High train and val results. Bad test and predict results

For my thesis project I've been trying to make a CNN for some challenging data. There's four classes with the following amount of images respectively [410, 410, 269, 206] = 1,295 total. Now I know ...
1
vote
0answers
16 views

Embedding Layer on unseen data

Let's say we have a categorical variables with 5 different categories (levels). I train and get a good model based on this dataset using embedding layer with, say, 3 embedding size and with some ...
1
vote
4answers
59 views

Large no of categorical variables with large no of categories

I'm working on a binary classification problem where the dataset is slightly imbalanced (30% class 0 | 70% class 1). Most of my features are categorical with large number of categories. For example: ...
0
votes
1answer
43 views

Categorical Variable and Target Variable

Though a similar question is answered here , but I wanted to take a different approach. Assuming that I have a binary target variable 1/0 and a categorical variable Gender M/F. From this, I can have a ...
1
vote
1answer
119 views

Feature Selection with one-hot-encoded categorical data

I have a dataset with 400+ columns. Almost 90% of these are categorical data with One-Hot-Encoding (OHE). I'm using the dataset for a classification problem. My professors asked me to perform feature ...
0
votes
0answers
21 views

Unsupervised learning/ clustering for data with multiple categorical variables

Dataset: I have been trying unsupervised clustering algorithms (K-modes & SOM) to cluster the students based on their grades in 3 exams. Should I one-hot encode the data (even though grades are ...
0
votes
0answers
16 views

How to use contrast coding schemes in the case of multiclass target variable? How to encode categorical features if contrast coding fails?

How do you deal with a dataset which only has categorical variables, all of whom have high cardinality? What is the right way to encode high cardinality categorical variables if the target variable ...
4
votes
3answers
63 views

Test dataset with categorical variable value not present in train dataset & transformer

I want to replace values of a categorical variable ( named 'six' ) by the mean of my target variable ( named 'target' ). I am fitting a transformer doing just that on a train dataset df and then ...
1
vote
2answers
115 views

Applying mean encoding before or after splitting into train and test set

I have a dataset of 50000 observations with columns of high cardinality. The best way to encode them is with mean encoding, then to use regularization. I will use CV rather than smoothing. But when I ...
0
votes
0answers
12 views

How to Decide Topics For the Documents using LDA

I am trying to Classify Topics From Documents using LDA. I want to get topics classified as human classify from words, https://medium.com/mlreview/topic-modeling-with-scikit-learn-e80d33668730 https:...
4
votes
1answer
406 views

How to handle columns with categorical data and many unique values

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world. I also have another column with 145 nunique values that I could also use ...
1
vote
0answers
35 views

When should embeddings not be used for categorical data? What are their limitations?

I recently came across the concept of embeddings so the concept is still new to me, but it is my understanding that embeddings convert one-hot encoded input data into a dense vector. Vectors ...
1
vote
4answers
90 views

How do I identify clusters that match on categorical data?

I am seeking some directions for a proper path to research the solve for this problem: My company made all our employees take a "StrengthFinders" test, which results in every employee being assigned ...
2
votes
0answers
34 views

How to deal with a potencially multiple categorical variable

I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring....
1
vote
1answer
155 views

Dealing with multiple distinct-value categorical variables

So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values. For instance, one column have more ...
2
votes
1answer
24 views

Positive semidefinite kernel matrix from Gower distance

I have a dataframe with continuous and categorical variables and I want to obtain a kernel matrix for classification. The kernel matrix must be symmetric and positive semidefinite, so that no ...
1
vote
0answers
42 views

Dummy variables for unseen data in R

I got the following problem: When I trained my model I created my dummy variables(before train-test split) in the following way: ...
1
vote
1answer
181 views

What are the approaches to aggregate categorical variables?

I am working on a clickstream dataset. I have come up with the following example dataset to explain my problem: ...
1
vote
1answer
44 views

EDA for analysis of nominal variable with high cardinality

I have a nominal variable (car model) with very high cardinality (~8500 labels) and I would like to analyse its relation with a binary target variable. While I can create logical groups and compare ...
0
votes
1answer
84 views

Categorical data into numeric in excel

I have a large dataset and I would like to convert these categorical data into numeric in binary form to perform k means clustering in R. However, I get an error in value. This is the formula that I ...
1
vote
1answer
47 views

Linear regression model with (categorical) predictor variables

I used LM model with (categorical) predictor variables on my data in r like this (I have count variable as dependent/target variable): ...
2
votes
2answers
76 views

Dealing with a dataset with a mix of continuous and categorical variables

How do the choice of machine learning algorithm and preprocessing change when some of the independent variables are categorical while others are continuous? Can such data be directly applied to the ...
0
votes
1answer
203 views

Converting continuous to categorical variable

What method must be chosen for converting a continuous variable(socio-economic ratio) into a categorical variable, the quantiles are as follows: ...
1
vote
1answer
102 views

Regression - Unbalanced Categorical Features

I have a data set that has some unbalanced categorical features. I would like to build a regression model to predict a label using machine learning (ML). How do I handle data imbalances in ...
0
votes
0answers
17 views

How to perform T-test and chi square test to my categorical variables like country, education and predict accuracy using logistic regression?

I'm new to Data science. I have been working on a classification project which has columns (Sex, Age, Occupation, Marital Status, education, country, relationship,capital gain, income). Here income('&...
0
votes
2answers
937 views

Correlation between nominal categorical variables

I have two arrays, whose values are nominal categorical variables. Each element represents a zone of a city: in the first vector we have the class each zone belongs to (so these might also be seen as ...
1
vote
1answer
106 views

How to handle different input sizes of an NN when One-Hot-Encoding a categorical input?

let's assume an input dataset that is a mix of categorical values and real values. When preprocessing this data into an appropriate NN input, OHE is recommended because it doesn't assume any order of ...
3
votes
1answer
2k views

Confusion about Entity Embeddings of Categorical Variables - Working Example!

Problem Statement: I have problem making the Entity Embedding of Categorical Variable works for a simple dataset. I have followed the original github, or paper, or other blogposts[1,2,or this 3], or ...
0
votes
1answer
39 views

Queries regarding feature importance for categorical features

Queries regarding feature importance for categorical features: Context: I have almost 185 categorical features and these categorical features have either 2 or 3 or 8 or 1 or sometimes 4 categories, ...
1
vote
3answers
55 views

Should I build a different model for each subset

I have a dataset which has categorical variable class. I am trying to solve a regression problem I am not understanding whether I should build a model on entire dataset and consider variable class as ...
0
votes
1answer
419 views

Regression in Python with many NaN values spread across all columns

I want to do a regression to predict "value" based on the other columns from below example table. The data was collected by single indicator and not across all data points, resulting in many NaN/blank ...
1
vote
1answer
46 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
1
vote
0answers
52 views

Input explanatory categorical variables along with time series into neural network

I want an advise on the ways to enter time series along with additional variables into convolutional neural network. Story first: I have a dataset of time series with daily energy consumption data (...
1
vote
1answer
105 views

Algorithm for purely categorical data

Looking for an algorithm to deal with purely categorical data. It was suggested to me to look into the K-medoids algorithm. Anyone know if there is a K-medoids algorithm R library(package)?
1
vote
1answer
46 views

Loss is bad, but accuracy increases?

I have a multicategorial classification problem for images. There are 5 (imbalanced) classes for which i use different class weights. In general there are only a few training images per class: ~56-238 ...
0
votes
1answer
14 views

Entropy loss from collapsing/merging two categories

Suppose I am counting occurrences in a sequence. For a classical example, let's say I'm counting how many of each kind of car comes down a highway. After keeping tally for a while, I see there are ...
0
votes
1answer
48 views

What is the the cost of combining categorical variables?

I have 2 categorical variables e.g. state and city. Missing are only in city. As opposed to throwing out all observations with missing values for city or throwing out city all together I was ...
0
votes
1answer
95 views

PANDAS Within Category Normalization

I'm want to normalize sales data of multiple point of sales (POS), Products and weeks. The dataframe looks like this: ...
2
votes
0answers
54 views

“Binary Encoding” in “Decision Tree” / “Random Forest” Algorithms

Is it OK to use Binary Encoding in a dataset containing categorical columns with very high cardinalities? Some facts about my dataset: My dataset has ~170000 rows One of the categoric variables has ...