Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
1
vote
1answer
1k views

High train and val results. Bad test and predict results

For my thesis project I've been trying to make a CNN for some challenging data. There's four classes with the following amount of images respectively [410, 410, 269, 206] = 1,295 total. Now I know ...
1
vote
0answers
240 views

Embedding Layer on unseen data

Let's say we have a categorical variables with 5 different categories (levels). I train and get a good model based on this dataset using embedding layer with, say, 3 embedding size and with some ...
4
votes
3answers
151 views

Test dataset with categorical variable value not present in train dataset & transformer

I want to replace values of a categorical variable ( named 'six' ) by the mean of my target variable ( named 'target' ). I am fitting a transformer doing just that on a train dataset df and then ...
3
votes
2answers
3k views

Applying mean encoding before or after splitting into train and test set

I have a dataset of 50000 observations with columns of high cardinality. The best way to encode them is with mean encoding, then to use regularization. I will use CV rather than smoothing. But when I ...
5
votes
2answers
3k views

One hot encoding large dataset

Initially, I have a dataset where for each row there is user_id and product_ids he bought. In that dataset there are 478191 products bought by different users. In order to discover frequent items ...
5
votes
2answers
469 views

How to continue incremental learning when a categorical variable has been assigned additional category labels?

Please help answer this question or point me to any resource. There is a model in an environment where training happens with new data and the data is discarded after training is completed. This keeps ...
1
vote
1answer
2k views

customer segmentation with categorical variables

I was adviced to write in this group regarding my question about modeling categorical database. I have a customer dataset, which is a survey result. I have 1595 obs. and about 200 columns(200 because ...
5
votes
3answers
843 views

Quasi-categorical variables - any ideas?

Let's say I'm trying to predict a person's electricity consumption, using the time of day as a predictor (hours 00-23), and further assume I have a hefty but finite amount of historical measurements. ...
5
votes
1answer
3k views

How to handle columns with categorical data and many unique values

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world. I also have another column with 145 nunique values that I could also use ...
1
vote
4answers
103 views

How do I identify clusters that match on categorical data?

I am seeking some directions for a proper path to research the solve for this problem: My company made all our employees take a "StrengthFinders" test, which results in every employee being assigned ...
2
votes
1answer
251 views

EDA for analysis of nominal variable with high cardinality

I have a nominal variable (car model) with very high cardinality (~8500 labels) and I would like to analyse its relation with a binary target variable. While I can create logical groups and compare ...
1
vote
0answers
108 views

When should embeddings not be used for categorical data? What are their limitations?

I recently came across the concept of embeddings so the concept is still new to me, but it is my understanding that embeddings convert one-hot encoded input data into a dense vector. Vectors ...
0
votes
1answer
1k views

Categorical data into numeric in excel

I have a large dataset and I would like to convert these categorical data into numeric in binary form to perform k means clustering in R. However, I get an error in value. This is the formula that I ...
2
votes
0answers
108 views

How to deal with a potencially multiple categorical variable

I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring....
2
votes
1answer
71 views

Positive semidefinite kernel matrix from Gower distance

I have a dataframe with continuous and categorical variables and I want to obtain a kernel matrix for classification. The kernel matrix must be symmetric and positive semidefinite, so that no ...
4
votes
2answers
3k views

Categorical data for sklearns Isolation Forrest

I'm trying to do anomaly detection with Isolation Forests (IF) in sklearn. Except for the fact that it is a great method of anomaly detection, I also want to use it because about half of my features ...
1
vote
0answers
86 views

Dummy variables for unseen data in R

I got the following problem: When I trained my model I created my dummy variables(before train-test split) in the following way: ...
2
votes
1answer
2k views

What are the approaches to aggregate categorical variables?

I am working on a clickstream dataset. I have come up with the following example dataset to explain my problem: ...
2
votes
3answers
1k views

Best practices for selecting categorical features

I'm trying to create a classifier that will predict whether someone will attend an interview or not. Each data point is for a single candidate and contains details such as the location of the ...
0
votes
1answer
1k views

Converting continuous to categorical variable

What method must be chosen for converting a continuous variable(socio-economic ratio) into a categorical variable, the quantiles are as follows: ...
1
vote
1answer
792 views

Regression - Unbalanced Categorical Features

I have a data set that has some unbalanced categorical features. I would like to build a regression model to predict a label using machine learning (ML). How do I handle data imbalances in ...
10
votes
1answer
7k views

Confusion about Entity Embeddings of Categorical Variables - Working Example!

Problem Statement: I have problem making the Entity Embedding of Categorical Variable works for a simple dataset. I have followed the original github, or paper, or other blogposts[1,2,or this 3], or ...
1
vote
1answer
178 views

Algorithm for purely categorical data

Looking for an algorithm to deal with purely categorical data. It was suggested to me to look into the K-medoids algorithm. Anyone know if there is a K-medoids algorithm R library(package)?
1
vote
3answers
371 views

Should I build a different model for each subset

I have a dataset which has categorical variable class. I am trying to solve a regression problem I am not understanding whether I should build a model on entire dataset and consider variable class as ...
0
votes
1answer
2k views

Regression in Python with many NaN values spread across all columns

I want to do a regression to predict "value" based on the other columns from below example table. The data was collected by single indicator and not across all data points, resulting in many NaN/blank ...
0
votes
1answer
2k views

PANDAS Within Category Normalization

I'm want to normalize sales data of multiple point of sales (POS), Products and weeks. The dataframe looks like this: ...
1
vote
0answers
232 views

Input explanatory categorical variables along with time series into neural network

I want an advise on the ways to enter time series along with additional variables into convolutional neural network. Story first: I have a dataset of time series with daily energy consumption data (...
1
vote
1answer
365 views

Loss is bad, but accuracy increases?

I have a multicategorial classification problem for images. There are 5 (imbalanced) classes for which i use different class weights. In general there are only a few training images per class: ~56-238 ...
0
votes
1answer
225 views

What is the the cost of combining categorical variables?

I have 2 categorical variables e.g. state and city. Missing are only in city. As opposed to throwing out all observations with missing values for city or throwing out city all together I was ...
1
vote
1answer
1k views

Prediction with unseen values in categorical variables

I have created an Artificial Neural Network with 4 features. I am at the point where I want to test the model with a live sample of a malicious file path/exe using: ...
0
votes
1answer
94 views

250 Categorical values

I have a dataset which has only categorical values. As I came across a few articles people suggested that KNN / Random forest would work for dataset like this. Though in R it couldn't handle as if ...
12
votes
3answers
9k views

How can I do classification with categorical data which is not fixed?

I have a classification problem with both categorical and numerical data. The problem I'm facing is that my categorical data is not fixed, that means that the new candidate whose label I want to ...
6
votes
3answers
2k views

Classifier and Technique to use for large number of categories

I am designing a scikit learn classifier for a sequence labelling task which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year. I have ...
1
vote
0answers
26 views

Suggestions on using model in production 1 test at a time

I have created an Artificial Neural Network with 4 categorical features and a binary outcome either 1 for suspicious or 0 for non-suspicious: ...
2
votes
2answers
162 views

Dummy variable for Categorical values

The question is in reference to solution of Titanic survival predictionat kaggle . As many have did the similar kind of feature extraction, They have converted some of the numerical features (Age, ...
2
votes
1answer
98 views

Selecting the right time series model [closed]

Using Python, I am trying to predict the future sales count of a product, using historical sales data. I am also trying to predict these counts for various groups of products. For example, my columns ...
9
votes
4answers
12k views

How to combine PCA and MCA on mixed data?

Suppose I have mixed data and (python) code which is capable of doing PCA (principal component analysis) on continuous predictors and MCA (multiple correspondence analysis) on nominal predictors. Is ...
8
votes
1answer
9k views

Keras categorical_crossentropy loss (and accuracy)

When training a neural network with keras for the categorical_crossentropy loss, how exactly is the loss defined? I expect it to be the average over all samples of $$\textstyle\text{loss}(p^\text{true}...
1
vote
0answers
3k views

How to plot a heatmap-like plot for categorical features?

I would greatly appreciate let me know how to plot a heatmap-like plot for categorical features? In fact, based on this post, the association between categorical ...
1
vote
3answers
292 views

What is the best way to visualize the relationship two categorical variables

I am currently working on an ambulance dataset and one of my tasks is to find when a patient was misdiagnosed by the call dispatcher. I have two codes; a dispatch code(what the dispatcher believes is ...
0
votes
1answer
713 views

Convert nominal to numeric variables?

I am trying to develeop an algorithm with sklearn and Tensorflow to predict which car can be offer to each customer. To do that I have a database with the answers of one survey to 1000 customers. An ...
2
votes
1answer
59 views

Same predictors in test set but I want different outputs

I have a (training) dataset about what TV spectators are watching and for how long. The goal (at new set - the test set) is to predict for how long the TV spectators will watch a specific channel and ...
3
votes
1answer
3k views

Data scaling before PCA: how to deal with categorical values?

I have to apply PCA on a dataset, which contains both numerical and categorical values. In the preprocessing phase, I converted all the categorical values in numerical, so that the software can deal ...
1
vote
0answers
1k views

What is the relationship between correlation ratio and one-way Anova?

According to the answer to this post, it is recommended to use one-way anova to compute the dependence between a categorical and a numerical variable. Besides, the second answer to this post says ...
1
vote
3answers
2k views

How to best forecast simple binary data? [closed]

I have a set of timeseries binary (boolean) data, with intervals of 1 day. Each day can either be 1 or 0 (true/false). What is the best way to forecast the next day/week's data based on the data I ...
12
votes
3answers
26k views

Mass convert categorical columns in Pandas (not one-hot encoding)

I have pandas dataframe with tons of categorical columns, which I am planning to use in decision tree with scikit-learn. I need to convert them to numerical values (not one hot vectors). I can do it ...
1
vote
1answer
275 views

Dropping less frequently used categorical data?

I'm new to the datascience field and working on an assignment. I have a dataset with 150K rows with a categorical and numerical data, the target is a boolean. A categorical column consist of quite ...
1
vote
2answers
206 views

Is there a quick way to check for multicollinearity between categorical variables in R?

I have a large amount of categorical and dummy variables (36) and I would like to remove a number of them based on their multicollinearity (or just collinearity). Instead of using Chi Square tests ...

1
3 4
5
6 7