Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
2
votes
1answer
3k views

Updating One-Hot Encoding to account for new categories

My question is focused around how to appropriately update an encoded feature set when a new category is introduced by the test data. I use the data in logistic regression and I know it is not a 'live' ...
2
votes
1answer
2k views

Multidimensional Scaling with Categorical Data

I have read the following about MDS in a book: using MDS requires an understanding of the individual feature's units; maybe we are using features that cannot be compared using the Euclidean ...
2
votes
1answer
362 views

Why is there such a mismatch between the Model's predicted probability and theoretical probability in logistic regression?

I am trying to do Logistic Regression using SAS Enterprise Miner. My Independent variables are ...
2
votes
1answer
32 views

Two questions about one-hot encoding: drop first? and features with thousands of categories [closed]

I have two questions about one-hot feature encoding: (1) Is it considered a "best practice" to drop the first (or at least one) one-hot encoded feature when one-hot encoding, like you would ...
2
votes
1answer
27 views

What is the best alternative for Fisher's Exact test for contigency tables that are NOT 2x2?

I am a newbie to data mining. I am trying to find associations between two categorical variables. Since more than 20% of my expected frequencies are less than 5, I wanted to use Fisher exact test but ...
2
votes
1answer
184 views

Categories with the same mean in target encoding

While doing target encoding it can happen that two categories have the same target mean. This is bad because there will be no difference in the new feature in it and we will lose some information. ...
2
votes
1answer
63 views

Clustering with Only Categorical Features

I am trying to do clustering with a bunch (24) of categorical features. I have done some research and found a lot of people recommending something such as K-Modes. I tried running K-Modes on my data ...
2
votes
1answer
411 views

ML methods for prediction, using categorical variables and time

Most of the time series analysis tutorials/textbooks I found time series data, usually deal with continuous numerical variables. I am currently trying to solve a problem that deals with multivariate ...
2
votes
1answer
55 views

Different encoders applied to a dataset

I have a dataset which have both categorical features with high cardinality (>8000) and low cardinality (4 or 5). Would that be ok to encode the high cardinality ones with one encoder (target encoder,...
2
votes
2answers
218 views

Categorical features preprocessing for clustering

Can anyone tell suggest the best practice for clustering data with mixtured features (both with categorical and continuous). I am struggling with a problem; I realized that for all metrics algorithms ...
2
votes
1answer
2k views

What are the approaches to aggregate categorical variables?

I am working on a clickstream dataset. I have come up with the following example dataset to explain my problem: ...
2
votes
1answer
250 views

EDA for analysis of nominal variable with high cardinality

I have a nominal variable (car model) with very high cardinality (~8500 labels) and I would like to analyse its relation with a binary target variable. While I can create logical groups and compare ...
2
votes
2answers
693 views
2
votes
1answer
59 views

Same predictors in test set but I want different outputs

I have a (training) dataset about what TV spectators are watching and for how long. The goal (at new set - the test set) is to predict for how long the TV spectators will watch a specific channel and ...
2
votes
1answer
615 views

Nominal categorical variable with two levels: Label Encoding or One Hot encoding?

For a nominal categorical variable that has two levels, e.g. Gender (levels = Male,Female), is it feasible to use label encoding instead of One Hot encoding ? If it is, are there any implications of ...
2
votes
1answer
1k views

Outlier detection on categorical network log data

I am working with a completely categorical network log data that consists of source ip address, destination ip address, source port, destination port, protocol. Data Preprocessing performed : ...
2
votes
2answers
292 views

Feature Engineering

I have a data frame with around 37,000 rows and 54 columns. Out of these 54 columns, two columns namely 'user_id' and 'mail_id' are provided in a very creepy format as shown below: ...
2
votes
2answers
8k views

Checking Correlation of Categorical variables in SPSS

I am building a predictive model for a classification problem using SPSS. Of the Independent variables, I have both Continuous and Categorical variables. SPSS gives only correlation between continuous ...
2
votes
1answer
105 views

Clustering with constraints

I have a dataset of this form: chrX posX labelX where chrX refers to the chromosome number, ...
2
votes
3answers
1k views

Choosing the right data mining method to find the effect of each parameter over the target

I am dealing with a lot of categorical data right now and I would like to use an appropriate data mining method in any tool [preferably R] to find the effect of each parameter [categorical parameters] ...
2
votes
1answer
35 views

Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality

When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less ...
2
votes
0answers
27 views

How to get the maximum likelihood estimate of the categorical distribution parameters using Lagrange optimization?

Let's say our data is discrete-valued and belongs to one of $K$ classes. The underlying probability distribution is assumed to be a categorical/multinoulli distribution given as $p(\textbf{x}) = \...
2
votes
2answers
583 views

Why is count encoding effective in improving accuracy? [duplicate]

Can someone please explain why/how Count encoding of categorical features improve accuracy in classification when compared to simply label encoding them ? I found one explanation in kaggle " ...
2
votes
0answers
52 views

Regression dataset with categorical features

I have thought of a regression technique that I want to try on several datasets. I would like these datasets to have the following properties: Be a tabular dataset (no images). Have at least 20k rows,...
2
votes
2answers
322 views

Clustering mixed data types - numeric, categorical, arrays, and text

I have a dataset with 4 types of data columns: ...
2
votes
0answers
889 views

Dealing with categorical variables in Isolation Forest

Isolation Forest is widely used when dealing with outlier/anomaly detection when we have no labels. The theory behind is that making random split at random points and counting how many splits you do ...
2
votes
0answers
34 views

ML recommendation system with items organized in a tree

I would like to develop a recommendation system (probably hybrid, user-based and feature-based) for items which are organized in a tree (there are categories, divided in sub-categories, divided in sub-...
2
votes
1answer
32 views

How to retrain a K-Modes model based on daily data?

I have read that retraining a model depends highly on what you are trying to achieve. I am conscious that maybe I need to retrain my model daily and after a certain time I have to train the model ...
2
votes
1answer
124 views

Logistic Regression Model for categorical features with multiple values in each category

I am working on an insurance use case to build a logistic regression classifier to predict if a policy will lapse or not. The dataset has more than 20 categorical features for a policy. Each ...
2
votes
0answers
278 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
2
votes
2answers
95 views

How to leverage description data in multi-class classification (dimensionality reduction)

I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as ...
2
votes
1answer
1k views

Naive Bayes for Categorical Features (Non Binary)

How do i use Naive Bayes Classifier (Using sklearn) for a Dataset considering that my feature set is categorical, ie more than 2 categories per feature are present. I've looked everywhere, some ...
2
votes
1answer
130 views

How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline

Let us say we have a dataset with a feature such as Surname. arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith'] And I want to encode this categorical info ...
2
votes
0answers
52 views

Why don't Target/LeaveOneOut Encoders work well for Regression Tasks?

In this review of categorical encoding, it states early on that For regression tasks, Target and LeaveOneOut probably won’t work well and later repeats that Target/LeaveOneOut (Owen Zhang's ...
2
votes
0answers
107 views

How to deal with a potencially multiple categorical variable

I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring....
2
votes
1answer
71 views

Positive semidefinite kernel matrix from Gower distance

I have a dataframe with continuous and categorical variables and I want to obtain a kernel matrix for classification. The kernel matrix must be symmetric and positive semidefinite, so that no ...
2
votes
1answer
98 views

Selecting the right time series model [closed]

Using Python, I am trying to predict the future sales count of a product, using historical sales data. I am also trying to predict these counts for various groups of products. For example, my columns ...
2
votes
1answer
27 views

Individual differences in response time (RT) experiment - searching for the right test

Given a distribution of response times to different categorical variables, what's the best way to test for individual differences? or more specifically: There are 100 people pressing buttons in 10 ...
2
votes
2answers
485 views

Transform Categorical Variables into Numerical

I'm very new to machine learning approaches. I'm reading a tutorial for build a predictive model using random forests. One of the transformations implemented was transform categorical variables to ...
2
votes
0answers
1k views

How to choose the optimal k in k-protoypes?

To analyze a dataset from banking I have both numerical and categorical values. I transform them to analyze with k-prototypes. The original dataset: The modified dataset: E.g.: Job (for 1 to 12 '...
2
votes
0answers
103 views

Type of Test to Determine Correlation in R

I have a dataset of approximately 48,000 rows each one a click of a an article, some of these clicks were also comments. For each article I have the country and subject of the article and name of ...
2
votes
0answers
65 views

Tag categorizer

I'm using Google prediction API and I need to perform tag categorization for free text. So when I receive a text like "Cant update application" is tagged with #Apps #General #CantUpdate. My ...
1
vote
3answers
3k views

Transformation of categorical variables

I have a data with continous variables and categorical variables. I am using Random Forest and have made my continues variables Gaussian by transformation and have standardized it. Should categorical ...
1
vote
3answers
3k views

What should I use if I have millions of possible values for a feature in a sklearn predictive model?

I am trying to create a large model. One of the features is categorical, and it has almost 100 million entries. I have looked at sklearn LabelEncoder, but I am concerned that it will still create an ...
1
vote
3answers
240 views

Correlation between categorical variables based on the target distribution

Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way ...
1
vote
3answers
371 views

Should I build a different model for each subset

I have a dataset which has categorical variable class. I am trying to solve a regression problem I am not understanding whether I should build a model on entire dataset and consider variable class as ...
1
vote
3answers
292 views

What is the best way to visualize the relationship two categorical variables

I am currently working on an ambulance dataset and one of my tasks is to find when a patient was misdiagnosed by the call dispatcher. I have two codes; a dispatch code(what the dispatcher believes is ...
1
vote
2answers
72 views

Clustering on categorical attributes

I have a dataset with only 2 categorical attributes out of 9. How can I get a clustering analysis on it? I am using R. Do you have any advices about instructions, how to do it, topics, ...? here's my ...
1
vote
1answer
46 views

What are the best practises to decide whether a variable is categorical?

What are some of the systematic ways to categorise variables into categorical or numeric? I believe using only intuition in such scenarios can many-a-times lead to major irreversible errors. What are ...
1
vote
4answers
103 views

How do I identify clusters that match on categorical data?

I am seeking some directions for a proper path to research the solve for this problem: My company made all our employees take a "StrengthFinders" test, which results in every employee being assigned ...

1 2
3
4 5
7