Questions tagged [categorical-data]

Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.

Filter by
Sorted by
Tagged with
2
votes
2answers
320 views
0
votes
1answer
62 views

What to do if a specific label of a category appears only a few times?

Let's say I am trying to predict whether a car will be auctioned or not (not what I'm actually trying to do, but it represents it pretty well) using tabular data. I have the year the car was made, its ...
4
votes
2answers
256 views

Anomaly detection using clustering of highly correlated Categorical data

My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else it's Anomaly. Likewise, there are thousands ...
1
vote
1answer
53 views

DBSCAN on textual and numerical columns

I have a dataset which has two columns: title price sentence1 12 sentence2 13 I have used doc2vec to convert the ...
2
votes
2answers
95 views

How to leverage description data in multi-class classification (dimensionality reduction)

I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as ...
2
votes
1answer
407 views

ML methods for prediction, using categorical variables and time

Most of the time series analysis tutorials/textbooks I found time series data, usually deal with continuous numerical variables. I am currently trying to solve a problem that deals with multivariate ...
0
votes
1answer
46 views

Separate numerical and categorical variables

I have a dataset (42000, 10) which contains 7 categorical features and 3 numerical. I would like to separate both the numerical and categorical features into 2 different data frames i.e I would like 2 ...
1
vote
3answers
82 views

logistic regression or density estimation for binary dependent variable and binary (or categorical) features

I have a binary dependent variable $t$ and categorical features. We can even simplify to binary features since I can one-hot encode the categorical variables. In practice the one-hot encoding induces ...
3
votes
2answers
5k views

Imputation of missing values and dealing with categorical values

I have a dataset (10 million rows, 55 columns) with many missing values. I need to predict those values somehow using other non-missing values, i.e. replace them with something that is not NaN. Mean ...
3
votes
1answer
139 views

PCA and k-means for categorical variables?

I have a clustering task at hand. The data that I have contains only categorical variables. So, k-modes seemed like the best option. But I am not sure what are the data pre processing steps required ...
0
votes
3answers
37 views

Transforming Categorical to Numerical variable

I have a categorical variable with 4 levels ('8 c', '6 c','NAN','Others') and I want to convert it to numerical form. an Obvious way is to simply remove the 'c' part from the first two categories and ...
0
votes
0answers
10 views

What is the recommended embedding for a categorical variable with more than 40000 thousand categories?

I have a feature called Planning_id with more than 40000 categories. What is the recommended embedding size? I read that: embedding_dimension = # categories * 0.25 is a good rule of thumb, but I still ...
1
vote
1answer
149 views

Association between Categorical Variables and regression

We perform data analysis and build models. Say, for example, I built a regression model that has more than one predictor (multiple regression). We then check many things: normality, multicollinearity, ...
-1
votes
0answers
17 views

How to use one hot encoding with time series data ( arima eg) [closed]

I have cumulative number of medical cases weekly for 60 weeks and categorical data on week wise events that occurred. I’m trying to analyse which event may increase or decrease the cumulative cases. <...
0
votes
0answers
12 views

How to treat a column that contains a list of categorical, high cardinality values for a classification problem?

The list cannot be exploded into several columns because this will result in very high dimensionality. I would like to know the following: How to treat this kind of column in a dataframe? Can I keep ...
1
vote
1answer
32 views

How to build multiple variable regression having a mix of numerical & categorical features?

There is a need to estimate Annual Average Daily Traffic Volume (AADT). We have bunch of data about vehicles' speeds during several years. It is noticed that AADT depends on the average number of such ...
0
votes
0answers
26 views

How to train a GAN to generate categorical variable

I am trying to train a simple GAN to generate a categorical variable size, which takes discrete values between 1-100. I am looking for some tips or directions on ...
1
vote
1answer
232 views

Encode missing data and unseen data

Let's assume that I have a classification problem and all my features are categorical data. I have missing data (and I do not want to do any imputation). Also, I know that I will have some unseen ...
0
votes
1answer
43 views

How to encode high cardinality categorical data?

I have a dataset of 1600 rows and 28 columns. Only one column is partially complete with 1300 records. The rest is NaN. I did a value count of this columns and it has 84 different categories that are ...
3
votes
1answer
43 views

How to identify if there is a relationship between 5 categorical independent variables to a binary dependent variable?

My dataset has 5 independent variables, each with a value of either Large, Medium or None and a binary dependent variable. The dataset has 67 rows with a split of 17:50. I would like to identify if ...
153
votes
4answers
95k views

When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?

I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building ...
0
votes
0answers
26 views

How to run PCA when data contains some categorial features?

Assume that we have a dataset with various features, and some of the features are categorial. And PCA dosn't work good on categorical features. How should I handle such datasets using PCA, what is ...
2
votes
1answer
31 views

How to retrain a K-Modes model based on daily data?

I have read that retraining a model depends highly on what you are trying to achieve. I am conscious that maybe I need to retrain my model daily and after a certain time I have to train the model ...
2
votes
1answer
31 views

Two questions about one-hot encoding: drop first? and features with thousands of categories [closed]

I have two questions about one-hot feature encoding: (1) Is it considered a "best practice" to drop the first (or at least one) one-hot encoded feature when one-hot encoding, like you would ...
0
votes
1answer
878 views

Multiple variable as input and output

I'm trying to predict the possible diagnosis given a consultation reason. I have ID's for all the data. So my data kind of looks like below ...
0
votes
2answers
5k views

Correlation between nominal categorical variables

I have two arrays, whose values are nominal categorical variables. Each element represents a zone of a city: in the first vector we have the class each zone belongs to (so these might also be seen as ...
3
votes
2answers
569 views

How to Set the Same Categorical Codes to Train and Test data? Python-Pandas

NOTE: If someone else it's wondering about this topic, I understand you're getting deeper in the Data Analysis world, so I did this question before to learn that: You encode categorical values as ...
2
votes
1answer
124 views

Logistic Regression Model for categorical features with multiple values in each category

I am working on an insurance use case to build a logistic regression classifier to predict if a policy will lapse or not. The dataset has more than 20 categorical features for a policy. Each ...
1
vote
1answer
183 views

Is there any way to collect categorical features quickly in Julia DataFrames?

I'm using Julia 0.6.3 with Dataframes.jl I was wondering if there was any way to get categorial features easily in Julia? For large datasets it can be impossible to enter everything by hand. My ...
1
vote
0answers
23 views

When doing feature selection, are features like “year”, “month” considered as ordinal features or should I convert them to strings?

I am working on a hotel reservation dataset that has both categorical and numerical (continuous and discrete) features (26 columns, 30244 rows). Target is also categorical and it says if the user &...
0
votes
2answers
40 views

How would I classify this variable?

I am learning about the difference between categorical, ordinal and numerical variables. From what I understand: Categorical variables have 2+ categories without any intrinsic order. Ordinal ...
1
vote
0answers
21 views

Feature Selection Statistical Test for Nominal Response Vs Continuous Predictors? (in R)

I cannot find much information on this, none so far useful. I have a sparse data set with 17K+ columns of continuous gene expressions, an example of a typical column: (3.15, 0, 7.1294, 0, 0, 0, 2300.2,...
1
vote
1answer
652 views

faster alternatives to sparse.model.matrix?

I have a large dataset that is entirely categorical. I'm trying to train with it using xgboost, so I must first convert this categorical data to numerical. So far I've been using sparse.model.matrix() ...
1
vote
1answer
29 views

Handling nominal category features in decision tree

I have been reading some stackoverflow questions on how to handle nominal features for decision tree (sklearn implementation). One of the answer states that : Using a OneHotEncoder is the only current ...
1
vote
0answers
10 views

spark ml StringIndexer vs OneHotEncoder, when to use which?

Confused as to when to use StringIndexer vs StringIndexer+OneHotEncoder. The OneHotEncoder docs say For string type input data, it is common to encode categorical features using StringIndexer first. ...
0
votes
1answer
27 views

Entropy loss from collapsing/merging two categories

Suppose I am counting occurrences in a sequence. For a classical example, let's say I'm counting how many of each kind of car comes down a highway. After keeping tally for a while, I see there are ...
1
vote
0answers
24 views

An Unsupervised learning method suitable for large categorical data sets

I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
1
vote
1answer
79 views

Queries regarding feature importance for categorical features

Queries regarding feature importance for categorical features: Context: I have almost 185 categorical features and these categorical features have either 2 or 3 or 8 or 1 or sometimes 4 categories, ...
0
votes
1answer
16 views

Handling categorical data with more over 100 unique classes

I am working with a pure categorical data set. And some classes have more than 100 unique values. I could not find any appropriate encoding possibility. So I created a SQL table, where each value got ...
0
votes
1answer
34 views

Model for predicting duration based on categorical data

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
1
vote
1answer
99 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
0
votes
1answer
1k views

Should the type of Boolean categorical features be numerical or categorical after encoding?

There are categorical features which have two different value in my dataframe next to numerical features. I've converted these categorical values to 0 or 1. I will apply correalation elimination on ...
1
vote
2answers
23 views

How do I assign specific values to categorical variables

I have a Pandas data frame with columns within a survey with the following categorical values - "Increased, Decreased, Neutral". My question is how can I assign specific numerical values to ...
1
vote
2answers
259 views

What are some good methods to forecast future revenue on categorical and value based data?

I have monthly snapshots (3 years) of all the contract data. It includes following information: Contract status [Categorical]: Proposed, tracked, submitted, won, lost, etc Contract stages [...
1
vote
1answer
331 views

Aggregating multiple encoded categorical values

I am trying find commonly used techniques when dealing with high cardinality multi-valued categorical variables. I am currently using a dataset with a feature CATEGORY which has a cardinality of ~20,...
1
vote
2answers
2k views

Dealing with multiple distinct-value categorical variables

So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values. For instance, one column have more ...
0
votes
0answers
31 views

Encoding Categorical Data

I am trying to predict a car price from a given dataset. When I am trying to encode the categorical data I am getting an error. I am attaching an image with the error below. Can someone help me know ...
8
votes
4answers
7k views

One Hot encoding for large number of values

How do we use one hot encoding if the number of values which a categorical variable can take is large ? In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary ...
2
votes
5answers
193 views

How do you predict a continuous variable when all your independent variables are categorical

I am new to data science and ML. Recently I have been given a sales dataset which contains weekly sales of a fashion brand. It has information about the product like category(t shirt, polo shirt, ...
2
votes
1answer
63 views

Clustering with Only Categorical Features

I am trying to do clustering with a bunch (24) of categorical features. I have done some research and found a lot of people recommending something such as K-Modes. I tried running K-Modes on my data ...

1
2 3 4 5
7