Questions tagged [categorical-data]
Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values "label", they do not "measure". Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too.
390
questions
1
vote
1
answer
142
views
Modeling the Price Movement- What analysis should be used
I am still not confident on my analysis and I am really confused that what could be the best way to model such a problem.
Thanks in advance for your help.
- 31
171
votes
4
answers
117k
views
When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?
I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building ...
- 3
2
votes
0
answers
200
views
How to get the maximum likelihood estimate of the categorical distribution parameters using Lagrange optimization?
Let's say our data is discrete-valued and belongs to one of $K$ classes.
The underlying probability distribution is assumed to be a categorical/multinoulli distribution given as $p(\textbf{x}) = \...
- 19.4k
1
vote
1
answer
217
views
How to get dummy variables from "first name"
I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage.
Since the datasets are so big (...
- 19.4k
0
votes
1
answer
34
views
What's the minimum percentage of categories should be present in the categorical variable for to ignore the variable entirely
For example, if i have a feature "colour_codes" that has close to 5000 distinct color codes inside it. And the number of samples/rows is 10 million. Then should I ignore the feature "...
- 904
0
votes
1
answer
32
views
Over-sampling when predicting a contionuous variable
Lets say i am predicting house selling prices (continuous) and therefore have multiple independent variables (numerical and categorical). Is it common practice to balance the dataset when the ...
- 266
0
votes
1
answer
119
views
Separating numerical and categorical features in a binary classification problem
I have a dataset with employee data with around 9500 rows, and have to predict if the target is 0 or 1.
Some of my features are the department of an employee, gender, salary, review_score(numerical),...
- 11
0
votes
2
answers
2k
views
Frequency/Count encoding
How do I perform frequency/count encoding for a train and test set?
The implementations of this encoding I've seen simply frequency encode the categorical variables on a particular dataset (no ...
- 2,846
15
votes
2
answers
5k
views
Why does frequency encoding work?
Frequency encoding is a widely used technique in Kaggle competitions, and many times proves to be a very reasonable way of dealing with categorical features with high cardinality. I really don't ...
- 2,846
2
votes
2
answers
497
views
Categorical to One hot encoding - Big data [closed]
I have a sales dataset which consists of binary label as output - "Business win" and "Business loss" of our products.
We have a set of 1st level customers (lets call that group as ...
- 1,321
0
votes
3
answers
138
views
Transforming Categorical to Numerical variable
I have a categorical variable with 4 levels ('8 c', '6 c','NAN','Others') and I want to convert it to numerical form. an Obvious way is to simply remove the 'c' part from the first two categories and ...
- 2,664
0
votes
2
answers
102
views
Activation and Loss Function not chosen correctly when use Neural Network
I have three classes for my text dataset before.
These are my classes:
0 = Cat
1 = Not Both
2 = Dog
Then I use this code:
...
- 1,595
2
votes
0
answers
39
views
(R) can I convert a categorical variable into a numeric equivalent in linear regression to predict a continuous variable?
Specifically, I have an item code as one of the independent variables that can have several hundred possible values results in underfitting when predicting the projected availability of that item. I'd ...
- 1,595
0
votes
1
answer
26
views
How to make model not too dependent on one variable?
Let's suppose I have a generic model:
Variable A | Variable B | Variable C | Variable D
Variable Dis a categorical variable. ( ...
- 74
1
vote
1
answer
572
views
How to make predictions on unseen data with different cardinality using xgboost
I am training an XGBoost regression model on a feature set $X$ that includes a feature $x_k$ with high cardinality (~100). First, I am using one-hot-encoding to convert $x_k$ and then split the set ...
- 414
0
votes
1
answer
37
views
Entropy loss from collapsing/merging two categories
Suppose I am counting occurrences in a sequence. For a classical example, let's say I'm counting how many of each kind of car comes down a highway.
After keeping tally for a while, I see there are ...
CommunityBot
- 1
1
vote
0
answers
101
views
Add noise to categorical data in tf / Keras?
Just being curious about adding noise to categorical variables. I suspect that adding gaussian noise over a one hot encoded variable wouldn't be enough. I was thinking about adding noise after ...
- 2,359
3
votes
4
answers
564
views
How can I deal with tiny categories?
I'm playing around with UCI Bank Marketing Dataset. So, there is a categorical variable named default which tells us if client "has credit in default". ...
- 10.8k
2
votes
2
answers
318
views
Feature Engineering
I have a data frame with around 37,000 rows and 54 columns. Out of these 54 columns, two columns namely 'user_id' and 'mail_id' are provided in a very creepy format as shown below:
...
CommunityBot
- 1
1
vote
1
answer
23
views
How to test whether data is clustered wrt. subcategories?
I have a dataset of about 2000 entries, containing two numerical values, one categorical and one sub-categorical label for each entry. The data is from chemistry lab data, but for the purpose of this ...
- 24.5k
2
votes
0
answers
1k
views
How to choose the optimal k in k-protoypes?
To analyze a dataset from banking I have both numerical and categorical values. I transform them to analyze with k-prototypes.
The original dataset:
The modified dataset:
E.g.: Job (for 1 to 12 '...
- 1
0
votes
3
answers
1k
views
Dealing with Extra Categories in Test Set
Suppose I have a data set which consists a dependent variable y and independent variables X. Suppose that there is a specific ...
- 376
0
votes
0
answers
2k
views
Should the type of Boolean categorical features be numerical or categorical after encoding?
There are categorical features which have two different value in my dataframe next to numerical features. I've converted these categorical values to 0 or 1.
I will apply correalation elimination on ...
CommunityBot
- 1
1
vote
0
answers
36
views
Why is a column detected as text and not as categorical when opening excel file?
I have the following values in an excel sheet
Aguascalientes
Baja California
Baja California Sur
Campeche
Chiapas
Chihuahua
Ciudad de México
Coahuila
Colima
Durango
Estado de México
Guanajuato
...
- 19.4k
2
votes
1
answer
28
views
Individual differences in response time (RT) experiment - searching for the right test
Given a distribution of response times to different categorical variables, what's the best way to test for individual differences?
or more specifically:
There are 100 people pressing buttons in 10 ...
CommunityBot
- 1
1
vote
1
answer
293
views
Association between Categorical Variables and regression
We perform data analysis and build models. Say, for example, I built a regression model that has more than one predictor (multiple regression). We then check many things: normality, multicollinearity, ...
CommunityBot
- 1
2
votes
2
answers
482
views
How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline
Let us say we have a dataset with a feature such as Surname:
arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith']
And I would like to encode this categorical ...
- 2,361
0
votes
0
answers
30
views
How to consider the change in categorical variable in multiple linear regression?
I am building a multiple linear regression model to predict the mileage of tires and one of the independent variables is the wheel position. It is categorical and I could encode it to run the model ...
- 1,595
2
votes
1
answer
946
views
ML methods for prediction, using categorical variables and time
Most of the time series analysis tutorials/textbooks I found time series data, usually deal with continuous numerical variables. I am currently trying to solve a problem that deals with multivariate ...
CommunityBot
- 1
0
votes
1
answer
4k
views
Separate numerical and categorical variables
I have a dataset (42000, 10) which contains 7 categorical features and 3 numerical. I would like to separate both the numerical and categorical features into 2 different data frames i.e I would like 2 ...
- 1,119
0
votes
2
answers
213
views
logistic regression or density estimation for binary dependent variable and binary (or categorical) features [closed]
I have a binary dependent variable $t$ and categorical features. We can even simplify to binary features since I can one-hot encode the categorical variables. In practice the one-hot encoding induces ...
- 56
4
votes
2
answers
5k
views
Imputation of missing values and dealing with categorical values
I have a dataset (10 million rows, 55 columns) with many missing values. I need to predict those values somehow using other non-missing values, i.e. replace them with something that is not NaN. Mean ...
- 1,321
2
votes
1
answer
199
views
Two questions about one-hot encoding: drop first? and features with thousands of categories [closed]
I have two questions about one-hot feature encoding:
(1) Is it considered a "best practice" to drop the first (or at least one) one-hot encoded feature when one-hot encoding, like you would ...
- 2,246
3
votes
2
answers
3k
views
How to Set the Same Categorical Codes to Train and Test data? Python-Pandas
NOTE:
If someone else it's wondering about this topic, I understand you're getting deeper in the Data Analysis world, so I did this question before to learn that:
You encode categorical values as ...
- 21
1
vote
0
answers
72
views
When doing feature selection, are features like "year", "month" considered as ordinal features or should I convert them to strings?
I am working on a hotel reservation dataset that has both categorical and numerical (continuous and discrete) features (26 columns, 30244 rows). Target is also categorical and it says if the user &...
- 73
1
vote
0
answers
27
views
Feature Selection Statistical Test for Nominal Response Vs Continuous Predictors? (in R)
I cannot find much information on this, none so far useful.
I have a sparse data set with 17K+ columns of continuous gene expressions, an example of a typical column: (3.15, 0, 7.1294, 0, 0, 0, 2300.2,...
1
vote
1
answer
366
views
Handling nominal category features in decision tree
I have been reading some stackoverflow questions on how to handle nominal features for decision tree (sklearn implementation).
One of the answer states that :
Using a OneHotEncoder is the only current ...
- 1,178
0
votes
1
answer
132
views
Handling categorical data with more over 100 unique classes
I am working with a pure categorical data set. And some classes have more than 100 unique values.
I could not find any appropriate encoding possibility. So I created a SQL table, where each value got ...
- 765
0
votes
0
answers
35
views
Encoding Categorical Data
I am trying to predict a car price from a given dataset. When I am trying to encode the categorical data I am getting an error. I am attaching an image with the error below. Can someone help me know ...
- 1
8
votes
4
answers
9k
views
One Hot encoding for large number of values
How do we use one hot encoding if the number of values which a categorical variable can take is large ?
In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary ...
- 189
2
votes
4
answers
3k
views
How do you predict a continuous variable when all your independent variables are categorical
I am new to data science and ML. Recently I have been given a sales dataset which contains weekly sales of a fashion brand. It has information about the product like category(t shirt, polo shirt, ...
- 1,012
2
votes
1
answer
82
views
Clustering with Only Categorical Features
I am trying to do clustering with a bunch (24) of categorical features. I have done some research and found a lot of people recommending something such as K-Modes. I tried running K-Modes on my data ...
CommunityBot
- 1
15
votes
3
answers
30k
views
How to convert categorical data to numerical data in Pyspark
I am using Ipython notebook to work with pyspark applications. I have a CSV file with lots of categorical columns to determine whether the income falls under or over the 50k range. I would like to ...
- 1,595
1
vote
1
answer
457
views
one hot encoding target variable in tree and non tree (knn) methods
I am learning about label encoders, one hot encoding etc applied to datasets for classification via KNN and XGBoost type trees. However, I am a bit confused as to whether the target variable should be ...
- 765
1
vote
2
answers
104
views
Linear Regression with Category variables
I'm currently learning and exploring machine learning and understand the basics of linear regression based on two numerical variables, but now I wish to go a little further and need some guidance ...
- 7,182
1
vote
0
answers
31
views
How to get correlation between the categories of two categorical variable?
I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua&...
- 1,859
8
votes
2
answers
23k
views
Always drop the first column after performing One Hot Encoding?
Since one of the columns can be generated completely from the others, and hence retaining this extra column does not add any new information for the modelling process, would it be good practice to ...
1
vote
2
answers
434
views
Different number of features in train vs test when using Label Encoding
This is not a duplicate of Different number of features in train vs test
There are some categorical columns in my data, and the cardinality for each of them is large, so I chose to use LabelEncoding ...
- 111
0
votes
0
answers
34
views
Equitable selection of users through ranking
I am looking to take a dataset largely derived of user input in categorical form, this sign up sheet asks for many data points such as age group, race, sign up date, as well as a few others. My goal ...
1
vote
1
answer
95
views
Anomaly detection on sparse categorical data
I have a big dataset with a column "clientid" and a categorical column "choice".
I want to find out what are the clients that have strange combinations of choices (less frequent ...
- 691