Questions tagged [dummy-variables]
The dummy-variables tag has no usage guidance.
43
questions
-1
votes
1
answer
43
views
Beginner data scientist looking for help. (apologies for the vagueness ) [closed]
I am analyzing airbnb data to understand which variable is most correlated and in doing so build a model. Upon tackling qualitative variables I decided to use dummies to assign quantitative values to ...
0
votes
0
answers
350
views
How to do DBSCAN clustering with mixed variables (numerical features and binary/ordinal variables)?
I have a question written at the end of the post which refers to the "Distances" paragraph. The other first two paragraphs give additional info.
Context
I'm working on a project where I have ...
- 13
2
votes
1
answer
514
views
Dummy Variable trap in Linear Regression
The dummy variable trap is a common problem with linear regression when dealing with categorical variables, since one hot encoding introduces redundancy, so if we have m categories in our categorical ...
- 35
0
votes
0
answers
27
views
Dealing with conditional variables in anomaly detection
I am working on an anomaly detection model and I have a conditional variable, i.e., it is zero or it has an amount like below histogram. Suppose the variable shows the time when a machine is not ...
- 11
0
votes
1
answer
140
views
Is there a way to forecast a time series multiple linear regression using externally made dummy variables?
This question concerns question 4h of this textbook exercise. It asks to make future predictions based on a chosen TSLM model which involves an endogenously (if i'm using this right) made dummy ...
- 21
0
votes
0
answers
16
views
Dummy Predictors / Continuous Dependent Variable
I have a dataset with 50+ dummy coded variables that represent the purchases of an individual customer. Columns represent the products and the cell values 0 or 1, whether the product has been ...
1
vote
2
answers
24
views
Should I create single feature for each specific word which i find in text or one for all them?
I am doing feature engineering right now for my classification task. In my dataframe I have a column with text messages. I decided to create a binary feature which depends on whether or not in this ...
- 183
1
vote
1
answer
273
views
How to get dummy variables from "first name"
I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage.
Since the datasets are so big (...
- 111
1
vote
2
answers
4k
views
inconsistency between y and x numbers in the Split into train and test sets
I am new to the field to the data science, and need help in the following:
I am working on a data set that consists of both categorical and numerical values, first I have concatenate the two files (...
- 11
1
vote
2
answers
193
views
what would be the correct representation of categorical variables like sex?
I have a doubt about what will be the right way to use or represent categorical variables with only two values like "sex". I have checked it up from different sources, but I was not able to ...
- 217
1
vote
1
answer
91
views
Use dummy variables to create a rank variable. R
I have a series of multiple response (dummy) variables describing causes for a canceled visits. A visit can have multiple reasons for the cancelation. My goal is to create a single mutually exclusive ...
- 37
0
votes
1
answer
36
views
Dummies Variables and Scaling in Regression Problems
I was wondering if having dummies variable and scaling other variables could joke my model. In particular, I have implemented a Random Forest Regressor by using scikit-learn, but my data model is ...
- 11
2
votes
1
answer
341
views
What exactly is a dummy trap? Is dropping one dummy feature really a good practice?
So I'm going through a Machine Learning course, and this course explains that to avoid the dummy trap, a common practice is to drop one column. It also explains that since the info on the dropped ...
1
vote
1
answer
782
views
Should I include all dummy variables or N-1 dummy variables (keep one as reference) in neural networks
I have a categorical variable with N factor levels (e.g. gender has two levels) in classification problem. I have converted it into dummy variables (male and female).
I have to use neural network (...
- 125
0
votes
1
answer
58
views
how do tree based methods deal with missing feature columns?
all,
i have trained a model using xgboost. Some of the features are one hot encoded e.g. currency where it is either gbp or usd. it seems that when i output the feature importance gbp and usd were in ...
- 506
1
vote
2
answers
131
views
Problem with converting string to dummy variables
I'm new in data science, I have data which want to work on it, I omitted extra columns and convert it to 4 columns ( Product, Date, Market, Demand ) . in this data Product and Market are string, I ...
- 13
1
vote
1
answer
68
views
How to handle fixed values for variables in pre-processing
I have a dataset which contains few variables whose values do not change. Some of the variables are non-numeric (for example all values for that variable contain the value 5) and few variables are ...
- 521
0
votes
2
answers
877
views
Pandas get_dummies() rows dropping after joining back with X
I'm having an issue that I can't explain and am hoping I am missing something simple.
I have a large dataset of shape(45Million+, 51) and am loading it in for some analyses (classifiers, deep learning,...
- 101
1
vote
1
answer
725
views
onehotencoder random forest
In a Random Forest context, do I need to setup dummies/OnehotEncoder in a dataset where features/varibles are numerical but refer to some kind of category?
Let's say I have the following variables:
...
- 113
0
votes
1
answer
260
views
Creating dummy variables [closed]
I have a Eurobarometer dataset and want to eventually create a logistic regression model and a linear probability model using a set of dates as the dependent variable.
However, the dates in the ...
2
votes
1
answer
137
views
Dummy variable only for character value in a column (Neglecting float and integers)
My dataset consists of 3000 rows and 50 columns, out of which one column (ESTIMATE_FAMILY_CONTRIBUTION) contains all numerical value(around 2000 different values like 20,30,32....) but got one value ...
1
vote
1
answer
648
views
Random Forest in R with only character variables
I am new to using random forest in R an my goal is to identify the independent variables which have the highest impact on the dependent variables. I am looking at sales data, and sales is my dependent ...
- 53
1
vote
1
answer
145
views
Handling Numerical Categorical Column in ML models in Python
When I was exploring the titanic dataset to estimate the probability of a person of surviving using the Logistic Model, I realized there are two ways of handling numerical categorical variables :
Use ...
- 115
2
votes
1
answer
1k
views
How to interpret dummy variable in ML prediction?
I am working on a binary classification problem where I have a mix of continuous and categorical variables.
Categorical variables were created by me using ...
- 2,555
1
vote
2
answers
305
views
Dummy encoding the categorical variables using the changed version of OneHotEncoder [duplicate]
This is my code, I was trying to dummy encode the first column of X using OneHotEncoder but it was showing error and the documentation page of OneHotEncoder says that it has been changed and I wasn't ...
- 11
1
vote
1
answer
314
views
Create new variable in python for each iteration
I have two different dataframes like below.
And I need to make a dataframe like the following one.
Can you please help on how to that in python?
Thanks
2
votes
3
answers
3k
views
How to handle "year" variable for Machine Learning models
I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable?
Thanks ...
- 21
0
votes
1
answer
117
views
Frequency of occurrence - dummy variables
I am thinking about it not the first time, namely if I have a variable that I want to convert later to the variable dummy (cities in this case), should I delete lines that occur less often than N ...
- 177
0
votes
2
answers
921
views
How to Keep Missing Values in Ordinal Logistic Regression
I’m using mord package in python to do ordinal logit regression (predict response to movie rating 1-5 stars).
One of my predictor variables is also ordinal but ...
- 181
1
vote
0
answers
157
views
Same coefficient in multivariate regression with dummy variables
Hello Data Science community,
I have a model with 1 quantitative variable (y) and 2 categorical variables. In order to work with the categorical variables I have created dummy variables (binary) for ...
1
vote
1
answer
25
views
Handling hierarchical category independent variables
I have data with huge categorical attributes.
For example, main_column, sub_column1, sub_column2 are 3 hierarchical attributes.
If if take dummy variable on these columns the column count is ...
- 309
2
votes
2
answers
6k
views
Prediction after one hot encoding
I have a regression model that I want to make prediction based on values that I will get from an end user.
In my dataset, I have one categorical variable region ...
- 33
5
votes
3
answers
8k
views
Obtaining consistent one-hot encoding of train / production data
I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination ...
- 308
0
votes
1
answer
599
views
convert keywords in one column into several dummy columns
My dataframe has some free text fields named: {'title', 'description', 'location'}
I prepared this text column by: concatenating all into a new column, dropping ...
6
votes
3
answers
2k
views
How to give a higher importance to certain features in a (k-means) clustering model?
I am clustering data with numeric and categorical variables. To process the categorical variables for the cluster model, I create dummy variables. However, I feel like this results in a higher ...
- 71
1
vote
1
answer
281
views
How Dummy Variables Should Be Modeled In A Linear Regression Model?
I've a cross sectional model where I want predict number of users that take specific service, to make it I've many variables but have specifically two nominal: isWorkday(0 or 1) and weeday(1,2,3,...,7)...
1
vote
1
answer
247
views
caret dummyVars on unseen data
I created my dummy variables, trained my model and tested it as below:
...
- 531
2
votes
1
answer
187
views
How to deal with a potencially multiple categorical variable
I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring....
- 191
1
vote
1
answer
947
views
Can you apply PCA to part of your dataset?
I am working with kaggle dataset that has over 130 features composed of 116 categorical and 14 continuous features. I plotted the heatmap for the 14 continuous variables and found that most of them ...
- 13
8
votes
2
answers
8k
views
In which cases shouldn't we drop the first level of categorical variables?
Beginner in machine learning, I'm looking into the one-hot encoding concept.
Unlike in statistics when you always want to drop the first level to have k-1 dummies (...
- 331
3
votes
1
answer
4k
views
Using pandas get_dummies() on real world unseen data
I made a ML model, trained and tested it with my data containing categorical variables.
To create dummy variables I used pd.get_dummies() before the split.
I now ...
- 531
1
vote
1
answer
126
views
Dummy variables for unseen data in R
I got the following problem: When I trained my model I created my dummy variables(before train-test split) in the following way:
...
- 531
3
votes
3
answers
2k
views
How to obtain original feature names after using one-hot encoding
This question is on an implementation aspect of scikit-learn's DecisionTreeClassifier().
How do I get the feature names ranked in descending order, from the ...
- 51