Questions tagged [dummy-variables]
The dummy-variables tag has no usage guidance.
43
questions
0
votes
0
answers
59
views
How to do DBSCAN clustering with mixed variables (numerical features and binary/ordinal variables)?
I have a question written at the end of the post which refers to the "Distances" paragraph. The other first two paragraphs give additional info.
Context
I'm working on a project where I have ...
2
votes
1
answer
169
views
Dummy Variable trap in Linear Regression
The dummy variable trap is a common problem with linear regression when dealing with categorical variables, since one hot encoding introduces redundancy, so if we have m categories in our categorical ...
- 35
0
votes
0
answers
17
views
OHE vs dummy variables
Could someone explain to me the difference between creating categorical “embeddings” with StringIndexer and OneHotEncoder, vs just creating dummy variables for each category? Aside from it being more ...
- 1
0
votes
0
answers
22
views
Dealing with conditional variables in anomaly detection
I am working on an anomaly detection model and I have a conditional variable, i.e., it is zero or it has an amount like below histogram. Suppose the variable shows the time when a machine is not ...
- 11
0
votes
1
answer
64
views
Is there a way to forecast a time series multiple linear regression using externally made dummy variables?
This question concerns question 4h of this textbook exercise. It asks to make future predictions based on a chosen TSLM model which involves an endogenously (if i'm using this right) made dummy ...
- 21
0
votes
0
answers
15
views
Dummy Predictors / Continuous Dependent Variable
I have a dataset with 50+ dummy coded variables that represent the purchases of an individual customer. Columns represent the products and the cell values 0 or 1, whether the product has been ...
1
vote
2
answers
16
views
Should I create single feature for each specific word which i find in text or one for all them?
I am doing feature engineering right now for my classification task. In my dataframe I have a column with text messages. I decided to create a binary feature which depends on whether or not in this ...
- 183
1
vote
1
answer
220
views
How to get dummy variables from "first name"
I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage.
Since the datasets are so big (...
- 111
1
vote
2
answers
3k
views
inconsistency between y and x numbers in the Split into train and test sets
I am new to the field to the data science, and need help in the following:
I am working on a data set that consists of both categorical and numerical values, first I have concatenate the two files (...
- 11
1
vote
2
answers
112
views
what would be the correct representation of categorical variables like sex?
I have a doubt about what will be the right way to use or represent categorical variables with only two values like "sex". I have checked it up from different sources, but I was not able to ...
- 217
1
vote
1
answer
68
views
Use dummy variables to create a rank variable. R
I have a series of multiple response (dummy) variables describing causes for a canceled visits. A visit can have multiple reasons for the cancelation. My goal is to create a single mutually exclusive ...
- 37
0
votes
1
answer
28
views
Dummies Variables and Scaling in Regression Problems
I was wondering if having dummies variable and scaling other variables could joke my model. In particular, I have implemented a Random Forest Regressor by using scikit-learn, but my data model is ...
- 11
1
vote
1
answer
228
views
What exactly is a dummy trap? Is dropping one dummy feature really a good practice?
So I'm going through a Machine Learning course, and this course explains that to avoid the dummy trap, a common practice is to drop one column. It also explains that since the info on the dropped ...
1
vote
1
answer
557
views
Should I include all dummy variables or N-1 dummy variables (keep one as reference) in neural networks
I have a categorical variable with N factor levels (e.g. gender has two levels) in classification problem. I have converted it into dummy variables (male and female).
I have to use neural network (...
- 125
0
votes
1
answer
45
views
how do tree based methods deal with missing feature columns?
all,
i have trained a model using xgboost. Some of the features are one hot encoded e.g. currency where it is either gbp or usd. it seems that when i output the feature importance gbp and usd were in ...
- 486
1
vote
2
answers
102
views
Problem with converting string to dummy variables
I'm new in data science, I have data which want to work on it, I omitted extra columns and convert it to 4 columns ( Product, Date, Market, Demand ) . in this data Product and Market are string, I ...
- 13
1
vote
1
answer
52
views
How to handle fixed values for variables in pre-processing
I have a dataset which contains few variables whose values do not change. Some of the variables are non-numeric (for example all values for that variable contain the value 5) and few variables are ...
- 491
0
votes
2
answers
678
views
Pandas get_dummies() rows dropping after joining back with X
I'm having an issue that I can't explain and am hoping I am missing something simple.
I have a large dataset of shape(45Million+, 51) and am loading it in for some analyses (classifiers, deep learning,...
- 101
1
vote
1
answer
647
views
onehotencoder random forest
In a Random Forest context, do I need to setup dummies/OnehotEncoder in a dataset where features/varibles are numerical but refer to some kind of category?
Let's say I have the following variables:
...
- 113
0
votes
1
answer
254
views
Creating dummy variables [closed]
I have a Eurobarometer dataset and want to eventually create a logistic regression model and a linear probability model using a set of dates as the dependent variable.
However, the dates in the ...
2
votes
1
answer
129
views
Dummy variable only for character value in a column (Neglecting float and integers)
My dataset consists of 3000 rows and 50 columns, out of which one column (ESTIMATE_FAMILY_CONTRIBUTION) contains all numerical value(around 2000 different values like 20,30,32....) but got one value ...
1
vote
1
answer
500
views
Random Forest in R with only character variables
I am new to using random forest in R an my goal is to identify the independent variables which have the highest impact on the dependent variables. I am looking at sales data, and sales is my dependent ...
- 53
1
vote
1
answer
111
views
Handling Numerical Categorical Column in ML models in Python
When I was exploring the titanic dataset to estimate the probability of a person of surviving using the Logistic Model, I realized there are two ways of handling numerical categorical variables :
Use ...
- 115
2
votes
1
answer
1k
views
How to interpret dummy variable in ML prediction?
I am working on a binary classification problem where I have a mix of continuous and categorical variables.
Categorical variables were created by me using ...
- 2,449
1
vote
2
answers
291
views
Dummy encoding the categorical variables using the changed version of OneHotEncoder [duplicate]
This is my code, I was trying to dummy encode the first column of X using OneHotEncoder but it was showing error and the documentation page of OneHotEncoder says that it has been changed and I wasn't ...
- 11
1
vote
1
answer
283
views
Create new variable in python for each iteration
I have two different dataframes like below.
And I need to make a dataframe like the following one.
Can you please help on how to that in python?
Thanks
2
votes
3
answers
2k
views
How to handle "year" variable for Machine Learning models
I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable?
Thanks ...
- 21
0
votes
1
answer
90
views
Frequency of occurrence - dummy variables
I am thinking about it not the first time, namely if I have a variable that I want to convert later to the variable dummy (cities in this case), should I delete lines that occur less often than N ...
- 167
0
votes
2
answers
765
views
How to Keep Missing Values in Ordinal Logistic Regression
I’m using mord package in python to do ordinal logit regression (predict response to movie rating 1-5 stars).
One of my predictor variables is also ordinal but ...
- 171
1
vote
0
answers
95
views
Same coefficient in multivariate regression with dummy variables
Hello Data Science community,
I have a model with 1 quantitative variable (y) and 2 categorical variables. In order to work with the categorical variables I have created dummy variables (binary) for ...
1
vote
1
answer
23
views
Handling hierarchical category independent variables
I have data with huge categorical attributes.
For example, main_column, sub_column1, sub_column2 are 3 hierarchical attributes.
If if take dummy variable on these columns the column count is ...
- 299
2
votes
2
answers
6k
views
Prediction after one hot encoding
I have a regression model that I want to make prediction based on values that I will get from an end user.
In my dataset, I have one categorical variable region ...
- 33
5
votes
3
answers
7k
views
Obtaining consistent one-hot encoding of train / production data
I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination ...
- 308
0
votes
1
answer
552
views
convert keywords in one column into several dummy columns
My dataframe has some free text fields named: {'title', 'description', 'location'}
I prepared this text column by: concatenating all into a new column, dropping ...
6
votes
3
answers
2k
views
How to give a higher importance to certain features in a (k-means) clustering model?
I am clustering data with numeric and categorical variables. To process the categorical variables for the cluster model, I create dummy variables. However, I feel like this results in a higher ...
- 71
1
vote
1
answer
260
views
How Dummy Variables Should Be Modeled In A Linear Regression Model?
I've a cross sectional model where I want predict number of users that take specific service, to make it I've many variables but have specifically two nominal: isWorkday(0 or 1) and weeday(1,2,3,...,7)...
1
vote
1
answer
203
views
caret dummyVars on unseen data
I created my dummy variables, trained my model and tested it as below:
...
- 531
2
votes
1
answer
175
views
How to deal with a potencially multiple categorical variable
I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring....
- 191
1
vote
1
answer
806
views
Can you apply PCA to part of your dataset?
I am working with kaggle dataset that has over 130 features composed of 116 categorical and 14 continuous features. I plotted the heatmap for the 14 continuous variables and found that most of them ...
- 13
7
votes
2
answers
7k
views
In which cases shouldn't we drop the first level of categorical variables?
Beginner in machine learning, I'm looking into the one-hot encoding concept.
Unlike in statistics when you always want to drop the first level to have k-1 dummies (...
- 321
3
votes
1
answer
4k
views
Using pandas get_dummies() on real world unseen data
I made a ML model, trained and tested it with my data containing categorical variables.
To create dummy variables I used pd.get_dummies() before the split.
I now ...
- 531
1
vote
1
answer
115
views
Dummy variables for unseen data in R
I got the following problem: When I trained my model I created my dummy variables(before train-test split) in the following way:
...
- 531
3
votes
3
answers
2k
views
How to obtain original feature names after using one-hot encoding
This question is on an implementation aspect of scikit-learn's DecisionTreeClassifier().
How do I get the feature names ranked in descending order, from the ...
- 51