Questions tagged [dummy-variables]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
46 views

When would pandas.get_dummies()'s parameter of drop_first=False be appropriate to use?

While working on some case studies that use various machine learning models, I came across a project for predicting churn in the telecom industry. The Jupyter Notebook I saw had the following lines of ...
Rakesh Poluri's user avatar
-1 votes
1 answer
44 views

Beginner data scientist looking for help. (apologies for the vagueness ) [closed]

I am analyzing airbnb data to understand which variable is most correlated and in doing so build a model. Upon tackling qualitative variables I decided to use dummies to assign quantitative values to ...
user152192's user avatar
2 votes
1 answer
647 views

Dummy Variable trap in Linear Regression

The dummy variable trap is a common problem with linear regression when dealing with categorical variables, since one hot encoding introduces redundancy, so if we have m categories in our categorical ...
AAA's user avatar
  • 35
0 votes
0 answers
27 views

Dealing with conditional variables in anomaly detection

I am working on an anomaly detection model and I have a conditional variable, i.e., it is zero or it has an amount like below histogram. Suppose the variable shows the time when a machine is not ...
8Simon8's user avatar
  • 11
0 votes
1 answer
164 views

Is there a way to forecast a time series multiple linear regression using externally made dummy variables?

This question concerns question 4h of this textbook exercise. It asks to make future predictions based on a chosen TSLM model which involves an endogenously (if i'm using this right) made dummy ...
Cameron's user avatar
  • 21
0 votes
0 answers
17 views

Dummy Predictors / Continuous Dependent Variable

I have a dataset with 50+ dummy coded variables that represent the purchases of an individual customer. Columns represent the products and the cell values 0 or 1, whether the product has been ...
user133793's user avatar
1 vote
2 answers
26 views

Should I create single feature for each specific word which i find in text or one for all them?

I am doing feature engineering right now for my classification task. In my dataframe I have a column with text messages. I decided to create a binary feature which depends on whether or not in this ...
Ir8_mind's user avatar
  • 183
1 vote
1 answer
286 views

How to get dummy variables from "first name"

I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage. Since the datasets are so big (...
Raha Moosavi's user avatar
1 vote
2 answers
4k views

inconsistency between y and x numbers in the Split into train and test sets

I am new to the field to the data science, and need help in the following: I am working on a data set that consists of both categorical and numerical values, first I have concatenate the two files (...
Rasha Abdin's user avatar
1 vote
2 answers
213 views

what would be the correct representation of categorical variables like sex?

I have a doubt about what will be the right way to use or represent categorical variables with only two values like "sex". I have checked it up from different sources, but I was not able to ...
Lila's user avatar
  • 217
1 vote
1 answer
102 views

Use dummy variables to create a rank variable. R

I have a series of multiple response (dummy) variables describing causes for a canceled visits. A visit can have multiple reasons for the cancelation. My goal is to create a single mutually exclusive ...
Mar355's user avatar
  • 37
0 votes
1 answer
44 views

Dummies Variables and Scaling in Regression Problems

I was wondering if having dummies variable and scaling other variables could joke my model. In particular, I have implemented a Random Forest Regressor by using scikit-learn, but my data model is ...
fflpdqqoeit's user avatar
2 votes
1 answer
457 views

What exactly is a dummy trap? Is dropping one dummy feature really a good practice?

So I'm going through a Machine Learning course, and this course explains that to avoid the dummy trap, a common practice is to drop one column. It also explains that since the info on the dropped ...
UchuuStranger's user avatar
1 vote
1 answer
929 views

Should I include all dummy variables or N-1 dummy variables (keep one as reference) in neural networks

I have a categorical variable with N factor levels (e.g. gender has two levels) in classification problem. I have converted it into dummy variables (male and female). I have to use neural network (...
SiH's user avatar
  • 125
0 votes
1 answer
60 views

how do tree based methods deal with missing feature columns?

all, i have trained a model using xgboost. Some of the features are one hot encoded e.g. currency where it is either gbp or usd. it seems that when i output the feature importance gbp and usd were in ...
Maths12's user avatar
  • 516
1 vote
2 answers
141 views

Problem with converting string to dummy variables

I'm new in data science, I have data which want to work on it, I omitted extra columns and convert it to 4 columns ( Product, Date, Market, Demand ) . in this data Product and Market are string, I ...
ramin's user avatar
  • 13
1 vote
1 answer
79 views

How to handle fixed values for variables in pre-processing

I have a dataset which contains few variables whose values do not change. Some of the variables are non-numeric (for example all values for that variable contain the value 5) and few variables are ...
Sm1's user avatar
  • 531
0 votes
2 answers
940 views

Pandas get_dummies() rows dropping after joining back with X

I'm having an issue that I can't explain and am hoping I am missing something simple. I have a large dataset of shape(45Million+, 51) and am loading it in for some analyses (classifiers, deep learning,...
datahappy's user avatar
  • 101
1 vote
1 answer
748 views

onehotencoder random forest

In a Random Forest context, do I need to setup dummies/OnehotEncoder in a dataset where features/varibles are numerical but refer to some kind of category? Let's say I have the following variables: ...
Pithit's user avatar
  • 113
0 votes
1 answer
262 views

Creating dummy variables [closed]

I have a Eurobarometer dataset and want to eventually create a logistic regression model and a linear probability model using a set of dates as the dependent variable. However, the dates in the ...
user14031981's user avatar
2 votes
1 answer
142 views

Dummy variable only for character value in a column (Neglecting float and integers)

My dataset consists of 3000 rows and 50 columns, out of which one column (ESTIMATE_FAMILY_CONTRIBUTION) contains all numerical value(around 2000 different values like 20,30,32....) but got one value ...
SHUBHAM KUMAR's user avatar
1 vote
1 answer
687 views

Random Forest in R with only character variables

I am new to using random forest in R an my goal is to identify the independent variables which have the highest impact on the dependent variables. I am looking at sales data, and sales is my dependent ...
ColRow's user avatar
  • 53
1 vote
1 answer
166 views

Handling Numerical Categorical Column in ML models in Python

When I was exploring the titanic dataset to estimate the probability of a person of surviving using the Logistic Model, I realized there are two ways of handling numerical categorical variables : Use ...
Kuljeet Keshav's user avatar
2 votes
1 answer
1k views

How to interpret dummy variable in ML prediction?

I am working on a binary classification problem where I have a mix of continuous and categorical variables. Categorical variables were created by me using ...
The Great's user avatar
  • 2,565
1 vote
2 answers
306 views

Dummy encoding the categorical variables using the changed version of OneHotEncoder [duplicate]

This is my code, I was trying to dummy encode the first column of X using OneHotEncoder but it was showing error and the documentation page of OneHotEncoder says that it has been changed and I wasn't ...
Mudit Gupta's user avatar
1 vote
1 answer
325 views

Create new variable in python for each iteration

I have two different dataframes like below. And I need to make a dataframe like the following one. Can you please help on how to that in python? Thanks
Umme A. Munira's user avatar
2 votes
3 answers
4k views

How to handle "year" variable for Machine Learning models

I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable? Thanks ...
Luis's user avatar
  • 21
0 votes
1 answer
124 views

Frequency of occurrence - dummy variables

I am thinking about it not the first time, namely if I have a variable that I want to convert later to the variable dummy (cities in this case), should I delete lines that occur less often than N ...
tbone's user avatar
  • 177
0 votes
2 answers
993 views

How to Keep Missing Values in Ordinal Logistic Regression

I’m using mord package in python to do ordinal logit regression (predict response to movie rating 1-5 stars). One of my predictor variables is also ordinal but ...
Insu Q's user avatar
  • 181
1 vote
0 answers
178 views

Same coefficient in multivariate regression with dummy variables

Hello Data Science community, I have a model with 1 quantitative variable (y) and 2 categorical variables. In order to work with the categorical variables I have created dummy variables (binary) for ...
Eduardo Martinez's user avatar
1 vote
1 answer
25 views

Handling hierarchical category independent variables

I have data with huge categorical attributes. For example, main_column, sub_column1, sub_column2 are 3 hierarchical attributes. If if take dummy variable on these columns the column count is ...
Wickkiey's user avatar
  • 309
2 votes
2 answers
7k views

Prediction after one hot encoding

I have a regression model that I want to make prediction based on values that I will get from an end user. In my dataset, I have one categorical variable region ...
IngridX's user avatar
  • 33
5 votes
3 answers
9k views

Obtaining consistent one-hot encoding of train / production data

I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination ...
Andrew Maurer's user avatar
0 votes
1 answer
610 views

convert keywords in one column into several dummy columns

My dataframe has some free text fields named: {'title', 'description', 'location'} I prepared this text column by: concatenating all into a new column, dropping ...
Cristian C. Bittel's user avatar
7 votes
3 answers
2k views

How to give a higher importance to certain features in a (k-means) clustering model?

I am clustering data with numeric and categorical variables. To process the categorical variables for the cluster model, I create dummy variables. However, I feel like this results in a higher ...
Eva's user avatar
  • 81
1 vote
1 answer
288 views

How Dummy Variables Should Be Modeled In A Linear Regression Model?

I've a cross sectional model where I want predict number of users that take specific service, to make it I've many variables but have specifically two nominal: isWorkday(0 or 1) and weeday(1,2,3,...,7)...
David Salgado's user avatar
1 vote
1 answer
251 views

caret dummyVars on unseen data

I created my dummy variables, trained my model and tested it as below: ...
3nomis's user avatar
  • 541
2 votes
1 answer
191 views

How to deal with a potencially multiple categorical variable

I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring....
Diogo Santos's user avatar
1 vote
1 answer
1k views

Can you apply PCA to part of your dataset?

I am working with kaggle dataset that has over 130 features composed of 116 categorical and 14 continuous features. I plotted the heatmap for the 14 continuous variables and found that most of them ...
JJtheNOOB's user avatar
9 votes
2 answers
8k views

In which cases shouldn't we drop the first level of categorical variables?

Beginner in machine learning, I'm looking into the one-hot encoding concept. Unlike in statistics when you always want to drop the first level to have k-1 dummies (...
Dan Chaltiel's user avatar
3 votes
1 answer
4k views

Using pandas get_dummies() on real world unseen data

I made a ML model, trained and tested it with my data containing categorical variables. To create dummy variables I used pd.get_dummies() before the split. I now ...
3nomis's user avatar
  • 541
1 vote
1 answer
131 views

Dummy variables for unseen data in R

I got the following problem: When I trained my model I created my dummy variables(before train-test split) in the following way: ...
3nomis's user avatar
  • 541
3 votes
3 answers
2k views

How to obtain original feature names after using one-hot encoding

This question is on an implementation aspect of scikit-learn's DecisionTreeClassifier(). How do I get the feature names ranked in descending order, from the ...
S Datta's user avatar
  • 51