Questions tagged [dummy-variables]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
59 views

How to do DBSCAN clustering with mixed variables (numerical features and binary/ordinal variables)?

I have a question written at the end of the post which refers to the "Distances" paragraph. The other first two paragraphs give additional info. Context I'm working on a project where I have ...
2 votes
1 answer
169 views

Dummy Variable trap in Linear Regression

The dummy variable trap is a common problem with linear regression when dealing with categorical variables, since one hot encoding introduces redundancy, so if we have m categories in our categorical ...
  • 35
0 votes
0 answers
17 views

OHE vs dummy variables

Could someone explain to me the difference between creating categorical “embeddings” with StringIndexer and OneHotEncoder, vs just creating dummy variables for each category? Aside from it being more ...
0 votes
0 answers
22 views

Dealing with conditional variables in anomaly detection

I am working on an anomaly detection model and I have a conditional variable, i.e., it is zero or it has an amount like below histogram. Suppose the variable shows the time when a machine is not ...
  • 11
0 votes
1 answer
64 views

Is there a way to forecast a time series multiple linear regression using externally made dummy variables?

This question concerns question 4h of this textbook exercise. It asks to make future predictions based on a chosen TSLM model which involves an endogenously (if i'm using this right) made dummy ...
  • 21
0 votes
0 answers
15 views

Dummy Predictors / Continuous Dependent Variable

I have a dataset with 50+ dummy coded variables that represent the purchases of an individual customer. Columns represent the products and the cell values 0 or 1, whether the product has been ...
1 vote
2 answers
16 views

Should I create single feature for each specific word which i find in text or one for all them?

I am doing feature engineering right now for my classification task. In my dataframe I have a column with text messages. I decided to create a binary feature which depends on whether or not in this ...
  • 183
1 vote
1 answer
220 views

How to get dummy variables from "first name"

I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage. Since the datasets are so big (...
1 vote
2 answers
3k views

inconsistency between y and x numbers in the Split into train and test sets

I am new to the field to the data science, and need help in the following: I am working on a data set that consists of both categorical and numerical values, first I have concatenate the two files (...
1 vote
2 answers
112 views

what would be the correct representation of categorical variables like sex?

I have a doubt about what will be the right way to use or represent categorical variables with only two values like "sex". I have checked it up from different sources, but I was not able to ...
  • 217
1 vote
1 answer
68 views

Use dummy variables to create a rank variable. R

I have a series of multiple response (dummy) variables describing causes for a canceled visits. A visit can have multiple reasons for the cancelation. My goal is to create a single mutually exclusive ...
  • 37
0 votes
1 answer
28 views

Dummies Variables and Scaling in Regression Problems

I was wondering if having dummies variable and scaling other variables could joke my model. In particular, I have implemented a Random Forest Regressor by using scikit-learn, but my data model is ...
1 vote
1 answer
228 views

What exactly is a dummy trap? Is dropping one dummy feature really a good practice?

So I'm going through a Machine Learning course, and this course explains that to avoid the dummy trap, a common practice is to drop one column. It also explains that since the info on the dropped ...
1 vote
1 answer
557 views

Should I include all dummy variables or N-1 dummy variables (keep one as reference) in neural networks

I have a categorical variable with N factor levels (e.g. gender has two levels) in classification problem. I have converted it into dummy variables (male and female). I have to use neural network (...
  • 125
0 votes
1 answer
45 views

how do tree based methods deal with missing feature columns?

all, i have trained a model using xgboost. Some of the features are one hot encoded e.g. currency where it is either gbp or usd. it seems that when i output the feature importance gbp and usd were in ...
  • 486
1 vote
2 answers
102 views

Problem with converting string to dummy variables

I'm new in data science, I have data which want to work on it, I omitted extra columns and convert it to 4 columns ( Product, Date, Market, Demand ) . in this data Product and Market are string, I ...
  • 13
1 vote
1 answer
52 views

How to handle fixed values for variables in pre-processing

I have a dataset which contains few variables whose values do not change. Some of the variables are non-numeric (for example all values for that variable contain the value 5) and few variables are ...
  • 491
0 votes
2 answers
678 views

Pandas get_dummies() rows dropping after joining back with X

I'm having an issue that I can't explain and am hoping I am missing something simple. I have a large dataset of shape(45Million+, 51) and am loading it in for some analyses (classifiers, deep learning,...
  • 101
1 vote
1 answer
647 views

onehotencoder random forest

In a Random Forest context, do I need to setup dummies/OnehotEncoder in a dataset where features/varibles are numerical but refer to some kind of category? Let's say I have the following variables: ...
  • 113
0 votes
1 answer
254 views

Creating dummy variables [closed]

I have a Eurobarometer dataset and want to eventually create a logistic regression model and a linear probability model using a set of dates as the dependent variable. However, the dates in the ...
2 votes
1 answer
129 views

Dummy variable only for character value in a column (Neglecting float and integers)

My dataset consists of 3000 rows and 50 columns, out of which one column (ESTIMATE_FAMILY_CONTRIBUTION) contains all numerical value(around 2000 different values like 20,30,32....) but got one value ...
1 vote
1 answer
500 views

Random Forest in R with only character variables

I am new to using random forest in R an my goal is to identify the independent variables which have the highest impact on the dependent variables. I am looking at sales data, and sales is my dependent ...
  • 53
1 vote
1 answer
111 views

Handling Numerical Categorical Column in ML models in Python

When I was exploring the titanic dataset to estimate the probability of a person of surviving using the Logistic Model, I realized there are two ways of handling numerical categorical variables : Use ...
2 votes
1 answer
1k views

How to interpret dummy variable in ML prediction?

I am working on a binary classification problem where I have a mix of continuous and categorical variables. Categorical variables were created by me using ...
  • 2,449
1 vote
2 answers
291 views

Dummy encoding the categorical variables using the changed version of OneHotEncoder [duplicate]

This is my code, I was trying to dummy encode the first column of X using OneHotEncoder but it was showing error and the documentation page of OneHotEncoder says that it has been changed and I wasn't ...
1 vote
1 answer
283 views

Create new variable in python for each iteration

I have two different dataframes like below. And I need to make a dataframe like the following one. Can you please help on how to that in python? Thanks
2 votes
3 answers
2k views

How to handle "year" variable for Machine Learning models

I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable? Thanks ...
  • 21
0 votes
1 answer
90 views

Frequency of occurrence - dummy variables

I am thinking about it not the first time, namely if I have a variable that I want to convert later to the variable dummy (cities in this case), should I delete lines that occur less often than N ...
  • 167
0 votes
2 answers
765 views

How to Keep Missing Values in Ordinal Logistic Regression

I’m using mord package in python to do ordinal logit regression (predict response to movie rating 1-5 stars). One of my predictor variables is also ordinal but ...
  • 171
1 vote
0 answers
95 views

Same coefficient in multivariate regression with dummy variables

Hello Data Science community, I have a model with 1 quantitative variable (y) and 2 categorical variables. In order to work with the categorical variables I have created dummy variables (binary) for ...
1 vote
1 answer
23 views

Handling hierarchical category independent variables

I have data with huge categorical attributes. For example, main_column, sub_column1, sub_column2 are 3 hierarchical attributes. If if take dummy variable on these columns the column count is ...
  • 299
2 votes
2 answers
6k views

Prediction after one hot encoding

I have a regression model that I want to make prediction based on values that I will get from an end user. In my dataset, I have one categorical variable region ...
  • 33
5 votes
3 answers
7k views

Obtaining consistent one-hot encoding of train / production data

I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination ...
0 votes
1 answer
552 views

convert keywords in one column into several dummy columns

My dataframe has some free text fields named: {'title', 'description', 'location'} I prepared this text column by: concatenating all into a new column, dropping ...
6 votes
3 answers
2k views

How to give a higher importance to certain features in a (k-means) clustering model?

I am clustering data with numeric and categorical variables. To process the categorical variables for the cluster model, I create dummy variables. However, I feel like this results in a higher ...
  • 71
1 vote
1 answer
260 views

How Dummy Variables Should Be Modeled In A Linear Regression Model?

I've a cross sectional model where I want predict number of users that take specific service, to make it I've many variables but have specifically two nominal: isWorkday(0 or 1) and weeday(1,2,3,...,7)...
1 vote
1 answer
203 views

caret dummyVars on unseen data

I created my dummy variables, trained my model and tested it as below: ...
  • 531
2 votes
1 answer
175 views

How to deal with a potencially multiple categorical variable

I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring....
1 vote
1 answer
806 views

Can you apply PCA to part of your dataset?

I am working with kaggle dataset that has over 130 features composed of 116 categorical and 14 continuous features. I plotted the heatmap for the 14 continuous variables and found that most of them ...
7 votes
2 answers
7k views

In which cases shouldn't we drop the first level of categorical variables?

Beginner in machine learning, I'm looking into the one-hot encoding concept. Unlike in statistics when you always want to drop the first level to have k-1 dummies (...
3 votes
1 answer
4k views

Using pandas get_dummies() on real world unseen data

I made a ML model, trained and tested it with my data containing categorical variables. To create dummy variables I used pd.get_dummies() before the split. I now ...
  • 531
1 vote
1 answer
115 views

Dummy variables for unseen data in R

I got the following problem: When I trained my model I created my dummy variables(before train-test split) in the following way: ...
  • 531
3 votes
3 answers
2k views

How to obtain original feature names after using one-hot encoding

This question is on an implementation aspect of scikit-learn's DecisionTreeClassifier(). How do I get the feature names ranked in descending order, from the ...
  • 51