Questions tagged [transformation]
The transformation tag has no usage guidance.
33
questions
1
vote
1
answer
24
views
Data Skewness is nan or inf
I have checked the Skewness of my data before applying a Log transformation using the next code :
print("Skewness: %f" % df['Wind Speed (km/h)'].skew())
...
- 165
0
votes
0
answers
12
views
Transform monthly data to the weekly data based on categorical value
I have monthly data like this which is hours:
month,hours
1,20
2,19.5
3,21
...
I have weekly data like this which is quantities of the asset:
...
0
votes
1
answer
61
views
Data transformation pipeline error
When I'm making data transformation pipeline on a dataset I keep getting error as " all the input array dimensions except for concatenation axis must match exactly, but along dimension 0, the ...
- 1
0
votes
0
answers
119
views
0
votes
0
answers
50
views
Is SVM rotation invariant?
Let's say we have some data X and we want to find a linear separator using soft SVM with l2 regularization, and then we want to solve the same problem after applying some rotation matrix Q to the data ...
0
votes
0
answers
13
views
Methods to appropriately capture equal ranges of turnover
Suppose I have a dataset with three columns, turnover, market, count, and turnover is the range of turnover in EUR. Something like:
...
- 1
0
votes
0
answers
30
views
How to multiply the boolean/binary values in 1 row in a dataframe by the chr values in another row with the product landing in the original row
Suppose that I already know which subset out of a set of 30 candidate regressor columns are the true regressors included in the structural equation describing that dataset (because I do by ...
- 157
1
vote
0
answers
67
views
How do SciKit-Learn Pipelines pass Data between Steps?
I would like to create some Custom Transformers and incorporate them in a SciKit-Learn Pipeline.
I'd like to pass more than just a Dataframe between the transformer ...
- 617
0
votes
0
answers
66
views
Should I apply log transform?
Hsre is my skewness and kurtosis value before applying log transform. For skewness:
I use -3 < x < 3 as acceptable value and kurtosis at -10 < x < 10 as acceptable value. Since my data has ...
1
vote
1
answer
23
views
Data transformation
Should data transformation techniques (ex: creating new features/ log attributes) be done before or after feature selection techniques (ex: mutual information feature selection)? Are there any ...
- 161
0
votes
1
answer
72
views
Handling data skewness and kurtosis
I have a dataset where the variables have high skewness (> ±1) and kurtosis (> ±5). I tried to remove outliers and log10 transformation the skewness and kurtosis are still high. Are there any ...
- 161
0
votes
2
answers
816
views
Should I deal with missing values first then transform the data or vice versa?
I am currently working on a project involving time series banking stock price data. I have around 3000 observations, some columns have a lot of missing values (null value); they can account for 5 to ...
0
votes
1
answer
15
views
How can I transform or plot my data to see power consumption more easily?
I have two heating tapes installed in my setup and they provide heat to maintain the reaction at a certain setpoint temperature. Basically, the heating tapes go on a cycle of on/off to maintain the ...
0
votes
1
answer
112
views
Why does log-transforming the target have a huge impact on MSE value?
I am doing linear regression using the Boston Housing data set, and the effect of applying $\log(y)$ has a huge impact on the MSE. Failing to do it gives MSE=34.94 ...
- 119
1
vote
1
answer
49
views
Effect of log odds on skewed data
Does taking the log of odds bring linearity between the odds of the dependent variable & the independent variables by removing skewness in the data? Is this one reason why we use log of odds in ...
- 275
1
vote
1
answer
554
views
Feature engineering before splitting
This is a sister post to the original closed post (here). Since the data transformation part is done after data spliting on the TRAINING data only, I wonder wouldn't such transformation has dependency ...
- 411
0
votes
1
answer
2k
views
Should i always transform data to normal distribution?
I am trying to understand transformations but this question seems to be in my and some people's mind. If we have a numeric variable in EVERY data science case. Transforming data(Log, power transforms) ...
- 91
0
votes
0
answers
152
views
Is there a Softmax-like transformation with scale-invariance and linarity?
At the moment I'm using XGBoost to generate a prediction of probabilities with a custom objective-function to build something like an expert system. To do so I need to transform the raw XGBoost ...
- 111
0
votes
1
answer
898
views
How to revert np.log(data) and data.diff()?
I have used np.log(data) and then applied data.diff() to transform my data in timeseries model. I have the predictions. How do I ...
- 153
0
votes
1
answer
87
views
Outlier treatment
I am working on a regression problem where I have a lot of outliers in multiple variables. As far as I can think of, there are 3 things I can do to outliers.
Remove them (least attractive option)
...
- 1,941
0
votes
1
answer
1k
views
Should one log transform discrete numerical variables?
I am working on a Linear Regression problem and one of the assumptions of a Linear Regression model is that the features should be Normally Distributed. Hence to convert my non linear features to ...
- 1,941
0
votes
1
answer
124
views
Un-Pivot Data in Tableau
Say you have data with fields named: A, B, C, KEY, VALUE.
And lets say the KEY field contains a discrete set of possible values like "X", "Y", and "Z".
How do you ...
- 101
0
votes
1
answer
126
views
How to feature engineering after getting test data in deployment?
I am kind of confuse about this topic of feature engineering. I am trying to make an web app in which people can upload test data as csv. Now I am confuse about how to do feature engineering after ...
- 193
0
votes
1
answer
203
views
Transforming time series into static features?
I'm working on a side project where I have a mixture of static data and time series, and the goal would be to perform clustering on the data. There's a bunch of data sources, but basically the main ...
- 1,310
0
votes
3
answers
224
views
Transforming Categorical to Numerical variable
I have a categorical variable with 4 levels ('8 c', '6 c','NAN','Others') and I want to convert it to numerical form. an Obvious way is to simply remove the 'c' part from the first two categories and ...
user120794
1
vote
1
answer
22
views
Should I apply a transformation to columns with INTEGERS, in case I want to reduce the skewness of that column?
I am performing EDA on a dataset of Hotel Reservations. Target is Categorical stating if a given customer will cancel the reservation or not. Dataset has 25 features, 30244 entries.
I have two ...
- 73
0
votes
2
answers
127
views
How to reshape or clean data to be able to visualize it with violin plots?
My end goal is to visualize some data using a violin plot or something similar using Python.
I have the following data in a file (test.csv). The first column is a ...
- 113
1
vote
1
answer
707
views
What if outliers still exist after variable transformation?
I have a variable with a skewed distribution.
I applied BoxCox transformation and now the variable follows a Gaussian distribution. But, as seen in the image below in the boxplot, outliers still ...
- 75
1
vote
0
answers
68
views
Multiple regression with non-normal data in minitab - help
I am aiming to assess the effect of BMI (continuous) on certain biomarkers (also continuous) whilst adjusting for several relevant variables (mixed categorical and continuous) using multiple ...
- 11
2
votes
1
answer
392
views
Are we allowed to transform the continuous target variable by creating a log transformation in order to have a normal distribution?
The following code gives the target variable Item_Outlet_Sales before transformation and Item_Outlet_Sales_log which is transformed
...
- 131
0
votes
1
answer
115
views
Condense matrix of values into one column?
I have a dataset where I have unnecessarily duplicated column variables that I want to condense down. I wish the output wasn't so clumsy and I've already had to do some work to transform it and make ...
- 1
1
vote
1
answer
176
views
What is the difference between normalization and re-scaling?
This site does not describe the nature of the normalization tag. Does it differ from re-scaling? Many authors use the two terms interchangeably.
- 605
1
vote
1
answer
90
views
Anonymize continuous variable for masking purposes
I am about to kick off a large hackathon event.
We have a dataset that is comprised of one continuous variable with high precision, and a number of categorical variables qualifying these data 3-levels ...
- 911