Questions tagged [transformation]
The transformation tag has no usage guidance.
23
questions
0
votes
0
answers
16
views
Finding data with transformation applied
Is there a way to find relatedness between data and the data obtained after some transformation applied to it? i.e. given a data I need to find the most related data(most of the values in that data ...
- 1
0
votes
1
answer
29
views
Why does log-transforming the target have a huge impact on MSE value?
I am doing linear regression using the Boston Housing data set, and the effect of applying $\log(y)$ has a huge impact on the MSE. Failing to do it gives MSE=34.94 ...
- 109
1
vote
1
answer
20
views
Effect of log odds on skewed data
Does taking the log of odds bring linearity between the odds of the dependent variable & the independent variables by removing skewness in the data? Is this one reason why we use log of odds in ...
- 233
1
vote
1
answer
26
views
Feature engineering before splitting
This is a sister post to the original closed post (here). Since the data transformation part is done after data spliting on the TRAINING data only, I wonder wouldn't such transformation has dependency ...
- 359
0
votes
0
answers
8
views
How to predict an outcome of the game (next row) based on all previous games (rows)?
I'm a data science student and I've come across a fairly unusual dataset (to me, which explains the vague title).
It's of the following form:
STAT_1
STAT_2
...
HOME
AWAY
NEXT_HOME
NEXT_AWAY
...
- 1
0
votes
1
answer
75
views
Should i always transform data to normal distribution?
I am trying to understand transformations but this question seems to be in my and some people's mind. If we have a numeric variable in EVERY data science case. Transforming data(Log, power transforms) ...
- 23
0
votes
0
answers
19
views
320GB `YYYY/MM/DD/HH/*.json.gz` -> `YYYY/MM/tenant_id=x/data.parquet`?
I have 1mil gzipped files which contain in total 350mil \n separated json objects. 26GB compressed, ~320GB uncompressed, representing 7 years of data for a multi-...
0
votes
0
answers
56
views
Is there a Softmax-like transformation with scale-invariance and linarity?
At the moment I'm using XGBoost to generate a prediction of probabilities with a custom objective-function to build something like an expert system. To do so I need to transform the raw XGBoost ...
- 11
0
votes
1
answer
70
views
How to revert np.log(data) and data.diff()?
I have used np.log(data) and then applied data.diff() to transform my data in timeseries model. I have the predictions. How do I ...
- 103
0
votes
1
answer
32
views
Outlier treatment
I am working on a regression problem where I have a lot of outliers in multiple variables. As far as I can think of, there are 3 things I can do to outliers.
Remove them (least attractive option)
...
- 1,241
0
votes
1
answer
276
views
Should one log transform discrete numerical variables?
I am working on a Linear Regression problem and one of the assumptions of a Linear Regression model is that the features should be Normally Distributed. Hence to convert my non linear features to ...
- 1,241
0
votes
1
answer
28
views
Un-Pivot Data in Tableau
Say you have data with fields named: A, B, C, KEY, VALUE.
And lets say the KEY field contains a discrete set of possible values like "X", "Y", and "Z".
How do you ...
- 101
0
votes
1
answer
44
views
How to feature engineering after getting test data in deployment?
I am kind of confuse about this topic of feature engineering. I am trying to make an web app in which people can upload test data as csv. Now I am confuse about how to do feature engineering after ...
- 23
0
votes
1
answer
41
views
Transforming time series into static features?
I'm working on a side project where I have a mixture of static data and time series, and the goal would be to perform clustering on the data. There's a bunch of data sources, but basically the main ...
- 1,248
0
votes
3
answers
74
views
Transforming Categorical to Numerical variable
I have a categorical variable with 4 levels ('8 c', '6 c','NAN','Others') and I want to convert it to numerical form. an Obvious way is to simply remove the 'c' part from the first two categories and ...
user120794
0
votes
0
answers
26
views
Algorithm for learning image distortion?
I'm looking for tools to characterize relationships between gridded outputs of multiple physical models as image distortions. For instance, given a 2-d picture of the temperature distribution in two ...
- 1
1
vote
1
answer
18
views
Should I apply a transformation to columns with INTEGERS, in case I want to reduce the skewness of that column?
I am performing EDA on a dataset of Hotel Reservations. Target is Categorical stating if a given customer will cancel the reservation or not. Dataset has 25 features, 30244 entries.
I have two ...
- 53
0
votes
2
answers
73
views
How to reshape or clean data to be able to visualize it with violin plots?
My end goal is to visualize some data using a violin plot or something similar using Python.
I have the following data in a file (test.csv). The first column is a ...
- 113
1
vote
1
answer
77
views
What if outliers still exist after variable transformation?
I have a variable with a skewed distribution.
I applied BoxCox transformation and now the variable follows a Gaussian distribution. But, as seen in the image below in the boxplot, outliers still ...
- 45
1
vote
0
answers
23
views
Multiple regression with non-normal data in minitab - help
I am aiming to assess the effect of BMI (continuous) on certain biomarkers (also continuous) whilst adjusting for several relevant variables (mixed categorical and continuous) using multiple ...
- 11
2
votes
1
answer
67
views
Are we allowed to transform the continuous target variable by creating a log transformation in order to have a normal distribution?
The following code gives the target variable Item_Outlet_Sales before transformation and Item_Outlet_Sales_log which is transformed
...
- 33
0
votes
1
answer
22
views
Condense matrix of values into one column?
I have a dataset where I have unnecessarily duplicated column variables that I want to condense down. I wish the output wasn't so clumsy and I've already had to do some work to transform it and make ...
- 1
1
vote
1
answer
45
views
Anonymize continuous variable for masking purposes
I am about to kick off a large hackathon event.
We have a dataset that is comprised of one continuous variable with high precision, and a number of categorical variables qualifying these data 3-levels ...
- 901