# Questions tagged [transformation]

The tag has no usage guidance.

23 questions
Filter by
Sorted by
Tagged with
16 views

### Finding data with transformation applied

Is there a way to find relatedness between data and the data obtained after some transformation applied to it? i.e. given a data I need to find the most related data(most of the values in that data ...
• 1
29 views

### Why does log-transforming the target have a huge impact on MSE value?

I am doing linear regression using the Boston Housing data set, and the effect of applying $\log(y)$ has a huge impact on the MSE. Failing to do it gives MSE=34.94 ...
• 109
1 vote
20 views

### Effect of log odds on skewed data

Does taking the log of odds bring linearity between the odds of the dependent variable & the independent variables by removing skewness in the data? Is this one reason why we use log of odds in ...
• 233
1 vote
26 views

### Feature engineering before splitting

This is a sister post to the original closed post (here). Since the data transformation part is done after data spliting on the TRAINING data only, I wonder wouldn't such transformation has dependency ...
• 359
8 views

### How to predict an outcome of the game (next row) based on all previous games (rows)?

I'm a data science student and I've come across a fairly unusual dataset (to me, which explains the vague title). It's of the following form: STAT_1 STAT_2 ... HOME AWAY NEXT_HOME NEXT_AWAY ...
75 views

### Should i always transform data to normal distribution?

I am trying to understand transformations but this question seems to be in my and some people's mind. If we have a numeric variable in EVERY data science case. Transforming data(Log, power transforms) ...
• 23
19 views

### 320GB YYYY/MM/DD/HH/*.json.gz -> YYYY/MM/tenant_id=x/data.parquet?

I have 1mil gzipped files which contain in total 350mil \n separated json objects. 26GB compressed, ~320GB uncompressed, representing 7 years of data for a multi-...
56 views

### Is there a Softmax-like transformation with scale-invariance and linarity?

At the moment I'm using XGBoost to generate a prediction of probabilities with a custom objective-function to build something like an expert system. To do so I need to transform the raw XGBoost ...
• 11
70 views

### How to revert np.log(data) and data.diff()?

I have used np.log(data) and then applied data.diff() to transform my data in timeseries model. I have the predictions. How do I ...
32 views

### Outlier treatment

I am working on a regression problem where I have a lot of outliers in multiple variables. As far as I can think of, there are 3 things I can do to outliers. Remove them (least attractive option) ...
• 1,241
276 views

### Should one log transform discrete numerical variables?

I am working on a Linear Regression problem and one of the assumptions of a Linear Regression model is that the features should be Normally Distributed. Hence to convert my non linear features to ...
• 1,241
28 views

### Un-Pivot Data in Tableau

Say you have data with fields named: A, B, C, KEY, VALUE. And lets say the KEY field contains a discrete set of possible values like "X", "Y", and "Z". How do you ...
• 101
44 views

### How to feature engineering after getting test data in deployment?

I am kind of confuse about this topic of feature engineering. I am trying to make an web app in which people can upload test data as csv. Now I am confuse about how to do feature engineering after ...
41 views

### Transforming time series into static features?

I'm working on a side project where I have a mixture of static data and time series, and the goal would be to perform clustering on the data. There's a bunch of data sources, but basically the main ...
• 1,248
74 views

### Transforming Categorical to Numerical variable

I have a categorical variable with 4 levels ('8 c', '6 c','NAN','Others') and I want to convert it to numerical form. an Obvious way is to simply remove the 'c' part from the first two categories and ...
26 views

### Algorithm for learning image distortion?

I'm looking for tools to characterize relationships between gridded outputs of multiple physical models as image distortions. For instance, given a 2-d picture of the temperature distribution in two ...
1 vote
18 views

### Should I apply a transformation to columns with INTEGERS, in case I want to reduce the skewness of that column?

I am performing EDA on a dataset of Hotel Reservations. Target is Categorical stating if a given customer will cancel the reservation or not. Dataset has 25 features, 30244 entries. I have two ...
73 views

### How to reshape or clean data to be able to visualize it with violin plots?

My end goal is to visualize some data using a violin plot or something similar using Python. I have the following data in a file (test.csv). The first column is a ...
• 113
1 vote
77 views

### What if outliers still exist after variable transformation?

I have a variable with a skewed distribution. I applied BoxCox transformation and now the variable follows a Gaussian distribution. But, as seen in the image below in the boxplot, outliers still ...
• 45
1 vote
23 views

### Multiple regression with non-normal data in minitab - help

I am aiming to assess the effect of BMI (continuous) on certain biomarkers (also continuous) whilst adjusting for several relevant variables (mixed categorical and continuous) using multiple ...
• 11
67 views

### Are we allowed to transform the continuous target variable by creating a log transformation in order to have a normal distribution?

The following code gives the target variable Item_Outlet_Sales before transformation and Item_Outlet_Sales_log which is transformed ...
• 33