Questions tagged [transformation]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

Finding data with transformation applied

Is there a way to find relatedness between data and the data obtained after some transformation applied to it? i.e. given a data I need to find the most related data(most of the values in that data ...
user avatar
  • 1
0 votes
1 answer
29 views

Why does log-transforming the target have a huge impact on MSE value?

I am doing linear regression using the Boston Housing data set, and the effect of applying $\log(y)$ has a huge impact on the MSE. Failing to do it gives MSE=34.94 ...
user avatar
  • 109
1 vote
1 answer
20 views

Effect of log odds on skewed data

Does taking the log of odds bring linearity between the odds of the dependent variable & the independent variables by removing skewness in the data? Is this one reason why we use log of odds in ...
user avatar
  • 233
1 vote
1 answer
26 views

Feature engineering before splitting

This is a sister post to the original closed post (here). Since the data transformation part is done after data spliting on the TRAINING data only, I wonder wouldn't such transformation has dependency ...
user avatar
  • 359
0 votes
0 answers
8 views

How to predict an outcome of the game (next row) based on all previous games (rows)?

I'm a data science student and I've come across a fairly unusual dataset (to me, which explains the vague title). It's of the following form: STAT_1 STAT_2 ... HOME AWAY NEXT_HOME NEXT_AWAY ...
user avatar
0 votes
1 answer
75 views

Should i always transform data to normal distribution?

I am trying to understand transformations but this question seems to be in my and some people's mind. If we have a numeric variable in EVERY data science case. Transforming data(Log, power transforms) ...
user avatar
  • 23
0 votes
0 answers
19 views

320GB `YYYY/MM/DD/HH/*.json.gz` -> `YYYY/MM/tenant_id=x/data.parquet`?

I have 1mil gzipped files which contain in total 350mil \n separated json objects. 26GB compressed, ~320GB uncompressed, representing 7 years of data for a multi-...
user avatar
0 votes
0 answers
56 views

Is there a Softmax-like transformation with scale-invariance and linarity?

At the moment I'm using XGBoost to generate a prediction of probabilities with a custom objective-function to build something like an expert system. To do so I need to transform the raw XGBoost ...
user avatar
0 votes
1 answer
70 views

How to revert np.log(data) and data.diff()?

I have used np.log(data) and then applied data.diff() to transform my data in timeseries model. I have the predictions. How do I ...
user avatar
0 votes
1 answer
32 views

Outlier treatment

I am working on a regression problem where I have a lot of outliers in multiple variables. As far as I can think of, there are 3 things I can do to outliers. Remove them (least attractive option) ...
user avatar
  • 1,241
0 votes
1 answer
276 views

Should one log transform discrete numerical variables?

I am working on a Linear Regression problem and one of the assumptions of a Linear Regression model is that the features should be Normally Distributed. Hence to convert my non linear features to ...
user avatar
  • 1,241
0 votes
1 answer
28 views

Un-Pivot Data in Tableau

Say you have data with fields named: A, B, C, KEY, VALUE. And lets say the KEY field contains a discrete set of possible values like "X", "Y", and "Z". How do you ...
user avatar
  • 101
0 votes
1 answer
44 views

How to feature engineering after getting test data in deployment?

I am kind of confuse about this topic of feature engineering. I am trying to make an web app in which people can upload test data as csv. Now I am confuse about how to do feature engineering after ...
user avatar
0 votes
1 answer
41 views

Transforming time series into static features?

I'm working on a side project where I have a mixture of static data and time series, and the goal would be to perform clustering on the data. There's a bunch of data sources, but basically the main ...
user avatar
  • 1,248
0 votes
3 answers
74 views

Transforming Categorical to Numerical variable

I have a categorical variable with 4 levels ('8 c', '6 c','NAN','Others') and I want to convert it to numerical form. an Obvious way is to simply remove the 'c' part from the first two categories and ...
user avatar
0 votes
0 answers
26 views

Algorithm for learning image distortion?

I'm looking for tools to characterize relationships between gridded outputs of multiple physical models as image distortions. For instance, given a 2-d picture of the temperature distribution in two ...
user avatar
  • 1
1 vote
1 answer
18 views

Should I apply a transformation to columns with INTEGERS, in case I want to reduce the skewness of that column?

I am performing EDA on a dataset of Hotel Reservations. Target is Categorical stating if a given customer will cancel the reservation or not. Dataset has 25 features, 30244 entries. I have two ...
user avatar
0 votes
2 answers
73 views

How to reshape or clean data to be able to visualize it with violin plots?

My end goal is to visualize some data using a violin plot or something similar using Python. I have the following data in a file (test.csv). The first column is a ...
user avatar
  • 113
1 vote
1 answer
77 views

What if outliers still exist after variable transformation?

I have a variable with a skewed distribution. I applied BoxCox transformation and now the variable follows a Gaussian distribution. But, as seen in the image below in the boxplot, outliers still ...
user avatar
  • 45
1 vote
0 answers
23 views

Multiple regression with non-normal data in minitab - help

I am aiming to assess the effect of BMI (continuous) on certain biomarkers (also continuous) whilst adjusting for several relevant variables (mixed categorical and continuous) using multiple ...
user avatar
2 votes
1 answer
67 views

Are we allowed to transform the continuous target variable by creating a log transformation in order to have a normal distribution?

The following code gives the target variable Item_Outlet_Sales before transformation and Item_Outlet_Sales_log which is transformed ...
user avatar
  • 33
0 votes
1 answer
22 views

Condense matrix of values into one column?

I have a dataset where I have unnecessarily duplicated column variables that I want to condense down. I wish the output wasn't so clumsy and I've already had to do some work to transform it and make ...
user avatar
  • 1
1 vote
1 answer
45 views

Anonymize continuous variable for masking purposes

I am about to kick off a large hackathon event. We have a dataset that is comprised of one continuous variable with high precision, and a number of categorical variables qualifying these data 3-levels ...
user avatar
  • 901