Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [transformation]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
54 views

Standard Scaling After Log Transformation

I have a quick question about whether or not to standardize features after a log transformation. I have one feature that is heavily skewed and requires the log transformation, for the other features I'...
atn291's user avatar
  • 11
0 votes
0 answers
18 views

SAS: how to replace a value in column two in a dataset for which column one = x?

So, total SAS noob here (and I don't like it al all so far). I tried 4 different methods, which all do not work. So I guess in all of these code, something similar is wrong, something rather ...
babipsylon's user avatar
0 votes
0 answers
50 views

How to find adjacent neighbours using Python?

Full problem description at stackoverflow I need to find the adjacent neighbours (not necessarily nearest neighbours) to a given point in a multidimensional space. As shown in the screenshot below, I ...
skm's user avatar
  • 101
1 vote
1 answer
100 views

Data Skewness is nan or inf

I have checked the Skewness of my data before applying a Log transformation using the next code : print("Skewness: %f" % df['Wind Speed (km/h)'].skew()) ...
baddy's user avatar
  • 165
0 votes
0 answers
12 views

Transform monthly data to the weekly data based on categorical value

I have monthly data like this which is hours: month,hours 1,20 2,19.5 3,21 ... I have weekly data like this which is quantities of the asset: ...
Muhammad Ikhwan Perwira's user avatar
0 votes
1 answer
75 views

Data transformation pipeline error

When I'm making data transformation pipeline on a dataset I keep getting error as " all the input array dimensions except for concatenation axis must match exactly, but along dimension 0, the ...
Amy's user avatar
  • 1
0 votes
0 answers
65 views

Is SVM rotation invariant?

Let's say we have some data X and we want to find a linear separator using soft SVM with l2 regularization, and then we want to solve the same problem after applying some rotation matrix Q to the data ...
user3917631's user avatar
1 vote
0 answers
103 views

How do SciKit-Learn Pipelines pass Data between Steps?

I would like to create some Custom Transformers and incorporate them in a SciKit-Learn Pipeline. I'd like to pass more than just a Dataframe between the transformer ...
Connor's user avatar
  • 651
0 votes
0 answers
81 views

Should I apply log transform?

Hsre is my skewness and kurtosis value before applying log transform. For skewness: I use -3 < x < 3 as acceptable value and kurtosis at -10 < x < 10 as acceptable value. Since my data has ...
UrDailyCS's user avatar
1 vote
1 answer
29 views

Data transformation

Should data transformation techniques (ex: creating new features/ log attributes) be done before or after feature selection techniques (ex: mutual information feature selection)? Are there any ...
Kusisi Karem's user avatar
0 votes
1 answer
93 views

Handling data skewness and kurtosis

I have a dataset where the variables have high skewness (> ±1) and kurtosis (> ±5). I tried to remove outliers and log10 transformation the skewness and kurtosis are still high. Are there any ...
Kusisi Karem's user avatar
0 votes
2 answers
1k views

Should I deal with missing values first then transform the data or vice versa?

I am currently working on a project involving time series banking stock price data. I have around 3000 observations, some columns have a lot of missing values (null value); they can account for 5 to ...
MINH NHỰT NGUYỄN TRẦN's user avatar
0 votes
1 answer
16 views

How can I transform or plot my data to see power consumption more easily?

I have two heating tapes installed in my setup and they provide heat to maintain the reaction at a certain setpoint temperature. Basically, the heating tapes go on a cycle of on/off to maintain the ...
ScepticalChymist's user avatar
0 votes
1 answer
145 views

Why does log-transforming the target have a huge impact on MSE value?

I am doing linear regression using the Boston Housing data set, and the effect of applying $\log(y)$ has a huge impact on the MSE. Failing to do it gives MSE=34.94 ...
Caterina's user avatar
  • 119
2 votes
1 answer
65 views

Effect of log odds on skewed data

Does taking the log of odds bring linearity between the odds of the dependent variable & the independent variables by removing skewness in the data? Is this one reason why we use log of odds in ...
Apoorva's user avatar
  • 297
1 vote
1 answer
662 views

Feature engineering before splitting

This is a sister post to the original closed post (here). Since the data transformation part is done after data spliting on the TRAINING data only, I wonder wouldn't such transformation has dependency ...
Student's user avatar
  • 411
0 votes
1 answer
3k views

Should i always transform data to normal distribution?

I am trying to understand transformations but this question seems to be in my and some people's mind. If we have a numeric variable in EVERY data science case. Transforming data(Log, power transforms) ...
canP's user avatar
  • 101
0 votes
0 answers
153 views

Is there a Softmax-like transformation with scale-invariance and linarity?

At the moment I'm using XGBoost to generate a prediction of probabilities with a custom objective-function to build something like an expert system. To do so I need to transform the raw XGBoost ...
Someone2's user avatar
  • 111
0 votes
1 answer
1k views

How to revert np.log(data) and data.diff()?

I have used np.log(data) and then applied data.diff() to transform my data in timeseries model. I have the predictions. How do I ...
Sandhya Indurkar's user avatar
0 votes
1 answer
89 views

Outlier treatment

I am working on a regression problem where I have a lot of outliers in multiple variables. As far as I can think of, there are 3 things I can do to outliers. Remove them (least attractive option) ...
spectre's user avatar
  • 2,105
0 votes
1 answer
1k views

Should one log transform discrete numerical variables?

I am working on a Linear Regression problem and one of the assumptions of a Linear Regression model is that the features should be Normally Distributed. Hence to convert my non linear features to ...
spectre's user avatar
  • 2,105
0 votes
1 answer
175 views

Un-Pivot Data in Tableau

Say you have data with fields named: A, B, C, KEY, VALUE. And lets say the KEY field contains a discrete set of possible values like "X", "Y", and "Z". How do you ...
vicatcu's user avatar
  • 101
0 votes
1 answer
134 views

How to feature engineering after getting test data in deployment?

I am kind of confuse about this topic of feature engineering. I am trying to make an web app in which people can upload test data as csv. Now I am confuse about how to do feature engineering after ...
Pritam Sinha's user avatar
0 votes
1 answer
298 views

Transforming time series into static features?

I'm working on a side project where I have a mixture of static data and time series, and the goal would be to perform clustering on the data. There's a bunch of data sources, but basically the main ...
lte__'s user avatar
  • 1,320
0 votes
3 answers
277 views

Transforming Categorical to Numerical variable

I have a categorical variable with 4 levels ('8 c', '6 c','NAN','Others') and I want to convert it to numerical form. an Obvious way is to simply remove the 'c' part from the first two categories and ...
user avatar
1 vote
1 answer
22 views

Should I apply a transformation to columns with INTEGERS, in case I want to reduce the skewness of that column?

I am performing EDA on a dataset of Hotel Reservations. Target is Categorical stating if a given customer will cancel the reservation or not. Dataset has 25 features, 30244 entries. I have two ...
leahnanno's user avatar
0 votes
2 answers
147 views

How to reshape or clean data to be able to visualize it with violin plots?

My end goal is to visualize some data using a violin plot or something similar using Python. I have the following data in a file (test.csv). The first column is a ...
zmike's user avatar
  • 113
1 vote
1 answer
865 views

What if outliers still exist after variable transformation?

I have a variable with a skewed distribution. I applied BoxCox transformation and now the variable follows a Gaussian distribution. But, as seen in the image below in the boxplot, outliers still ...
Joe's user avatar
  • 85
1 vote
0 answers
74 views

Multiple regression with non-normal data in minitab - help

I am aiming to assess the effect of BMI (continuous) on certain biomarkers (also continuous) whilst adjusting for several relevant variables (mixed categorical and continuous) using multiple ...
shar6580's user avatar
5 votes
1 answer
812 views

Linear Regression bad results after log transformation

I have a dataset that has the following columns: The variable I'm trying to predict is "rent". My dataset looks a lot similar to what happens in this notebook. I tried to normalize the rent ...
Caldass_'s user avatar
  • 167
2 votes
1 answer
417 views

Are we allowed to transform the continuous target variable by creating a log transformation in order to have a normal distribution?

The following code gives the target variable Item_Outlet_Sales before transformation and Item_Outlet_Sales_log which is transformed ...
spchaane's user avatar
  • 131
0 votes
1 answer
150 views

Condense matrix of values into one column?

I have a dataset where I have unnecessarily duplicated column variables that I want to condense down. I wish the output wasn't so clumsy and I've already had to do some work to transform it and make ...
aep58's user avatar
  • 1
1 vote
1 answer
230 views

What is the difference between normalization and re-scaling?

This site does not describe the nature of the normalization tag. Does it differ from re-scaling? Many authors use the two terms interchangeably.
Subhash C. Davar's user avatar
1 vote
1 answer
92 views

Anonymize continuous variable for masking purposes

I am about to kick off a large hackathon event. We have a dataset that is comprised of one continuous variable with high precision, and a number of categorical variables qualifying these data 3-levels ...
HEITZ's user avatar
  • 911