Questions tagged [transformation]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
24 views

Data Skewness is nan or inf

I have checked the Skewness of my data before applying a Log transformation using the next code : print("Skewness: %f" % df['Wind Speed (km/h)'].skew()) ...
baddy's user avatar
  • 165
0 votes
0 answers
12 views

Transform monthly data to the weekly data based on categorical value

I have monthly data like this which is hours: month,hours 1,20 2,19.5 3,21 ... I have weekly data like this which is quantities of the asset: ...
Muhammad Ikhwan Perwira's user avatar
0 votes
1 answer
61 views

Data transformation pipeline error

When I'm making data transformation pipeline on a dataset I keep getting error as " all the input array dimensions except for concatenation axis must match exactly, but along dimension 0, the ...
Amy's user avatar
  • 1
0 votes
0 answers
119 views

einstein sum subscripts string contains too many subscripts for operand 4

...
Asta Surviliene's user avatar
0 votes
0 answers
50 views

Is SVM rotation invariant?

Let's say we have some data X and we want to find a linear separator using soft SVM with l2 regularization, and then we want to solve the same problem after applying some rotation matrix Q to the data ...
user3917631's user avatar
0 votes
0 answers
13 views

Methods to appropriately capture equal ranges of turnover

Suppose I have a dataset with three columns, turnover, market, count, and turnover is the range of turnover in EUR. Something like: ...
Emil11's user avatar
  • 1
0 votes
0 answers
30 views

How to multiply the boolean/binary values in 1 row in a dataframe by the chr values in another row with the product landing in the original row

Suppose that I already know which subset out of a set of 30 candidate regressor columns are the true regressors included in the structural equation describing that dataset (because I do by ...
Marlen's user avatar
  • 157
1 vote
0 answers
67 views

How do SciKit-Learn Pipelines pass Data between Steps?

I would like to create some Custom Transformers and incorporate them in a SciKit-Learn Pipeline. I'd like to pass more than just a Dataframe between the transformer ...
Connor's user avatar
  • 617
0 votes
0 answers
66 views

Should I apply log transform?

Hsre is my skewness and kurtosis value before applying log transform. For skewness: I use -3 < x < 3 as acceptable value and kurtosis at -10 < x < 10 as acceptable value. Since my data has ...
UrDailyCS's user avatar
1 vote
1 answer
23 views

Data transformation

Should data transformation techniques (ex: creating new features/ log attributes) be done before or after feature selection techniques (ex: mutual information feature selection)? Are there any ...
Kusisi Karem's user avatar
0 votes
1 answer
72 views

Handling data skewness and kurtosis

I have a dataset where the variables have high skewness (> ±1) and kurtosis (> ±5). I tried to remove outliers and log10 transformation the skewness and kurtosis are still high. Are there any ...
Kusisi Karem's user avatar
0 votes
2 answers
816 views

Should I deal with missing values first then transform the data or vice versa?

I am currently working on a project involving time series banking stock price data. I have around 3000 observations, some columns have a lot of missing values (null value); they can account for 5 to ...
MINH NHỰT NGUYỄN TRẦN's user avatar
0 votes
1 answer
15 views

How can I transform or plot my data to see power consumption more easily?

I have two heating tapes installed in my setup and they provide heat to maintain the reaction at a certain setpoint temperature. Basically, the heating tapes go on a cycle of on/off to maintain the ...
ScepticalChymist's user avatar
0 votes
1 answer
112 views

Why does log-transforming the target have a huge impact on MSE value?

I am doing linear regression using the Boston Housing data set, and the effect of applying $\log(y)$ has a huge impact on the MSE. Failing to do it gives MSE=34.94 ...
Caterina's user avatar
  • 119
1 vote
1 answer
49 views

Effect of log odds on skewed data

Does taking the log of odds bring linearity between the odds of the dependent variable & the independent variables by removing skewness in the data? Is this one reason why we use log of odds in ...
Apoorva's user avatar
  • 275
1 vote
1 answer
554 views

Feature engineering before splitting

This is a sister post to the original closed post (here). Since the data transformation part is done after data spliting on the TRAINING data only, I wonder wouldn't such transformation has dependency ...
Student's user avatar
  • 411
0 votes
1 answer
2k views

Should i always transform data to normal distribution?

I am trying to understand transformations but this question seems to be in my and some people's mind. If we have a numeric variable in EVERY data science case. Transforming data(Log, power transforms) ...
canP's user avatar
  • 91
0 votes
0 answers
152 views

Is there a Softmax-like transformation with scale-invariance and linarity?

At the moment I'm using XGBoost to generate a prediction of probabilities with a custom objective-function to build something like an expert system. To do so I need to transform the raw XGBoost ...
Someone2's user avatar
  • 111
0 votes
1 answer
898 views

How to revert np.log(data) and data.diff()?

I have used np.log(data) and then applied data.diff() to transform my data in timeseries model. I have the predictions. How do I ...
Sandhya Indurkar's user avatar
0 votes
1 answer
87 views

Outlier treatment

I am working on a regression problem where I have a lot of outliers in multiple variables. As far as I can think of, there are 3 things I can do to outliers. Remove them (least attractive option) ...
spectre's user avatar
  • 1,941
0 votes
1 answer
1k views

Should one log transform discrete numerical variables?

I am working on a Linear Regression problem and one of the assumptions of a Linear Regression model is that the features should be Normally Distributed. Hence to convert my non linear features to ...
spectre's user avatar
  • 1,941
0 votes
1 answer
124 views

Un-Pivot Data in Tableau

Say you have data with fields named: A, B, C, KEY, VALUE. And lets say the KEY field contains a discrete set of possible values like "X", "Y", and "Z". How do you ...
vicatcu's user avatar
  • 101
0 votes
1 answer
126 views

How to feature engineering after getting test data in deployment?

I am kind of confuse about this topic of feature engineering. I am trying to make an web app in which people can upload test data as csv. Now I am confuse about how to do feature engineering after ...
Pritam Sinha's user avatar
0 votes
1 answer
203 views

Transforming time series into static features?

I'm working on a side project where I have a mixture of static data and time series, and the goal would be to perform clustering on the data. There's a bunch of data sources, but basically the main ...
lte__'s user avatar
  • 1,310
0 votes
3 answers
224 views

Transforming Categorical to Numerical variable

I have a categorical variable with 4 levels ('8 c', '6 c','NAN','Others') and I want to convert it to numerical form. an Obvious way is to simply remove the 'c' part from the first two categories and ...
user avatar
1 vote
1 answer
22 views

Should I apply a transformation to columns with INTEGERS, in case I want to reduce the skewness of that column?

I am performing EDA on a dataset of Hotel Reservations. Target is Categorical stating if a given customer will cancel the reservation or not. Dataset has 25 features, 30244 entries. I have two ...
leahnanno's user avatar
0 votes
2 answers
127 views

How to reshape or clean data to be able to visualize it with violin plots?

My end goal is to visualize some data using a violin plot or something similar using Python. I have the following data in a file (test.csv). The first column is a ...
zmike's user avatar
  • 113
1 vote
1 answer
707 views

What if outliers still exist after variable transformation?

I have a variable with a skewed distribution. I applied BoxCox transformation and now the variable follows a Gaussian distribution. But, as seen in the image below in the boxplot, outliers still ...
Joe's user avatar
  • 75
1 vote
0 answers
68 views

Multiple regression with non-normal data in minitab - help

I am aiming to assess the effect of BMI (continuous) on certain biomarkers (also continuous) whilst adjusting for several relevant variables (mixed categorical and continuous) using multiple ...
shar6580's user avatar
2 votes
1 answer
392 views

Are we allowed to transform the continuous target variable by creating a log transformation in order to have a normal distribution?

The following code gives the target variable Item_Outlet_Sales before transformation and Item_Outlet_Sales_log which is transformed ...
SP_'s user avatar
  • 131
0 votes
1 answer
115 views

Condense matrix of values into one column?

I have a dataset where I have unnecessarily duplicated column variables that I want to condense down. I wish the output wasn't so clumsy and I've already had to do some work to transform it and make ...
aep58's user avatar
  • 1
1 vote
1 answer
176 views

What is the difference between normalization and re-scaling?

This site does not describe the nature of the normalization tag. Does it differ from re-scaling? Many authors use the two terms interchangeably.
Subhash C. Davar's user avatar
1 vote
1 answer
90 views

Anonymize continuous variable for masking purposes

I am about to kick off a large hackathon event. We have a dataset that is comprised of one continuous variable with high precision, and a number of categorical variables qualifying these data 3-levels ...
HEITZ's user avatar
  • 911