# Extracting linear trends from a dataset

Consider a sensor measurement f that varies with both temperature T and the properties of the fluid being measured. The temperature changes through each day and the fluid properties can be assumed to vary less frequently. If I cross plot the data in Excel then by eye I can very easily draw a straight line through some points and translate that line horizontally and "voila" that same line fits through other clusters of plots. So if that line has slope -1/a then all I need to do instead of plotting f versus time I actually plot f + a T versus time I get a curve which does not have the temperature dependence and is now indicative of fluid property. Cool.

So my question is how to automate this extraction of a. My plan today is to set objective function as the L1 norm of the derivative of the time series and minimize over a as that might give a time series of many points with almost zero derivative and a few jump discontinuities where it decides the fluid has changed [as opposed to an L2 that would smear everything out].

But my thoughts are that this feature extraction is likely already covered in some text book somewhere and what I am really missing is the better vocabulary to look it up :-)

Any suggestions? Thanks

OK, I am happy with this. I define a "neuron" to be 2/piarctan(lambda(date-date0)) and have about 99 neurons with steadily increasing date0. If I write any time series as a sum of these neurons then effectively that creates a smooth, piecewise constant approximation to that time series, which is what I'm after. So I add to those neurons an extra "neuron" which is measured temperature data giving me 100 neurons and then a simple L2 optimization to find the best fit of the whole shebang.

In case you can treat the properties as discrete variables (aka "dummies", "factors", "one-hot"), you can use linear regression to decompose the effects. Viz. you estimate a "straight line" for each of the properties.

# Data
df = data.frame(y=c(1,2,3,5,6, 10,12,13,15,16),prop=c(0,0,0,0,0,1,1,1,1,1),temp=c(5,6,7,8,9,6,7,8,9,10))

# Plot data
plot(df$$temp[df$$prop==1], df$$y[df$$prop==1],xlim=c(4,10),ylim=c(0,16),xlab="temp",ylab="y")
lines(df$$temp[df$$prop==1], df$$y[df$$prop==1])
lines(df$$temp[df$$prop==0], df$$y[df$$prop==0], col="blue")


# Linear regression
reg = lm(y~temp+prop,data=df)
summary(reg)

Residuals:
Min     1Q Median     3Q    Max
-0.4   -0.2    0.0    0.2    0.4

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -6.40000    0.55032  -11.63 7.85e-06 ***
temp         1.40000    0.07559   18.52 3.32e-07 ***
prop         8.40000    0.22678   37.04 2.72e-09 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.3381 on 7 degrees of freedom
Multiple R-squared:  0.9971,    Adjusted R-squared:  0.9963
F-statistic:  1222 on 2 and 7 DF,  p-value: 1.245e-09

# Prediction for "prop" 1, 0
pred0 = predict(reg,newdata=df[df$$prop==0,]) pred1 = predict(reg,newdata=df[df$$prop==1,])

lines(df$$temp[df$$prop==1], pred1, col="red")
lines(df$$temp[df$$prop==0], pred0, col="purple")


So you get one "predicted line" per prop.

For prop=0 this would be calculated like $$-6.4 + 1.4 * temp + 0 * 8.4$$.

For prop=1 this would be calculated like $$-6.4 + 1.4 * temp + 1 * 8.4$$.

Essentially: same intercept and slope. Only the "niveau" of the lines is shifted according to the prop indicator.

• Thanks for the feedback, but I don't know the prop indicator. They are also a time series. Indeed determining that is the goal. E.g., you have Freq(time) = Prop(time) + Temp(time)*Coeff + C where Temp(Time) and F(Time) are known. Goal is to choose a Coeff to get Prop(time) to be largely constant and cluster around a few values. Given a value of Coeff, I can define Prop(time) from that equation and I tried minimizing L1 norm of the derivative of Prop(time) and that didnt work. Nov 2, 2021 at 17:06
• Tried something else that didn't work, for a given slope "Coeff" and intercept "C" I can compute a sort of Radon transform by summing all the datapoints that lie close to that line. I was hoping the result would cluster in the Radon transformed space and I could Kmean or something. But nope.... No obvious clustering. I think my problem is that the "eye" does a great job of finding line through the outliers but lots of lines fat through the middle of the blob have plenty of intersection too, maybe more, so the Radon fails. Nov 3, 2021 at 16:42