Let's assume that I have a linear model with $k$ variables:
$y = \beta_0 + \beta_1\cdot x_1 + \dots + \beta_k \cdot x_k$.
Now, I want to add variable $x_{k+1}$, but, according to domain knowledge, the dependency of $y$ onto $x_{k+1}$ is not linear, but rather "S-shaped". To capture this dependency, a well-established method is to use parametrised arcus tangens function: $D \cdot \arctan{\frac{x_{k+1} - A}{B}} + C$. To include it in the linear model we can safely ignore $D$ and $C$ parameters (as they will contribute to $\beta_{k+1}$ and $\beta_0$ respectively), so eventually I'd end up with the model:
$y = \beta_0 + \beta_1\cdot x_1 + \dots + \beta_k \cdot x_k + \beta_{k+1}\cdot \arctan{\frac{x_{k+1} - A}{B}}$.
What I would like to do is to find a way to test different transformations (with different $A$ and $B$ parameters automatically. Firstly, I went for searching the grid of different $A$ and $B$ values, i.e. fitting the model to different transformations of $x_{k+1}$ and returning list of models, sorted by $R^2$, $RMSE$ or $AIC$. However, this usually favours very sharp curves:
This doesn't make much sense in the eye test though. I know that eye-test may be biased but to me, more propable dependency would be captured by a shape as below:
One could already noticed that there are plenty of observations where $x_{k+1}$ is or is near to $0$, which is actually a characteristic of those variables that will be approximated by $arctan$. That may probably affect the fitting, but still something clearly doesn't work here.
Another approach that I considered is to fit the model using gradient descent search rather than using standard software (like lm
from R). The problem is that with this $arctan$ transformation, we don't have a convex objective function anymore.
How can I then automatically test for the best parameters of the $arctan$ transformation? Am I on the right track with either idea or can I approach this problem differently?r