I am working on a dataset for classification where each observation is a series of values of a certain measurement $Y$ for a fixed range of values of measurement $X$ (i.e. a discrete mapping from $A \subseteq X$ to $Y$). (I'm not allowed to reveal what $X$ and $Y$ are due to confidentiality issues.) I am doing some single-variable regression on this relationship to obtain a smooth $X$ $\to$ $Y$ function for each observation. The basis functions I'm using for this are

$$\left\{x\mapsto 1,\ x \mapsto \frac{1}{\sqrt{x}},\ x\mapsto e^{-x \cdot 10^{-3}}\right\}$$

This is done by applying each of these functions to the range of $X$ values; then this transformed feature set is passed to a standard linear regressor. (The $10^{-3}$ factor in the exponential is to avoid numerical issues; if I just use $e^{-x}$ the coefficient for it becomes 0 due to the $e^{-x}$ features vanishing to 0 for large values of $x$.) This essentially amounts to finding the best-fitting linear combination of the basis functions.

It seems like this choice of basis functions yields curves (dashed) that fit pretty well to the original data (solid):

regression results

(Again, I'm not labelling the axes due to confidentiality issues.)

When I went to inspect how well the regression coefficients separated instances of each class, I found this:

scatter plot of regression coefficients for 1 and 1/sqrt(x)

The coefficient of 1 (that is, the independent term) and that of $\frac{1}{\sqrt{x}}$ seem to be linearly correlated. This lead me to think that perhaps one of the two basis functions was redundant. However, when I removed either, the regression performed worse. For example, removing the independent term:

regression results after removing the independent term

So it is not necessarily true that one of these two basis functions are redundant. My question is: if the values of a subset of the regression coefficients seem to be linearly correlated, what can I infer from it, if anything?

  • 1
    $\begingroup$ no no the fact that coefficients between different basis functions are correlatd DOES NOT mean that one basis function is redudant. In fact it means that both are needed in a definite ratio one to the other $\endgroup$
    – Nikos M.
    Feb 25 '21 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.