Predictive power of a dataset

I am reading a book on machine learning for undergraduate. I am actually confused on linear regression flexibility as the say:

Occasionally, linear regression will fail to recover a good solution for a data set. While this may be because our data doesn’t actually have predictive power, it might also just indicate that our data is provided in a format unsuitable for linear regression.

I read some questions here related to predictive power and I noticed that it's all about the model produced. What do we mean when we talk about the predictive power of a dataset?

I think of it as there is no relationship between features (as linear regression is intented to learn the relationship between the inputs $$X$$ and the output $$Y$$). But I am not convinced of this answer yet.

Usually predictive power refers to the model, rather than the data. I've occasionally seen some people use it in the way that the author of your book uses it (see this for example).

In the context of your book, yes, predictive power refers to whether input can be mapped to target output $$X\rightarrow Y$$. We can infer a dataset's "predictive power" by trying to model it (e.g. linear regression). If the model performs poorly, then there are two possibilities as the book says: either the dataset is not predictive (i.e. it does not offer a clean mapping from input to target output) or the methods we are using are unsuitable to model the mapping.

Some examples of both situations:

• If you generated random data for $$X$$ and $$Y$$, the resulting dataset would (probably) have no predictive power as no model could reasonably generalize the mapping $$X\rightarrow Y$$.

• If you have a nonlinear mapping, then linear regression would not fit it well. For example, if our dataset was such that $$y_1$$ is mapped to by $$||\vec{x}||<\alpha$$ and all other inputs map to $$y_2$$, then our dataset is extremely predictive, but our linear regression model cannot fit it (since the mapping is nonlinear). In this toy example, it's easy to see the predictive power of the dataset, particularly if the input is in 2D/3D since we could just plot it. However, manually observing such trends in highly dimensional space using actual data can be very difficult, hence we use the tools that you are learning to help interpret the data. Also, when there's nonlinearity, it's difficult to statistically evaluate the dataset itself. Variables with linear relationships are simple to correlate (e.g. Pearson's correlation coefficient) but nonlinearities can make correlation difficult. I assume that this is why your book defers to vague terminology as it's probably for pedagogical, rather than pedantic, purposes. After all, it gets the point across without needing to discuss the ongoing research into quantifying nonlinear correlations.

• the mapping aspect is not particularly clear. Are you referring to correlation? Suggest to elaborate further on this point. – mnm Sep 7 '20 at 0:11
• @mnm what specifically is unclear? I'm referring to the fact that ML methods generally learn to map input to output. See this: en.wikipedia.org/wiki/… – Benji Albert Sep 7 '20 at 4:49
• reference to wikipedia article must be avoided at all costs especially when discussing mathematical modelling because such articles have no authenticity to them. And for this reason alone, similar references to articles published on mushroomed websites like medium be avoided too. What do you mean by mapping is the question. Basis of what logic is the mapping done? – mnm Sep 7 '20 at 6:39
• @mnm if you don't understand this, then you should post a question yourself. I encourage you to read (wikipedia lists references at the bottom), but in short, the ML model learns a mapping $f$ such that $f:X\rightarrow Y$ given training inputs $\{x_1,x_2,...,x_n\}\in X$ which respectively correspond to $\{y_1,y_2,...,y_n\}\in Y$. The goal is to map $x_i \rightarrow y_i$ with enough generalization that the end user can map previously unseen samples of $X$ to the correct $Y$. "Correct" being the true mapping function that which the ML model approximates – Benji Albert Sep 7 '20 at 13:28
• you've completely missed the point! Its not about my understanding, so lets get that out off the way. And the question still remains unanswered, how is the mapping done? What determines that there is a relationship between X and Y? What is the logic? – mnm Sep 7 '20 at 14:11

Rather than asking about the predictive power of a dataset, I think it's intuitive to ask about the predictive power of a model. My reasoning is as follows;

A dataset can be univariate, bivariate or multivariate types. The dataset can contain only numerical features or categorical features or both. Suppose there is a univariate dataset with a negative skewed distribution. In such a case the mean, median will be less than the mode. Now suppose this univariate dataset consist of continuous data type. Knowing that its distribution is negatively skewed has already given the analyst a clue about its symmetry or distribution. So basis of this brief introduction, as an analyst will I be interested to know the predictive power of a dataset or the model('s) that I build using this dataset, is a question worth discussing?

There have been several studies in literature that have discussed the model's predictive power like 1,2,3 (see references). In contrast, I have not come across any study that has discussed the predictive power of a dataset. Perhaps a future research direction.

However, I did find an article published on R-bloggers that discussed about a predictive power score, a concept somewhat similar to correlation coefficient.

And finally something about mapping. I think a better term could be "correlation" which at least quantifies the relationship between two variables X and Y.

Note

A similar question was asked on stats.stackexchange.com. The comments in it conform to my initial doubt, that there is no such thing as the predictive power of a dataset.

References

1. Lee, P. H. (2014). Resampling methods improve the predictive power of modeling in class-imbalanced datasets. International journal of environmental research and public health, 11(9), 9776-9789.
2. López‐López, J. A., Marín‐Martínez, F., Sánchez‐Meca, J., Van den Noortgate, W., & Viechtbauer, W. (2014). Estimation of the predictive power of the model in mixed‐effects meta‐regression: A simulation study. British Journal of Mathematical and Statistical Psychology, 67(1), 30-48.
3. Newson, R. B. (2010). Comparing the predictive powers of survival models using Harrell's C or Somers’ D. The Stata Journal, 10(3), 339-358.
• ah, I see what you were talking about in the comment thread. I've only rarely come across "predictive power" to indicate characteristics of the dataset rather than the model, but because it's inherently a vague term, some do use it in this manner ieeexplore.ieee.org/document/5616710. Thanks for taking the time to answer! – Benji Albert Sep 8 '20 at 6:13