I have a general question that comes to my mind, I'm doing machine learning projects and I took a look at many datasets and worked with, mostly there are already famous datasets that everyone uses.

Now the question comes to my mind, let's say I decided to make my own dataset, is there a possibility that my data are so random so that no relationship exist between my inputs and outputs ?? this is interesting because if this is possible, then no machine learning model will achieve to find an inputs outputs relationship in the data and will fail to solve the regression or classification problem.

Moreover, is it mathematically possible that some values have absolutely no relationship between them? in other words, there is no function (linear or nonlinear) that can map those inputs to the outputs.

Now I thought about this problem and concluded that if there is a possibility for this, then it will likely happen in regression because maybe the target outputs are in the same range and the same features values can correspond to the same output values and that will confuse the machine learning model.

What are your thoughts on this? did you had this problem in your daily life as a machine learning engineer, data scientist or hobbyist?


Yes and no


Let's consider at two distributions $F_X(x)$ and $G_Y(y)$ whose joint distribution is simply the product of the the two distributions: $H_{X,Y}(x,y) = F_X(x)G_Y(y)$. This means that the distributions are 100% independent: not just linearly (no correlation), but that there is absolutely no dependence between the two distributions. (Copula, Sklar's theorem, etc.)

The NO

For the above scenario to happen, there has to be absolutely no dependence between the two distributions. At the population level, this is calculated by Hoeffding's test of independence.

$$\theta =\int \bigg{[}H_{X,Y}(x,y) - F_X(x)G_Y(y)\bigg{]}^2 dH_{X,Y}(x,y) = 0$$

For empirical data, this would be tested with the hypothesis test $H_0: \theta=0, H_a: \theta\ne 0$.

Remember how we never accept a null hypothesis of equality because we know that it not quite true?

The YES (again)

It can be the case that, for all practical purposes, the distributions are unrelated. Yes, maybe if a butterfly flaps its wings in Ecuador it will be more likely that there will be an earthquake in Madagascar...but probably not much of an effect. This corresponds to $\theta\ne0$ but $\theta\approx0$ (equal for all practical purposes).

  • $\begingroup$ I have very little experience in formal statistics (as in, I only have small experience in statistical machine learning), what do you mean by "null hypothesis of equality", and why do we reject it? P.S. I hail from the AI.SE via this question: ai.stackexchange.com/questions/18395/…. does my answer to the same question there make any sense at all? $\endgroup$ – k.c. sayz 'k.c sayz' Mar 4 '20 at 18:52
  • $\begingroup$ I mean something like $H_0: \mu=6$ versus $H_a: \mu\ne 6$, except that we’re testing the Hoeffding value instead of the mean. I did not mention rejecting a null hypothesis, so do you mean why we never accept a null hypothesis? $\endgroup$ – Dave Mar 4 '20 at 19:04
  • $\begingroup$ i was referring to this part of your answer: "Remember how we never accept a null hypothesis of equality because we know that it not quite true?" $\endgroup$ – k.c. sayz 'k.c sayz' Mar 5 '20 at 1:37
  • $\begingroup$ The idea is that, unless it is truly the case that $\theta$ is absolutely, bang-on equal to zero, they null is at least a little bit false. In practice, we know that both distributions are influenced by the same forces of the universe, so a butterfly flapping its wings will indeed have ever so slight of an impact on the probability of an earthquake. (Chaos theory is what gets into how it could be a large impact.) $\endgroup$ – Dave Mar 5 '20 at 2:10

Of course you can.

import numpy as np
from sklearn.linear_model import LinearRegression

lr  = LinearRegression()

X = np.random.rand(1000,10)
y = np.random.rand(1000,1)

Out:   0.00009

In this case there is no relation.

For classification you can do the same:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

log = LogisticRegression()

X = np.random.rand(1000,10)
y = np.random.rand(1000,1)>0.5


Out: 0.53

In this case 0.5 is random

  • $\begingroup$ Thanks for your answer but it is not quite what I'm asking! The linear regression and classification model means nothing. I can show you many examples where linear regression fail but neural network succeed to find a relationship. My question is more about mathematics and data science than machine learning. The question is whether data exist that have no relationships, even a neural network model cant fit. In theory, neural networks are proven to model any nonlinearity, that confused me more $\endgroup$ – basilisk Mar 3 '20 at 20:53
  • $\begingroup$ you can substitute that linear model for whatever you want. If you do train test split you won't get better accuracy than random guessing. If it does, it is overfitting $\endgroup$ – Carlos Mougan Mar 3 '20 at 21:02
  • $\begingroup$ I might argue that if the model is able to predict out-of-sample, then perhaps it has latched onto whatever the computer is doing to generate those two sets of data. $\endgroup$ – Dave Mar 3 '20 at 23:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.