Is there a possibility that there is no relationship between some inputs and outputs?

Question

I have a general question that comes to my mind, I'm doing machine learning projects and I took a look at many datasets and worked with, mostly there are already famous datasets that everyone uses.

Now the question comes to my mind, let's say I decided to make my own dataset, is there a possibility that my data are so random so that no relationship exist between my inputs and outputs ?? this is interesting because if this is possible, then no machine learning model will achieve to find an inputs outputs relationship in the data and will fail to solve the regression or classification problem.

Moreover, is it mathematically possible that some values have absolutely no relationship between them? in other words, there is no function (linear or nonlinear) that can map those inputs to the outputs.

Now I thought about this problem and concluded that if there is a possibility for this, then it will likely happen in regression because maybe the target outputs are in the same range and the same features values can correspond to the same output values and that will confuse the machine learning model.

What are your thoughts on this? did you had this problem in your daily life as a machine learning engineer, data scientist or hobbyist?

Answer 1

Yes and no

The YES

Let's consider at two distributions $F_X(x)$ and $G_Y(y)$ whose joint distribution is simply the product of the the two distributions: $H_{X,Y}(x,y) = F_X(x)G_Y(y)$. This means that the distributions are 100% independent: not just linearly (no correlation), but that there is absolutely no dependence between the two distributions. (Copula, Sklar's theorem, etc.)

The NO

For the above scenario to happen, there has to be absolutely no dependence between the two distributions. At the population level, this is calculated by Hoeffding's test of independence.

$$\theta =\int \bigg{[}H_{X,Y}(x,y) - F_X(x)G_Y(y)\bigg{]}^2 dH_{X,Y}(x,y) = 0$$

For empirical data, this would be tested with the hypothesis test $H_0: \theta=0, H_a: \theta\ne 0$.

Remember how we never accept a null hypothesis of equality because we know that it not quite true?

The YES (again)

It can be the case that, for all practical purposes, the distributions are unrelated. Yes, maybe if a butterfly flaps its wings in Ecuador it will be more likely that there will be an earthquake in Madagascar...but probably not much of an effect. This corresponds to $\theta\ne0$ but $\theta\approx0$ (equal for all practical purposes).

Answer 2

Of course you can.

import numpy as np
from sklearn.linear_model import LinearRegression

lr  = LinearRegression()

X = np.random.rand(1000,10)
y = np.random.rand(1000,1)

lr.fit(X,y)
lr.score(X,y)
Out:   0.00009

In this case there is no relation.

For classification you can do the same:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

log = LogisticRegression()

X = np.random.rand(1000,10)
y = np.random.rand(1000,1)>0.5



log.fit(X,y)

roc_auc_score(log.predict(X),y)
Out: 0.53

In this case 0.5 is random