Let's say I have $n$ data points with just one categorical feature $x$ and a continuous target variable $y$. I want to divide the possible values of $x$ into subsets such that the value of $y$ doesn't vary much within a subset.

As an example, suppose there are only $5$ possible values of $x$: $x_1,\ldots,x_5$. We observe that the values of $y$ don't vary much across $x_1$ and $x_2$. $y$ doesn't vary much across $x_3$ and $x_5$, but is different from the typical $y$ value for $x_1$ and $x_2$. For $x_4$, $y$ values are quite different from the above groups. So we can say that $(x_1,x_2), (x_3,x_5)$ and $(x_4)$ can be considered as "clusters".

Now what's a concrete way of saying that $y$ doesn't "vary much across $x_1$ and $x_2$"? One natural way to define that is for $y$ to have the same distribution for $x_1$ and $x_2$, i.e. the same mean and standard deviation of $y$ values for both $x_1$ and $x_2$.

  • Is there a better way to characterize the fact that $y$ doesn't "vary much across $x_1$ and $x_2$? Maybe the way I defined it above is too idealistic and I need a criterion more suited to a real-life dataset?

  • Are there any popular existing methods or library functions to solve such kind of a problem (i.e. clustering categorical feature values on the basis of continuous target values)?


Since you are looking for a degree of similarity regarding $y$ and the values of $x_1,...,x_5$ do not matter you can view this as a clustering problem regarding $y$:

Let $y_1,...y_5$ be the target values with $f(x_i)=y_i$ for $i\in\{1,...5\}$ then you need to define a distance measure $d(y_i,y_j)$ which, since your variables $y$ are continuous, could be the euclidian distance: $d(y_i,y_j) = (y_i-y_j)^2$. But you could also choose the absolute difference. (note that I am assuming your $y_i$ to be one-dimensional here, i.e. $y_i\in \mathbb R$). That gives you a formula to measure "does not vary that much".

It also provides an answer to your second question: you can choose from a range of unsupervised ML algorithms to do clustering. The most popular one probably being $K$-means. The idea here is very straight forward:

1. select a number of clusters K
2. initialize the position of the K clusters randomly
3. Assign each y_i to the closest cluster
4. For each cluster k calculate the new cluster position as the mean of all y_i belonging to that cluster
5. repeat 3 and 4 until the assignments of y_i does not change anymore

Mathematically this gives you a mapping $c(y_i)=k$.

And eventually when you are done with your clustering you just pick a cluster $k$ and for each $y_i$ in that cluster you look up the corresponding $x_i$. Which will return what you asked for.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.