0
$\begingroup$

I have the following data frame with one categorical and two numerical columns:

    V1  V2  V3
1   A   1   3
2   A   3   5
3   B   3   3
4   C   2   3

I have turned this into the following dummy variables:

    V1.1  V1.2  V1.3 V2  V3
1   1     0     0    1   3
2   1     0     0    3   5
3   0     1     0    3   3
4   0     0     1    2   3

Now, I want to apply clustering to this latter set. I guess that I could get better results if I downweight the dummy variable columns (proportionally to their number) because with equal weights the distance based clusters will be distorted.

My question is, how could I get the following weight vector from the new data set:

0.33, 0.33, 0.33, 1, 1
$\endgroup$
  • $\begingroup$ How did you do the transformation from your first data set to the second? I strongly suspect the information you need will be available as part of that transformation. $\endgroup$ – Philip Kendall Sep 15 '16 at 5:37
  • $\begingroup$ I produce the dummy variables this way: data.frame(model.matrix(label~. -1, data)) $\endgroup$ – Hendrik Sep 15 '16 at 7:44
1
$\begingroup$

The information you need is in the assign attribute on the matrix returned by model.matrix. Something like this seems to work for me:

data <- data.frame(V1 = factor(c("A", "A", "B", "C")), V2 = c(1, 3, 3, 2), V3 = c(3, 5, 3, 3))
model <- model.matrix(~ . + 0, data = data)
assignments <- attr(model, "assign")
counts <- table(assignments)
weights <- sapply(assignments, function (x) { 1 / counts[[x]] })

weights is now (0.3333333, 0.3333333, 0.3333333, 1.0000000, 1.0000000).

| improve this answer | |
$\endgroup$
0
$\begingroup$

Actually a more reasonable weight vector would be

(0.5, 0.5, 0.5, 1, 1)

Why - because any two rows will be different in exactly 0 or 2 columns. With this weight, the one-hot encoded attribute yields a distance contribution of exactly 1 or exactly 0.

The easiest way is to encode your data using these weights instead of 0 and 1 rather than applying the weights in later calculations.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.