0
$\begingroup$

I have the following data frame with one categorical and two numerical columns:

    V1  V2  V3
1   A   1   3
2   A   3   5
3   B   3   3
4   C   2   3

I have turned this into the following dummy variables:

    V1.1  V1.2  V1.3 V2  V3
1   1     0     0    1   3
2   1     0     0    3   5
3   0     1     0    3   3
4   0     0     1    2   3

Now, I want to apply clustering to this latter set. I guess that I could get better results if I downweight the dummy variable columns (proportionally to their number) because with equal weights the distance based clusters will be distorted.

My question is, how could I get the following weight vector from the new data set:

0.33, 0.33, 0.33, 1, 1
$\endgroup$
2
  • $\begingroup$ How did you do the transformation from your first data set to the second? I strongly suspect the information you need will be available as part of that transformation. $\endgroup$ Sep 15, 2016 at 5:37
  • $\begingroup$ I produce the dummy variables this way: data.frame(model.matrix(label~. -1, data)) $\endgroup$
    – Hendrik
    Sep 15, 2016 at 7:44

2 Answers 2

1
$\begingroup$

The information you need is in the assign attribute on the matrix returned by model.matrix. Something like this seems to work for me:

data <- data.frame(V1 = factor(c("A", "A", "B", "C")), V2 = c(1, 3, 3, 2), V3 = c(3, 5, 3, 3))
model <- model.matrix(~ . + 0, data = data)
assignments <- attr(model, "assign")
counts <- table(assignments)
weights <- sapply(assignments, function (x) { 1 / counts[[x]] })

weights is now (0.3333333, 0.3333333, 0.3333333, 1.0000000, 1.0000000).

$\endgroup$
0
0
$\begingroup$

Actually a more reasonable weight vector would be

(0.5, 0.5, 0.5, 1, 1)

Why - because any two rows will be different in exactly 0 or 2 columns. With this weight, the one-hot encoded attribute yields a distance contribution of exactly 1 or exactly 0.

The easiest way is to encode your data using these weights instead of 0 and 1 rather than applying the weights in later calculations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.