5
$\begingroup$

I am working on a project and currently experimenting cluster analysis. The dataset is mainly categorical variables and discrete numbers. Please pardon my poor programming skills as I am not very familiar with MathJax, but I will try to summarize the data set in words in the following three examples.

  1. imagine column 1 is participants names of course, from column2 - 5, each column's value range from 1 (least important) - 5 (most important). so in this case, column 2-8 only have discrete data.
  2. column 6 for example, since this is a multiple choice question. row 1 chose "nice" as an answer, however, row 2 chose "poor". In this case, we have one column that contains multiple categorical answers.
  3. for column 7-9, this is another type of multiple choice questions. However this time, each column represents only one answer. column 7 only allows string values "true", column 8 only allows string "somewhat". So in this case, we have multiple columns that represent multiple answers of the same question.

Any ideas how to solve this? Thankful to any input!

A2       A3       A4       A5       A6       A7         A8         A9
1        4        5        4        nice     true       somewhate  false
2        4        3        1        poor     true                  false
1        5        2        1        nice                somewhate    
3        2        1        5        nice     true                  false
$\endgroup$
1
$\begingroup$

I assume your data set is sth like

A0 A1 A2 B0 B1 B2
1  0  0  1  1  0
0  1  0  0  0  1
0  1  0  0  1  1
      ...

where A0 refer to the answer of question A, choice 0 and question A is a single choice question, B is a question allowing multiple answers. Each row represents one record.

For this data format, each column can be considered as a dimension (if some columns can be quantified to a value, you may group them into one column, ex "Like = 1, Neutral = 0, Dislike = -1") Then applying clustering algorithms, such as K-mean to cluster it.

Please tell if your data set format is different from my assumption.

Updated on Jun 30:

The idea is to quantify the variables, For column 2-5, the values are already quantified to [1, 5] For column 5, it allows ("poor"/ "nice"). Although they are strings, they are representing the preference of different magnitude. They can also be converted to "poor" = 0, "nice" = 1. For column 7-9, because it is a binary option, they can also be easily converted to ex. "true" = 0, "" = 1.

Then, you will have a matrix:

1    4    5    4    1    1    1    1
2    4    3    1    0    1    0    1
1    5    2    1    1    0    1    1
3    2    1    5    1    1    0    1

Remark1: For binary options, such as column 9, only allows "False" or "null", choosing "False" = 1 or 0 does not matter. In classification, only the distance matters.

Remark2: If you have a column allows different strings, such as "apple", "banana", "orange", you can convert it to an one hot vector. Ex.

A
apple
orange
orange
banana

can be converted to

A0   A1   A2
1    0    0
0    0    1
0    0    1
0    1    0

where A0, A1, A2 represent "apple", "banana" and "orange".

Remark3: if there is a column allow storing multi answers, ex.

A
apple
apple orange
orange
orange banana

can be converted to A0 A1 A2 1 0 0 1 0 1 0 0 1 0 1 1

Remark4: before applying some classification algorithms such as K-mean, it is better to normalize the magnitude of each dimension. For example, value of column 9 is [0, 1] can be normalized by times 5 to [0, 5]. The factor/ the range of the dimension reflects the importance of that factor.

I hope my answer can help.

$\endgroup$
  • $\begingroup$ Thank you. I will try to explain my data in two more specific examples: 1. imagine column 1 is participants names of cuase, from column2 - 8, each column's value range from 1 (least important) - 5 (most important) $\endgroup$ – Jing Jun 29 '16 at 16:32
  • $\begingroup$ so in the above example, all numbers are discrete. $\endgroup$ – Jing Jun 29 '16 at 16:32
  • $\begingroup$ in the 2 example: column 9 for example, since this is a multiple choice question. row 1 chose "very successful" as an answer, however, row 2 chose "needs improvement" $\endgroup$ – Jing Jun 29 '16 at 16:36
  • $\begingroup$ so in the 2 example, we have a single column which it contains multiple strings as potential answers $\endgroup$ – Jing Jun 29 '16 at 16:38
  • $\begingroup$ I guess I will add another example: for column 10-15, this is another type of multiple choice questions. However this time, each column represents one answer. column 10 only allows string values "very true", column 11 only allows string "somewhat true" $\endgroup$ – Jing Jun 29 '16 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.