2
$\begingroup$

Having a list of triplets {X1,X2,Y} such as :

{pennsylvania, fever , malaria}
{pennsylvania, headache , malaria}
{arizona, ketone smell , flu}
{new york, fever , cancer}
{ohio, hand pain , trauma}

i have thousands of samples with states , symptoms and diagnoses. i need to aggregate the states to reduce the dimension into smaller set of states using the symptoms and the diagnose. Any idea for that?

i started using Kmode algorithm for clustering the data into several clusters but i'm not sure if it makes any sense doing that

$\endgroup$
1
$\begingroup$

You can cluster discrete data using Jaccard index as a similarity metric. States that share more symptoms and diagnoses will have higher Jaccard index values. The Jaccard index values can be thresholded to form clusters.

| improve this answer | |
$\endgroup$
1
$\begingroup$

Interesting problem...

If I understand correctly you'd like to obtain clusters of states which have similar patterns/proportions of symptom+diagnosis, right?

If yes, I would suggest you reorganize the data so that one instance represents a state, with its features being the frequency of each pair (symptom, diagnosis). Based on this representation you could cluster states which have similar prevalence for a pair.

The disadvantage of this idea is that it considers pairs (symptom, diagnosis) as distinct even if only one of the two is different. Of course the same process could be done by considering only symptoms or only diagnosis. There are probably better approaches but this simple one might already provide some insight.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Most of the commonly used machine learning algorithms like Decision Trees and Logistic regression convert the categorical variables into one-hot encoding. For e.g., if you have 20 states then you create 20 variables which has only 1 entry non-zero at a time. In such cases, you can reduce the dimension of the input space by training an embedding matrix E just like word embeddings are learned, e.g., word2vec. You can use a hidden dimension of say 15, to reduce your input space.

To find the embedding E, you create a neural network which predicts the embedding of the disease given the embedding of state and symptoms. Training this model with the above objective through back-propagation will allow the embedding matrix capture structure present in the state in the given data. Once the training is over, the i^{th} row of E gives the new representation of your i^{th} state.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.