Association rule learning for multi-classification suggestions

Question

The task is the following: given a training set of medical symptoms and an associated diagnosis, output a list of the most likely diagnosises for a combination of symptoms. As of now, a solution exists which makes use of association rule learning methods: we find rules on the attributes of the training dataset and infer the likely diagnosises and their probability from the confidence we have in said rules to hold true.

However, this method doesn't seem to be able to scale for large datasets because of the sheer number of different attributes ($10^4$) and possible classes ($10^3$). Hence my question: are association rules a viable solution for this problem? Are there any alternatives?

Answer 1

Assuming there are many repetitions of each attribute and class, an embedding space can be learned. Attributes and classes that co-occur together will be projected into nearby space. One example is affinity weighted embedding.

Then prediction becomes approximate nearest neighbor search. For a given set of attributes, find the nearest class(es). One example is locality-sensitive hashing.

The strategy of combining learned embedding spaces and approximate nearest neighbor search scales very well for prediction. Search time can remain constant if you are willing to accept a looser definition of "approximate" as the data grows.

Answer 2

I would try the following two approaches and both are equally interesting.

The first is: k-means clustering. Here is why: On the basis of a set of symptoms, mapping to one diagnosis is something we can try. Also, we can change the number of clusters and check if it improves the accuracy/ results .

The second is: recommender system. This works very closely with association learning.I would give an example: we watch movies on Netflix and it suggests movies which we might like. This is on the basis of which movies we watched earlier and which other movies were watched by other people who have a similar taste as ours. We can use the same logic here. $ people with one symptom sss had xyz diagnosis. If a symptom is similar to sss, they get xyz diagnosis.

I hope this helps!

Answer 3

You can try Multi linear regression analysis. This can be done using excel(data-->data analysis-->regression). Basically this model develops the relationship between multiple variable impacting one dependent variable. In the example above:

diagnosis = a + b1 medical symptom1 + b2 medical symptom 2+ .....+ e where, a = intercept of the equation b1, b2,...., bn = slope of each variable e = margin of error.

Read more on this article: https://www.investopedia.com/terms/m/mlr.asp