I am learning analytics online and have some quick questions.
Usually when we do analysis, why is that we usually ignore the items/data points that are less frequent?
Let's say for ex: we have is drug frequency data and no of patients who consumed that drug in a hospital. As an example, the data looks like as shown below but in real-time, I might even have millions of records
From the above screenshot, we can know that whatever analysis and insights we come up with the above data (including few more columns of data which aren't shown here), we will definitely not consider
Meaning we can't base our conclusion/insights that we derive from our data based on
Drug D because only 2 out of 5000 patients have it which is less than
0.05% of our data.
Through it seems to intuitively make sense because
0.05% is very less to have any impact on output.
Now my question is what about
Drug G. It occurs
1.14% times in our data?
How do I know it's okay to believe that
Drug D - 0.05% is very less to have any impact on output and can be ignored whereas
Drug G - 1.14% has to be retained?
I am not sure whether my English skills helped you understand what I am trying to convey.
EDIT - UPDATED (Apologies if my question wasn't clear earlier)
What I am trying to do is (not an ML task but a Data Preparation task), manually map the drug names to the terms available in the dictionary (
Data Preparation task). As you can see in the screenshot,
Drug A is mapped to
ABCDE A. Similarly, I have to manually map for all 50K drugs. However, my question is given below
a) I can't spend resources (money/people) to manually (as it cannot be automated) go through all the 50K drugs and map it to dict terms because no one is interested to do this job. Whoever is interested, is not willing and it would be impossible to do all the 50K drugs and it would incur so much money to pay them. So, I have to make sure that manual reviewers focus on important (high frequent) terms first and it's even okay to ignore
DRUG D or
DRUG G because they contribute very little value to the data (considering the full dataset of million records)? Question is mainly on decision making based on systematic approach/mathematical approach rather than my judgement/visual inspection/subjective..
b) Hence now, I am trying to know whether there is any objective/systematic/mathematical approach that can tell me, we can ignore all drugs below a certain
N% etc... Because I can't just say that through visual inspection I feel
Drug G and
Drug D can be ignored. If you are gonna suggest me
Statistical significance test, can you please guide me on how can I set this as a problem? Because I usually see, it is used in hypothesis testing. Can I kindly request you to guide me on this?