a data point - too little to spend resources during data preparation?

Question

I am learning analytics online and have some quick questions.

Usually when we do analysis, why is that we usually ignore the items/data points that are less frequent?

Let's say for ex: we have is drug frequency data and no of patients who consumed that drug in a hospital. As an example, the data looks like as shown below but in real-time, I might even have millions of records

From the above screenshot, we can know that whatever analysis and insights we come up with the above data (including few more columns of data which aren't shown here), we will definitely not consider Drug D.

Meaning we can't base our conclusion/insights that we derive from our data based on Drug D because only 2 out of 5000 patients have it which is less than 0.05% of our data.

Through it seems to intuitively make sense because 0.05% is very less to have any impact on output.

Now my question is what about Drug G. It occurs 1.14% times in our data?

How do I know it's okay to believe that Drug D - 0.05% is very less to have any impact on output and can be ignored whereas Drug G - 1.14% has to be retained?

I am not sure whether my English skills helped you understand what I am trying to convey.

EDIT - UPDATED (Apologies if my question wasn't clear earlier)

What I am trying to do is (not an ML task but a Data Preparation task), manually map the drug names to the terms available in the dictionary (Data Preparation task). As you can see in the screenshot, Drug A is mapped to ABCDE A. Similarly, I have to manually map for all 50K drugs. However, my question is given below

a) I can't spend resources (money/people) to manually (as it cannot be automated) go through all the 50K drugs and map it to dict terms because no one is interested to do this job. Whoever is interested, is not willing and it would be impossible to do all the 50K drugs and it would incur so much money to pay them. So, I have to make sure that manual reviewers focus on important (high frequent) terms first and it's even okay to ignore DRUG D or DRUG G because they contribute very little value to the data (considering the full dataset of million records)? Question is mainly on decision making based on systematic approach/mathematical approach rather than my judgement/visual inspection/subjective..

b) Hence now, I am trying to know whether there is any objective/systematic/mathematical approach that can tell me, we can ignore all drugs below a certain N% etc... Because I can't just say that through visual inspection I feel Drug G and Drug D can be ignored. If you are gonna suggest me Statistical significance test, can you please guide me on how can I set this as a problem? Because I usually see, it is used in hypothesis testing. Can I kindly request you to guide me on this?

Answer 1

Practically everything related to statistics (including Machine Learning) has to do with studying chance, i.e. trying to determine to what extent an observation is due to chance or not.

For example one might want to know whether a drug actually helps with a particular disease or not. If we observe that one patient improves after taking the drug, there's not enough evidence to conclude since many other factors could have caused the improvement. This is why one needs a very strict protocol in order to obtain a statistically meaningful observation (two groups of patients, placebo etc.). A reasonably high number of observations is needed, otherwise it's impossible to distinguish the effect of "chance" (any other factor) and the real effect of the drug.

a) Is there any systematic/mathematical/theoretical approach that can tell me anything less than N% is too little to impact/influence the output?

The standard method for knowing whether an observation is due to chance or not is to use the appropriate statistical significance test. There are many of them and they depend on what exactly is being tested.

b) How do you decide which items are too little to impact output. Do you go with your judgement which is a subjective approach?

In ML it's common to take a more experimental approach, for example trying with/without an observation or feature and then evaluate which versions works better. Of course it helps to have an intuition of what is more likely to work. In general including extremely rare observations is a bad idea because it's likely to cause overfit, i.e. when the model "learns" something which is actually due to chance.

---

[added following OP's update]

In this case this is a resource allocation problem, I don't think statistical significance is relevant here. Assuming that you want to optimize the use of manual labor based on how often a drug is used, i.e. the only thing to maximize is the sum of the frequencies of the drugs being labelled, then it's simple: rank all the drugs by their frequency in descending order, then proceed with manual annotation following this order. This way you're sure that the drugs which account for more patients are done first, so whenever manual annotation stops the largest possible amount has been labelled.

Answer 2

In addition to Erwan's answer, which gives great general advice, consider the following questions when you are deciding rather to keep data.

What question(s) are you trying to answer? What are you trying to learn from the data?

If you are trying to build a model that will predict patient recovery based on drug administered and a variety of other biomedical data on the patients, it might be best to exclude Drug G if it is truly a very rare treatment. Including them could lead to overfitting, especially if one has a large effect. Alternatively, your model may assign a very low importance to Drug G as a feature because of its low prevalence.

As Erwan notes, the best approach is experimental. See how your model performs with and without the data. However, leaving out Drug G has its own dangers. If drug G is a newer treatment, then the next round of data you throw at your model will likely have more drug G in it, and you model will perform poorly on those data. You can always revise your model in this case.

Consider a modified scenario. You work for the company that manufactures Drug G. Drug G is relatively new and has been approved to treat condition X. A number of patients with condition X also have condition Y, and taking Drug G also appears to help patients with condition Y improve. Your employer wants to know whether research on treating condition Y with Drug G is worth the investment to try to compete with other drugs in the market.

In this second scenario, you cannot omit the data for Drug G. However, because Drug G is underrepresented in the overall dataset, you will still run into the dangers of an overfitted model (a challenge in regression problems), a model that underemphasizes Drug G because it is underrepresented (a challenge in classification problems), or a statistically insignificant result (a challenge in general).