How do you apply hypothesis testing to your features?

Question

How do you apply hypothesis testing to your features in a ML model? Let say for example that I am doing a regression task and I want to cut some features (once I have trained my model) to increase performance. How do I apply hypothesis testing to decide whether that feature is useful or not? I am just a bit confused about what my null hypothesis would be, level of significance and how to run the experimentation to get the p-value of the feature (I have heard that a level of significance of 0.15 is a good threshold, but I am not sure).

For example. I am doing a regression task to predict the cost of my factory, considering the production of three machines (A,B,C). I make a linear regression with the data and I find out that the p-values of machine A is greater than my level of significance, hence, it is not statistically significant and I decide to discard that feature for my model.

The data looks something like this

d = ('Cost': [44439, 43936, 44464, 41533, 46343], 'A': [515, 929, 800, 979, 1165], 'B': [541, 710, 675, 1147, 939], 'C': [928, 711, 824, 758, 635, 901]) df = pd.DataFrame(data=d)

If you want to see the full data, I have taken this example from a video on Youtube. I couldn't really understand how he drew that conclusion about the level of significance and how he used hypothesis testing in this case.

The link is below, it starts from Min 4:00 (it is only 3 mins long after that)

Linear Regression using hypothesis testing

Answer 1

welcome to StackOverflow.

I will try to summarize as much as possible, but we have to cover a lot of concepts to properly answer your question, if something is not clear, please let a comment and I can change/correct the answer.

First of all, I will assume from the tone of your question that you know what a hypothesis test is.

Second, in the context of Linear Regression, you usually assume that your features are normally distributed, iid (identical independent distributed), constant error variance (that is, they do not vary with X attributes) and so on. These are strong assumptions, and it is important to have it in mind since we are calculating our regression coefficients using data that have these properties.

Put in simple terms, you have your set of features X, in this case A, B, and C, and you want to predict cost, therefore you have the following function:

Thus, you use above equation to minimize the RSS (Residual Sum of Squares) and find out what are your coefficients.

Now, in the context of hypothesis test, you want to verify if your coefficients are statistically relevant, that is, saying it on simple words, you want to check if they are far enough from zero in order you can say they are relevant, not a statistical incident. How exactly you do that? Creating a Null Hypothesis that you coefficient is zero, and an Alternative Hypothesis saying otherwise.

(Example to calculating Hypothesis Test for Beta_1 coefficient)

Then you calculate t-test for your coefficient:

So you are assuming your coefficient has a t-distribution and you want to test if it is far enough from zero to determine its relevancy.

You then calculate your t-test value and estimate a p-value. Therefore, in the end, you will ask the following question: What is the probability that my coefficient comes from a distribution centered around zero? If you have a high p-value, it shows you that probably your coefficient is zero, or close to it and not significant. Otherwise, if you have a low p-value, then it is not likely that it comes from a distribution around zero, then you should take this coefficient (and feature) in your analysis.

In the example from the video, he uses a rule that all coefficients above 0.05 are not statistically relevant. Here, 0.05 is widely used as a cutoff point, but you must be aware that there are a lot of issues regarding this choice and the debate inside scientific community is not settled.

If you really want to understanding all nuances involving Linear Regression, Statistical Tests and problems when modelling data using this technique, I highly recommend that you read the Introduction to Statistical Learning Book - Chapter 3. All images and insights were taken from there, and for me it is my reference book for Machine Learning techniques.

I hope this helps to clarify your question.

Answer 2

Removing variables is different from testing hypotheses. The test of an hypothesis is something you don't do with Machine Learning, but with statistical analysis. In a statistical model, each parameter can be thought as the test of an hypothesis, associated with a degree od significance. Moreover, hypothesis testing tells you something about the world based on how variables are associated with each other. You don't care about performance in this case, just about checking that an association has the sign you expected.

Whether it is the case to remove a variable or not, in order to improve your model's performance, it's a different problem. It strongly depends from the data you are working with, could you please provide a description of you task and your dataset?