2
$\begingroup$

Selecting a column from a Dataframe, plotting its Histogram using matplotlib and then finding the variance is the steps that I have to take for this part of a project.

The final goal of the project is detecting anomalies in data.

I understand the part of plotting the data in form of a graph to understand it better. On the other hand I’m not sure why finding the variance of this column is so important.

Would really appreciate if I get any information regarding this.

I know the definition of variance but just can’t seem to understand the usage and importance of it in this project.

$\endgroup$
2
$\begingroup$

Variance in a feature (defined as the average of the squared differences from the mean) is important in machine learning because variance impacts the capacity of the model to use that feature.

For example, if a feature has no variance (e.g., is not a random variable), the feature has no ability to contribute to task performance. A zero variance feature will be constant for different levels of the target.

Additionally, some machine learning models make strong assumptions about the distribution of features. Variance is one way to check for distributional assumptions.

$\endgroup$
1
  • $\begingroup$ thank you for your answer, maybe you can also give me some advice. So I have a csv file and I want to plot the values using matplotlib and find their variance. What would be the best type of graph to show this? Should I include finding mean also, is mean also important in this case ?. Goal is to sort of analyse and see how that column of integers are distributed $\endgroup$
    – E199504
    Jan 31 '20 at 14:00
0
$\begingroup$

The presence of variance is very important in your dataset because this will allow the model to learn about the different patterns hidden in the data.

But your model must not have a high variance which may cause the model to overfit. So there needs to be a balance maintained.

Read about bias-variance tradeoff - Link

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.