I am new to data science/ machine learning world. I know that in Statistics we assume that a certain event/ process has some particular distribution and the samples of that random process are part of some sampling distribution. The findings from the data could then be generalized by using confidence intervals and significance levels.
How do we generalize our findings once we "learn" the patterns in the data set? What is the alternative to confidence levels here?
Bias-Variance decomposition is one way, and V-C dimension/bound is another...
Both of these are metrics you can use to get a feeling of how confident you should be that your training results will generalize to out-of-sample.
V-C dimension focuses on the results of this learning algorithm outcome. Bias-Variance focuses on the expected outcome of the algorithm itself.
Pick your poison - I sincerely hope that this was helpful.
Essentially machine learning uses non parametric methods...The assumption is that you have enough data and (computation) time You Identify best model by cross validation (rather than eg assessing significance of coefficients) , and estimate prediction error by using test set. Confidence intervals can also be generated by bootstrapping.
