# Tag Info

An approach to overcome this 'zero frequency problem' in a Bayesian setting is to add one to the count for every attribute value-class combination when an attribute value doesn’t occur with every class value. So, for example, say your training data looked like this: $$\begin{array}{c|c|c|} & \text{Spam} = yes & \text{Spam} = no \\ \hline \text{... 9 In general, you have a choice when handling missing values hen training a naive Bayes classifier. You can either choose to either Omit records with any missing values, Omit only the missing attributes. I'll use the example linked to above to demonstrate these two approaches. Suppose we add one more training record to that example. Outlook Temperature ... 7 By doing so, the joint distribution can be found easily by just multiplying the probability of each feature whilst in the real world they may not be independent and you have to find the correct joint distribution. It is naive due to this simplification. 6 It is completely normal in some circumstances. If you consider the learning problem from a statistical perspective, learning is done by trying to estimate the conditional estimate of your output variable given input variables. When you are doing learning you basically have to main components: sample data from training and the learning algorithm, and you use ... 6 The differences in speed between Naive Bayes and SVM simply boils down to the formulation and the assumptions of each model, and has little to do with the particular library or implementation. Not only is naive bayes a simple probabilistic classifier, it also makes an additional assumption of independence between its features, so that parameter estimates ... 6 Is this normal? It is not surprising. First, you are using different measures of feature importance. It’s like measuring the importance of people (or simply sorting them) using their a) weight, b) height, c) wealth and d) IQ. With a and b you might get quite similar results, but these results are likely to be different from results obtained with c and d. ... 5 Naive Bayes should be able to handle imbalanced datasets. Recall that the Bayes formula is$$P(y \mid x) = \cfrac{P(x \mid y) \, P(y)}{P(x)} \propto P(x \mid y) \, P(y)$$So P(x \mid y) \, P(y) takes the prior P(y) into account. In your case maybe you overfit and need some smoothing? You can start with +1 smoothing and see if it gives any improvements.... 5 Although I haven't verified it, a first glance at the features and training set you used shows an obvious problem. You have just two data samples with the exact same features while you give it different labels. Although other types of models might not break and just give both samples an equal probability of being 0 or 1, something in the Naive Bayes ... 5 The difference boils down to "how we define P(x_i|C_k)?", where x_i is a single feature, and C_k is a class from a total of K classes. Discrete In discrete case, P(x_i|C_k) is represented by a table as follows: x_i P(x_i|C_k) a 0.5 b 0.2 c 0.3 We have one of these tables for each feature-class pair (i, k). Lets denote i-th ... 4 Let me try to answer your questions point by point. Perhaps you already solved your problem, but your questions are interesting and so perhaps other people can benefit from this discussion. Is Naive Bayes overfitting to the training set? If Naive Bayes is implemented correctly, I don't think it should be overfitting like this on a task that it's considered ... 4 Your training and test errors are affected by the size of the training. Take a look to this plot, usually known as a learning curve: In this example, we compute the training score and the test score (cross validation score) of a Naive Bayes model as we increase the number of examples in the training dataset. The higher the score is, the better the model is ... 4 The IDF part of TF-IDF gives less weight to a word if it occurs in a large fraction of the documents in your corpus. However, this doesn't necessarily mean that the word is unimportant for distinguishing your two classes. A word which is common in your corpus, but which also occurs substantially more often in one class than the other, could very well be ... 4 Your question is sensible. The way in which posterior probability is calculated in the classical Naive Bayes classifier (in sklearn) is like summation of the conditional probabilities of the all the features in the dataset. Even though the features are treated as conditionally independent, to learn the classification probability all the features are always ... 4 The reason NB is called "Naive" is that is makes the assumption that the predictive variables are all independent. This assumption usually skews the model scores (which, under the above naive assumption are unbiased probability estimates) towards 0 or 1. In your case, e.g., the presence of words flower and petal indicate gardening category, but, because the ... 4 Your formula is correct for one w_i, but if you want to classify a document, you need to compute P(c | w_1,\ldots,w_N). Then you have$$P(c | w_1,\ldots,w_N) = \frac{P(c)\cdot P(w_1,\ldots,w_N|c)}{P(w_1,\ldots,w_N)} = \frac{P(c) \cdot \prod_{i=1}^N P(w_i|c)}{P(w_1,\ldots,w_N)} \neq \prod_{i=1}^NP(c|w_i)$$where the second equation holds because of the ... 4 Bernoulli models the presence/absence of a feature. Multinomial models the number of counts of a feature. Here's a concise explanation. Wikipedia warns that Note that a naive Bayes classifier with a Bernoulli event model is not the same as a multinomial NB classifier with frequency counts truncated to one. To understand why, we should note that, as ... 4 Collecting your data From the comments you state that you wish to classify comments into a label (1-poor, 2-fair, 3-ok, 4-good, 5-very good). Thus you will be training a model that maps a set of words (the paragraph) to a number which represents the rating. I will assume that you also have labels for some of the comments which we will call your training set.... 4 Is it true that linear classifiers differ only in the Learning algorithm, but do they do the same during Prediction y = w1*x1 + w2*x2 + ... + c? Yes, all parametric linear classifiers try to predict the weights for the same equation y = w1*x1 + w2*x2 + ... + c and differ in how they try to optimize the cost function. If I used one method for ... 4 I do not think your formulation is correct. What you have described are just conditional distributions for each word in the sentence but not the joint conditional distribution, given a specific class. In your case, we have by Bayes rule:$$ Pr(spam | X) \propto Pr(X | spam) \times Pr(spam) = Pr(you, won, lottery, for, 1million | spam) \times Pr(spam).$$... 3 The correct term for what you're describing here is 'class imbalance' or 'class imbalance problem' . It'd be great if you could include that in the title to have a more meaningful title. Concerning your first question: Have you plotted a confusion matrix of the resulting classifications? Is the reason why the accuracy is not satisfying really that ... 3 Regarding your first question... Do you anticipate the majority category to be similarly over-represented in real-world data as it is in your training data? If so, perhaps you could perform two-step classification: Train a binary classifier (on all your training data) to predict membership (yes/no) in the majority class. Train a multi-class classifier (on ... 3 The standard answer is to work in log space, and manipulate the log of probabilities instead of probabilities, for exactly this reason. This classifier involves products of probabilities which just become sums of log probabilities. You allude to that already, but the problem you suggest isn't a problem. Internally you don't calculate a probability and then ... 3 A basic Naive Bayes is being used in this example. Each feature can have a number of different values within the ranges of 2 or 3. Bernoulli Naive Bayes requires that each feature be either true or false or 0 or 1. Multinomial Naive Bayes allows features to be of values 0+ as it is counting occurrences of features. Hope this helps. The Naive Bayes ... 3 Unbalanced class distributions First, unbalanced datasets will cause your model to have a bias towards the over-represented classes. If the distribution of the classes is not very drastic then this should not cause a significant problem with any algorithm you will employ. However, as the difference between the class distribution becomes more severe you ... 3 I can't tell you for sure without you describing your calculation more or showing code, but my guess is you're not actually calculating the posterior probability here. I bet this is just the conditional likelihood, or at best the unnormalized posterior. Remember: the posterior calculation has a division component. Does yours? You're probably forgetting to ... 3 Bernoulli Naive bayes does not assume gaussian distribution of all continuous features, because it does not make sense. Gaussian Naive Bayes assumes gaussian distribution for continuous features and it is the appropriate way for using Naive Bayes approach if you have continuous features. On the other hand, if you have binary categorical data then the ... 3 We are trying to select the optimal c, here d is fixed and hence P(d) and \frac1{P(d)} is just a positive constant. Multiplying an objective function with a positive constant doesn't change the optimal solution, hence we can drop P(d). 3 Because P(d) is constant in terms of c, so it doesn't affect the location of the maximum (only its size but we don't care about that). 3 I totally agree with Esmailian. Naive Bayes is Naive - Assumes Independence. Steps: Calculate Independently Smooth using Laplacian smoothing (to avoid zeroing the whole value) Additional Tip: Use log instead of multiplying the probabilities. (this will make sure that your values are not closing to zero, keeping some context.) Example:$$p(a) = p(x1)\...
This is just a basic property of conditional independence. If two events A and B are conditionally independent, given event C then: $$Pr(A \ and\ B \ |\ C) = Pr(A \ |\ C) * Pr(B \ | \ C)$$ or equivalently, $$Pr(A \ | \ B \ and\ C) = Pr(A \ |\ C).$$ That is, $$Pr(A \ and \ B | \ C)$$ $$=\frac{Pr(A \ and \ B \ and \ C)}{Pr(C)}$$ (definition ...