0
$\begingroup$

The company I work for has deployed a trained rule-based sentiment analyzer model vader to make predictions on customer's attitude. We import the model from nltk library directly, so we didn't train it. Then we have some prediction data. Now I need to evaluate how well the model performs on our data. We plan to annotate some data as ground truth labels to calculate metrics for evaluation. Does this method work? If so, what data size we need to annotate? We have around 3000 imbalanced predictions, and we want to annotate as little data as possible as we don't have many annotators.

$\endgroup$

1 Answer 1

2
$\begingroup$

If you want to avoid annotate the data yourself, you can try to evaluate the model on one or more benchmark datasets that are available for sentiment analysis. This may work unless the model is very specific.

Alternatively, as you said, you have to collect the data by yourself. Indeed, the more data you use to evaluate the model the more statistically plausible the obtained metrics values would be: maybe you can stop annotating when the variance of the metrics don't change much.. Also to reduce the annotation effort, you could select a small subset of the predictions that are particularly relevant. I mean that allow you to cover a vast space of scenarios.

$\endgroup$
4
  • $\begingroup$ Thanks a lot! You've really opened my eyes to a different way of looking at the data size challenge. I was kind of stuck trying to nail down a specific data size. Just want to be clear, when i select a subset of the data for annotation, do the data has to be balanced or in proportion to the prediction data (very imblanced)? If we want to see how the model performs on the current data we have, should we select a subset in proportion to the data distribution we have? If we just want to evaluate the model's performance in general, is it better to select balanced data covering a vast features? $\endgroup$
    – Shelby
    Commented Sep 23, 2023 at 11:59
  • $\begingroup$ @Shelby I'm glad the answer is helpful. In general, you want to select a test subset such that its distribution is the same (or very similar) to the actual data distribution (i.e., the data the model will predict.) So, if the actual data is imbalanced also the test-set should be imbalanced in the same way. $\endgroup$ Commented Sep 23, 2023 at 16:05
  • $\begingroup$ Thank you @ Luca Anzalone. Just one more question. We do not have the actual data distribution, so how to select test data based on actual data distribution? We only have predictions which I think do not necessarily represent the actual (ground truth) data distribution, right? Another question is when do we need to prepare the balanced data? For training if we don't know the actual data distribution? $\endgroup$
    – Shelby
    Commented Sep 24, 2023 at 21:43
  • $\begingroup$ @Shelby If you don't know the data distribution, you can make an initial assumption according your domain knowledge of the problem (say you expect the classes to be imbalanced). Then you can iteratively or periodically update your belief according to both the predictions and performance (e.g. the errors) of the model to get a more realistic estimate. Then balanced distribution during training is usually useful to not introduce biases but also if you don't make any particular assumption on the data. $\endgroup$ Commented Sep 25, 2023 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.