11

CLS stands for classification and its there to represent sentence-level classification. In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog where this is also covered in detail.


8

[CLS] stands for classification. It is added at the beginning because the training tasks here is sentence classification. And because they need an input that can represent the meaning of the entire sentence, they introduce a new tag. They can’t take any other word from the input sequence, because the output of that is the word representation. So they add a ...


7

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values). If you have many input classes, label smoothing can help. You can try a smaller model dimension. If you ...


5

In order to get accurate results, you should not oversample the test set! Otherwise you are simply evaluating on synthetic samples that you yourself have created. The support on your classification report should mirror the imbalance in your dataset. From what I understand you have 3500 samples, then you did some oversampling (probably brought them to around ...


5

Let me first clarify the general principle of classification with text data. Note that I'm assuming that you're using a "traditional" method (like decision trees), as opposed to Deep Learning (DL) method. As you correctly understand, each individual text document (instance) has to be represented as a vector of features, each feature representing a ...


4

Scikit-learn has compose.ColumnTransformer which allows different columns or column subsets of the input to be transformed separately and the features generated by each transformer will be concatenated to form a single feature space. This is useful for heterogeneous or columnar data, to combine several feature extraction mechanisms or transformations into a ...


4

You should go with the simpler model, the one that needs fewer features. Fewer features means quicker training cycles, better interpretibility and a faster forward pass. All of these are important considerations if you would like to productionize your model.


3

The nDCG depends on the relevance of each document as you can see on the Wikipedia definition. I guess you could use 0 and 1 as relevance scores, but then all relevant documents would have the same score of 1, and then it wouldn't make much sense to apply the nDCG penalty discounts. A similar measure often used with binary relevance scores is the mean ...


3

In order to better understand the role of [CLS] let's recall that BERT model has been trained on 2 main tasks: Masked language modeling: some random words are masked with [MASK] token, the model learns to predict those words during training. For that task we need the [MASK] token. Next sentence prediction: given 2 sentences, the model learns to predict if ...


3

To build on the previous answer: In transfer learning, the goal is to use a pre-trained model and tweak the model to then specialise it to suit a certain task. So, what we do is, as SrJ has eluded to, keep the main model's architecture in tact. So this would be the 6 CNN layers (and possibly the three linear layers, if they were also involved in pre-training)...


3

The correct way of calling the parameters inside Pipeline is using double underscore like named_step__parameter_name .So the first thing I noticed is in this line: parameters = {'vect__ngram_range': [(1, 1), (1, 2)],'tfidf__use_idf': (True, False),'clf__alpha': (1e-2, 1e-3) } You are calling vect__ngram_range but this should be tfidf__ngram_range Now this ...


2

In general this doesn't work well, since it's almost unavoidable that the classifier won't be able to distinguish all the categories from each other. I'd suggest tying to reduce the number of categories (for instance discard the least common ones). In any case I'm not aware of any specific model to deal with a high number of classes, it's regular text ...


2

For the most part, as you add more classes to your multiclass classification problem, it becomes more difficult to construct a model. All algorithms can run into trouble because there are fewer examples of each class to learn from. In case the 27 class target feature does not work well for you, you can decrease the number of classes even more. To do this: ...


2

What you need is simply a language model. This is a very common task so you should be able to find code and data easily. This question gives some pointers for Python (be careful, the accepted answer is incorrect according to the two other answers). Applying the language model to a sentence gives you a probability (or a perplexity score, which works the ...


2

Well, obviously the use cases depends on the industry. Also, I am assuming you are thinking of use cases that are somehow useful. But let's think of some examples: I once worked with a book distributor that tagged each book they sold with keywords (Fantasy, Horror, etc). You can automate the tagging process if you have a sufficiently large dataset of ...


2

First of all, you use two terms Clustering and Classification interchangably and I would like to draw your attention to this. Your problem is purely Clustering. Secondly, you asked for testing accuracy. As your problem is pure Clustering, there is no evaluation for that. The last but not least is the problem of "Short Text Understanding". In short ...


2

This is called an open-class text classification problem, it's used in particular for some author identification problems. I don't have any recent pointers but from a quick search I found this article: https://www.aclweb.org/anthology/N16-1061.pdf In the field of author classification there is a similar problem called author verification, which can be ...


2

There seems to be a mistake in your method: I read about the use of downsampling and upsampling, so I applied them before training and testing the dataset. It's incorrect to change the distribution of the test set. When resampling, the resampling should be applied only on the training set. The goal is to force the model to take into account the two classes,...


2

You could train a character-level language model, e.g. an LSTM, on the real short texts, and use the perplexity as the signal to know whether a piece of text is real or not. In order to find an appropriate perplexity threshold, you can have a look at the distribution of perplexities over a validation holdout dataset. UPDATE: There are multiple ...


2

What you are proposing is a heuristic method, because you define the rules manually in advance. From a Machine Learning (ML) point of view the "training" is the part where you observe some data and decide which rules to apply, and the "testing" is when you run a program which applies these rules to obtain a predicted label. As you ...


2

First, congratulations for thinking to do a qualitative analysis of the results :) I know it should be obvious, but so many people just assume that the system works and don't bother checking their output. Now, strictly speaking what you're seeing is not a bug. These are errors made by a statistical system. A statistical system is not meant to get everything ...


2

In order to detect overfitting you need to separate your data in a training set - that you use to estimate the parameters of you model - and a test set - where you evaluate your model keeping the parameters fixed, this is usually called cross-validation. I understand from your results that you are not doing such separation of data so, you're not able to ...


2

Very interesting question. Easy, but probably lazy answer When using pre-trained models, it is always advised to feed it data similar to what it was trained with. Basically, if it matters, don't remove them, and if it doesn't matter, it doesn't hurt to keep them in. Obviously, if you can, try with or without stopwords, and see what works best for your ...


2

That approach is reasonable. Short text inputs and multi-class classification outputs is a challenging problem. Genism's doc2vec hyperparameters probably matter less than collecting more data or reducing the number of labels. It might be useful to try more advanced models, such as Transformer or Switch Transformer.


2

I suggest you use the state of the art for this kind of problems: a BERT-based approach. This kind of approach is well documented and very accessible, given the large amount of examples available online. The approach consists of taking a pre-trained neural network model from the BERT family (Transformer encoders normally trained on a masked language model ...


2

If you want to be up to date with the new advancements, a good way is skimming through the accepted papers of the major NLP conferences, namely ACL, EMNLP, and the regional EACL, NAACL, AACL. If you want even more information, you can skim through the papers uploaded to the arxiv. One way to do that is via Twitter, by following bots that tweet papers in ...


2

Your problem as you said is a high level of syntax overlapping between your sentences. take a look at these two sentences: Work to live versus live to work. The earlier that you can allow yourself to enjoy other things in life, aside from your job while the latter means obtaining resources so that you can be a functional member of society, and to permit ...


2

For any kind of Machine Learning task or a NLP task (which is what you are doing), you need to convert string/text values to numeric values. The machine cannot uderstand or work with string values. It only understands numeric values. So for example if you are doing a machine learning task, you would use libraries like OneHotEncoder, LabelEncoder etc to ...


1

There are different ways to address the task that you describe: If the goal is simply to predict the author among a set of predefined authors, then this is not a clustering task (unsupervised) but a classification task (supervised). This implies that you would split the data between training and test set (or use cross-validation), train a model using the ...


1

Okay so keeping it very short and precisely in context of your question- Accuracy tells us, out of all the documents how many are classified correctly. Precision tells us out of all documents which are predicted in a category, how often its correct. Uni -gram- "nasa", "is" "space" , "agency" bi-gram- "nasa is", "space agency" Now lets go over the ...


Only top voted, non community-wiki answers of a minimum length are eligible