Hot answers tagged

3

You are currently using the fit_transform method on both your training dataset as well as your test set. This is incorrect since you should not fit the model on your test set as (depending on the model used) this would be overfitting, and it can give issues with dataset shapes when creating new columns based on the values in the data (count vectorizer, ...


3

In fact, you want to translate "yellow that is glossy and sorta dark" by (170,173,11). A good way to solve this, is by using a neural machine translation model. Therefore, you can use a encoder/decoder system like many translation models, but with 3 digits as output. To achieve this, you will want to have training data with plenty of text to color ...


2

It's common to see some confusion about TFIDF so thank you for asking this question :) TFIDF is not a metric, it's a weighting scheme This means that it's a way to represent a document, not to compare documents. TFIDF assumes a bag of words (BoW) representation, i.e. a document or sentence is represented as a set of words (their order doesn't matter). The ...


2

Text vectorisation is a good way to have a reliable classification. You have several libraries like doc2vec that you can use together with logistic regression or dimensional reduction technique like tSNE or UMAP. https://radimrehurek.com/gensim/auto_examples/tutorials/run_doc2vec_lee.html On the other hand, you can also use libraries like BERT or TF-IDF: ...


1

I guess there is no universal solution for that. But I can explain my roadmap as NLP scientist. Firstly, I try to find the most common dataset for my task (NER in your case). Then, I search for the leaderboard which shows the best papers/models for that dataset. Finally, I try to figure out which makes their models best in the leaderboard. For example, here ...


1

Well sure it's doable: the NLP part is just speech recognition + extracting the formal command from the text, it's very similar to "virtual assistants" like Apple Siri, Amazon Alexa, Ok Google. However the hard part is to formalize all the possible commands that can be given, and then train a model to correctly map voice commands to software ...


1

Assuming your goal is to infer whether a sentence is: positive already happened, positive likely to happen, negative already happened or negative likely to happen; you end up with a 4-classes classification problem, which you need to label in advance (this would be, if feasible, the tedious-human work). After that, you can also apply word embedding layers to ...


1

So, Clustering is "Unsupervised" learning : You make groups in which elements look like each-other. In Unsupervised learning, you don't have a Label that you look for. Here, your problem is to Classify text between 3 categories : Sports, Foreign, Local. Those 3 categories ARE labels : You know you have news about those 3 subjects, and want to make ...


1

"promotional" is not an inflected form of "promotion", therefore "promotion" is not the lemma of "promotional". Actually, "promotion" is a noun and "promotional" is an adjective. Maybe what you actually want to do is not lemmatisation but stemming. Note that the stem is the root of the word and, ...


1

One way to understand how ELMo's character convolutions work is by directly inspecting the source code. There, in the forward method, you can see that the input to the network is a tensor of dimensions (batch_size, sequence_length, 50), where 50 is the maximum number of characters per word. Therefore, before passing the text to the network, it is segmented ...


1

The BIO format (and its variants) is a standard format for training a sequence labeling model, in particular a Named Entity Recognition (NER) model. Sequence labeling consists in assigning a label to every token in the sequence, so at the "low level" stages of training and predicting the system must deal with the token and its label, as well as (...


1

you can start by using torchscript, it may require changing ur whole code, and switching to transformers( by loading the backbone of the model and the last layers) so basically u get out from GIL interpreter, coz it does not support multithreading. by with torchscript u can run ur model in c++ env, there's also onnx which I believe it enhances performance. ...


1

Since you mention deep learning, one option is to embedded the documents and then cluster the documents. Each cluster could be labeled as "Good" or "Not Good". The labeling could be done by hand or automatically by voting with existing labels (e.g., if a majority of the documents are "Good" then the entire cluster is "Good&...


1

In general, you need labeled data to perform Sentiment Analysis. In case you don't have, you need to improvise. I found one article where the author claims that his implementation of unsupervised learning worked adequately. The post: Unsupervised Sentiment Analysis I quote some parts of it: The main idea behind this approach is that negative and positive ...


1

First I think it's worth mentioning that in the context of an exploratory study with a small dataset, manual analysis is certainly as useful as applying NLP methods (if not more) since: Small size is an advantage for manual study and a disadvantage for automatic methods. There's no particular goal other than uncovering general patterns or insights, so it's ...


1

You can first detect the "out of vocabulary" words, and check if they are part of a location dataset. There are locations datasets that you can use and adapt them to be not case sensitive. Here are the ones for the cities: https://simplemaps.com/data/world-cities About streets, you can use the world roads dataset and apply the same logic: https://...


1

TL;DR: A theoretical/mathematical explanation for why word2vec/GloVe embeddings of analogies appear to form parallelograms, and so can be "solved" by adding/subtracting embeddings, is given here, as summarised in this blog. More explanation of w2v is given here. The dimensions of word2vec (or GloVe, etc) word embeddings are not directly ...


1

Precision and recall are "hard" metrics. They are measure if the model's prediction is exactly the same as the target label. Often times systems like yours can use a more flexible metric such as top-5 error rate, the model is considered to have generated the correct response if the target label is one of the model’s top 5 predictions.


1

Extensible Markup Language (XML) is a way to encoding documents in a format that is both human-readable and machine-readable. It is also relatively simple and commonly used. It allows data and metadata to be linked. Those are the reasons that is often used for treebank data.


Only top voted, non community-wiki answers of a minimum length are eligible