New answers tagged nlp
0
Neural networks have a higher learning capacity than statistical modeling. In other words, the more data you give a neural network the better it will perform. Statistical models have a lower asymptotic performance, even as the amount of data increases.
In that paper, they use backtranslation to synthetically generate more data. The increase in data allows a ...
1
Can you use two different datasets as train and test sets with countVectorizer and test_train_split?
You're doing a big mistake in your code, which is applying the vectoriser before the train/test splitting. The vectoriser should be fit only on the training dataset, then the learned counts should be applied to the test set. Instead you applied the vectoriser to the whole data which you then splitter into train and test.
# THIS IS OK
# train_data = np....
0
You can't fit X_train into y_train without encoding.
Try something like this for labels:
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown='ignore')
enc.fit(X)
and label encoding for labels.
1
count_vectorizer.vocabulary_.items() and count_vectorizer.vocabulary_ Inconsistent number of returns
For future readers: as Ben wrote in the comments, it is really hard in general that len(my_dict) != len(my_dict.items()). When these kind of strage behaviours happen, it is always a good practise to perform some routine checks:
Clean your environment from every variable, even better restart the kernel and then run again your code.
Check your code for ...
0
"probabilities of finding a neighboring word given a word"
here you refer to the Skip-Gram architecture, where given the center word you predict the surrounding words.
This extract from these notes might clarify your question. Note that by assuming the conditional independence the total probability factors into a product.
"As in CBOW, we need to ...
0
The past token internal states are reused both in GPT-2 and any other Transformer decoder.
For example, in fairseq's implementation of the transformer, these previous states are received in TransformerDecoder.forward in parameter incremental_state(see the source code).
Remember that there is a mask in the self-attention blocks in the decoder that prevents ...
-1
There isn't a correct way to approach this problem. One common way is what you are doing, i.e. check for various values of $k$ and have a heuristic tell me the best value. Some such methods are are the elbow, silhouette and the gap statistic that you're using. Determining the number of clusters via such a method is perfectly valid; in fact that's what they'...
0
GloVe Will "Most Likely" Work For Your Purposes
I found myself with a question similar to yours about 1 month ago. I met with some fellow data scientists that had more experience with NLP word vectorization than me. After reviewing many options, I felt that Global Vectors (GloVe) would work best for me. It is doing well for my purposes, and, for my purposes,...
1
Question 1:
To do so, I would use the Gensim wrapper of FastText because Gensim has a predict_output_word which does exactly what you want. Given a list of context words, it provides the most fitting words.
Question 2:
It is up to the user. FastText isn't inherently CBOW or Skipgram. See: https://fasttext.cc/docs/en/python-module.html
Question 3:
Yes, ...
0
You can import the pre-trained bert model by using the below mentioned lined of code
pip install pytorch_pretrained_bert
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForNextSentencePrediction
BERT_CLASS = BertForNextSentencePrediction
PRE_TRAINED_MODEL_NAME_OR_PATH = '/path/to/the/files/containing/models/files' # make sure all the ...
0
For synonyms I would directly use WordNet.
[added] For contextually similar words the traditional approach is to extract a context vector for every target word:
for every occurrence of a target word extract the words within a -/+ N window (e.g. N=5).
for every target word aggregate all its context words in a single context vector over the whole vocabulary.
...
1
I think the task you're looking for is called "Punctuation restoration" or "Punctuation prediction". An example of this is described in this paper, or this other paper.
Unfortunately I am not aware of available per-trained model that you might apply to your dataset. Honestly I think that it is unlikely that something trained specifically for costumer service ...
0
One option, which I have discovered, is back-translation via the Unsupervised Data Augmentation repository made public by Google Research. This is based on this paper.
1
There are several ways you can obtain document embeddings. If you want to obtain a vector of a document that is not part of the trained doc2vec model, gensim provides a method called infer_vector which allows to you map embeddings.
You can also use bert-as-service to generate sentence level embeddings. I would recommend using Google's Universal Sentence ...
0
RNN/LSTM is designed for series (data has time step) like data(E. g. a sentence ) which has dependency between different parts of the data. In English, some words in a sentence have a dependency on previous words.
To carry the dependency information and ignore the non-important information until the end of the sentence RNN/LSTM was introduced.
If you use ...
0
I gave a lot of thought about the question. I agree with you. But the slight difference might come if there are any random variable operation happens during the training. What model are you using for training?
2
This problem is multiple choice answering question. I can see you have already tried gensim, doc2vec etc. You can try pytorch based transformer solution. Here is the link: multiple-choice . You can create your data in swag format and remove --do_train in below code for prediction on your dataset.
It has been trained on swag dataset and has given decent ...
1
Firstly, fixing the input size of a model is more of an architectural decision than a problem. By fixing input size we:
Limit the GPU memory usage while training
Reduce the training time per epoch
Reduce evaluation time per input sample
If you want to, you can train your own transformer model by increasing the size of the input or increasing the number of ...
1
If I got your problem description correct, you are looking for a recommender system like for example used by Netflix or Amazon. State of the art solution would be to use Latent Dirichlet Allocation topic modeling to make recommendations based on topics (in your case, topics would be the tags). Here is a very good video tutorial on this topic: https://youtu....
4
The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it.
The amount of memory needed by the self-attention in the Transformer is quadratic on the length of the input. This means that increasing the maximum length of the input, increases drastically the needed memory for self-...
1
One solution could be to train a single embedding space, StarSpace is one such implementation. That single embedding space would contain all users, documents, and tags. Then it is a nearest neighbor search to recommend any combination. Given a user, find the nearest documents. Given a tag, find the nearest tags …
For new entities (i.e., users, documents, or ...
1
This is an NER problem. Rather than you splitting your sentence to words and finding the right word from dict, I suggest you use an NER (may be spacy NER mentioned by @jindrich).
This NER will point out right block of information from the text your sentences.
Once you get an Entity then you can parse its value. If it is quantitative then it is easy to ...
0
You should at least try pre-trained embedding vectors. TfidVectorizer is particularly sensitive to out-of-vocabulary words, which are likely to appear if you're trying transfer learning to a new domain. The GloVe embeddings [1] have a dictionary of 400k vectors, unless you're working with documents from a technical domain they should provide some improvement....
1
There's a lot of ways to do this, one approach is to use token-based matching. You can use this to easily find any "tokens" in the text, like names, places, or just plain words.
Methodology
I'd recommend using Rule-based Entity Recognition in spaCy. You'll define the "rules" of what the entity looks like, here's the example from the docs where we define ...
0
For the sentences you provided the nltk sentence tokeniser would work just fine.
from nltk.tokenize import sent_tokenize
sentences = ['Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too','Can company1 buy company2? What will be their total turnover then?','Coronavirus was ...
-1
Embedding layer, is used in auto-encoders to construct word2vec.
Embedding layer, are a type of layer, used in Deep Learning. You can find others here. Auto-encoders, are a type of architecture, where embedding layers are used. Using these architectures, one can calculate word2vec. word2vec values are calculated when words are fed into the auto-encoders.
2
You'll probably find pointers in the literature searching for "offensive text detection". There are many variants/overlaps with related tasks such as detecting bullying. There are probably also annotated datasets around, in case you want to use these as training data.
2
One way to approach it is to split the sentence into tokens and count the number of tokens that are profanities.
import re
def tokenize(text):
return re.findall(r'\w+', text.lower())
profane_tokens = {"nerfherder"}
sentence = "Why you stuck-up, half-witted, scruffy-looking nerfherder!"
tokens = tokenize(sentence)
# Rate: number of occurrences ...
0
I think you can find this answer a great solution to your problem.
I've used it successfully in my case.
0
First, some background context:
Non-contextual word embeddings (e.g. word2vec) only reflect co-occurrence statistics. The similarity between two embedded vectors may only be loosely related to their semantics (e.g. the representations for country names like "france" and "italy" may be close) or there may even be negative correlation (antonyms may be very ...
-1
Gensim has a built in functionality to find similar words, using Word2vec. You can train a Word2Vec model using gensim:
model = Word2Vec(sentences, size=100, window=5, min_count=5, workers=4)
You can make use of the most_similar function to find the top n similar words. It allows you to input a list of positive and negative words to tackle the 'good' and ...
1
You have to give your training set to the model to be trained
_= pipe.fit(triningSet.data, triningSet.target)
I don't see any training dataset here. you have to fit the CountVectoriser to your data set.
1
An automatic system for finding synonyms using word embeddings is not possible. Word embeddings find co-occurrence. For example, "good" and "bad" co-occur together in a corpus, thus are near each other in the embedding space. However, "good" and "bad" are antonyms.
A copilot system could work. Word embeddings can find a set of candidate synonyms by finding ...
0
A direct way would be to encode any binary input features as embedded vectors and add them together as the initial hidden state for the LSTM, and then you train it as a normal language model.
The "little manual introduction" could be supplied to the language model (together with the initial hidden state created from the binary features) at inference time ...
1
I'm not sure that building a model using CRF is the best approach here. It's going to require quite a bit of training data and effort to get it working like you want. Dates are pretty structured in most cases so there are more straightforward approaches to extracting them. Stanford's SUTime library, for instance, does exactly what you're describing in your ...
1
One idea that could be leveraged - use all the model answers and learning material, find some sort of semantic similarity between all that text and the student answer. The higher the similarity with a model answer with a specific grade tag, the higher the probability of assigning that answer that particular grade. You can couple this with a metric of ...
1
Limit outputs od decoder to N. Not sure how easy it would be, probably a bit digging into official implementation but after that the main "skeleton" of the GPT2 is usable, meaning that all of the pre-training can be reused to produce meaningful sentences.
1
The problem is not really "new text", since by definition any classification model for text is meant to be applied to some new text. The problem is out of vocabulary words (OOV): the model will not be able to represent words that it didn't see in the training data.
The most simple way (and probably the most standard way) to deal with OOV in the test data is ...
0
You can use Spacy's named entity recognition. Since I feels it is good in accuracy as well as multi-processing approach is also possible.
https://spacy.io/usage/linguistic-features#named-entities
import spacy
from statistics import mode
nlp = spacy.load("en_core_web_md")
Q1 = "where is Texas ?"
Q2 = "where is California?"
Q3 = "where is NASA?"
Q4 = "who ...
0
There is very cool tool called bert-as-service (https://github.com/hanxiao/bert-as-service) which does the job for you. It maps a sentence to a fixed length word embeddings based on the pre trained model you use. It also allows a lot of parameter tweaking which is covered extensively in the documentation.
Top 50 recent answers are included
