New answers tagged nlp
1
Here's what you should do
Prepare your dataset: Follow similar instructions as described in the paper and preprocess your dataset. This will be your major task as after this you will only have to fine-tune the model. If you don't have a dataset, you can use the dataset used in this research paper, which can be downloaded from here.
Download the pre-trained ...
2
First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples. In my opinion "noisy" is when positive and negative cases are mixed together in a way that makes it difficult (or impossible) to distinguish between them. I think this matters because you're more likely to find ...
4
The reason you're seeing BERT and its derivatives as benchmarks is probably because it is newer than the other models mentioned and shows state-of-the-art performance on many NLP tasks. Thus, when researchers publish new models they normally want to compare them to the current leading models out there (i.e BERT).
I don't know if there has been a study on the ...
0
I recommend this paper. The authors treat the size of embeddings as a hyperparameter and provide a detailed study on it. They show that this dimensionality should depend on the corpus.
0
BERT is trained on a combination of the losses for masked language modeling and next sentence prediction. For this, BERT receives as input the concatenation of the special token [CLS], the first sentence tokens, the special token [SEP], the second sentence tokens and a final [SEP].
[CLS] | First sentence tokens | [SEP] | Second sentence tokens | [SEP]
...
1
Let's take the common translation task which transformers can be used for as an example: If you would like to translate English to German one example of your training data could be
("the cat is black", "die Katze ist schwarz").
In this case your target is simply the German sentence "die Katze ist schwarz" (which is of course not processed as a string but ...
1
You cannot run the universal sentence encoder in reverse. There is no practical way to take an arbitrary embedding vector and get a sentence.
My suggestion would instead be to find the sentence in your data with the embedding closest to your center. Euclidean distance works well, specially if you used K-means or another euclidean method to create your ...
1
Welcome to the community,
I do not know about other libraries, but gensim has a very good API to create word2vec models. In order to preprocess data, you have to decide first what things you are gonna keep in your vocab and whatnot. for ex:- Punctuations, numbers, alphanumeric words(ex - 42nd) etc.
In my knowledge, the most generic preprocessing pipeline ...
2
Anyways is this problem I am trying to solve fit into the general NLP ML space?
Generally speaking, feeding the source data in bulk to a ML system is unlikely to give the kind of structured output you expect. It's likely that you would have to somehow guide the process in the direction of what you want to obtain, and this might take a lot of time and ...
3
Some points first:
BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which gives you a richer embedding of words in a context but in classic embeddings (yes, after BERT we can call others "classic"!) you mostly deal with neighborhood i.e. ...
0
When you run BERT, you get one vector per input token + 1 special token called [CLS] + 1 special token called [SEP]. Maybe more precise than calling BERT embeddings as embeddings, would be calling them hidden states of BERT. The contextual information get into the embeddings via 12 layers of self-attentive neural network.
However, the tokenization is tricky ...
0
TF-IDF is the most simple and starting point of training the embedding for paragraph. SIF and doc2vec provide alternative methods for the embeddings too. Skip thought use encoder to train the embeddings. There are multiple ways of getting the embeddings.
0
Yes there multiples ways to do it. Simple ones TF-IDF, SIF and quick/skip thought use encoder-decoder structure and the output of encoder is the embedding. Then the similarity between documents is simply the cos of embeddings. Doc2vec ultimately generate embeddings too.
0
Try the simplest approach first - deterministic check looking for intersection overlap between the set of fruit names and the set of items bought.
Set comparisons are scalable because the look-up time for each item is constant.
If scaling is an issue with regular set membership check, bloom filter is an option.
0
One of the default algorithms to use for this use case (set of search strings to be searched simultaneously in a text) is Aho Corasick. From the Wikipedia page: "The complexity of the algorithm is linear in the length of the strings plus the length of the searched text plus the number of output matches." Implementations of this algorithm exist in all common ...
2
Using NER (more generally sequence labeling) means classifying every token in the sentence, so if the goal is only to label every sentence there's no strong need for it in your case.
However NER might be more appropriate in case the order of the words is important, because sequence labeling models take it into account whereas traditional text classification ...
0
Yes, your intuition about the definition of online learning in Topic modeling(LDA) is correct :
"The model can also be updated with new documents for online training."
However, I would quote the standard definition of online learning in machine learning :
It is a method in machine learning in which data becomes available in sequential order and is ...
1
TF-IDF is a vectorization technique used to convert documents (a single tweet in your case is a document) to vectors. After you train the TF-IDF model, the only words/vocabulary it has learnt, would be from the set of documents (aka corpus, the entire set of 3k tweets).
Since you mentioned that there were 570 unique feature words after TF-IDF, that would be ...
4
BERT generates contextualized word embeddings, which means that BERTprovides the most accurate embeddings when a word is in a sentence(context).
For each of the words within the sentence, BERT will generate a vector of numbers. In your case, you will have a good representation of the word "bank".
So if you have a sentence for all the other words that you ...
1
I can give you some hint of doing so with deep learning approaches.
It's easy to use gensim and sklearn python libraries. First, you need to extract the word embeddings which are vector of numbers to represent a word, and then take the average of the words within a sentence is a way of fining that vector representation for your sentence.
So extract ...
0
Here are a few general thoughts about your project:
As far as I understand, you're trying to extract very specific information from a semi-structured database using free-form natural language queries, correct? If yes it's important that you realize that this is a quite ambitious project, reaching a decent stage of quality is probably going to take a lot of ...
0
The question may be too old but I think the BigBadMe answer is not true. As the keras docs said:
units: Positive integer, dimensionality of the output space.
The number of units actually is the dimension of the hidden state (or the output).
For example, in the image below, the hidden state (the red circles) has length 2. The number of units is the ...
0
Maybe too late, but you could have a look at HeidelTime. It is a Java library that can detect dates, times, durations and time sets in texts for many languages. The downside of HeidelTime is that it seems to be not maintained since 2018. Also, you need the TreeTagger tool in order to use it. But the README of HeidelTime explains the necessary steps quite ...
0
In your example, the hidden state corresponding to the first token ([CLS]) in hidden_reps can be used as a sentence embedding.
By contrast, the pooled output (mistakenly referred to as hidden states of each [cls] in your code) proved a bad proxy for a sentence embedding in my experiments.
0
The previous answer was wrong so I removed it.
Here goes my second attempt after reading Speech and Language processing by daniel Jurafsky and James H Martin(good book to read).
The 39 features associated with an observation/acoustic is considered to have come from mixtures of multivariate gaussian.
Why Mixture of MV gaussian ? Assuming a single MV ...
0
Assuming that you have read some stuff on DL already. You might want to refer to these couple of videos
Introduction:
https://www.youtube.com/watch?v=aircAruvnKk
How a neural network learns:
https://www.youtube.com/watch?v=IHZwWFHWa-w
They have a good visualization and helps in understanding how a neural network learns.
Hope that helps.
We begin by ...
1
Y = Wx + b
Here, I am assuming that the h is in subscript :
Y is the value that we want to predict
x is the input
b is the bias
W is the weight
Now, the question that you asked If I understood clearly :
You want to know, How can you get the weight and Bias of your model ? And are the W input to your machine learning model or is it the neural networks ...
1
$X$ is your design matrix (data), so can get large, both in number of data points and number of predictors. The issues are indeed:
Forming $XX^T$ may be computationally expensive
Computing it explicitly may be inaccurate due to floating-point roundoff, as you're adding lots of products
Inverting a matrix can be expensive here if the number of predictors is ...
0
Your problem is indeed a typical one-class classification problem, and as far as I know one-class SVM is usually a good option for that.
I think you should investigate what causes the poor performance:
Evaluating with accuracy is probably not informative enough, you would need to find out at least whether the errors tend to be mostly false positive or ...
2
Paraphrase detection is still a very active and very challenging research area, so it's unlikely that there are full-fledged standard libraries for this task since there is still no clear "best solution" to this problem.
In order to build a corpus you might want to look at how shared tasks/competitions have done it before. I know at least of SemEval which ...
0
Yes, I think that's a sound approach and a good way to compare different systems.
A ROC curve comparison is usually more informative than the raw performance scores, but it's still quite general. In case you want to observe even more detail, you could also try to look at specific groups of instances. One way to do that is to count for every instance how ...
2
Word2Vec algorithms (Skip Gram and CBOW) treat each word equally,
because their goal to compute word embeddings. The distinction
becomes important when one needs to work with sentences or document
embeddings; not all words equally represent the meaning of a
particular sentence. And here different weighting strategies are
applied, TF-IDF is one of those ...
1
To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the encoding. Here is the PE lookup table generation rewritten in C as a for-for loop:
int d_model = 512, max_len = 5000;
double pe[max_len][d_model];
for (int i = ...
2
The issue is due to your lamda function with the tokenizer key word argument.
>>> from sklearn.feature_extraction.text import TfidfVectorizer
>>> from joblib import dump
>>> t = TfidfVectorizer()
>>> dump(t, 'tfidf.pkl')
['tfidf.pkl']
No issues. Now let's pass a lambda function to tokenizer
>>> t = ...
1
Attention weights are learned through backpropagation, just like canonical layer weights.
The hard part about attention models is to learn how the math underlying alignment works. Different formulations of attention compute alignment scores in different ways. The main is Bahdanau attention, formulated here. The other is Luong's, provided in several variants ...
1
An increase in validation loss while training loss is decreasing is an indicator that your model overfits. Check out this article for an easy to read general explanation.
In the context of autoencoders this means your neural net almost reproduces the input image. Try to reduce overfit by applying regularization, e.g. add dropout, add input noise, use less ...
2
There's no simple answer to this question. As far as I know in general the choice depends mostly on the type of classification:
Bag of Words (usually with tf-idf weights) is a simple but quite efficient representation for classification based on the text topic or similar, assuming the classes are reasonably distinct from each other.
Word embeddings are a ...
0
I suggest to check out some introductory books on NLP, e.g. Natural Language Processing with Python. This is a very accessible and practical introduction and useful even if you won't be working in Python. Another, more detailed text book is Speech and Language Processing by Jurafsky and Martin.
You need to understand the basics first, and these are ...
0
A common approach for this is LDA (Latent Dirichlet Allocation), which not only gives you the groups, but also a way to identify the topics of the groups by giving you the most common or distinctive words for each topic.
0
I am answering to the question in your title ("removing junk sentences"), ignoring your final goal of finding promises within the corpus.
One thing I would try is to treat this as a classification problem (junk vs non-junk). You can train a model based on a labelled set (i.e. you need to label some subset of your dataset) and then classify the rest of the ...
1
I did something similar a while ago. We wanted to classify several types of pdf.
We first extracted the text of the documents.
We created NLP features with the text
Then added pdf metadata: size of the file, number of pages, name of the document...
We then built a classification model with a few samples and did Active Learning
I guess that you could also ...
0
I have checked/read/watched countless sources on NLP, but at the end only two really made the difference:
The best book ever on NLP: Speech and Language Processing by Dan Jurafsky and James H. Martin. The authors are making all its content available for free on their academic website. This contains 99.999% of the NLP notions needed in a whole ML career, and ...
1
Natural Language Processing in Action: Understanding, Analyzing, and Generating Text with Python is a new practical textbook that covers all the latest (2019) topics.
1
That would be a sequence labeling task, the most common type is Named Entity Recognition, you'll find many examples about it but you can train a custom system with your data. The traditional method is Conditional Random Fields, there are a good few libraries available.
Side note: usually a single CRF model is used to do both detecting and labeling at once (...
0
Word2Vec algorithm does not go inside words. Word “king” is never used as a gerund, so there is no reason why it should be similar to gerunds.
My guesses are:
Your corpus might by wrongly tokenized. Maybe there are some OCR-related errors with word splitting something like “li-↲ king”.
You might be using a different algorithm for getting the embeddings (e....
0
I think these parameters are mostly used when you combine the vectorizer and a machine learning model in a pipeline. Therefore, you should tune these parameters based on the outcome of your model training.
For example, if your task is to classify input texts, you may want to tune the max_features parameter such that the number of features is not too large, ...
1
That would depend on the exact goal of the task and the specifics of the dataset, but in general I would say that it's always better to use the information specifically provided with the data if it's relevant for the task. In this case the rating for the product is indeed very likely to reflect the sentiment of the text, so I would go with it. Notice that ...
Top 50 recent answers are included
