New answers tagged nlp
0
votes
How to evaluate machine translations of long documents?
Partial answer: I think I can implement calculate_ter like this:
...
- 101
0
votes
LLAMA MODEL WITHOUT USING HUGGINGFACE API
The license for Llama here https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/LICENSE is from Meta, and doesn't require HuggingFace API.
- 535
0
votes
Accepted
Text segmentation problem
The problem you are describing is not a classic NLP problem.
There is a similar classic NLP problem called "topic modelling", which consists of discovering topics in a collection of text ...
- 20.5k
2
votes
Open source NLP annotation tool/library supports active learning
For my projects I use NLP Lab, by John Snow Labs. It provides automated annotation and model training, saves time compared to other tools, and is completely free of charge. Another impressive thing ...
0
votes
Get the keywords from positive and negative reviews
Have a look at keybert https://github.com/MaartenGr/KeyBERT which extract keywords
- 146
0
votes
About the last decoder layer in transformer architecture
I understand that we are talking about inference time (i.e. decoding), not training.
At each decoding step, all the predicted tokens are passed as input to the decoder, not only the last one. There is ...
- 20.5k
3
votes
Some simple questions about confusion matrix and metrics in general
Most of your questions cannot be objectively answered.
Whether or not a model is good depends on what is the use for it.
Seeing how your classes are imbalanced, it definitely affects the metrics you ...
- 548
0
votes
How to fetch text from pdf to further proceed with question answer based model from the same document?
You can do this by using OCR engines like pytesseract. Once the text has been extracted you can either use custom NLP rules for framing the questions and their ...
- 1,576
1
vote
Extract phrases/keywords that are SIMILAR to a python list of keyword/phrases, from a document
You could use sentence transformer library to calculate the similarity between different phrases. It also works for multi worded tokens.
...
- 36
4
votes
Accepted
Some simple questions about confusion matrix and metrics in general
The first model where the f1_score is around 61% can not be considered as a good model. You can achieve much better results than that. This can be seen in the ...
- 1,576
0
votes
BIO Format (Skills,Qualification,Experience)
You need to manually annotate a large sample of your input text like this:
...
- 24.6k
1
vote
How do people usually handle creating an embedding vector of longer texts (32000 characters?
When dealing with longer texts, you can use a technique called "sliding window" to break the text into smaller segments. This involves taking a window of fixed size and sliding it along the ...
- 81
1
vote
Accepted
Why my sentiment analysis model is overfitting?
There might be multiple reasons that might be the reason for overfitting some of which are:
1.) Scaling the data
2.) You have not mentioned which parameter values you have selected in the Tfidf ...
- 1,576
0
votes
Using BERT to extract a list of words and phrases from documents
You have a list of keyphrases that you need to extract from documents. You could use NER for this purpose. But looking at the size of the keyphrases (3000) it will be a difficult task because you ...
- 1,576
0
votes
What are the approaches for extracting an injury and its description from a paragraph?
Given the unstructured nature of your injury descriptions, I don't think this is doable by means of classical NLP techniques. I suggest you use a large language model (LLM), either OpenAi's GPT family ...
- 20.5k
1
vote
Load an LLM in multiple GPUs
Using multiple GPUs usually means that the whole model is copied into the memory of each of them. In Pytorch this is achieved with nn.DataParallel or nn.parallel.DistributedDataParallel. This however, ...
- 20.5k
1
vote
LLM powered chat bot enhanced by NER
It's probably best to look into some research but here is some information which might help in the meantime based on some general thoughts;
Adding NER to a LLM can enhance the models ability to ...
- 81
4
votes
Sentence tokenization for sentence without punctuation
You can apply a previous step to add punctuation and proper casing to the text, and then segment the sentences. For this, you may use Re-punctuate. When applied to your text, Re-punctuate gives the ...
- 20.5k
2
votes
Transformers doubt
We should not mistake the K, Q and V vectors received by the multi-head attention block with those received by the scaled dot-product block.
The K, Q and V vectors that are fed to the multi-head ...
- 20.5k
1
vote
Deduplication using NLP
You could try the following:
Similarity Metrics: There are several similarity metrics that can be used to compare the similarity between two texts. For example, the Jaccard similarity coefficient, ...
1
vote
Accepted
2 basic doubts on time series
Regarding Doubt 1:
Yes, if you shuffle those for training the order within each row will be preserved. If you shuffle the rows, that's good, and the LSTM models will consider each row as a separate ...
- 342
0
votes
Why is the decoder not a part of BERT architecture?
In short, Bidirectional Encoder Representations from Transformers (BERT) is not designed for decorder-related tasks.
I can't see how BERT makes predictions without using a decoder unit, which was a ...
- 1
3
votes
Accepted
Easy question on autoregressive LLM
At t=1: K, Q, V are the projections of the sequence "today is a good".
However, given that the computations of the first tokens were already done in the previous step, usually, there is some ...
- 20.5k
Top 50 recent answers are included