As of May 31, 2023, we have updated our Code of Conduct.

New answers tagged

0 votes

How to evaluate machine translations of long documents?

Partial answer: I think I can implement calculate_ter like this: ...
Paul Prescod's user avatar
0 votes

LLAMA MODEL WITHOUT USING HUGGINGFACE API

The license for Llama here https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/LICENSE is from Meta, and doesn't require HuggingFace API.
brewmaster321's user avatar
0 votes
Accepted

Text segmentation problem

The problem you are describing is not a classic NLP problem. There is a similar classic NLP problem called "topic modelling", which consists of discovering topics in a collection of text ...
noe's user avatar
  • 20.5k
2 votes

Open source NLP annotation tool/library supports active learning

For my projects I use NLP Lab, by John Snow Labs. It provides automated annotation and model training, saves time compared to other tools, and is completely free of charge. Another impressive thing ...
Divyanshu Katiyar's user avatar
0 votes

Get the keywords from positive and negative reviews

Have a look at keybert https://github.com/MaartenGr/KeyBERT which extract keywords
Ggjj11's user avatar
  • 146
0 votes

About the last decoder layer in transformer architecture

I understand that we are talking about inference time (i.e. decoding), not training. At each decoding step, all the predicted tokens are passed as input to the decoder, not only the last one. There is ...
noe's user avatar
  • 20.5k
3 votes

Some simple questions about confusion matrix and metrics in general

Most of your questions cannot be objectively answered. Whether or not a model is good depends on what is the use for it. Seeing how your classes are imbalanced, it definitely affects the metrics you ...
liakoyras's user avatar
  • 548
0 votes

How to fetch text from pdf to further proceed with question answer based model from the same document?

You can do this by using OCR engines like pytesseract. Once the text has been extracted you can either use custom NLP rules for framing the questions and their ...
spectre's user avatar
  • 1,576
1 vote

Extract phrases/keywords that are SIMILAR to a python list of keyword/phrases, from a document

You could use sentence transformer library to calculate the similarity between different phrases. It also works for multi worded tokens. ...
shrey jain's user avatar
4 votes
Accepted

Some simple questions about confusion matrix and metrics in general

The first model where the f1_score is around 61% can not be considered as a good model. You can achieve much better results than that. This can be seen in the ...
spectre's user avatar
  • 1,576
0 votes

BIO Format (Skills,Qualification,Experience)

You need to manually annotate a large sample of your input text like this: ...
Erwan's user avatar
  • 24.6k
1 vote

How do people usually handle creating an embedding vector of longer texts (32000 characters?

When dealing with longer texts, you can use a technique called "sliding window" to break the text into smaller segments. This involves taking a window of fixed size and sliding it along the ...
RegressIt's user avatar
1 vote
Accepted

Why my sentiment analysis model is overfitting?

There might be multiple reasons that might be the reason for overfitting some of which are: 1.) Scaling the data 2.) You have not mentioned which parameter values you have selected in the Tfidf ...
spectre's user avatar
  • 1,576
0 votes

Using BERT to extract a list of words and phrases from documents

You have a list of keyphrases that you need to extract from documents. You could use NER for this purpose. But looking at the size of the keyphrases (3000) it will be a difficult task because you ...
spectre's user avatar
  • 1,576
0 votes

What are the approaches for extracting an injury and its description from a paragraph?

Given the unstructured nature of your injury descriptions, I don't think this is doable by means of classical NLP techniques. I suggest you use a large language model (LLM), either OpenAi's GPT family ...
noe's user avatar
  • 20.5k
1 vote

Load an LLM in multiple GPUs

Using multiple GPUs usually means that the whole model is copied into the memory of each of them. In Pytorch this is achieved with nn.DataParallel or nn.parallel.DistributedDataParallel. This however, ...
noe's user avatar
  • 20.5k
1 vote

LLM powered chat bot enhanced by NER

It's probably best to look into some research but here is some information which might help in the meantime based on some general thoughts; Adding NER to a LLM can enhance the models ability to ...
RegressIt's user avatar
4 votes

Sentence tokenization for sentence without punctuation

You can apply a previous step to add punctuation and proper casing to the text, and then segment the sentences. For this, you may use Re-punctuate. When applied to your text, Re-punctuate gives the ...
noe's user avatar
  • 20.5k
2 votes

Transformers doubt

We should not mistake the K, Q and V vectors received by the multi-head attention block with those received by the scaled dot-product block. The K, Q and V vectors that are fed to the multi-head ...
noe's user avatar
  • 20.5k
1 vote

Deduplication using NLP

You could try the following: Similarity Metrics: There are several similarity metrics that can be used to compare the similarity between two texts. For example, the Jaccard similarity coefficient, ...
Vinita Silaparasetty's user avatar
1 vote
Accepted

2 basic doubts on time series

Regarding Doubt 1: Yes, if you shuffle those for training the order within each row will be preserved. If you shuffle the rows, that's good, and the LSTM models will consider each row as a separate ...
m13op22's user avatar
  • 342
0 votes

Why is the decoder not a part of BERT architecture?

In short, Bidirectional Encoder Representations from Transformers (BERT) is not designed for decorder-related tasks. I can't see how BERT makes predictions without using a decoder unit, which was a ...
We.Me.ii's user avatar
3 votes
Accepted

Easy question on autoregressive LLM

At t=1: K, Q, V are the projections of the sequence "today is a good". However, given that the computations of the first tokens were already done in the previous step, usually, there is some ...
noe's user avatar
  • 20.5k

Top 50 recent answers are included