# Tag Info

4

The answer to your needs is called "bucketing". It consists of creating batches of sequences with similar length, to minimize the needed padding. In tensorflow, you can do it with tf.data.experimental.bucket_by_sequence_length. Take into account that previously it was in a different python package (tf.contrib.data.bucket_by_sequence_length), so the ...

3

Maybe you don't have a positive and a negative class. Your input are word vectors. Unless you trained your word vectors before with explicit positive and negative labels, it is very unlikely that your KMeans learned that difference. If you used pre-trained word vectors, your KMeans could have learned an arbitrary difference between cluster 0 and cluster 1. ...

2

BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, BERT only gives you the subword vectors. Subwords are used for representing both the input text and the output tokens. When an unseen word is presented to BERT, it ...

2

The GridsearchCV object in sklearn does a cross-validation on the data you feed it during your fit. In your case you have specified cv=5: this means GridSearchCV splits your data into train/test splits 5 times and reports on the mean performance over those 5 trials to be 0.868. You asked why GridSearchCV knows the best parameters without feeding it testing ...

1

The name provides a clue. BERT (Bidirectional Encoder Representations from Transformers): So basically BERT = Transformer Minus the Decoder BERT ends with the final representation of the words after the encoder is done processing it. In Transformer, the above is used in the decoder. That piece of architecture is not there in BERT

1

Found a solution, which is to pass a custom batch generator of type keras.utils.Sequence to the model.fit function (where one can write any logic to construct batches and to modify/augment training data) instead of passing the entire dataset in one go. Relevant code for reference: # Must implement the __len__ function returning the number # of batches in ...

1

No, as there is randomness in the method implementation, for example here (in LdaModel of the gensim library). Hence, it can affect your final result in each run. Therefore, if you want to keep the result reproducible, you can set the random_state property of the model to a constant seed (see the documentation for more details).

1

The key is precisely in the definition of the loss: loss_object = tf.keras.losses.SparseCategoricalCrossentropy( from_logits=True, reduction='none') As you can see, the loss is created with the flag from_logits=True which means that the input to the loss is not a probability distribution, but unnormalized log probabilities, namely "logits", ...

1

You are describing semi-supervised learning where the training dataset is only partially labeled. One common set of techniques to solve that problem is active learning. In active learning, there is a learning loop where the algorithm makes predictions and a human corrects those predictions. Pre-clustering is a specific active learning technique to address ...

1

To answer in the simplest way possible - let the model learn the attention weights by training itself. We do that by defining a Dense single layer MLP with 1 unit which 'transforms' each word in the input sentence in such a way that when a dot product of this transformation with the last decoder state is taken, the resulting value is high if the word in ...

1

NLTK has a sentiment module. You can try and check the statistics of positive vs negative for each text in the clusters.

1

It sounds like you are looking for an unsupervised learning approach (meaning you don't need to manually label your data). Something like k-means clustering could work well. This would allow you to group you comments into k distinct clusters. You could then view counts of comments in those clusters and explore the clusters to determine their meaning. In ...

1

This is the standard TF-IDF feature extraction: you transform the document counts. It just looks odd to separate the two steps like this. sklearn provides both TfidfTransformer and TfidfVectorizer; note the documentation of the latter: Equivalent to CountVectorizer followed by TfidfTransformer.

1

There's no algorithm intended specifically for this task, you need to design the process yourself (like for most tasks btw). Given that the goal would be to use a person's name as an indication, I'd suggest you represent a name as a vector of characters n-grams in the features. Example with bigrams ($n=2$): "Braund" = [ #B, Br, ra, au, un, nd, d# ] ...

1

There is a token vocabulary, that is, the set of all possible tokens that can be handled by BERT. You can find the vocabulary used by one of the variants of BERT (BERT-base-uncased) here. You can see that it contains one token per line, with a total of 30522 tokens. The softmax is computed over them. The token granularity in the BERT vocabulary is subwords. ...

1

The basic concept of transfer learning is: Storing knowledge gained from solving one problem and applying it to different but related problem I guess to be precise this is called Transductive Transfer Learning. In this we learn from the already observed training dataset and then predict the labels of the testing dataset. Even though we do not know the labels ...

1

To provide a simplistic and less mathematical reasons. You can assume like this: In a simple feed-forward neural network (a black-box of course), you shall learn the set of weights, learning a function to map inputs to outputs. But, in the transformers based architecture, you have Attentions. Here, the weights are structured into Query, Key and Value (Q,K,V)....

1

It looks to me like what you propose makes sense, but there has been some research done around these questions of time representation already. I'd suggest you check the state of the art in this domain, if only not to reinvent the wheel or miss important cases. I'm not very knowledgeable about it but I can at least point you to TimeML and the related ...

1

As Erwan said in the comments, it depends. In my experience, it depends specifically on two things: Tokenization method: The length of a document in number of tokens will vary considerably depending on how you split it up. Splitting your text into individual characters will result in a longer document than splitting it into sub-word units (e.g. WordPiece), ...

1

Before talking about the solution, why don't you focus on the content instead? I think it would be more helpful to solve your problem, considering that most of the email addresses end with the sender's sign, Name Surname. Also, the probability of failing to obtain this information from an email address is much higher than the probability of failing to get it ...

Only top voted, non community-wiki answers of a minimum length are eligible