New answers tagged text-classification
0
You're getting the error because your code is written using the variable cluster as if it's both a numpy array (cluster.shape[0]) and a list (cluster.append(vectors[target]). A numpy array doesn't have an append() method and a list doesn't have a shape attribute. To fix this you need to decide which data structures you're going to use and make sure that the ...
1
Assuming that the "human readable" texts are more likely to contain actual words, you could count the number of dictionary words that occur in each.
You could use Wordnet for example.
The number or proportion of word hits, and their length, could be features for a model or maybe it would be enough with a simple cutoff rule.
You might want to ...
2
You could train a character-level language model, e.g. an LSTM, on the real short texts, and use the perplexity as the signal to know whether a piece of text is real or not.
In order to find an appropriate perplexity threshold, you can have a look at the distribution of perplexities over a validation holdout dataset.
UPDATE: There are multiple ...
0
Try making features like vowel_count, consonant_count, digitcount , vowel_density(vowel_count/total_length_of_words)
Another wild thing -
split the strigns with numbers and _ using regex and try to see if they are english words or not, use a pretrained model like spacy.english or nltk.words to check, make a column representing english words count if any.
...
1
Following discussion with Erwan: one of his previous answer partially has answered my question. However I would like to understand the following. One needs to have a corpus, then label news/tweets in fake/not fake, then run the model. But how the algorithm works on texts and takes relevant words or features for detecting fake news?
First, let me emphasize ...
1
The nDCG depends on the relevance of each document as you can see on the Wikipedia definition. I guess you could use 0 and 1 as relevance scores, but then all relevant documents would have the same score of 1, and then it wouldn't make much sense to apply the nDCG penalty discounts.
A similar measure often used with binary relevance scores is the mean ...
0
I would go for this:
data -> tokenize -> tfidf* -> neural net
But in tfidf vectorizer, you could actually regularize the number of terms used, say for example restricting the minimum number of occurrences of a term and/or defining the max_number of features so that you only keep the ones that have the highest importance according to Tfidf.
If you ...
3
Scikit-learn has compose.ColumnTransformer which
allows different columns or column subsets of the input to be
transformed separately and the features generated by each transformer
will be concatenated to form a single feature space. This is useful
for heterogeneous or columnar data, to combine several feature
extraction mechanisms or transformations into a ...
Top 50 recent answers are included
