2 votes

Is it good practice to remove the numeric values from the text data during preprocessing?

To build on Prashant's answer, it will depend on your problem. If you think those values are important to your task, you might try to extract them and tack them onto the end of your data (I'm thinking ...
  • 391
2 votes
Accepted

Why I would use TF-IDF after Bag-of-Words (CountVectorizer)?

This is the standard TF-IDF feature extraction: you transform the document counts. It just looks odd to separate the two steps like this. sklearn provides both <...
  • 10.2k
2 votes

How to process the hyphenated english words for any nlp problem?

They all sound like interesting approaches. The first one is better I think because it allows for unseen hyphenated words to be somewhat understood (as e.g. well + known ~= well-known). For a tfidf ...
2 votes

How to decide to go with BOW or TFIDF

It depends on the problem you are trying to solve. If you know the signal in the dataset already, the words which decide your decision then go with Bag of Words. This is useful when you are doing ...
1 vote

Which phrase should be returned in case of multiple matches when comparing text?

This is usually done by carefully choosing two things: The sentence representation. Word count is the most simple option but there can be many others: TFIDF weights, with/without removing stop words, ...
  • 22.8k
1 vote
Accepted

Word representation that gives more weight to terms frequent in corpus?

I'm not aware of any standard representation which increases the importance of document-frequent words, but IDF can simply be reverted: instead of the usual $$idf(w,D)=\log\left(\frac{N}{|d\in D\ |\ w ...
  • 22.8k
1 vote
Accepted

One-hot vector for fixed vocabulary

...
  • 6,962
1 vote

Machine learning algorithms for forming Homophones from input dataset word

I have very limited knowledge about homophones generator. I feel to make a homophone detector, one should focus more on the phonetics of the word rather than the spellings. try to make a word-...
1 vote

Machine learning algorithms for correct words formation from jumbled words

...
1 vote

Is it good practice to remove the numeric values from the text data during preprocessing?

Is it recommended to discard this numerics before creating a vectorizer(bow/tf-idf) for any model(classification/regression) development? It depends on the problem statement for example year could be ...
  • 1,401
1 vote

How to process the hyphenated english words for any nlp problem?

1 and 3 would be nice. Separating "well-known" to "well" and "known" would not be a good idea because you lost an information and/or have an erroneous/unuseful counts.
  • 106
1 vote

How to process the hyphenated english words for any nlp problem?

I agree with Nicholas' answer, a few more thoughts: you could use a standard English tokenizer (e.g. nltk, Spacy), if only to see how they process hyphenated words. Similarly you could check how it's ...
  • 22.8k

Only top scored, non community-wiki answers of a minimum length are eligible