DaL
  • Member for 6 years, 3 months
  • Last seen this week
What is the difference between NLP and text mining?
Accepted answer
12 votes

I agree with Sean's answer. NLP and text mining are usually used for different goals. Also, there is indeed an overlap and both definitions are vogue. Other than the difference in goal, there is a ...

View answer
Is feature selection necessary?
Accepted answer
12 votes

Feature selection might be consider a stage to avoid. You have to spend computation time in order to remove features and actually lose data and the methods that you have to do feature selection are ...

View answer
Why do we need to handle data imbalance?
Accepted answer
8 votes

You need to deal with imbalanced data set when the value of finding the minority class is much higher than that of finding the majority. Let say that 1% of the population have that rare disease. ...

View answer
Why don't tree ensembles require one-hot-encoding?
Accepted answer
7 votes

The encoding leads to a question of representation and the way that the algorithms cope with the representation. Let's consider 3 methods of representing n categorial values of a feature: A single ...

View answer
Is Overfitting a problem in Unsupervised learning?
Accepted answer
7 votes

Overfitting happens when the model fits the training dataset more than it fits the underlying distribution. In a way, it models the specific sample rather than producing a more general model of the ...

View answer
Stopwords for programming languages (for, while, print,...)
6 votes

Stop words are common words in a language and they are usually removed when there appearance is not indicative to the analysis goal. Suppose our goal is to text mine and find out if a given text is ...

View answer
feature selection techniques
Accepted answer
6 votes

Doing that is a very good idea. The problem is that doing that is very hard. Feature selection is a NP-complete problem. The practical meaning as that we don't know any fast algorithm that can select ...

View answer
How to start prediction from dataset?
Accepted answer
6 votes

The problem you are facing is a time series problem. Your events are categorial which is a specific case (so most common techniques like arima and Fourier transform are irrelevant). Before getting ...

View answer
Feature selection by overfitting a small sample size
Accepted answer
5 votes

Congratulations! You have suggested independently the Wrapper method for feature selection. Yes, you can use this method. However, consider that the wrapper method is slow since you have to train a ...

View answer
Why do popular search engines not follow the usual AND, OR logic for queries?
5 votes

Nice question! An exact answer should be given by looking in the search engine source code but here is a possible explanation. I run the queries at Google burglar 33,800,000 burglar AND burglar 29,...

View answer
Signs there are too many class labels
5 votes

A class taxonomy should: Serve the business needs Be learnable There is a potential tradeoff here. The more exact and specific the taxonomy, the more you'll know about the entities and you'll be ...

View answer
Should I go for a 'balanced' dataset or a 'representative' dataset?
5 votes

Separate the operational and the training scenarios. The operational scenario is the one in which your classifier will be measure on. This is where you should perform well. Use should have a dataset ...

View answer
Perform classification on market basket analysis
5 votes

In case that the number of items is quite small, turning the problem into a classification problem will be the most convenient solution. Use each item as a feature, the class as a concept. Now you can ...

View answer
Application of Machine learning or Neural Networks for automatic Time table scheduling
Accepted answer
4 votes

Scheduling problems might be NP-Complete problems. It is not clear what are the specific details. You might get lucky and have specific constraints that are leading to an easy sub problem or just an ...

View answer
Which Outlier Detection Method? Why?
Accepted answer
4 votes

You can justify your choices by using data. Treat the anomaly detection like a supervised learning problem where the concept is being anomaly. Then you'll be able to present - for each method - its ...

View answer
Sentiment Analysis: Creating dictionary from dataset
4 votes

When looking on the probability of word occurrence, you will get stop words and other popular words. You are interested in words that appear more in the comments (assumed hate related) than in normal ...

View answer
NLP - Is Gazetteer a cheat?
4 votes

Using a list of entities has few disadvantages: The list is closed The list is not context sensitive. You need context in order to differ between "a white house" and "the white house". List building ...

View answer
Build a binary classifier with only positive and unlabeled data
4 votes

Your problem belongs to the framework of PU learning (only positives, a lot of unlabelled). It is also close to the more common frameworks of Semi supervised learning (few positives and negatives, a ...

View answer
Advantages of AUC vs standard accuracy
4 votes

I'd like to refer to how you should choose a performance measure. Before that I'll refer to the specific question of accuracy and AUC. As answered before, on imbalanced dataset using the majority run ...

View answer
How big is big data?
4 votes

I think that big data starts at the point where the size prevents you from doing what you want to. In most scenarios, there is a limit on the running time that is considered feasible. In some cases it ...

View answer
Data balance -before or after feature selection/engineering
3 votes

Balancing is using which of the samples to consider in the data set (adding/reducing rows). Features selection and feature engineering is removing and adding information about each sample (adding/...

View answer
Is machine learning successful in solving combinatorial optimisation problems in NP-hard? Discuss problem of scheduling using machine learning
3 votes

You don't have to get to EXP-complete in order to get a hard problem. NP-Complete is bad enough... Cryptographic assumptions (e.g., the existence of one way functions) are also a good way to create ...

View answer
Training data from different sources
Accepted answer
3 votes

It seems that you have a domain adaptation problem. The samples from the two sources behaves differently. I suggest reading Frustratingly Easy Domain Adaptation. As the name hints, the solution is ...

View answer
When should we consider a dataset as imbalanced?
3 votes

Imbalance is not defined formally but a ratio of 1 to 10 is usually imbalanced enough to benefit from using balancing technique. There are two type of imbalance, relative and absolute. In the ...

View answer
Dissmissing features based on correlation with target variable
3 votes

Please note that Pearson correlation (and mutual information) considers the concept and the single feature. There are cases in which a single feature is useless but given more features it becomes ...

View answer
How to find similarity/distance matrix with mixed Continuous and Categorical data?
3 votes

Similarity measures are subjective and so are they ways to combine them. You should decide what is your subjective definition of similarity and then find a way to combine them that fit your definition....

View answer
How to grow a list of related words based on initial keywords?
3 votes

This is one of the nice problem where the scope might vary from an homework assignment to a Google size project. Indeed, you can start with co-occurrence of the words (e.g., conditional probability). ...

View answer
How to evaluate data capability to train a model?
Accepted answer
2 votes

1.2M is about 2^16. You have 512 features plus the concept so the number of possibilities is much larger. Therefore, you can claim that the number of samples you have is too small. Though that, ...

View answer
What is PAC learning?
Accepted answer
2 votes

PAC stands for Probably Approximately Correct. It was a very common research area in computer science looking for proof of learnability of certain hypothesis sets. The usual hypothesis sets were not ...

View answer
Regarding "modification" of feature columns in supervised learning
2 votes

If you are considering the information theoretical point of view, given x3and x4 then x3/x4 adds no information. However, before we rush into conclusions, one must recall that there are more aspects. ...

View answer