59 votes
Accepted

GBM vs XGBOOST? Key differences?

Quote from the author of xgboost: Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. ...
user avatar
  • 4,016
51 votes

When is a Model Underfitted?

A model underfits when it is too simple with regards to the data it is trying to model. One way to detect such situation is to use the bias–variance approach, which can represented like this: Your ...
user avatar
37 votes
Accepted

Adaboost vs Gradient Boosting

Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion. Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more ...
user avatar
  • 5,987
33 votes
Accepted

Is logistic regression actually a regression algorithm?

Logistic regression is regression, first and foremost. It becomes a classifier by adding a decision rule. I will give an example that goes backwards. That is, instead of taking data and fitting a ...
user avatar
  • 1,146
23 votes

Item based and user based recommendation difference in Mahout

Item Based Algorithm ...
user avatar
  • 1,025
22 votes

Are decision tree algorithms linear or nonlinear

A decision tree is a non-linear mapping of X to y. This is easy to see if you take an arbitrary function and create a tree to ...
user avatar
  • 361
18 votes

GBM vs XGBOOST? Key differences?

In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in ...
user avatar
16 votes

When to choose linear regression or Decision Tree or Random Forest regression?

Let me explain it using some examples for clear intuition: When do you use linear regression vs Decision Trees? Linear regression is a linear model, which means it works really nicely when the ...
user avatar
  • 8,016
15 votes

Are decision tree algorithms linear or nonlinear

Recently a friend of mine was asked whether decision tree algorithm a linear or nonlinear algorithm in an interview Decision trees is a non-linear classifier like the neural networks, etc. It is ...
user avatar
  • 8,016
15 votes

In industry, what type of new data science algorithms does one develop?

I am no data scientist, only an aspiring one for two years, moving from my background in software engineering and mathematics. So I took some courses, had some interviews, read a lot on the subject ...
user avatar
  • 1,021
14 votes

GBM vs XGBOOST? Key differences?

One very important difference is xgboost has implemented DART, the dropout regularization for regression trees. References Rashmi, K. V., & Gilad-Bachrach, ...
user avatar
  • 1,320
13 votes

Why does Gradient Boosting regression predict negative values when there are no negative y-values in my training set?

Remember that the GradientBoostingRegressor (assuming a squared error loss function) successively fits regression trees to the residuals of the previous stage. Now ...
user avatar
13 votes
Accepted

Algorithms for text clustering

Check the Stanford NLP Group's open source software, in particular, Stanford Classifier. The software is written in Java, which will likely delight you, but also ...
user avatar
13 votes
Accepted

How to determine if character sequence is English word or noise

During NLP and text analytics, several varieties of features can be extracted from a document of words to use for predictive modeling. These include the following. ngrams Take a random sample of ...
user avatar
12 votes
Accepted

K-means vs. online K-means

Online k-means (more commonly known as sequential k-means) and traditional k-means are very similar. The difference is that online k-means allows you to update the model as new data is received. ...
user avatar
12 votes
Accepted

Item based and user based recommendation difference in Mahout

You are correct that both models work on the same data without any problem. Both items operate on a matrix of user-item ratings. In the user-based approach the algorithm produces a rating for an item <...
user avatar
  • 286
11 votes

Clustering based on similarity scores

I think a number of clustering algorithms that normally use a metric, do not actually rely on the metric properties (other than commutativity, but I think you'd have that here). For example, DBSCAN ...
user avatar
  • 3,062
11 votes

When is a Model Underfitted?

To answer your question it is important to understand the frame of reference you are looking for, if you are looking for what philosophically you are trying to achieve in model fitting, check out ...
user avatar
  • 891
11 votes
Accepted

Solving a system of equations with sparse data

If I understand you correctly, this is the case of multiple linear regression with sparse data (sparse regression). Assuming that, I hope you will find the following resources useful. 1) NCSU lecture ...
user avatar
10 votes

Is logistic regression actually a regression algorithm?

Short Answer Yes, logistic regression is a regression algorithm and it does predict a continuous outcome: the probability of an event. That we use it as a binary classifier is due to the ...
user avatar
10 votes

Clustering based on similarity scores

Alex made a number of good points, though I might have to push back a bit on his implication that DBSCAN is the best clustering algorithm to use here. Depending on your implementation, and whether or ...
user avatar
  • 4,179
10 votes

In industry, what type of new data science algorithms does one develop?

In industry its usually variations (but important ones) of the ground ideas. Look at this boosting timeline: (Ada)Boosting Formally by two profesors in 2003 xgboost by DLMC Distributed Machine ...
user avatar
  • 5,291
9 votes
Accepted

Does reinforcement learning require the help of other learning algorithms?

You do not need additional learning algorithms to perform reinforcement learning in simple systems where you can explore all states. For those, simple iterative Q-learning can do very well - as well ...
user avatar
  • 27.3k
9 votes
Accepted

Algorithms and techniques for spell checking

Here is what I built... Step 1: Store all the words in a Trie data structure. Wiki about trie. Step 2: Train an RNN or RNTN to get seq2seq mapping for words and store the model Step 3: Retrieve top n ...
user avatar
9 votes

What is the difference between outlier detection and anomaly detection?

(I actually wanted to write this as an answer to the Cross Validated question: Difference between Anomaly and Outlier, but the question is protected - I think answering it here should be fine, despite ...
user avatar
  • 400
9 votes
Accepted

Decision Trees - how does split for categorical features happen?

You are right on all counts: If DT splits a node with the above algorithm and treat those 10 values are true numeric values, will it not lead to wrong/misinterpreted splits? Yes absolutely, ...
user avatar
  • 21.8k
8 votes

When is a Model Underfitted?

Models are but abstractions of what is seen in real life. They are designed in order to abstract-away nitty-gritties of the real system in observation, while keeping sufficient information to support ...
user avatar
  • 4,027
8 votes
Accepted

Classifying Java exceptions

First of all, some basics of classification (and in general any supervised ML tasks), just to make sure we have same set of concepts in mind. Any supervised ML algorithm consists of at least 2 ...
user avatar
  • 2,771
8 votes
Accepted

Gradient boosting algorithm example

I tried to construct the following simple example (mostly for my self-understanding) which I hope could be useful for you. If someone else notices any mistake please let me know. This is somehow based ...
user avatar
8 votes
Accepted

What knowledge do I need in order to write a simple AI program to play a game?

There are multiple ways to approach solving game playing problems. Some games can be solved by search algorithms for example. This works well for card and board games up to some level of complexity. ...
user avatar
  • 27.3k

Only top scored, non community-wiki answers of a minimum length are eligible