Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
67 votes
Accepted

GBM vs XGBOOST? Key differences?

Quote from the author of xgboost: Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. ...
Icyblade's user avatar
  • 4,336
42 votes
Accepted

Adaboost vs Gradient Boosting

Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion. Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more ...
oW_'s user avatar
  • 6,357
20 votes

GBM vs XGBOOST? Key differences?

In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in ...
Sandeep S. Sandhu's user avatar
16 votes

Why does Gradient Boosting regression predict negative values when there are no negative y-values in my training set?

Remember that the GradientBoostingRegressor (assuming a squared error loss function) successively fits regression trees to the residuals of the previous stage. Now ...
Milad Shahidi's user avatar
15 votes

GBM vs XGBOOST? Key differences?

One very important difference is xgboost has implemented DART, the dropout regularization for regression trees. References Rashmi, K. V., & Gilad-Bachrach, ...
horaceT's user avatar
  • 1,360
15 votes

In industry, what type of new data science algorithms does one develop?

I am no data scientist, only an aspiring one for two years, moving from my background in software engineering and mathematics. So I took some courses, had some interviews, read a lot on the subject ...
Pieter21's user avatar
  • 1,041
13 votes
Accepted

How to determine if character sequence is English word or noise

During NLP and text analytics, several varieties of features can be extracted from a document of words to use for predictive modeling. These include the following. ngrams Take a random sample of ...
Brandon Loudermilk's user avatar
10 votes
Accepted

Algorithms and techniques for spell checking

Here is what I built... Step 1: Store all the words in a Trie data structure. Wiki about trie. Step 2: Train an RNN or RNTN to get seq2seq mapping for words and store the model Step 3: Retrieve top n ...
Rahul Reddy Vemireddy's user avatar
10 votes

In industry, what type of new data science algorithms does one develop?

In industry its usually variations (but important ones) of the ground ideas. Look at this boosting timeline: (Ada)Boosting Formally by two profesors in 2003 xgboost by DLMC Distributed Machine ...
Noah Weber's user avatar
  • 5,679
9 votes

What is the difference between outlier detection and anomaly detection?

(I actually wanted to write this as an answer to the Cross Validated question: Difference between Anomaly and Outlier, but the question is protected - I think answering it here should be fine, despite ...
Marco13's user avatar
  • 400
9 votes
Accepted

Decision Trees - how does split for categorical features happen?

You are right on all counts: If DT splits a node with the above algorithm and treat those 10 values are true numeric values, will it not lead to wrong/misinterpreted splits? Yes absolutely, ...
Erwan's user avatar
  • 25.5k
8 votes
Accepted

Gradient boosting algorithm example

I tried to construct the following simple example (mostly for my self-understanding) which I hope could be useful for you. If someone else notices any mistake please let me know. This is somehow based ...
antonioACR1's user avatar
8 votes
Accepted

What knowledge do I need in order to write a simple AI program to play a game?

There are multiple ways to approach solving game playing problems. Some games can be solved by search algorithms for example. This works well for card and board games up to some level of complexity. ...
Neil Slater's user avatar
7 votes

What knowledge do I need in order to write a simple AI program to play a game?

It highly depends on the type of game and the information about the state of the game that is available to your AI. Some of the most famous game playing AIs from last few years are based on deep ...
noe's user avatar
  • 26.9k
7 votes

What is the rationale for discretization of continuous features and when should it be done?

One reason to discretize continuous features is to improve the signal-to-noise ratio. Fitting a model using features that have been binned reduces the impact that small fluctuations in the data have ...
Brian Spiering's user avatar
7 votes

What are the most suitable machine learning algorithms according to type of data?

Given the list of algorithms you provided, these falls under 3 major classification of ML algorithms. 1) Classification Algorithms - Naive Bayes Classification, Decision Tree, Random Forest, kNN, ...
Kshitiz 's user avatar
7 votes

Algorithms for aggregating duplicate identities based on non-numerical data?

I haven't yet successfully solved my record linkage problem, but I wanted to share some of the stuff I've found in the process in case it's of use to anyone else. This is a work in progress based here ...
ropeladder's user avatar
7 votes

What is the difference between outlier detection and anomaly detection?

Fundamentally there is no difference. Say you have data and you want to build a model of it. As the name suggests, modeling is about finding a model, that is, a simplified representation of your data....
tom's user avatar
  • 2,248
6 votes

Naive about which Naive Bayes in article

First off, yes there are different Naive Bayes algorithms. But they are all based on the same principle, namely Bayes Theorem where the features are assumed to be independent. Here is a short guide ...
Valentin Calomme's user avatar
6 votes
Accepted

Proper Understanding of Condensed Nearest Neighbor

Condensed Nearest Neighbors algorithm helps to reduce the dataset X for k-NN classification. It constructs a subset of examples which are able to correctly classify the original data set using a 1-NN ...
Anastasiia Shalygina's user avatar
5 votes

Are decision tree algorithms linear or nonlinear

As many pointed out, a regression/decision tree is a non-linear model. Note however that it is a piecewise linear model: in each neighborhood (defined in a non-linear way), it is linear. In fact, the ...
Matifou's user avatar
  • 149
5 votes
Accepted

Logbook: Machine Learning approaches

How are you solving this? How are you keeping track of the work done? What's your logbook tool? This might not be the best approach. But, this is how my team does it. We believe that for pulling ...
Dawny33's user avatar
  • 8,296
5 votes

Does logistic regression can only solve binary classification problem?

No, multiclass classification is also possible. Try reading up on 'One vs All' multiclass classification. https://en.wikipedia.org/wiki/Multiclass_classification Multiclass, One vs All ...
Anshul G.'s user avatar
  • 535
5 votes

Decision Trees - how does split for categorical features happen?

Yes, it will add a certain bias given by the fact that we are inserting an ordering that is not intrinsic to the categories Not really. The natural way to deal with a categorical feature that has L ...
Davide ND's user avatar
  • 181
5 votes

Is Annoy a machine learning algorithm to find nearest neighbor ? and is it similar to K nearest neighbor algorithm?

Don't have enough reputation to comment to a resource, so answering this myself. About Annoy Annoy is a library being used here for finding approximate nearest neighbours, approximate being the key ...
Bhavul's user avatar
  • 151
5 votes

What is the difference between AI, ML, NN and DL?

AI: a super general term, means a bit of everything... and nothing at the same time. It's all about building intelligent machines, even though its meaning is not fully developed. It's not used in a ...
Leevo's user avatar
  • 6,255
4 votes

Are decision tree algorithms linear or nonlinear

A decision tree is a non-linear classifier. If your dataset contains consistent samples, namely you don't have the same input features and contradictory labels, decision trees can classify the data ...
Green Falcon's user avatar
  • 14.1k
4 votes

k-means clustering data with large number of meaningless values

One thing you could do is apply some dimensional reduction algorithm (such as PCA) so you can get the columns with high variance, then run k-means on that data set. However, I suggest against using k-...
masotann's user avatar
  • 171
4 votes

Which algorithm to use to predict the duration of some task

What others have said is accurate, that you need to build a regression model of some sort. Depending on the scale of the duration of your task, you will have to model it slightly differently. In ...
franciscojavierarceo's user avatar
4 votes

What knowledge do I need in order to write a simple AI program to play a game?

What you are looking for is called Reinforcement Learning. At my university, there is a complete course ($15 \cdot 3h = 45h$) only to introduce students to this topic. Here are my (mostly german) ...
Martin Thoma's user avatar
  • 18.9k

Only top scored, non community-wiki answers of a minimum length are eligible