56

Quote from the author of xgboost: Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. Specifically, xgboost used a more regularized model formalization to control over-fitting, which gives it better performance. We have updated a comprehensive tutorial on introduction to the model, which ...


50

A model underfits when it is too simple with regards to the data it is trying to model. One way to detect such situation is to use the bias–variance approach, which can represented like this: Your model is underfitted when you have a high bias. To know whether you have a too high bias or a too high variance, you view the phenomenon in terms of training ...


34

Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion. Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more weight on misclassified samples and less weight on correctly classified samples. The final prediction is a weighted average of all the weak learners, where more ...


29

Logistic regression is regression, first and foremost. It becomes a classifier by adding a decision rule. I will give an example that goes backwards. That is, instead of taking data and fitting a model, I'm going to start with the model in order to show how this is truly a regression problem. In logistic regression, we are modeling the log odds, or logit, ...


22

Item Based Algorithm for every item i that u has no preference for yet for every item j that u has a preference for compute a similarity s between i and j add u's preference for j, weighted by s, to a running average return the top items, ranked by weighted average User Based Algorithm for every item i that u has no preference for yet for ...


21

A decision tree is a non-linear mapping of X to y. This is easy to see if you take an arbitrary function and create a tree to its maximum depth. For example: if x = 1, y = 1 if x = 2, y = 15 if x = 3, y = 3 if x = 4, y = 27 ... Of course, this is a completely over-fit tree and won't generalize. But it demonstrates why a decision tree is a non-linear ...


17

In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in speed and memory utilization: Use of sparse matrices with sparsity aware algorithms Improved data structures for better processor cache utilization which makes ...


15

I am no data scientist, only an aspiring one for two years, moving from my background in software engineering and mathematics. So I took some courses, had some interviews, read a lot on the subject online. My opinion: New algorithms are developed in research centers and at universities. And even then, most algorithms used in companies are already developed, ...


14

Recently a friend of mine was asked whether decision tree algorithm a linear or nonlinear algorithm in an interview Decision trees is a non-linear classifier like the neural networks, etc. It is generally used for classifying non-linearly separable data. Even when you consider the regression example, decision tree is non-linear. For example, a linear ...


14

Let me explain it using some examples for clear intuition: When do you use linear regression vs Decision Trees? Linear regression is a linear model, which means it works really nicely when the data has a linear shape. But, when the data has a non-linear shape, then a linear model cannot capture the non-linear features. So in this case, you can use the ...


14

One very important difference is xgboost has implemented DART, the dropout regularization for regression trees. References Rashmi, K. V., & Gilad-Bachrach, R. (2015). Dart: Dropouts meet multiple additive regression trees. arXiv preprint arXiv:1505.01866.


13

Check the Stanford NLP Group's open source software, in particular, Stanford Classifier. The software is written in Java, which will likely delight you, but also has bindings for some other languages. Note, the licensing - if you plan to use their code in commercial products, you have to acquire commercial license. Another interesting set of open source ...


13

During NLP and text analytics, several varieties of features can be extracted from a document of words to use for predictive modeling. These include the following. ngrams Take a random sample of words from words.txt. For each word in sample, extract every possible bi-gram of letters. For example, the word strength consists of these bi-grams: {st, tr, re, ...


12

You are correct that both models work on the same data without any problem. Both items operate on a matrix of user-item ratings. In the user-based approach the algorithm produces a rating for an item i by a user u by combining the ratings of other users u' that are similar to u. Similar here means that the two user's ratings have a high Pearson correlation ...


11

Remember that the GradientBoostingRegressor (assuming a squared error loss function) successively fits regression trees to the residuals of the previous stage. Now if the tree in stage i predicts a value larger than the target variable for a particular training example, the residual of stage i for that example is going to be negative, and so the regression ...


11

I think a number of clustering algorithms that normally use a metric, do not actually rely on the metric properties (other than commutativity, but I think you'd have that here). For example, DBSCAN uses epsilon-neighborhoods around a point; there is nothing in there that specifically says the triangle inequality matters. So you can probably use DBSCAN, ...


11

Online k-means (more commonly known as sequential k-means) and traditional k-means are very similar. The difference is that online k-means allows you to update the model as new data is received. Online k-means should be used when you expect the data to be received one by one (or maybe in chunks). This allows you to update your model as you get more ...


11

If I understand you correctly, this is the case of multiple linear regression with sparse data (sparse regression). Assuming that, I hope you will find the following resources useful. 1) NCSU lecture slides on sparse regression with overview of algorithms, notes, formulas, graphics and references to literature: http://www.stat.ncsu.edu/people/zhou/courses/...


10

Short Answer Yes, logistic regression is a regression algorithm and it does predict a continuous outcome: the probability of an event. That we use it as a binary classifier is due to the interpretation of the outcome. Detail Logistic regression is a type of generalize linear regression model. In an ordinary linear regression model, a continuous outcome, ...


10

Alex made a number of good points, though I might have to push back a bit on his implication that DBSCAN is the best clustering algorithm to use here. Depending on your implementation, and whether or not you're using accelerated indices (many implementations do not), your time and space complexity will both be O(n2), which is far from ideal. Personally, my ...


10

To answer your question it is important to understand the frame of reference you are looking for, if you are looking for what philosophically you are trying to achieve in model fitting, check out Rubens answer he does a good job of explaining that context. However, in practice your question is almost entirely defined by business objectives. To give a ...


10

In industry its usually variations (but important ones) of the ground ideas. Look at this boosting timeline: (Ada)Boosting Formally by two profesors in 2003 xgboost by DLMC Distributed Machine Learning Community in 2014 lightgbm by Microsoft in 2017 catboost by yandex in 2017 +- all the variations in between that did not caught up Building on "basic" idea ...


9

Here is what I built... Step 1: Store all the words in a Trie data structure. Wiki about trie. Step 2: Train an RNN or RNTN to get seq2seq mapping for words and store the model Step 3: Retrieve top n words with levenshtein distance with the. Wiki about LD with the word that you are trying to correct. Naive Approach: Calculating the edit distance between ...


9

You are right on all counts: If DT splits a node with the above algorithm and treat those 10 values are true numeric values, will it not lead to wrong/misinterpreted splits? Yes absolutely, for exactly the reason you mention below: Should it rather perform the split based on == and != for this variable? But then, how will the algorithm know ...


8

In general regression models (any) can behave in an arbitrary way beyond the domain spanned by training samples. In particular, they are free to assume linearity of the modeled function, so if you for instance train a regression model with points: X Y 10 0 20 1 30 2 it is reasonable to build a model f(x) = x/10-1, which for x<10 returns ...


8

First of all, some basics of classification (and in general any supervised ML tasks), just to make sure we have same set of concepts in mind. Any supervised ML algorithm consists of at least 2 components: Dataset to train and test on. Algorithm(s) to handle these data. Training dataset consists of a set of pairs (x, y), where x is a vector of features ...


8

You do not need additional learning algorithms to perform reinforcement learning in simple systems where you can explore all states. For those, simple iterative Q-learning can do very well - as well as a variety of similar techniques, such as Temporal Difference, SARSA. All these can be used without neural networks, provided your problem is not too big (...


8

I tried to construct the following simple example (mostly for my self-understanding) which I hope could be useful for you. If someone else notices any mistake please let me know. This is somehow based on the following nice explanation of gradient boosting http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/ The example aims to ...


8

There are multiple ways to approach solving game playing problems. Some games can be solved by search algorithms for example. This works well for card and board games up to some level of complexity. For instance, IBM's Deep Blue was essentially a fast heuristic-driven search for optimal moves. However, probably the most generic machine learning algorithm ...


8

(I actually wanted to write this as an answer to the Cross Validated question: Difference between Anomaly and Outlier, but the question is protected - I think answering it here should be fine, despite the lower visibility) People occasionally argue that there is no difference between an outlier and an anomaly by citing Charu Aggarwal, author of the Book "...


Only top voted, non community-wiki answers of a minimum length are eligible