46

A model underfits when it is too simple with regards to the data it is trying to model. One way to detect such situation is to use the bias–variance approach, which can represented like this: Your model is underfitted when you have a high bias. To know whether you have a too high bias or a too high variance, you view the phenomenon in terms of training ...


38

Quote from the author of xgboost: Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. Specifically, xgboost used a more regularized model formalization to control over-fitting, which gives it better performance. We have updated a comprehensive tutorial on introduction to the model, ...


23

Logistic regression is regression, first and foremost. It becomes a classifier by adding a decision rule. I will give an example that goes backwards. That is, instead of taking data and fitting a model, I'm going to start with the model in order to show how this is truly a regression problem. In logistic regression, we are modeling the log odds, or logit, ...


21

Item Based Algorithm for every item i that u has no preference for yet for every item j that u has a preference for compute a similarity s between i and j add u's preference for j, weighted by s, to a running average return the top items, ranked by weighted average User Based Algorithm for every item i that u has no preference for yet for ...


18

A decision tree is a non-linear mapping of X to y. This is easy to see if you take an arbitrary function and create a tree to its maximum depth. For example: if x = 1, y = 1 if x = 2, y = 15 if x = 3, y = 3 if x = 4, y = 27 ... Of course, this is a completely over-fit tree and won't generalize. But it demonstrates why a decision tree is a non-linear ...


14

Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion. Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more weight on misclassified samples and less weight on correctly classified samples. The final prediction is a weighted average of all the weak learners, where more ...


13

Check the Stanford NLP Group's open source software (http://www-nlp.stanford.edu/software), in particular, Stanford Classifier (http://www-nlp.stanford.edu/software/classifier.shtml). The software is written in Java, which will likely delight you, but also has bindings for some other languages. Note, the licensing - if you plan to use their code in ...


13

During NLP and text analytics, several varieties of features can be extracted from a document of words to use for predictive modeling. These include the following. ngrams Take a random sample of words from words.txt. For each word in sample, extract every possible bi-gram of letters. For example, the word strength consists of these bi-grams: {st, tr, re, ...


13

In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in speed and memory utilization: Use of sparse matrices with sparsity aware algorithms Improved data structures for better processor cache utilization which makes ...


11

Online k-means (more commonly known as sequential k-means) and traditional k-means are very similar. The difference is that online k-means allows you to update the model as new data is received. Online k-means should be used when you expect the data to be received one by one (or maybe in chunks). This allows you to update your model as you get more ...


11

If I understand you correctly, this is the case of multiple linear regression with sparse data (sparse regression). Assuming that, I hope you will find the following resources useful. 1) NCSU lecture slides on sparse regression with overview of algorithms, notes, formulas, graphics and references to literature: http://www.stat.ncsu.edu/people/zhou/courses/...


11

You are correct that both models work on the same data without any problem. Both items operate on a matrix of user-item ratings. In the user-based approach the algorithm produces a rating for an item i by a user u by combining the ratings of other users u' that are similar to u. Similar here means that the two user's ratings have a high Pearson correlation ...


10

Short Answer Yes, logistic regression is a regression algorithm and it does predict a continuous outcome: the probability of an event. That we use it as a binary classifier is due to the interpretation of the outcome. Detail Logistic regression is a type of generalize linear regression model. In an ordinary linear regression model, a continuous outcome, ...


10

To answer your question it is important to understand the frame of reference you are looking for, if you are looking for what philosophically you are trying to achieve in model fitting, check out Rubens answer he does a good job of explaining that context. However, in practice your question is almost entirely defined by business objectives. To give a ...


10

Recently a friend of mine was asked whether decision tree algorithm a linear or nonlinear algorithm in an interview Decision trees is a non-linear classifier like the neural networks, etc. It is generally used for classifying non-linearly separable data. Even when you consider the regression example, decision tree is non-linear. For example, a linear ...


10

One very important difference is xgboost has implemented DART, the dropout regularization for regression trees. References Rashmi, K. V., & Gilad-Bachrach, R. (2015). Dart: Dropouts meet multiple additive regression trees. arXiv preprint arXiv:1505.01866.


9

I think a number of clustering algorithms that normally use a metric, do not actually rely on the metric properties (other than commutativity, but I think you'd have that here). For example, DBSCAN uses epsilon-neighborhoods around a point; there is nothing in there that specifically says the triangle inequality matters. So you can probably use DBSCAN, ...


9

Let me explain it using some examples for clear intuition: When do you use linear regression vs Decision Trees? Linear regression is a linear model, which means it works really nicely when the data has a linear shape. But, when the data has a non-linear shape, then a linear model cannot capture the non-linear features. So in this case, you can use the ...


9

I am no data scientist, only an aspiring one for two years, moving from my background in software engineering and mathematics. So I took some courses, had some interviews, read a lot on the subject online. My opinion: New algorithms are developed in research centers and at universities. And even then, most algorithms used in companies are already developed, ...


8

In general regression models (any) can behave in an arbitrary way beyond the domain spanned by training samples. In particular, they are free to assume linearity of the modeled function, so if you for instance train a regression model with points: X Y 10 0 20 1 30 2 it is reasonable to build a model f(x) = x/10-1, which for x<10 returns ...


8

Alex made a number of good points, though I might have to push back a bit on his implication that DBSCAN is the best clustering algorithm to use here. Depending on your implementation, and whether or not you're using accelerated indices (many implementations do not), your time and space complexity will both be O(n2), which is far from ideal. Personally, my ...


8

There are multiple ways to approach solving game playing problems. Some games can be solved by search algorithms for example. This works well for card and board games up to some level of complexity. For instance, IBM's Deep Blue was essentially a fast heuristic-driven search for optimal moves. However, probably the most generic machine learning algorithm ...


8

Here is what I built... Step 1: Store all the words in a Trie data structure. Wiki about trie. Step 2: Train an RNN or RNTN to get seq2seq mapping for words and store the model Step 3: Retrieve top n words with levenshtein distance with the. Wiki about LD with the word that you are trying to correct. Naive Approach: Calculating the edit distance between ...


7

Models are but abstractions of what is seen in real life. They are designed in order to abstract-away nitty-gritties of the real system in observation, while keeping sufficient information to support desired analysis. If a model is overfit, it takes into account too many details about what is being observed, and small changes on such object may cause the ...


7

First off, if your data has as many variations (in function of time, context, and others) as to make it hard to apply a single strategy to cope with it, you may be interested in doing a prior temporal/contextual/... characterization of the dataset. Characterizing data, i.e., extracting information about how the volume or specifics of the content varies ...


7

The original MacQueen k-means publication (the first to use the name "kmeans") is an online algorithm. MacQueen, J. B. (1967). "Some Methods for classification and Analysis of Multivariate Observations". Proceedings of 5th Berkeley Symposium on Mathematical Statistics and Probability 1. University of California Press. pp. 281–297 After assigning each ...


7

Aleksandr's answer is completely correct. However, the way the question is posed implies that this is a straightforward ordinary least squares regression question: minimizing the sum of squared residuals between a dependent variable and a linear combination of predictors. Now, while there may be many zeros in your design matrix, your system as such is not ...


7

First of all, some basics of classification (and in general any supervised ML tasks), just to make sure we have same set of concepts in mind. Any supervised ML algorithm consists of at least 2 components: Dataset to train and test on. Algorithm(s) to handle these data. Training dataset consists of a set of pairs (x, y), where x is a vector of features ...


7

I would take a look at t-digest algorithm. It's been merged into mahout and also a part of some other libraries for big data streaming. You can get more about this algorithm particularly and big data anomaly detection in general in next resources: Practical machine learning anomaly detection book. Webinar: Anomaly Detection When You Don't Know What You Need ...


7

What you have is a sequence of events according to time so do not hesitate to call it Time Series! Clustering in time series has 2 different meanings: Segmentation of time series i.e. you want to segment an individual time series into different time intervals according to internal similarities. Time series clustering i.e. you have several time series and ...


Only top voted, non community-wiki answers of a minimum length are eligible