136

The standard k-means algorithm isn't directly applicable to categorical data, for various reasons. The sample space for categorical data is discrete, and doesn't have a natural origin. A Euclidean distance function on such a space isn't really meaningful. As someone put it, "The fact a snake possesses neither wheels nor legs allows us to say nothing about ...


50

SVM is a powerful classifier. It has some nice advantages (which I guess were responsible for its popularity)... These are: Efficiency: Only the support vectors play a role in determining the classification boundary. All other points from the training set needn't be stored in memory. The so-called power of kernels: With appropriate kernels you can transform ...


49

There's a number of different ways of going about this depending on exactly how much semantic information you want to retain and how easy your documents are to tokenize (html documents would probably be pretty difficult to tokenize, but you could conceivably do something with tags and context.) Some of them have been mentioned by ffriend, and the paragraph ...


42

First of all, dimensionality reduction is used when you have many covariated dimensions and want to reduce problem size by rotating data points into new orthogonal basis and taking only axes with largest variance. With 8 variables (columns) your space is already low-dimensional, reducing number of variables further is unlikely to solve technical issues with ...


39

It's easier to start with your second question and then go to the first. Bagging Random Forest is a bagging algorithm. It reduces variance. Say that you have very unreliable models, such as Decision Trees. (Why unreliable? Because if you change your data a little bit, the decision tree created can be very different.) In such a case, you can build a robust ...


36

Anomaly Detection or Event Detection can be done in different ways: Basic Way Derivative! If the deviation of your signal from its past & future is high you most probably have an event. This can be extracted by finding large zero crossings in derivative of the signal. Statistical Way Mean of anything is its usual, basic behavior. if something ...


30

No, despite their names they are not equivalent or even that similar. Gini impurity is a measure of misclassification, which applies in a multiclass classifier context. Gini coefficient applies to binary classification and requires a classifier that can in some way rank examples according to the likelihood of being in a positive class. Both could be ...


27

Why to use deep networks? Let's first try to solve very simple classification task. Say, you moderate a web forum which is sometimes flooded with spam messages. These messages are easily identifiable - most often they contain specific words like "buy", "porn", etc. and a URL to outer resources. You want to create filter that will alert you about such ...


25

@statsRus starts to lay the groundwork for your answer in another question https://datascience.stackexchange.com/questions/1/what-characterises-the-difference-between-data-science-and-statistics: Data collection: web scraping and online surveys Data manipulation: recoding messy data and extracting meaning from linguistic and social network data ...


25

At the expense of over-simplication, latent features are 'hidden' features to distinguish them from observed features. Latent features are computed from observed features using matrix factorization. An example would be text document analysis. 'words' extracted from the documents are features. If you factorize the data of words you can find 'topics', where '...


23

In my opinion, there are solutions to deal with categorical data in clustering. R comes with a specific distance for categorical data. This distance is called Gower (http://www.rdocumentation.org/packages/StatMatch/versions/1.2.0/topics/gower.dist) and it works pretty well.


22

Well the names are pretty straight-forward and should give you a clear idea of vector representations. The Word2Vec Algorithm builds distributed semantic representation of words. There are two main approaches to training, Distributed Bag of Words and The skip gram model. One involves predicting the context words using a centre word, while the other ...


21

Item Based Algorithm for every item i that u has no preference for yet for every item j that u has a preference for compute a similarity s between i and j add u's preference for j, weighted by s, to a running average return the top items, ranked by weighted average User Based Algorithm for every item i that u has no preference for yet for ...


20

(In addition to the excellent answer by Tim Goodman) The choice of k-modes is definitely the way to go for stability of the clustering algorithm used. The clustering algorithm is free to choose any distance metric / similarity score. Euclidean is the most popular. But any other metric can be used that scales according to the data distribution in each ...


18

This question seems really about representation, and not so much about clustering. Categorical data is a problem for most algorithms in machine learning. Suppose, for example, you have some categorical variable called "color" that could take on the values red, blue, or yellow. If we simply encode these numerically as 1,2, and 3 respectively, our algorithm ...


15

After reading your question, I became curious about the topic of time series clustering and dynamic time warping (DTW). So, I have performed a limited search and came up with basic understanding (for me) and the following set of IMHO relevant references (for you). I hope that you'll find this useful, but keep in mind that I have intentionally skipped ...


15

h2o has an anomaly detection module and traditionally the code is available in R.However beyond version 3 it has similar module available in python as well,and since h2o is open source it might fit your bill. You can see an working example over here import sys sys.path.insert(1,"../../../") import h2o def anomaly(ip, port): h2o.init(ip, port) ...


15

They are probably using "leave one out encoding" to refer to Owen Zhang's strategy. From: https://www.kaggle.com/c/caterpillar-tube-pricing/forums/t/15748/strategies-to-encode-categorical-variables-with-many-categories The encoded column is not a conventional dummy variable, but instead is the mean response over all rows for this categorical level, ...


14

News outlets tend to use "Big Data" pretty loosely. Vendors usually provide case studies surrounding their specific products. There aren't a lot out there for open source implementations, but they do get mentioned. For instance, Apache isn't going to spend a lot of time building a case study on hadoop, but vendors like Cloudera and Hortonworks probably ...


14

Yes, you are correct that the dominant difference between the area under the curve of a receiver operator characteristic curve (ROC-AUC) and the area under the curve of a Precision-Recall curve (PR-AUC) lies in its tractability for unbalanced classes. They are very similar and have been shown to contain essentially the same information, however PR curves ...


14

For a specific model you feed it data, choose the features, choose hyperparameters etcetera. Compared to the reality it makes a three types of mistakes: Bias (due to too low model complexity, a sampling bias in your data) Variance (due to noise in your data, overfitting of your data) Randomness of the reality you are trying to predict (or lack of ...


13

I recently developed a toolbox: Python Outlier Detection toolbox (PyOD). See GitHub. It is designed for identifying outlying objects in data with both unsupervised and supervised approaches. PyOD is featured for: Unified APIs, detailed documentation, and interactive examples across various algorithms. Advanced models, including Neural Networks/Deep ...


13

One of the benefits of decision trees is that ordinal (continuous or discrete) input data does not require any significant preprocessing. In fact, the results should be consistent regardless of any scaling or translational normalization, since the trees can choose equivalent splitting points. The best preprocessing for decision trees is typically whatever is ...


12

Instead of scraping, you might try to get the data directly here: http://www.imdb.com/interfaces. It looks like they have data available via ftp for movies, actors, etc.


12

An approach that yields more consistent results is K-means++. This approach acknowledges that there is probably a better choice of initial centroid locations than simple random assignment. Specifically, K-means tends to perform better when centroids are seeded in such a way that doesn't clump them together in space. In short, the method is as follows: ...


11

What @Clayton posted seems about right to me, for those terms, and for "data mining" being one tool of the data scientist. However, I haven't really used the term "data collection," and it doesn't strike me as synonymous with "data mining." My own answer to your question: no, the terms aren't the same. Definitions may be loose in this field, but I haven't ...


11

For the record, I think this is the type of question that's perfect for the data science Stack Exchange. I hope we get a bunch of real world examples of data problems and several perspectives on how best to solve them. I would encourage you not to use p-values as they can be pretty misleading (1, 2). My approach hinges on you being able to summarize traffic ...


11

I will try to answer your questions, but before I'd like to note that using term "large dataset" is misleading, as "large" is a relative concept. You have to provide more details. If you're dealing with bid data, then this fact will most likely affect selection of preferred tools, approaches and algorithms for your data analysis. I hope that the following ...


11

You are correct that both models work on the same data without any problem. Both items operate on a matrix of user-item ratings. In the user-based approach the algorithm produces a rating for an item i by a user u by combining the ratings of other users u' that are similar to u. Similar here means that the two user's ratings have a high Pearson correlation ...


11

Right now, I only have time for a very brief answer, but I'll try to expand on it later on. What you want to do is a clustering, since you want to discover some labels for your data. (As opposed to a classification, where you would have labels for at least some of the data and you would like to label the rest). In order to perform a clustering on your ...


Only top voted, non community-wiki answers of a minimum length are eligible