Questions tagged [machine-learning]

Methods and principles of building "computer systems that automatically improve with experience."

Filter by
Sorted by
Tagged with
6
votes
2answers
326 views

Statistical Commute Analysis in Java

I have a rather large commute every day - it ranges between about an hour and about an hour and half of driving. I have been tracking my driving times, and want to continue to do so. I am capturing ...
6
votes
2answers
419 views

Looking for algebras designed to transform time series

I am looking for information on (formal) algebraic systems that can be used to transform time-series - in either a practical or academic context. I hope that there exists (at least one) small, ...
29
votes
4answers
12k views

Quick guide into training highly imbalanced data sets

I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
8
votes
2answers
369 views

How to build a textual search engine?

I am having an HTML string and want to find out if a word I supply is relevant in that string. Relevancy could be measured based on frequency in the text. An example to illustrate my problem: ...
11
votes
2answers
660 views

Neural net for server monitoring

I'm looking at pybrain for taking server monitor alarms and determining the root cause of a problem. I'm happy with training it using supervised learning and curating the training data sets. The data ...
2
votes
1answer
167 views

Creating obligatory combinations of variables for drawing by random forest

Problem For my machine learning task, I create a set of predictors. Predictors come in "bundles" - multi-dimensional measurements (3 or 4 - dimensional in my case). The hole "bundle" makes sense ...
14
votes
1answer
2k views

Machine learning libraries for Ruby

Are there any machine learning libraries for Ruby that are relatively complete (including a wide variety of algorithms for supervised and unsupervised learning), robustly tested, and well-documented? ...
3
votes
2answers
170 views

non query-based document ranking

We have ~500 biomedical documents each of some 1-2 MB. We want to use a non query-based method to rank the documents in order of their unique content score. I'm calling it "unique content" because our ...
2
votes
0answers
396 views

ANOVA RBF kernel returns very poor results

I was curious about the ANOVA RBF kernel provided by kernlab package available in R. I tested it with a numeric dataSet of 34 input variables and one output variable. For each variable I have 700 ...
12
votes
3answers
6k views

Unstructured text classification

I'm going to classify unstructured text documents, namely web sites of unknown structure. The number of classes to which I am classifying is limited (at this point, I believe there is no more than ...
5
votes
1answer
5k views

Polynomial Kernel Parameters in SVMs

In SVMs the polynomial kernel is defined as: (scale * crossprod(x, y) + offset)^degree How do the scale and offset parameters affect the model and what range should they be in? (intuitively please) ...
10
votes
2answers
3k views

Libraries for Online Machine Learning

I am looking for packages (either in python, R, or a standalone package) to perform online learning to predict stock data. I have found and read about Vowpal Wabbit (https://github.com/JohnLangford/...
6
votes
1answer
174 views

Kappa From Combined Confusion Matrices

I am trying to evaluate and compare several different machine learning models built with different parameters (i.e. downsampling, outlier removal) and different classifiers (i.e. Bayes Net, SVM, ...
-5
votes
2answers
275 views

which programming language has a large library that can do machine learning algorithm, R, matlab or python [closed]

As what I described in the title, we are especially interested in those for dealing with big data----ts efficiency and stability, and used in industry not in experiment or university. Thanks!
13
votes
2answers
4k views

What features are generally used from Parse trees in classification process in NLP?

I am exploring different types of parse tree structures. The two widely known parse tree structures are a) Constituency based parse tree and b) Dependency based parse tree structures. I am able to ...
-2
votes
2answers
1k views

Which one will be the dominating programming language for next 5 years for analytics , machine learning . R or python or SAS [closed]

Which one will be the dominating programming language for next 5 years for analytics , machine learning . R verses python verses SAS. Advantage and disadvantage.
24
votes
4answers
28k views

Do Random Forest overfit?

I have been reading around about Random Forests but I cannot really find a definitive answer about the problem of overfitting. According to the original paper of Breiman, they should not overfit when ...
10
votes
2answers
454 views

implementing temporal difference in chess

I have been developing a chess program which makes use of alpha-beta pruning algorithm and an evaluation function that evaluates positions using the following features namely material, kingsafety, ...
3
votes
0answers
108 views

How can I model open environment in reinforcement learning? [closed]

I'm studying reinforcement learning in order to implement a kind of time series pattern analyzer such as market. The most examples I have seen are based on the maze environment. But in real market ...
3
votes
1answer
212 views

Query similarity: how much data is used in practice?

I recently read Similarity Measures for Short Segments of Text (Metzler et al.). It describes basic methods for measuring query similarity, and in the paper, the data consists of queries and their ...
10
votes
4answers
776 views

What initial steps should I use to make sense of large data sets, and what tools should I use?

Caveat: I am a complete beginner when it comes to machine learning, but eager to learn. I have a large dataset and I'm trying to find pattern in it. There may / may not be correlation across the data,...
19
votes
2answers
11k views

Text categorization: combining different kind of features

The problem I am tackling is categorizing short texts into multiple classes. My current approach is to use tf-idf weighted term frequencies and learn a simple linear classifier (logistic regression). ...
4
votes
4answers
7k views

Can I use unsupervised learning followed by supervised learning?

I have a question about classifying documents using supervised learning and unsupervised learning. For example: - I have a bunch of documents talking about football. As we know, football has a ...
11
votes
1answer
236 views

Solutions for Continuous Online Cluster Identification?

Let me show you an example of a hypothetical online clustering application: At time n points 1,2,3,4 are allocated to the blue cluster A and points b,5,6,7 are allocated to the red cluster B. At ...
17
votes
3answers
6k views

Nearest neighbors search for very high dimensional data

I have a big sparse matrix of users and items they like (in the order of 1M users and 100K items, with a very low level of sparsity). I'm exploring ways in which I could perform kNN search on it. ...
12
votes
2answers
5k views

Sentiment data for Emoji

For experimenting we'd like to use the Emoji embedded in many Tweets as a ground truth/training data for simple quantitative senitment analysis. Tweets are usually too unstructured for NLP to work ...
6
votes
1answer
553 views

Coreference Resolution for German Texts

Does anyone know a libarary for performing coreference resolution on German texts? As far as I know, OpenNLP and Stanford NLP are not able to perform coreference resolution for German Texts. The ...
5
votes
1answer
1k views

Classifying Java exceptions

We have a classification algorithm to categorize Java exceptions in Production. This algorithm is based on hierarchical human defined rules so when a bunch of text forming an exception comes up, it ...
4
votes
4answers
286 views

Handling huge dataset imbalance (2 class values) and appropriate ML algorithm

I have train and test sets of chronological data consisting of 305000 instances and 70000,appropriately. There are 15 features in each instance and only 2 possible class values ( NEW,OLD). The problem ...
3
votes
1answer
58 views

Can I classify set of documents using classifying method using limited number of concepts ?

I have set of documents and I want classify them to true and false My question is I have to take the whole words in the documents then I classify them depend on the similarity words in these ...
11
votes
2answers
2k views

Solving a system of equations with sparse data

I am attempting to solve a set of equations which has 40 independent variables (x1, ..., x40) and one dependent variable (y). The total number of equations (number of rows) is ~300, and I want to ...
4
votes
3answers
454 views

When is there enough data for generalization?

Are there any general rules that one can use to infer what can be learned/generalized from a particular data set? Suppose the dataset was taken from a sample of people. Can these rules be stated as ...
12
votes
3answers
956 views

Predicting next medical condition from past conditions in claims data

I am currently working with a large set of health insurance claims data that includes some laboratory and pharmacy claims. The most consistent information in the data set, however, is made up of ...
23
votes
3answers
42k views

Data Science Project Ideas [closed]

I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion. I have just started with Data Science and ...
48
votes
5answers
11k views

Should I go for a 'balanced' dataset or a 'representative' dataset?

My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
10
votes
3answers
2k views

Statistics + Computer Science = Data Science? [closed]

i want to become a data scientist. I studied applied statistics (actuarial science), so i have a great statistical background (regression, stochastic process, time series, just for mention a few). But ...
66
votes
3answers
51k views

Advantages of AUC vs standard accuracy

I was starting to look into area under curve(AUC) and am a little confused about its usefulness. When first explained to me, AUC seemed to be a great measure of performance but in my research I've ...
7
votes
4answers
1k views

Job title similarity

I'm trying to define a metric between job titles in IT field. For this I need some metric between words of job titles that are not appearing together in the same job title, e.g. metric between the ...
6
votes
2answers
836 views

Why should I care about seasonal data when I forecast?

I have a timeseries with hourly gas consumption. I want to use ARMA/ARIMA to forecast the consumption on the next hour, basing on the previous. Why should I analyze/find the seasonality (with Seasonal ...
10
votes
1answer
4k views

t-SNE Python implementation: Kullback-Leibler divergence

t-SNE, as in [1], works by progressively reducing the Kullback-Leibler (KL) divergence, until a certain condition is met. The creators of t-SNE suggests to use KL divergence as a performance criterion ...
53
votes
8answers
57k views

Clustering geo location coordinates (lat,long pairs)

What is the right approach and clustering algorithm for geolocation clustering? I'm using the following code to cluster geolocation coordinates: ...
16
votes
2answers
9k views

How to increase accuracy of classifiers?

I am using OpenCV letter_recog.cpp example to experiment on random trees and other classifiers. This example has implementations of six classifiers - random trees, boosting, MLP, kNN, naive Bayes and ...
24
votes
4answers
11k views

Meaning of latent features?

I am learning about matrix factorization for recommender systems and I am seeing the term latent features occurring too frequently but I am unable to understand ...
3
votes
2answers
2k views

ARMA/ARIMA on energy forecasts timeseries: strange prediction

I'm trying to use ARMA/ARIMA with the statsmodel Python package, in order to predict the gas consumption. I tried with a dataset of this format: Using only the gas column. ...
16
votes
2answers
5k views

How to choose the features for a neural network?

I know that there is no a clear answer for this question, but let's suppose that I have a huge neural network, with a lot of data and I want to add a new feature in input. The "best" way would be to ...
6
votes
2answers
1k views

Efficient dynamic clustering

I have a set of datapoints from the unit interval (i.e. 1-dimensional dataset with numerical values). I receive some additional datapoints online, and moreover the value of some datapoints might ...
132
votes
17answers
110k views

Best python library for neural networks

I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?
4
votes
3answers
3k views

Stochastic gradient descent in logistic regression

I am very new to machine learning and in my first project have stumbled across a lot of issues which I really want to get through. I'm using logistic regression with R's ...
8
votes
2answers
178 views

What are some standard ways of computing the distance between individual search queries?

I made a similar question asking about distance between "documents" (Wikipedia articles, news stories, etc.). I made this a separate question because search queries are considerably smaller than ...
34
votes
5answers
10k views

What are some standard ways of computing the distance between documents?

When I say "document", I have in mind web pages like Wikipedia articles and news stories. I prefer answers giving either vanilla lexical distance metrics or state-of-the-art semantic distance metrics,...