Questions tagged [machine-learning]

Machine Learning is a subfield of computer science that draws on elements from algorithmic analysis, computational statistics, mathematics, optimization, etc. It is mainly concerned with the use of data to construct models that have high predictive/forecasting ability. Topics include modeling building, applications, theory, etc.

Filter by
Sorted by
Tagged with
6
votes
1answer
189 views

Kappa From Combined Confusion Matrices

I am trying to evaluate and compare several different machine learning models built with different parameters (i.e. downsampling, outlier removal) and different classifiers (i.e. Bayes Net, SVM, ...
-6
votes
2answers
287 views

which programming language has a large library that can do machine learning algorithm, R, matlab or python [closed]

As what I described in the title, we are especially interested in those for dealing with big data----ts efficiency and stability, and used in industry not in experiment or university. Thanks!
13
votes
2answers
4k views

What features are generally used from Parse trees in classification process in NLP?

I am exploring different types of parse tree structures. The two widely known parse tree structures are a) Constituency based parse tree and b) Dependency based parse tree structures. I am able to ...
-3
votes
2answers
1k views

Which one will be the dominating programming language for next 5 years for analytics , machine learning . R or python or SAS [closed]

Which one will be the dominating programming language for next 5 years for analytics , machine learning . R verses python verses SAS. Advantage and disadvantage.
32
votes
4answers
44k views

Do Random Forest overfit?

I have been reading around about Random Forests but I cannot really find a definitive answer about the problem of overfitting. According to the original paper of Breiman, they should not overfit when ...
10
votes
2answers
560 views

implementing temporal difference in chess

I have been developing a chess program which makes use of alpha-beta pruning algorithm and an evaluation function that evaluates positions using the following features namely material, kingsafety, ...
3
votes
0answers
112 views

How can I model open environment in reinforcement learning? [closed]

I'm studying reinforcement learning in order to implement a kind of time series pattern analyzer such as market. The most examples I have seen are based on the maze environment. But in real market ...
3
votes
1answer
215 views

Query similarity: how much data is used in practice?

I recently read Similarity Measures for Short Segments of Text (Metzler et al.). It describes basic methods for measuring query similarity, and in the paper, the data consists of queries and their ...
10
votes
4answers
1k views

What initial steps should I use to make sense of large data sets, and what tools should I use?

Caveat: I am a complete beginner when it comes to machine learning, but eager to learn. I have a large dataset and I'm trying to find pattern in it. There may / may not be correlation across the data,...
23
votes
2answers
15k views

Text categorization: combining different kind of features

The problem I am tackling is categorizing short texts into multiple classes. My current approach is to use tf-idf weighted term frequencies and learn a simple linear classifier (logistic regression). ...
4
votes
4answers
9k views

Can I use unsupervised learning followed by supervised learning?

I have a question about classifying documents using supervised learning and unsupervised learning. For example: - I have a bunch of documents talking about football. As we know, football has a ...
11
votes
1answer
269 views

Solutions for Continuous Online Cluster Identification?

Let me show you an example of a hypothetical online clustering application: At time n points 1,2,3,4 are allocated to the blue cluster A and points b,5,6,7 are allocated to the red cluster B. At ...
20
votes
3answers
7k views

Nearest neighbors search for very high dimensional data

I have a big sparse matrix of users and items they like (in the order of 1M users and 100K items, with a very low level of sparsity). I'm exploring ways in which I could perform kNN search on it. ...
14
votes
2answers
5k views

Sentiment data for Emoji

For experimenting we'd like to use the Emoji embedded in many Tweets as a ground truth/training data for simple quantitative senitment analysis. Tweets are usually too unstructured for NLP to work ...
8
votes
1answer
720 views

Coreference Resolution for German Texts

Does anyone know a libarary for performing coreference resolution on German texts? As far as I know, OpenNLP and Stanford NLP are not able to perform coreference resolution for German Texts. The ...
5
votes
1answer
2k views

Classifying Java exceptions

We have a classification algorithm to categorize Java exceptions in Production. This algorithm is based on hierarchical human defined rules so when a bunch of text forming an exception comes up, it ...
4
votes
4answers
310 views

Handling huge dataset imbalance (2 class values) and appropriate ML algorithm

I have train and test sets of chronological data consisting of 305000 instances and 70000,appropriately. There are 15 features in each instance and only 2 possible class values ( NEW,OLD). The problem ...
3
votes
1answer
58 views

Can I classify set of documents using classifying method using limited number of concepts ?

I have set of documents and I want classify them to true and false My question is I have to take the whole words in the documents then I classify them depend on the similarity words in these ...
11
votes
2answers
3k views

Solving a system of equations with sparse data

I am attempting to solve a set of equations which has 40 independent variables (x1, ..., x40) and one dependent variable (y). The total number of equations (number of rows) is ~300, and I want to ...
4
votes
3answers
964 views

When is there enough data for generalization?

Are there any general rules that one can use to infer what can be learned/generalized from a particular data set? Suppose the dataset was taken from a sample of people. Can these rules be stated as ...
12
votes
3answers
994 views

Predicting next medical condition from past conditions in claims data

I am currently working with a large set of health insurance claims data that includes some laboratory and pharmacy claims. The most consistent information in the data set, however, is made up of ...
25
votes
3answers
43k views

Data Science Project Ideas [closed]

I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion. I have just started with Data Science and ...
53
votes
6answers
14k views

Should I go for a 'balanced' dataset or a 'representative' dataset?

My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
11
votes
3answers
2k views

Statistics + Computer Science = Data Science? [closed]

i want to become a data scientist. I studied applied statistics (actuarial science), so i have a great statistical background (regression, stochastic process, time series, just for mention a few). But ...
89
votes
4answers
78k views

Advantages of AUC vs standard accuracy

I was starting to look into area under curve(AUC) and am a little confused about its usefulness. When first explained to me, AUC seemed to be a great measure of performance but in my research I've ...
8
votes
4answers
1k views

Job title similarity

I'm trying to define a metric between job titles in IT field. For this I need some metric between words of job titles that are not appearing together in the same job title, e.g. metric between the ...
6
votes
2answers
993 views

Why should I care about seasonal data when I forecast?

I have a timeseries with hourly gas consumption. I want to use ARMA/ARIMA to forecast the consumption on the next hour, basing on the previous. Why should I analyze/find the seasonality (with Seasonal ...
11
votes
1answer
5k views

t-SNE Python implementation: Kullback-Leibler divergence

t-SNE, as in [1], works by progressively reducing the Kullback-Leibler (KL) divergence, until a certain condition is met. The creators of t-SNE suggests to use KL divergence as a performance criterion ...
62
votes
8answers
81k views

Clustering geo location coordinates (lat,long pairs)

What is the right approach and clustering algorithm for geolocation clustering? I'm using the following code to cluster geolocation coordinates: ...
20
votes
2answers
10k views

How to increase accuracy of classifiers?

I am using OpenCV letter_recog.cpp example to experiment on random trees and other classifiers. This example has implementations of six classifiers - random trees, boosting, MLP, kNN, naive Bayes and ...
31
votes
4answers
15k views

Meaning of latent features?

I am learning about matrix factorization for recommender systems and I am seeing the term latent features occurring too frequently but I am unable to understand ...
3
votes
2answers
2k views

ARMA/ARIMA on energy forecasts timeseries: strange prediction

I'm trying to use ARMA/ARIMA with the statsmodel Python package, in order to predict the gas consumption. I tried with a dataset of this format: Using only the gas column. ...
18
votes
2answers
6k views

How to choose the features for a neural network?

I know that there is no a clear answer for this question, but let's suppose that I have a huge neural network, with a lot of data and I want to add a new feature in input. The "best" way ...
7
votes
2answers
2k views

Efficient dynamic clustering

I have a set of datapoints from the unit interval (i.e. 1-dimensional dataset with numerical values). I receive some additional datapoints online, and moreover the value of some datapoints might ...
147
votes
17answers
122k views

Best python library for neural networks

I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?
5
votes
3answers
4k views

Stochastic gradient descent in logistic regression

I am very new to machine learning and in my first project have stumbled across a lot of issues which I really want to get through. I'm using logistic regression with R's ...
8
votes
2answers
190 views

What are some standard ways of computing the distance between individual search queries?

I made a similar question asking about distance between "documents" (Wikipedia articles, news stories, etc.). I made this a separate question because search queries are considerably smaller than ...
37
votes
5answers
10k views

What are some standard ways of computing the distance between documents?

When I say "document", I have in mind web pages like Wikipedia articles and news stories. I prefer answers giving either vanilla lexical distance metrics or state-of-the-art semantic distance metrics,...
8
votes
2answers
169 views

Linearly increasing data with manual reset

I have a linearly increasing time series dataset of a sensor, with value ranges between 50 and 150. I've implemented a Simple Linear Regression algorithm to fit a regression line on such data, and I'm ...
11
votes
1answer
598 views

Fisher Scoring v/s Coordinate Descent for MLE in R

R base function glm() uses Fishers Scoring for MLE, while the glmnet appears to use the coordinate descent method to solve the ...
29
votes
4answers
28k views

What algorithms should I use to perform job classification based on resume data?

Note that I am doing everything in R. The problem goes as follow: Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
7
votes
1answer
1k views

Linear Regression in R Mapreduce(RHadoop)

I m new to RHadoop and also to RMR... I had an requirement to write a Mapreduce Job in R Mapreduce. I have Tried writing but While executing this it gives an Error. Tring to read the file from hdfs ...
10
votes
1answer
170 views

Prediction with non-atomic features

I would like to use non-atomic data, as a feature for a prediction. Suppose I have a Table with these features: ...
3
votes
1answer
1k views

Data preparation and machine learning algorithm for click prediction

I am new to machine learning. I have a task at hand of predicting click probability given user information like city, state, OS version, OS family, device, browser family, browser version, etc. I have ...
10
votes
3answers
141 views

Handling a regularly increasing feature set

I'm working on a fraud detection system. In this field, new frauds appear regularly, so that new features have to be added to the model on ongoing basis. I wonder what is the best way to handle it (...
2
votes
1answer
852 views

Difference Between Hadoop Mapreduce(Java) and RHadoop mapreduce

I understand Hadoop MapReduce and its features but I am confused about R MapReduce. One difference I have read is that R utilizes maximum RAM. So do perform parallel processing integrated R with ...
6
votes
1answer
2k views

How to normalize results of Singular Value Decomposition (SVD) between 0 and 1?

I'm building a recommender system and using SVD as one of the preprocessing techniques. However, I want to normalize all my preprocessed data between 0 and 1 because all of my similarity measures (...
2
votes
3answers
2k views

How to use neural networks with large and variable number of inputs?

I'm new to machine learning, but I have an interesting problem. I have a large sample of people and visited sites. Some people have indicated gender, age, and other parameters. Now I want to restore ...
5
votes
4answers
579 views

How does Google categorize results from its image search?

While doing a Google image search, the page displays some figured out categories for the images of the topic being searched for. I'm interested in learning how this works, and how it chooses and ...
10
votes
3answers
12k views

Why does Gradient Boosting regression predict negative values when there are no negative y-values in my training set?

As I increase the number of trees in scikit learn's GradientBoostingRegressor, I get more negative predictions, even though there are no negative values in my ...