Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [machine-learning]

Methods and principles of building "computer systems that automatically improve with experience."

11
votes
1answer
220 views

Solutions for Continuous Online Cluster Identification?

Let me show you an example of a hypothetical online clustering application: At time n points 1,2,3,4 are allocated to the blue cluster A and points b,5,6,7 are allocated to the red cluster B. At ...
15
votes
3answers
4k views

Nearest neighbors search for very high dimensional data

I have a big sparse matrix of users and items they like (in the order of 1M users and 100K items, with a very low level of sparsity). I'm exploring ways in which I could perform kNN search on it. ...
12
votes
2answers
4k views

Sentiment data for Emoji

For experimenting we'd like to use the Emoji embedded in many Tweets as a ground truth/training data for simple quantitative senitment analysis. Tweets are usually too unstructured for NLP to work ...
5
votes
1answer
956 views

Classifying Java exceptions

We have a classification algorithm to categorize Java exceptions in Production. This algorithm is based on hierarchical human defined rules so when a bunch of text forming an exception comes up, it ...
4
votes
4answers
273 views

Handling huge dataset imbalance (2 class values) and appropriate ML algorithm

I have train and test sets of chronological data consisting of 305000 instances and 70000,appropriately. There are 15 features in each instance and only 2 possible class values ( NEW,OLD). The problem ...
3
votes
1answer
53 views

Can I classify set of documents using classifying method using limited number of concepts ?

I have set of documents and I want classify them to true and false My question is I have to take the whole words in the documents then I classify them depend on the similarity words in these ...
11
votes
2answers
2k views

Solving a system of equations with sparse data

I am attempting to solve a set of equations which has 40 independent variables (x1, ..., x40) and one dependent variable (y). The total number of equations (number of rows) is ~300, and I want to ...
4
votes
3answers
296 views

When is there enough data for generalization?

Are there any general rules that one can use to infer what can be learned/generalized from a particular data set? Suppose the dataset was taken from a sample of people. Can these rules be stated as ...
11
votes
3answers
873 views

Predicting next medical condition from past conditions in claims data

I am currently working with a large set of health insurance claims data that includes some laboratory and pharmacy claims. The most consistent information in the data set, however, is made up of ...
22
votes
3answers
40k views

Data Science Project Ideas [closed]

I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion. I have just started with Data Science and ...
39
votes
5answers
8k views

Should I go for a 'balanced' dataset or a 'representative' dataset?

My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
11
votes
3answers
2k views

Statistics + Computer Science = Data Science? [closed]

i want to become a data scientist. I studied applied statistics (actuarial science), so i have a great statistical background (regression, stochastic process, time series, just for mention a few). But ...
43
votes
3answers
31k views

Advantages of AUC vs standard accuracy

I was starting to look into area under curve(AUC) and am a little confused about its usefulness. When first explained to me, AUC seemed to be a great measure of performance but in my research I've ...
6
votes
4answers
895 views

Job title similarity

I'm trying to define a metric between job titles in IT field. For this I need some metric between words of job titles that are not appearing together in the same job title, e.g. metric between the ...
5
votes
2answers
739 views

Why should I care about seasonal data when I forecast?

I have a timeseries with hourly gas consumption. I want to use ARMA/ARIMA to forecast the consumption on the next hour, basing on the previous. Why should I analyze/find the seasonality (with Seasonal ...
9
votes
1answer
4k views

t-SNE Python implementation: Kullback-Leibler divergence

t-SNE, as in [1], works by progressively reducing the Kullback-Leibler (KL) divergence, until a certain condition is met. The creators of t-SNE suggests to use KL divergence as a performance criterion ...
38
votes
8answers
36k views

Clustering geo location coordinates (lat,long pairs)

What is the right approach and clustering algorithm for geolocation clustering? I'm using the following code to cluster geolocation coordinates: ...
17
votes
2answers
7k views

How to increase accuracy of classifiers?

I am using OpenCV letter_recog.cpp example to experiment on random trees and other classifiers. This example has implementations of six classifiers - random trees, boosting, MLP, kNN, naive Bayes and ...
19
votes
4answers
6k views

Meaning of latent features?

I am learning about matrix factorization for recommender systems and I am seeing the term latent features occurring too frequently but I am unable to understand ...
3
votes
2answers
2k views

ARMA/ARIMA on energy forecasts timeseries: strange prediction

I'm trying to use ARMA/ARIMA with the statsmodel Python package, in order to predict the gas consumption. I tried with a dataset of this format: Using only the gas column. ...
16
votes
2answers
4k views

How to choose the features for a neural network?

I know that there is no a clear answer for this question, but let's suppose that I have a huge neural network, with a lot of data and I want to add a new feature in input. The "best" way would be to ...
6
votes
2answers
802 views

Efficient dynamic clustering

I have a set of datapoints from the unit interval (i.e. 1-dimensional dataset with numerical values). I receive some additional datapoints online, and moreover the value of some datapoints might ...
121
votes
17answers
90k views

Best python library for neural networks

I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?
5
votes
3answers
2k views

Stochastic gradient descent in logistic regression

I am very new to machine learning and in my first project have stumbled across a lot of issues which I really want to get through. I'm using logistic regression with R's ...
8
votes
2answers
173 views

What are some standard ways of computing the distance between individual search queries?

I made a similar question asking about distance between "documents" (Wikipedia articles, news stories, etc.). I made this a separate question because search queries are considerably smaller than ...
30
votes
5answers
9k views

What are some standard ways of computing the distance between documents?

When I say "document", I have in mind web pages like Wikipedia articles and news stories. I prefer answers giving either vanilla lexical distance metrics or state-of-the-art semantic distance metrics,...
8
votes
2answers
134 views

Linearly increasing data with manual reset

I have a linearly increasing time series dataset of a sensor, with value ranges between 50 and 150. I've implemented a Simple Linear Regression algorithm to fit a regression line on such data, and I'm ...
11
votes
1answer
392 views

Fisher Scoring v/s Coordinate Descent for MLE in R

R base function glm() uses Fishers Scoring for MLE, while the glmnet appears to use the coordinate descent method to solve the ...
26
votes
4answers
20k views

What algorithms should I use to perform job classification based on resume data?

Note that I am doing everything in R. The problem goes as follow: Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
7
votes
1answer
1k views

Linear Regression in R Mapreduce(RHadoop)

I m new to RHadoop and also to RMR... I had an requirement to write a Mapreduce Job in R Mapreduce. I have Tried writing but While executing this it gives an Error. Tring to read the file from hdfs ...
10
votes
1answer
144 views

Prediction with non-atomic features

I would like to use non-atomic data, as a feature for a prediction. Suppose I have a Table with these features: ...
3
votes
1answer
916 views

Data preparation and machine learning algorithm for click prediction

I am new to machine learning. I have a task at hand of predicting click probability given user information like city, state, OS version, OS family, device, browser family, browser version, etc. I have ...
10
votes
3answers
109 views

Handling a regularly increasing feature set

I'm working on a fraud detection system. In this field, new frauds appear regularly, so that new features have to be added to the model on ongoing basis. I wonder what is the best way to handle it (...
3
votes
1answer
824 views

Difference Between Hadoop Mapreduce(Java) and RHadoop mapreduce

I understand Hadoop MapReduce and its features but I am confused about R MapReduce. One difference I have read is that R utilizes maximum RAM. So do perform parallel processing integrated R with ...
6
votes
1answer
1k views

How to normalize results of Singular Value Decomposition (SVD) between 0 and 1?

I'm building a recommender system and using SVD as one of the preprocessing techniques. However, I want to normalize all my preprocessed data between 0 and 1 because all of my similarity measures (...
3
votes
3answers
1k views

How to use neural networks with large and variable number of inputs?

I'm new to machine learning, but I have an interesting problem. I have a large sample of people and visited sites. Some people have indicated gender, age, and other parameters. Now I want to restore ...
5
votes
4answers
420 views

How does Google categorize results from its image search?

While doing a Google image search, the page displays some figured out categories for the images of the topic being searched for. I'm interested in learning how this works, and how it chooses and ...
8
votes
3answers
5k views

Why does Gradient Boosting regression predict negative values when there are no negative y-values in my training set?

As I increase the number of trees in scikit learn's GradientBoostingRegressor, I get more negative predictions, even though there are no negative values in my ...
16
votes
5answers
3k views

Detecting cats visually by means of anomaly detection

I have a hobby project which I am contemplating committing to as a way of increasing my so far limited experience of machine learning. I have taken and completed the Coursera MOOC on the topic. My ...
9
votes
4answers
3k views

Feature Extraction Technique - Summarizing a Sequence of Data

I often am building a model (classification or regression) where I have some predictor variables that are sequences and I have been trying to find technique recommendations for summarizing them in the ...
2
votes
2answers
309 views

What are your favorite sources for news about Machine Learning and Data Science? [closed]

Data Science and Machine Learning include a lot of different topics and it´s hard to stay up-to-date about all the news about papers, researches or new tutorials and tools. What sources do you use to ...
14
votes
1answer
5k views

Binary classification model for sparse / biased data

I have a dataset with following specifications: Training dataset with 193176 samples with 2821 positives Test Dataset with 82887 samples with 673 positives There are 10 features. I want to perform a ...
5
votes
2answers
862 views

How to select algorithms for ensemble methods?

There is a general recommendation that algorithms in ensemble learning combinations should be different in nature. Is there a classification table, a scale or some rules that allow to evaluate how far ...
-5
votes
1answer
151 views

Online k-means explanation [closed]

Please, could someone recommend a paper or blog post that describes the online k-means algorithm.
18
votes
3answers
464 views

What statistical model should I use to analyze the likelihood that a single event influenced longitudinal data

I am trying to find a formula, method, or model to use to analyze the likelihood that a specific event influenced some longitudinal data. I am having difficultly figuring out what to search for on ...
23
votes
4answers
9k views

Word2Vec for Named Entity Recognition

I'm looking to use google's word2vec implementation to build a named entity recognition system. I've heard that recursive neural nets with back propagation through structure are well suited for named ...
11
votes
4answers
8k views

Is GLM a statistical or machine learning model?

I thought that generalized linear model (GLM) would be considered a statistical model, but a friend told me that some papers classify it as a machine learning technique. Which one is true (or more ...
9
votes
4answers
2k views

Suggest text classifier training datasets

Which freely available datasets can I use to train a text classifier? We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified ...
13
votes
4answers
2k views

What are the implications for training a Tree Ensemble with highly biased datasets?

I have a highly biased binary dataset - I have 1000x more examples of the negative class than the positive class. I would like to train a Tree Ensemble (like Extra Random Trees or a Random Forest) on ...
9
votes
1answer
174 views

Learning signal encoding

I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there ...