Questions tagged [machine-learning]
Machine Learning is a subfield of computer science that draws on elements from algorithmic analysis, computational statistics, mathematics, optimization, etc. It is mainly concerned with the use of data to construct models that have high predictive/forecasting ability. Topics include modeling building, applications, theory, etc.
11,119
questions
5
votes
1
answer
2k
views
Classifying Java exceptions
We have a classification algorithm to categorize Java exceptions in Production.
This algorithm is based on hierarchical human defined rules so when a bunch of text forming an exception comes up, it ...
- 153
4
votes
4
answers
328
views
Handling huge dataset imbalance (2 class values) and appropriate ML algorithm
I have train and test sets of chronological data consisting of 305000 instances and 70000,appropriately. There are 15 features in each instance and only 2 possible class values ( NEW,OLD). The problem ...
- 803
3
votes
1
answer
58
views
Can I classify set of documents using classifying method using limited number of concepts ?
I have set of documents and I want classify them to true and false
My question is I have to take the whole words in the documents then I classify them depend on the similarity words in these ...
- 33
12
votes
2
answers
3k
views
Solving a system of equations with sparse data
I am attempting to solve a set of equations which has 40 independent variables (x1, ..., x40) and one dependent variable (y). The total number of equations (number of rows) is ~300, and I want to ...
- 933
4
votes
3
answers
2k
views
When is there enough data for generalization?
Are there any general rules that one can use to infer what can be learned/generalized from a particular data set? Suppose the dataset was taken from a sample of people. Can these rules be stated as ...
- 811
12
votes
3
answers
1k
views
Predicting next medical condition from past conditions in claims data
I am currently working with a large set of health insurance claims data that includes some laboratory and pharmacy claims. The most consistent information in the data set, however, is made up of ...
- 123
28
votes
3
answers
43k
views
Data Science Project Ideas [closed]
I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion.
I have just started with Data Science and ...
- 383
58
votes
6
answers
15k
views
Should I go for a 'balanced' dataset or a 'representative' dataset?
My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
- 683
11
votes
3
answers
2k
views
Statistics + Computer Science = Data Science? [closed]
i want to become a data scientist. I studied applied statistics (actuarial science), so i have a great statistical background (regression, stochastic process, time series, just for mention a few). But ...
- 163
97
votes
4
answers
101k
views
Advantages of AUC vs standard accuracy
I was starting to look into area under curve(AUC) and am a little confused about its usefulness. When first explained to me, AUC seemed to be a great measure of performance but in my research I've ...
- 1,073
8
votes
4
answers
2k
views
Job title similarity
I'm trying to define a metric between job titles in IT field. For this I need some metric between words of job titles that are not appearing together in the same job title, e.g. metric between the ...
- 181
6
votes
2
answers
1k
views
Why should I care about seasonal data when I forecast?
I have a timeseries with hourly gas consumption. I want to use ARMA/ARIMA to forecast the consumption on the next hour, basing on the previous. Why should I analyze/find the seasonality (with Seasonal ...
- 1,667
11
votes
1
answer
6k
views
t-SNE Python implementation: Kullback-Leibler divergence
t-SNE, as in [1], works by progressively reducing the Kullback-Leibler (KL) divergence, until a certain condition is met.
The creators of t-SNE suggests to use KL divergence as a performance criterion ...
- 113
66
votes
9
answers
99k
views
Clustering geo location coordinates (lat,long pairs)
What is the right approach and clustering algorithm for geolocation clustering?
I'm using the following code to cluster geolocation coordinates:
...
- 803
22
votes
2
answers
12k
views
How to increase accuracy of classifiers?
I am using OpenCV letter_recog.cpp example to experiment on random trees and other classifiers. This example has implementations of six classifiers - random trees, boosting, MLP, kNN, naive Bayes and ...
- 323
37
votes
4
answers
19k
views
Meaning of latent features?
I am learning about matrix factorization for recommender systems and I am seeing the term latent features occurring too frequently but I am unable to understand ...
- 719
3
votes
2
answers
2k
views
ARMA/ARIMA on energy forecasts timeseries: strange prediction
I'm trying to use ARMA/ARIMA with the statsmodel Python package, in order to predict the gas consumption. I tried with a dataset of this format:
Using only the gas column.
...
- 1,667
22
votes
2
answers
7k
views
How to choose the features for a neural network?
I know that there is no a clear answer for this question, but let's suppose that I have a huge neural network, with a lot of data and I want to add a new feature in input. The "best" way ...
- 1,667
7
votes
2
answers
3k
views
Efficient dynamic clustering
I have a set of datapoints from the unit interval (i.e. 1-dimensional dataset with numerical values). I receive some additional datapoints online, and moreover the value of some datapoints might ...
- 83
150
votes
17
answers
126k
views
Best python library for neural networks
I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?
5
votes
3
answers
5k
views
Stochastic gradient descent in logistic regression
I am very new to machine learning and in my first project have stumbled across a lot of issues which I really want to get through.
I'm using logistic regression with R's ...
- 81
8
votes
2
answers
194
views
What are some standard ways of computing the distance between individual search queries?
I made a similar question asking about distance between "documents" (Wikipedia articles, news stories, etc.). I made this a separate question because search queries are considerably smaller than ...
- 811
39
votes
5
answers
11k
views
What are some standard ways of computing the distance between documents?
When I say "document", I have in mind web pages like Wikipedia articles and news stories. I prefer answers giving either vanilla lexical distance metrics or state-of-the-art semantic distance metrics,...
- 811
8
votes
2
answers
202
views
Linearly increasing data with manual reset
I have a linearly increasing time series dataset of a sensor, with value ranges between 50 and 150. I've implemented a Simple Linear Regression algorithm to fit a regression line on such data, and I'm ...
- 183
11
votes
1
answer
677
views
Fisher Scoring v/s Coordinate Descent for MLE in R
R base function glm() uses Fishers Scoring for MLE, while the glmnet appears to use the coordinate descent method to solve the ...
- 111
31
votes
4
answers
29k
views
What algorithms should I use to perform job classification based on resume data?
Note that I am doing everything in R.
The problem goes as follow:
Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
- 431
7
votes
1
answer
1k
views
Linear Regression in R Mapreduce(RHadoop)
I m new to RHadoop and also to RMR...
I had an requirement to write a Mapreduce Job in R Mapreduce. I have Tried writing but While executing this it gives an Error.
Tring to read the file from hdfs
...
- 101
10
votes
1
answer
178
views
Prediction with non-atomic features
I would like to use non-atomic data, as a feature for a prediction.
Suppose I have a Table with these features:
...
- 101
3
votes
1
answer
1k
views
Data preparation and machine learning algorithm for click prediction
I am new to machine learning. I have a task at hand of predicting click probability given user information like city, state, OS version, OS family, device, browser family, browser version, etc. I have ...
- 81
10
votes
3
answers
151
views
Handling a regularly increasing feature set
I'm working on a fraud detection system. In this field, new frauds appear regularly, so that new features have to be added to the model on ongoing basis.
I wonder what is the best way to handle it (...
- 103
2
votes
1
answer
857
views
Difference Between Hadoop Mapreduce(Java) and RHadoop mapreduce
I understand Hadoop MapReduce and its features but I am confused about R MapReduce.
One difference I have read is that R utilizes maximum RAM. So do perform parallel processing integrated R with ...
- 101
6
votes
1
answer
3k
views
How to normalize results of Singular Value Decomposition (SVD) between 0 and 1?
I'm building a recommender system and using SVD as one of the preprocessing techniques.
However, I want to normalize all my preprocessed data between 0 and 1 because all of my similarity measures (...
- 283
2
votes
3
answers
2k
views
How to use neural networks with large and variable number of inputs?
I'm new to machine learning, but I have an interesting problem. I have a large sample of people and visited sites. Some people have indicated gender, age, and other parameters. Now I want to restore ...
- 41
5
votes
4
answers
645
views
How does Google categorize results from its image search?
While doing a Google image search, the page displays some figured out categories for the images of the topic being searched for. I'm interested in learning how this works, and how it chooses and ...
- 51
10
votes
3
answers
18k
views
Why does Gradient Boosting regression predict negative values when there are no negative y-values in my training set?
As I increase the number of trees in scikit learn's GradientBoostingRegressor, I get more negative predictions, even though there are no negative values in my ...
- 209
17
votes
5
answers
3k
views
Detecting cats visually by means of anomaly detection
I have a hobby project which I am contemplating committing to as a way of increasing my so far limited experience of machine learning. I have taken and completed the Coursera MOOC on the topic. My ...
- 273
11
votes
4
answers
4k
views
Feature Extraction Technique - Summarizing a Sequence of Data
I often am building a model (classification or regression) where I have some predictor variables that are sequences and I have been trying to find technique recommendations for summarizing them in the ...
- 702
5
votes
2
answers
364
views
What are your favorite sources for news about Machine Learning and Data Science? [closed]
Data Science and Machine Learning include a lot of different topics and it´s hard to stay up-to-date about all the news about papers, researches or new tutorials and tools.
What sources do you use to ...
- 75
16
votes
2
answers
6k
views
Binary classification model for unbalanced data
I have a dataset with the following specifications:
Training dataset with 193,176 samples with 2,821 positives
Test Dataset with 82,887 samples with 673 positives
There are 10 features.
I want to ...
- 3,895
5
votes
2
answers
1k
views
How to select algorithms for ensemble methods?
There is a general recommendation that algorithms in ensemble learning combinations should be different in nature. Is there a classification table, a scale or some rules that allow to evaluate how far ...
- 313
-6
votes
1
answer
227
views
Online k-means explanation [closed]
Please, could someone recommend a paper or blog post that describes the online k-means algorithm.
- 121
23
votes
4
answers
708
views
What statistical model should I use to analyze the likelihood that a single event influenced longitudinal data
I am trying to find a formula, method, or model to use to analyze the likelihood that a specific event influenced some longitudinal data. I am having difficultly figuring out what to search for on ...
- 333
27
votes
4
answers
14k
views
Word2Vec for Named Entity Recognition
I'm looking to use google's word2vec implementation to build a named entity recognition system. I've heard that recursive neural nets with back propagation through structure are well suited for named ...
- 2,029
12
votes
4
answers
15k
views
Is GLM a statistical or machine learning model?
I thought that generalized linear model (GLM) would be considered a statistical model, but a friend told me that some papers classify it as a machine learning technique. Which one is true (or more ...
- 313
9
votes
4
answers
3k
views
Suggest text classifier training datasets
Which freely available datasets can I use to train a text classifier?
We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified ...
- 121
16
votes
4
answers
2k
views
What are the implications for training a Tree Ensemble with highly biased datasets?
I have a highly biased binary dataset - I have 1000x more examples of the negative class than the positive class. I would like to train a Tree Ensemble (like Extra Random Trees or a Random Forest) on ...
- 418
9
votes
1
answer
244
views
Learning signal encoding
I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there ...
- 1,824
14
votes
4
answers
2k
views
Looking for example infrastructure stacks/workflows/pipelines
I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
- 143
10
votes
4
answers
2k
views
Online machine learning tutorial
Does anyone know some good tutorials on online machine learning technics?
I.e. how it can be used in real-time environments, what are key differences compared to normal machine learning methods etc.
...
- 1,071
16
votes
3
answers
12k
views
Best way to classify datasets with mixed types of attributes
I would like to know what is the best way to classify a data set composed of mixed types of attributes, for example, textual and numerical. I know I can convert textual to boolean, but the vocabulary ...
- 161