All Questions
35,985
questions
11
votes
4
answers
4k
views
Feature Extraction Technique - Summarizing a Sequence of Data
I often am building a model (classification or regression) where I have some predictor variables that are sequences and I have been trying to find technique recommendations for summarizing them in the ...
- 702
0
votes
4
answers
999
views
First steps on a new cleaned dataset
What is the very first thing you do when you get your hands on a new data set (assuming it is cleaned and well structured)? Please share sample code snippets as I am sure this would be extremely ...
- 397
5
votes
2
answers
364
views
What are your favorite sources for news about Machine Learning and Data Science? [closed]
Data Science and Machine Learning include a lot of different topics and it´s hard to stay up-to-date about all the news about papers, researches or new tutorials and tools.
What sources do you use to ...
- 75
6
votes
7
answers
354
views
Good practices for manual modifications of data
More often than not, data I am working with is not 100% clean. Even if it is reasonably clean, still there are portions that need to be fixed.
When a fraction of data needs it, I write a script and ...
- 746
16
votes
2
answers
6k
views
Binary classification model for unbalanced data
I have a dataset with the following specifications:
Training dataset with 193,176 samples with 2,821 positives
Test Dataset with 82,887 samples with 673 positives
There are 10 features.
I want to ...
- 3,895
5
votes
2
answers
1k
views
How to select algorithms for ensemble methods?
There is a general recommendation that algorithms in ensemble learning combinations should be different in nature. Is there a classification table, a scale or some rules that allow to evaluate how far ...
- 313
9
votes
3
answers
2k
views
Score matrix string similarity
I have a load of documents, which have a load of key value pairs in them. The key might not be unique so there might be multiple keys of the same type with different values.
I want to compare the ...
- 95
-6
votes
1
answer
227
views
Online k-means explanation [closed]
Please, could someone recommend a paper or blog post that describes the online k-means algorithm.
- 121
5
votes
3
answers
295
views
Efficient solution of fmincg without providing gradient?
I'm working on multiclass logistic regression model with a large number of features (numFeatures > 100). Using a Maximum Likelihood Estimation based on the cost function and gradient, the fmincg ...
- 53
13
votes
2
answers
11k
views
Cross-validation: K-fold vs Repeated random sub-sampling
I wonder which type of model cross-validation to choose for classification problem: K-fold or random sub-sampling (bootstrap sampling)?
My best guess is to use 2/3 of the data set (which is ~1000 ...
- 5,424
7
votes
1
answer
236
views
Dealing with diverse text data
I'm currently working with a dataset with a wide range of document lengths -- anywhere from a single word to a full page of text. In addition, the grammatical structure and use of punctuation varies ...
- 2,029
23
votes
4
answers
708
views
What statistical model should I use to analyze the likelihood that a single event influenced longitudinal data
I am trying to find a formula, method, or model to use to analyze the likelihood that a specific event influenced some longitudinal data. I am having difficultly figuring out what to search for on ...
- 333
5
votes
2
answers
225
views
Techniques for trend extraction from unbalanced panel data
My data set is formatted like this:
User-id | Threat_score
aaa 45
bbb 32
ccc 20
The list contains the top 100 users with the highest threat ...
- 59
27
votes
4
answers
14k
views
Word2Vec for Named Entity Recognition
I'm looking to use google's word2vec implementation to build a named entity recognition system. I've heard that recursive neural nets with back propagation through structure are well suited for named ...
- 2,029
12
votes
4
answers
15k
views
Is GLM a statistical or machine learning model?
I thought that generalized linear model (GLM) would be considered a statistical model, but a friend told me that some papers classify it as a machine learning technique. Which one is true (or more ...
- 313
4
votes
1
answer
675
views
Network structure: k-cliques vs. p-cliques
In network structure, what is the difference between k-cliques and p-cliques, can anyone give a brief explaination with examples? Thanks in advanced!
============================
EDIT:
I found an ...
- 151
22
votes
4
answers
12k
views
Is logistic regression actually a regression algorithm?
The usual definition of regression (as far as I am aware) is predicting a continuous output variable from a given set of input variables.
Logistic regression is a binary classification algorithm, so ...
- 516
11
votes
4
answers
2k
views
Learning ordinal regression in R?
I'm working on a project and need resources to get me up to speed.
The dataset is around 35000 observations on 30 or so variables. About half the variables are categorical with some having many ...
- 111
10
votes
4
answers
784
views
Gas consumption outliers detection - Neural network project. Bad results
I tried to detect outliers in the energy gas consumption of some dutch buildings, building a neural network model. I have very bad results, but I can't find the reason.
I am not an expert so I would ...
- 1,667
2
votes
1
answer
720
views
Preference Matching Algorithm [duplicate]
There's this side project I'm working on where I need to structure a solution to the following problem.
I have two groups of people (clients). Group "A" intends to buy and group "B" intends to sell a ...
- 141
12
votes
2
answers
2k
views
Preference Matching Algorithm
There's this side project I'm working on where I need to structure a solution to the following problem.
I have two groups of people (clients). Group A intends to ...
- 141
17
votes
2
answers
10k
views
K-means vs. online K-means
K-means is a well known algorithm for clustering, but there is also an online variation of such algorithm (online K-means). What are the pros and cons of these approaches, and when should each be ...
- 4,097
9
votes
4
answers
3k
views
Suggest text classifier training datasets
Which freely available datasets can I use to train a text classifier?
We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified ...
- 121
16
votes
4
answers
2k
views
What are the implications for training a Tree Ensemble with highly biased datasets?
I have a highly biased binary dataset - I have 1000x more examples of the negative class than the positive class. I would like to train a Tree Ensemble (like Extra Random Trees or a Random Forest) on ...
- 418
13
votes
1
answer
521
views
What is the difference between global and universal compression methods?
I understand that compression methods may be split into two main sets:
global
local
The first set works regardless of the data being processed, i.e., they do not rely on any characteristic of the ...
- 4,097
5
votes
2
answers
905
views
How to speedup message passing between computing nodes
I'm developing a distributed application, and as it's been designed, there'll be a great load of communication during the processing. Since the communication is already as much spread along the entire ...
- 4,097
9
votes
1
answer
244
views
Learning signal encoding
I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there ...
- 1,824
34
votes
5
answers
13k
views
What are the use cases for Apache Spark vs Hadoop
With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've ...
- 521
5
votes
1
answer
154
views
What are good sources to learn about Bootstrap?
I think that Bootstrap can be useful in my work, where we have a lot a variables that we don't know the distribution of it. So, simulations could help.
What are good sources to learn about Bootstrap/...
9
votes
1
answer
329
views
Relational Data Mining without ILP
I have a huge dataset from a relational database which I need to create a classification model for. Normally for this situation I would use Inductive Logic Programming (ILP), but due to special ...
- 259
14
votes
4
answers
2k
views
Looking for example infrastructure stacks/workflows/pipelines
I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
- 143
5
votes
2
answers
2k
views
Does anyone use Julia programming language? [closed]
Is anyone using Julia (http://julialang.org/) for professional jobs?
Or using it instead of R, Matlab, or Mathematica?
Is it a good language?
If you have to ...
- 187
23
votes
3
answers
5k
views
How to grow a list of related words based on initial keywords?
I recently saw a cool feature that was once available in Google Sheets: you start by writing a few related keywords in consecutive cells, say: "blue", "green", "yellow", and it automatically generates ...
- 587
4
votes
2
answers
780
views
How to measure execution time on distributed system
I'm planning to run experiments with large datasets on distributed system in order to evaluate efficiency gains in comparison with previous proposals.
I have limited number of machines nearly ten ...
- 4,097
28
votes
7
answers
27k
views
Publicly available social network datasets/APIs
As an extension to our great list of publicly available datasets, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside ...
- 4,097
10
votes
4
answers
2k
views
Online machine learning tutorial
Does anyone know some good tutorials on online machine learning technics?
I.e. how it can be used in real-time environments, what are key differences compared to normal machine learning methods etc.
...
- 1,071
16
votes
3
answers
12k
views
Best way to classify datasets with mixed types of attributes
I would like to know what is the best way to classify a data set composed of mixed types of attributes, for example, textual and numerical. I know I can convert textual to boolean, but the vocabulary ...
- 161
44
votes
6
answers
6k
views
How can I transform names in a confidential data set to make it anonymous, but preserve some of the characteristics of the names?
Motivation
I work with datasets that contain personally identifiable information (PII) and sometimes need to share part of a dataset with third parties, in a way that doesn't expose PII and subject ...
- 822
11
votes
3
answers
7k
views
Best languages for scientific computing [closed]
It seems as though most languages have some number of scientific computing libraries available.
Python has Scipy
Rust has <...
- 1,824
114
votes
10
answers
121k
views
Choosing a learning rate
I'm currently working on implementing Stochastic Gradient Descent, SGD, for neural nets using back-propagation, and while I understand its purpose I have some ...
- 1,824
21
votes
7
answers
12k
views
How can I predict traffic based on previous time series data?
If I have a retail store and have a way to measure how many people enter my store every minute, and timestamp that data, how can I predict future foot traffic?
I have looked into machine learning ...
- 311
2
votes
1
answer
411
views
Why is there such a mismatch between the Model's predicted probability and theoretical probability in logistic regression?
I am trying to do Logistic Regression using SAS Enterprise Miner.
My Independent variables are
...
8
votes
3
answers
988
views
What to consider before learning a new language for data analysis
I'm currently in the very early stages of preparing a new research-project (still at the funding-application stage), and expect that data-analysis and especially visualisation tools will play a role ...
- 191
8
votes
2
answers
372
views
Filtering spam from retrieved data
I once heard that filtering spam by using blacklists is not a good approach, since some user searching for entries in your dataset may be looking for particular information from the sources blocked. ...
- 4,097
20
votes
5
answers
28k
views
Choose binary classification algorithm
I have a binary classification problem:
Approximately 1000 samples in training set
10 attributes, including binary, numeric and categorical
Which algorithm is the best choice for this type of ...
- 5,424
10
votes
4
answers
406
views
How to debug data analysis?
I've came across the following problem, that I recon is rather typical.
I have some large data, say, a few million rows. I run some non-trivial analysis on it, e.g. an SQL query consisting of several ...
8
votes
3
answers
172
views
How to compare experiments run over different infrastructures
I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
- 4,097
4
votes
2
answers
2k
views
Amazon S3 vs Google Drive [closed]
The majority of people use S3. However, Google Drive seems a promising alternative solution for storing large amounts of data. Are there specific reasons why one is better than the other?
- 599
12
votes
4
answers
10k
views
How to process natural language queries?
I'm curious about natural language querying. Stanford has what looks to be a strong set of software for processing natural language. I've also seen the Apache OpenNLP library, and the General ...
- 3,118
11
votes
3
answers
2k
views
Data Science oriented dataset/research question for Statistics MSc thesis
I'd like to explore 'data science'. The term seems a little vague to me, but I expect it to require:
machine learning (rather than traditional statistics);
a large enough dataset that you have to run ...
- 113