All Questions

Filter by
Sorted by
Tagged with
4 votes
2 answers
2k views

Amazon S3 vs Google Drive [closed]

The majority of people use S3. However, Google Drive seems a promising alternative solution for storing large amounts of data. Are there specific reasons why one is better than the other?
iliasfl's user avatar
  • 609
12 votes
4 answers
10k views

How to process natural language queries?

I'm curious about natural language querying. Stanford has what looks to be a strong set of software for processing natural language. I've also seen the Apache OpenNLP library, and the General ...
Steve Kallestad's user avatar
11 votes
3 answers
2k views

Data Science oriented dataset/research question for Statistics MSc thesis

I'd like to explore 'data science'. The term seems a little vague to me, but I expect it to require: machine learning (rather than traditional statistics); a large enough dataset that you have to run ...
user3279453's user avatar
9 votes
2 answers
4k views

Difference between using RMSE and nDCG to evaluate Recommender Systems

What kind of error measures do RMSE and nDCG give while evaluating a recommender system, and how do I know when to use one over the other? If you could give an example of when to use each, that would ...
covfefe's user avatar
  • 293
65 votes
6 answers
43k views

When is a Model Underfitted?

Logic often states that by underfitting a model, it's capacity to generalize is increased. That said, clearly at some point underfitting a model cause models to become worse regardless of the ...
blunders's user avatar
  • 1,932
6 votes
2 answers
142 views

How to best accomplish high speed comparison of like data?

I attack this problem frequently with inefficiency because it's always pretty low on the priority list and my clients are resistant to change until things break. I would like some input on how to ...
Steve Kallestad's user avatar
7 votes
2 answers
289 views

Data Science as a Social Scientist?

as I am very interested in programming and statistics, Data Science seems like a great career path to me - I like both fields and would like to combine them. Unfortunately, I have studied political ...
Christian Sauer's user avatar
10 votes
1 answer
297 views

How do I figure out if subsets can be plotted in a normal Euler diagram?

In some cases, it may be impossible to draw Euler diagrams with overlapping circles to represent all the overlapping subsets in the correct proportions. This type of data then requires using polygons ...
asheeshr's user avatar
  • 654
0 votes
2 answers
3k views

Example tasks of a data scientist and the necessary knowledge

Could you give some examples of typical tasks that a data scientist does in his daily job, and the must-know minimum for each of the levels (like junior, senior, etc. if there are any)? If possible, ...
zeki's user avatar
  • 41
37 votes
13 answers
20k views

What do you think of Data Science certifications?

I've now seen two data science certification programs - the John Hopkins one available at Coursera and the Cloudera one. I'm sure there are others out there. The John Hopkins set of classes is ...
127 votes
14 answers
120k views

Python vs R for machine learning

I'm just starting to develop a machine learning application for academic purposes. I'm currently using R and training myself in it. However, in a lot of places, I have seen people using Python. What ...
-2 votes
1 answer
626 views

How can I extract news about a particular company from various websites using RODBC package in R? And perform sentiment analysis on the data? [closed]

I would like to extract news about a company from online news by using the RODBC package in R. I would then like to use the extracted data for sentiment analysis. I want to accomplish this in such a ...
user714's user avatar
  • 11
6 votes
1 answer
10k views

How can we calculate AUC for a simple decision tree?

The setup is simple: binary classification using a simple decision tree, each node of the tree has a single threshold applied on a single feature. In general, building a ROC curve requires moving a ...
iliasfl's user avatar
  • 609
10 votes
2 answers
2k views

Debugging Neural Networks

I've built an artificial neural network in python using the scipy.optimize.minimize (Conjugate gradient) optimization function. I've implemented gradient checking, double checked everything etc and I'...
user3726050's user avatar
29 votes
4 answers
3k views

Books about the "Science" in Data Science? [closed]

What are the books about the science and mathematics behind data science? It feels like so many "data science" books are programming tutorials and don't touch things like data generating processes and ...
Anton's user avatar
  • 399
17 votes
3 answers
3k views

One-Class discriminatory classification with imbalanced, heterogenous Negative background?

I'm working on improving an existing supervised classifier, for classifying {protein} sequences as belonging to a specific class (Neuropeptide hormone precursors), or not. There are about 1,150 known ...
GrimSqueaker's user avatar
14 votes
4 answers
4k views

Big data case study or use case example

I have read lot of blogs\article on how different type of industries are using Big Data Analytic. But most of these article fails to mention What kinda data these companies used. What was the size of ...
Brown_Dynamite's user avatar
12 votes
3 answers
3k views

Does Amazon RedShift replace Hadoop for ~1XTB data?

There is plenty of hype surrounding Hadoop and its eco-system. However, in practice, where many data sets are in the terabyte range, is it not more reasonable to use Amazon RedShift for querying ...
trienism's user avatar
  • 253
10 votes
5 answers
597 views

Qualifications for PhD Programs

Yann LeCun mentioned in his AMA that he considers having a PhD very important in order to get a job at a top company. I have a masters in statistics and my undergrad was in economics and applied math,...
bstockton's user avatar
  • 212
8 votes
3 answers
169 views

Feature selection for tracking user activity within an application

I am developing a system that is intended to capture the "context" of user activity within an application; it is a framework that web applications can use to tag user activity based on requests made ...
Joshua Barron's user avatar
12 votes
9 answers
3k views

What are some easy to learn machine-learning applications? [closed]

Being new to machine-learning in general, I'd like to start playing around and see what the possibilities are. I'm curious as to what applications you might recommend that would offer the fastest ...
Steve Kallestad's user avatar
42 votes
10 answers
46k views

Can machine learning algorithms predict sports scores or plays?

I have a variety of NFL datasets that I think might make a good side-project, but I haven't done anything with them just yet. Coming to this site made me think of machine learning algorithms and I ...
Steve Kallestad's user avatar
40 votes
4 answers
27k views

What are the advantages of HDF compared to alternative formats?

What are the advantages of HDF compared to alternative formats? What are the main data science tasks where HDF is really suitable and useful?
IgorS's user avatar
  • 5,474
40 votes
10 answers
21k views

Do I need to learn Hadoop to be a Data Scientist?

An aspiring data scientist here. I don't know anything about Hadoop, but as I have been reading about Data Science and Big Data, I see a lot of talk about Hadoop. Is it absolutely necessary to learn ...
Pensu's user avatar
  • 591
3 votes
1 answer
884 views

HBase connector - Thrift or REST

There seem to be at least 2 ways to connect to HBase from external application, with language other then Java (i.e. Python): HBase Thrift API HBase Stargate (REST API) Does anyone know which one ...
Igor Bobriakov's user avatar
3 votes
5 answers
2k views

Are there any python based data visualization toolkits?

Data visualization is an important sub-field of data science and python programmers need to have available toolkits for them. Is there a Python API to Tableau? Are there any Python based data ...
5 votes
5 answers
472 views

Data science Ph.D. program, what do you think? [closed]

As Yann LeCun mentioned, a number of PhD programs in data science will be popping up in the next few years. NYU already have one, where Prof.LeCun is at right now. A statistics or cs PhD in machine ...
user13985's user avatar
  • 275
10 votes
2 answers
7k views

How to get an aggregate confusion matrix from n different classifications

I want to test the accuracy of a methodology. I ran it ~400 times, and I got a different classification for each run. I also have the ground truth, i.e., the real classification to test against. For ...
gc5's user avatar
  • 879
8 votes
1 answer
78 views

Cascaded Error in Apache Storm

Going through the presentation and material of Summingbird by Twitter, one of the reasons that is mentioned for using Storm and Hadoop clusters together in Summingbird is that processing through Storm ...
mbbce's user avatar
  • 347
12 votes
2 answers
7k views

Tradeoffs between Storm and Hadoop (MapReduce)

Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing ...
mbbce's user avatar
  • 347
4 votes
2 answers
363 views

How to get phrase tables from word alignments?

The output of my word alignment file looks as such: ...
alvas's user avatar
  • 2,410
23 votes
5 answers
4k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
Amir Ali Akbari's user avatar
3 votes
2 answers
2k views

built-in cov in pandas DataFrame results ValueError array is too big [closed]

So, I have a dataset with 39.949 variables and 180 rows. dataset is successfully saved in DataFrame but when I try to find cov() it result an error. here is the code ...
Aldy syahdeini's user avatar
16 votes
2 answers
1k views

Where in the workflow should we deal with missing data?

I'm building a workflow for creating machine learning models (in my case, using Python's pandas and sklearn packages) from data ...
Therriault's user avatar
9 votes
3 answers
5k views

Human activity recognition using smartphone data set problem

I'm new to this community and hopefully my question will well fit in here. As part of my undergraduate data analytics course I have choose to do the project on human activity recognition using ...
Jakubee's user avatar
  • 401
10 votes
1 answer
1k views

How should one deal with implicit data in recommendation

A recommendation system keeps a log of what recommendations have been made to a particular user and whether that user accepts the recommendation. It's like ...
wdg's user avatar
  • 203
12 votes
3 answers
2k views

Instances vs. cores when using EC2

Working on what could often be called "medium data" projects, I've been able to parallelize my code (mostly for modeling and prediction in Python) on a single system across anywhere from 4 to 32 cores....
Therriault's user avatar
25 votes
2 answers
36k views

What does the alpha and beta hyperparameters contribute to in Latent Dirichlet allocation?

LDA has two hyperparameters, tuning them changes the induced topics. What does the alpha and beta hyperparameters contribute to LDA? How does the topic change if one or the other hyperparameters ...
alvas's user avatar
  • 2,410
13 votes
4 answers
7k views

Algorithm for generating classification rules

So we have potential for a machine learning application that fits fairly neatly into the traditional problem domain solved by classifiers, i.e., we have a set of attributes describing an item and a "...
super_seabass's user avatar
8 votes
3 answers
525 views

What are the most popular data science application use cases for consumer web companies

The most popular use case seem to be recommender systems of different kinds (such as recommending shopping items, users in social networks etc.). But what are other typical data science applications, ...
Igor Bobriakov's user avatar
8 votes
2 answers
1k views

Multi layer back propagation Neural network for classification

Can someone explain me, how to classify a data like MNIST with MLBP-Neural network if I make more than one output (e.g 8), I mean if I just use one output I can easily classify the data, but if I use ...
Aldy syahdeini's user avatar
4 votes
1 answer
984 views

Open source solver for large mixed integer programming task?

I'm currently using General Algebraic Modeling System (GAMS), and more specifically CPLEX within GAMS, to solve a very large mixed integer programming problem. This allows me to parallelize the ...
rnorberg's user avatar
  • 203
9 votes
1 answer
353 views

Using SVM as a binary classifier, is the label for a data point chosen by consensus?

I'm learning Support Vector Machines, and I'm unable to understand how a class label is chosen for a data point in a binary classifier. Is it chosen by consensus with respect to the classification in ...
gc5's user avatar
  • 879
9 votes
3 answers
521 views

Google prediction API: What training/prediction methods Google Prediction API employs?

The details of the Google Prediction API are on this page, but I am not able to find any details about the prediction algorithms running behind the API. So far I have gathered that they let you ...
Tahir Akhtar's user avatar
12 votes
3 answers
5k views

How to animate growth of a social network?

I am seeking for a library/tool to visualize how social network changes when new nodes/edges are added to it. One of the existing solutions is SoNIA: Social Network Image Animator. It let's you make ...
Wojciech Walczak's user avatar
3 votes
1 answer
64 views

Can metadata be used to adapt parsing for an unescaped in field use of the delimiter?

I have data coming from a source system that is pipe delimited. Pipe was selected over comma since it was believed no pipes appeared in field, while it was known that commas do occur. After ingesting ...
Chris Simokat's user avatar
27 votes
4 answers
21k views

Is there a straightforward way to run pandas.DataFrame.isin in parallel?

I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of ...
Therriault's user avatar
15 votes
4 answers
6k views

How to specify important attributes?

Assume a set of loosely structured data (e.g. Web tables/Linked Open Data), composed of many data sources. There is no common schema followed by the data and each source can use synonym attributes to ...
vefthym's user avatar
  • 503
6 votes
1 answer
564 views

Is Data Science just a trend or is a long term concept? [closed]

I see a lot of courses in Data Science emerging in the last 2 years. Even big universities like Stanford and Columbia offers MS specifically in Data Science. But as long as I see, it looks like data ...
Filipe Ferminiano's user avatar
202 votes
35 answers
33k views

Publicly Available Datasets

One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...

15 30 50 per page