All Questions

Filter by
Sorted by
Tagged with
-2
votes
1answer
611 views

How can I extract news about a particular company from various websites using RODBC package in R? And perform sentiment analysis on the data? [closed]

I would like to extract news about a company from online news by using the RODBC package in R. I would then like to use the extracted data for sentiment analysis. I want to accomplish this in such a ...
5
votes
1answer
8k views

How can we calculate AUC for a simple decision tree?

The setup is simple: binary classification using a simple decision tree, each node of the tree has a single threshold applied on a single feature. In general, building a ROC curve requires moving a ...
10
votes
2answers
2k views

Debugging Neural Networks

I've built an artificial neural network in python using the scipy.optimize.minimize (Conjugate gradient) optimization function. I've implemented gradient checking, double checked everything etc and I'...
28
votes
4answers
3k views

Books about the "Science" in Data Science? [closed]

What are the books about the science and mathematics behind data science? It feels like so many "data science" books are programming tutorials and don't touch things like data generating processes and ...
17
votes
3answers
3k views

One-Class discriminatory classification with imbalanced, heterogenous Negative background?

I'm working on improving an existing supervised classifier, for classifying {protein} sequences as belonging to a specific class (Neuropeptide hormone precursors), or not. There are about 1,150 known ...
14
votes
4answers
3k views

Big data case study or use case example

I have read lot of blogs\article on how different type of industries are using Big Data Analytic. But most of these article fails to mention What kinda data these companies used. What was the size of ...
12
votes
3answers
3k views

Does Amazon RedShift replace Hadoop for ~1XTB data?

There is plenty of hype surrounding Hadoop and its eco-system. However, in practice, where many data sets are in the terabyte range, is it not more reasonable to use Amazon RedShift for querying ...
10
votes
5answers
567 views

Qualifications for PhD Programs

Yann LeCun mentioned in his AMA that he considers having a PhD very important in order to get a job at a top company. I have a masters in statistics and my undergrad was in economics and applied math,...
8
votes
3answers
156 views

Feature selection for tracking user activity within an application

I am developing a system that is intended to capture the "context" of user activity within an application; it is a framework that web applications can use to tag user activity based on requests made ...
12
votes
9answers
3k views

What are some easy to learn machine-learning applications? [closed]

Being new to machine-learning in general, I'd like to start playing around and see what the possibilities are. I'm curious as to what applications you might recommend that would offer the fastest ...
41
votes
10answers
45k views

Can machine learning algorithms predict sports scores or plays?

I have a variety of NFL datasets that I think might make a good side-project, but I haven't done anything with them just yet. Coming to this site made me think of machine learning algorithms and I ...
37
votes
4answers
17k views

What are the advantages of HDF compared to alternative formats?

What are the advantages of HDF compared to alternative formats? What are the main data science tasks where HDF is really suitable and useful?
37
votes
9answers
21k views

Do I need to learn Hadoop to be a Data Scientist?

An aspiring data scientist here. I don't know anything about Hadoop, but as I have been reading about Data Science and Big Data, I see a lot of talk about Hadoop. Is it absolutely necessary to learn ...
3
votes
1answer
820 views

HBase connector - Thrift or REST

There seem to be at least 2 ways to connect to HBase from external application, with language other then Java (i.e. Python): HBase Thrift API HBase Stargate (REST API) Does anyone know which one ...
2
votes
5answers
2k views

Are there any python based data visualization toolkits?

Data visualization is an important sub-field of data science and python programmers need to have available toolkits for them. Is there a Python API to Tableau? Are there any Python based data ...
5
votes
5answers
449 views

Data science Ph.D. program, what do you think? [closed]

As Yann LeCun mentioned, a number of PhD programs in data science will be popping up in the next few years. NYU already have one, where Prof.LeCun is at right now. A statistics or cs PhD in machine ...
10
votes
2answers
5k views

How to get an aggregate confusion matrix from n different classifications

I want to test the accuracy of a methodology. I ran it ~400 times, and I got a different classification for each run. I also have the ground truth, i.e., the real classification to test against. For ...
8
votes
1answer
71 views

Cascaded Error in Apache Storm

Going through the presentation and material of Summingbird by Twitter, one of the reasons that is mentioned for using Storm and Hadoop clusters together in Summingbird is that processing through Storm ...
12
votes
2answers
7k views

Tradeoffs between Storm and Hadoop (MapReduce)

Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing ...
4
votes
2answers
298 views

How to get phrase tables from word alignments?

The output of my word alignment file looks as such: ...
21
votes
5answers
3k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
3
votes
2answers
2k views

built-in cov in pandas DataFrame results ValueError array is too big [closed]

So, I have a dataset with 39.949 variables and 180 rows. dataset is successfully saved in DataFrame but when I try to find cov() it result an error. here is the code ...
16
votes
2answers
1k views

Where in the workflow should we deal with missing data?

I'm building a workflow for creating machine learning models (in my case, using Python's pandas and sklearn packages) from data ...
9
votes
3answers
5k views

Human activity recognition using smartphone data set problem

I'm new to this community and hopefully my question will well fit in here. As part of my undergraduate data analytics course I have choose to do the project on human activity recognition using ...
10
votes
1answer
1k views

How should one deal with implicit data in recommendation

A recommendation system keeps a log of what recommendations have been made to a particular user and whether that user accepts the recommendation. It's like ...
12
votes
3answers
2k views

Instances vs. cores when using EC2

Working on what could often be called "medium data" projects, I've been able to parallelize my code (mostly for modeling and prediction in Python) on a single system across anywhere from 4 to 32 cores....
23
votes
2answers
30k views

What does the alpha and beta hyperparameters contribute to in Latent Dirichlet allocation?

LDA has two hyperparameters, tuning them changes the induced topics. What does the alpha and beta hyperparameters contribute to LDA? How does the topic change if one or the other hyperparameters ...
13
votes
4answers
7k views

Algorithm for generating classification rules

So we have potential for a machine learning application that fits fairly neatly into the traditional problem domain solved by classifiers, i.e., we have a set of attributes describing an item and a "...
8
votes
3answers
505 views

What are the most popular data science application use cases for consumer web companies

The most popular use case seem to be recommender systems of different kinds (such as recommending shopping items, users in social networks etc.). But what are other typical data science applications, ...
8
votes
2answers
1k views

Multi layer back propagation Neural network for classification

Can someone explain me, how to classify a data like MNIST with MLBP-Neural network if I make more than one output (e.g 8), I mean if I just use one output I can easily classify the data, but if I use ...
4
votes
1answer
978 views

Open source solver for large mixed integer programming task?

I'm currently using General Algebraic Modeling System (GAMS), and more specifically CPLEX within GAMS, to solve a very large mixed integer programming problem. This allows me to parallelize the ...
9
votes
1answer
320 views

Using SVM as a binary classifier, is the label for a data point chosen by consensus?

I'm learning Support Vector Machines, and I'm unable to understand how a class label is chosen for a data point in a binary classifier. Is it chosen by consensus with respect to the classification in ...
9
votes
3answers
514 views

Google prediction API: What training/prediction methods Google Prediction API employs?

The details of the Google Prediction API are on this page, but I am not able to find any details about the prediction algorithms running behind the API. So far I have gathered that they let you ...
12
votes
3answers
4k views

How to animate growth of a social network?

I am seeking for a library/tool to visualize how social network changes when new nodes/edges are added to it. One of the existing solutions is SoNIA: Social Network Image Animator. It let's you make ...
3
votes
1answer
58 views

Can metadata be used to adapt parsing for an unescaped in field use of the delimiter?

I have data coming from a source system that is pipe delimited. Pipe was selected over comma since it was believed no pipes appeared in field, while it was known that commas do occur. After ingesting ...
26
votes
4answers
17k views

Is there a straightforward way to run pandas.DataFrame.isin in parallel?

I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of ...
15
votes
4answers
5k views

How to specify important attributes?

Assume a set of loosely structured data (e.g. Web tables/Linked Open Data), composed of many data sources. There is no common schema followed by the data and each source can use synonym attributes to ...
6
votes
1answer
542 views

Is Data Science just a trend or is a long term concept? [closed]

I see a lot of courses in Data Science emerging in the last 2 years. Even big universities like Stanford and Columbia offers MS specifically in Data Science. But as long as I see, it looks like data ...
198
votes
35answers
31k views

Publicly Available Datasets

One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...
10
votes
1answer
945 views

What is the most efficient data indexing technique

As we all know, there are some data indexing techniques, using by well-known indexing apps, like Lucene (for java) or Lucene.NET (for .NET), MurMurHash, B+Tree etc. For a No-Sql / Object Oriented ...
10
votes
4answers
216 views

Why is it hard to grant efficiency while using libraries?

Any small database processing can be easily tackled by Python/Perl/... scripts, that uses libraries and/or even utilities from the language itself. However, when it comes to performance, people tend ...
11
votes
3answers
693 views

Can map-reduce algorithms written for MongoDB be ported to Hadoop later?

In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select ...
69
votes
11answers
37k views

What is dimensionality reduction? What is the difference between feature selection and extraction?

From wikipedia: dimensionality reduction or dimension reduction is the process of reducing the number of random variables under consideration, and can be divided into feature selection and feature ...
10
votes
1answer
1k views

What is generative and discriminative model? How are they used in Natural Language Processing?

This question asks about generative vs. discriminative algorithm, but can someone give an example of the difference between these forms when applied to Natural Language Processing? How are generative ...
60
votes
6answers
28k views

Latent Dirichlet Allocation vs Hierarchical Dirichlet Process

Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Process (HDP) are both topic modeling processes. The major difference is LDA requires the specification of the number of topics, and HDP ...
1
vote
2answers
728 views

How to learn noSQL databases and how to know when SQL or noSQL is better [closed]

I want learn about NoSQL and when is better to use SQL or NoSQL. I know that this question depends on the case, but I'm asking for a good documentation on NoSQL, and some explanation of when is better ...
26
votes
6answers
3k views

Machine learning techniques for estimating users' age based on Facebook sites they like

I have a database from my Facebook application and I am trying to use machine learning to estimate users' age based on what Facebook sites they like. There are three crucial characteristics of my ...
15
votes
2answers
4k views

Is there any APIs for crawling abstract of paper?

If I have a very long list of paper names, how could I get abstract of these papers from internet or any database? The paper names are like "Assessment of Utility in Web Mining for the Domain of ...
13
votes
1answer
231 views

When a relational database has better performance than a no relational

When a relational database, like MySQL, has better performance than a no relational, like MongoDB? I saw a question on Quora other day, about why Quora still uses MySQL as their backend, and that ...
12
votes
2answers
139 views

Opensource tools for help in mining stream of leader board scores

Consider a stream containing tuples (user, new_score) representing users' scores in an online game. The stream could have 100-1,000 new elements per second. The ...

15 30 50 per page