All Questions

Filter by
Sorted by
Tagged with
127 votes
14 answers
120k views

Python vs R for machine learning

I'm just starting to develop a machine learning application for academic purposes. I'm currently using R and training myself in it. However, in a lot of places, I have seen people using Python. What ...
-2 votes
1 answer
624 views

How can I extract news about a particular company from various websites using RODBC package in R? And perform sentiment analysis on the data? [closed]

I would like to extract news about a company from online news by using the RODBC package in R. I would then like to use the extracted data for sentiment analysis. I want to accomplish this in such a ...
user714's user avatar
  • 11
6 votes
1 answer
10k views

How can we calculate AUC for a simple decision tree?

The setup is simple: binary classification using a simple decision tree, each node of the tree has a single threshold applied on a single feature. In general, building a ROC curve requires moving a ...
iliasfl's user avatar
  • 609
10 votes
2 answers
2k views

Debugging Neural Networks

I've built an artificial neural network in python using the scipy.optimize.minimize (Conjugate gradient) optimization function. I've implemented gradient checking, double checked everything etc and I'...
user3726050's user avatar
29 votes
4 answers
3k views

Books about the "Science" in Data Science? [closed]

What are the books about the science and mathematics behind data science? It feels like so many "data science" books are programming tutorials and don't touch things like data generating processes and ...
Anton's user avatar
  • 399
17 votes
3 answers
3k views

One-Class discriminatory classification with imbalanced, heterogenous Negative background?

I'm working on improving an existing supervised classifier, for classifying {protein} sequences as belonging to a specific class (Neuropeptide hormone precursors), or not. There are about 1,150 known ...
GrimSqueaker's user avatar
14 votes
4 answers
4k views

Big data case study or use case example

I have read lot of blogs\article on how different type of industries are using Big Data Analytic. But most of these article fails to mention What kinda data these companies used. What was the size of ...
Brown_Dynamite's user avatar
12 votes
3 answers
3k views

Does Amazon RedShift replace Hadoop for ~1XTB data?

There is plenty of hype surrounding Hadoop and its eco-system. However, in practice, where many data sets are in the terabyte range, is it not more reasonable to use Amazon RedShift for querying ...
trienism's user avatar
  • 253
10 votes
5 answers
596 views

Qualifications for PhD Programs

Yann LeCun mentioned in his AMA that he considers having a PhD very important in order to get a job at a top company. I have a masters in statistics and my undergrad was in economics and applied math,...
bstockton's user avatar
  • 212
8 votes
3 answers
169 views

Feature selection for tracking user activity within an application

I am developing a system that is intended to capture the "context" of user activity within an application; it is a framework that web applications can use to tag user activity based on requests made ...
Joshua Barron's user avatar
12 votes
9 answers
3k views

What are some easy to learn machine-learning applications? [closed]

Being new to machine-learning in general, I'd like to start playing around and see what the possibilities are. I'm curious as to what applications you might recommend that would offer the fastest ...
Steve Kallestad's user avatar
42 votes
10 answers
46k views

Can machine learning algorithms predict sports scores or plays?

I have a variety of NFL datasets that I think might make a good side-project, but I haven't done anything with them just yet. Coming to this site made me think of machine learning algorithms and I ...
Steve Kallestad's user avatar
40 votes
4 answers
26k views

What are the advantages of HDF compared to alternative formats?

What are the advantages of HDF compared to alternative formats? What are the main data science tasks where HDF is really suitable and useful?
IgorS's user avatar
  • 5,474
40 votes
10 answers
21k views

Do I need to learn Hadoop to be a Data Scientist?

An aspiring data scientist here. I don't know anything about Hadoop, but as I have been reading about Data Science and Big Data, I see a lot of talk about Hadoop. Is it absolutely necessary to learn ...
Pensu's user avatar
  • 591
3 votes
1 answer
881 views

HBase connector - Thrift or REST

There seem to be at least 2 ways to connect to HBase from external application, with language other then Java (i.e. Python): HBase Thrift API HBase Stargate (REST API) Does anyone know which one ...
Igor Bobriakov's user avatar
3 votes
5 answers
2k views

Are there any python based data visualization toolkits?

Data visualization is an important sub-field of data science and python programmers need to have available toolkits for them. Is there a Python API to Tableau? Are there any Python based data ...
5 votes
5 answers
472 views

Data science Ph.D. program, what do you think? [closed]

As Yann LeCun mentioned, a number of PhD programs in data science will be popping up in the next few years. NYU already have one, where Prof.LeCun is at right now. A statistics or cs PhD in machine ...
user13985's user avatar
  • 275
10 votes
2 answers
7k views

How to get an aggregate confusion matrix from n different classifications

I want to test the accuracy of a methodology. I ran it ~400 times, and I got a different classification for each run. I also have the ground truth, i.e., the real classification to test against. For ...
gc5's user avatar
  • 879
8 votes
1 answer
78 views

Cascaded Error in Apache Storm

Going through the presentation and material of Summingbird by Twitter, one of the reasons that is mentioned for using Storm and Hadoop clusters together in Summingbird is that processing through Storm ...
mbbce's user avatar
  • 347
12 votes
2 answers
7k views

Tradeoffs between Storm and Hadoop (MapReduce)

Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing ...
mbbce's user avatar
  • 347
4 votes
2 answers
363 views

How to get phrase tables from word alignments?

The output of my word alignment file looks as such: ...
alvas's user avatar
  • 2,410
23 votes
5 answers
4k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
Amir Ali Akbari's user avatar
3 votes
2 answers
2k views

built-in cov in pandas DataFrame results ValueError array is too big [closed]

So, I have a dataset with 39.949 variables and 180 rows. dataset is successfully saved in DataFrame but when I try to find cov() it result an error. here is the code ...
Aldy syahdeini's user avatar
16 votes
2 answers
1k views

Where in the workflow should we deal with missing data?

I'm building a workflow for creating machine learning models (in my case, using Python's pandas and sklearn packages) from data ...
Therriault's user avatar
9 votes
3 answers
5k views

Human activity recognition using smartphone data set problem

I'm new to this community and hopefully my question will well fit in here. As part of my undergraduate data analytics course I have choose to do the project on human activity recognition using ...
Jakubee's user avatar
  • 401
10 votes
1 answer
1k views

How should one deal with implicit data in recommendation

A recommendation system keeps a log of what recommendations have been made to a particular user and whether that user accepts the recommendation. It's like ...
wdg's user avatar
  • 203
12 votes
3 answers
2k views

Instances vs. cores when using EC2

Working on what could often be called "medium data" projects, I've been able to parallelize my code (mostly for modeling and prediction in Python) on a single system across anywhere from 4 to 32 cores....
Therriault's user avatar
25 votes
2 answers
35k views

What does the alpha and beta hyperparameters contribute to in Latent Dirichlet allocation?

LDA has two hyperparameters, tuning them changes the induced topics. What does the alpha and beta hyperparameters contribute to LDA? How does the topic change if one or the other hyperparameters ...
alvas's user avatar
  • 2,410
13 votes
4 answers
7k views

Algorithm for generating classification rules

So we have potential for a machine learning application that fits fairly neatly into the traditional problem domain solved by classifiers, i.e., we have a set of attributes describing an item and a "...
super_seabass's user avatar
8 votes
3 answers
525 views

What are the most popular data science application use cases for consumer web companies

The most popular use case seem to be recommender systems of different kinds (such as recommending shopping items, users in social networks etc.). But what are other typical data science applications, ...
Igor Bobriakov's user avatar
8 votes
2 answers
1k views

Multi layer back propagation Neural network for classification

Can someone explain me, how to classify a data like MNIST with MLBP-Neural network if I make more than one output (e.g 8), I mean if I just use one output I can easily classify the data, but if I use ...
Aldy syahdeini's user avatar
4 votes
1 answer
984 views

Open source solver for large mixed integer programming task?

I'm currently using General Algebraic Modeling System (GAMS), and more specifically CPLEX within GAMS, to solve a very large mixed integer programming problem. This allows me to parallelize the ...
rnorberg's user avatar
  • 203
9 votes
1 answer
352 views

Using SVM as a binary classifier, is the label for a data point chosen by consensus?

I'm learning Support Vector Machines, and I'm unable to understand how a class label is chosen for a data point in a binary classifier. Is it chosen by consensus with respect to the classification in ...
gc5's user avatar
  • 879
9 votes
3 answers
521 views

Google prediction API: What training/prediction methods Google Prediction API employs?

The details of the Google Prediction API are on this page, but I am not able to find any details about the prediction algorithms running behind the API. So far I have gathered that they let you ...
Tahir Akhtar's user avatar
12 votes
3 answers
5k views

How to animate growth of a social network?

I am seeking for a library/tool to visualize how social network changes when new nodes/edges are added to it. One of the existing solutions is SoNIA: Social Network Image Animator. It let's you make ...
Wojciech Walczak's user avatar
3 votes
1 answer
63 views

Can metadata be used to adapt parsing for an unescaped in field use of the delimiter?

I have data coming from a source system that is pipe delimited. Pipe was selected over comma since it was believed no pipes appeared in field, while it was known that commas do occur. After ingesting ...
Chris Simokat's user avatar
27 votes
4 answers
20k views

Is there a straightforward way to run pandas.DataFrame.isin in parallel?

I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of ...
Therriault's user avatar
15 votes
4 answers
6k views

How to specify important attributes?

Assume a set of loosely structured data (e.g. Web tables/Linked Open Data), composed of many data sources. There is no common schema followed by the data and each source can use synonym attributes to ...
vefthym's user avatar
  • 503
6 votes
1 answer
564 views

Is Data Science just a trend or is a long term concept? [closed]

I see a lot of courses in Data Science emerging in the last 2 years. Even big universities like Stanford and Columbia offers MS specifically in Data Science. But as long as I see, it looks like data ...
Filipe Ferminiano's user avatar
202 votes
35 answers
33k views

Publicly Available Datasets

One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...
10 votes
1 answer
1k views

What is the most efficient data indexing technique

As we all know, there are some data indexing techniques, using by well-known indexing apps, like Lucene (for java) or Lucene.NET (for .NET), MurMurHash, B+Tree etc. For a No-Sql / Object Oriented ...
sihirbazzz's user avatar
10 votes
4 answers
226 views

Why is it hard to grant efficiency while using libraries?

Any small database processing can be easily tackled by Python/Perl/... scripts, that uses libraries and/or even utilities from the language itself. However, when it comes to performance, people tend ...
Rubens's user avatar
  • 4,107
11 votes
3 answers
710 views

Can map-reduce algorithms written for MongoDB be ported to Hadoop later?

In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select ...
Amir Ali Akbari's user avatar
70 votes
11 answers
40k views

What is dimensionality reduction? What is the difference between feature selection and extraction?

From wikipedia: dimensionality reduction or dimension reduction is the process of reducing the number of random variables under consideration, and can be divided into feature selection and feature ...
alvas's user avatar
  • 2,410
10 votes
1 answer
2k views

What is generative and discriminative model? How are they used in Natural Language Processing?

This question asks about generative vs. discriminative algorithm, but can someone give an example of the difference between these forms when applied to Natural Language Processing? How are generative ...
alvas's user avatar
  • 2,410
62 votes
6 answers
33k views

Latent Dirichlet Allocation vs Hierarchical Dirichlet Process

Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Process (HDP) are both topic modeling processes. The major difference is LDA requires the specification of the number of topics, and HDP ...
alvas's user avatar
  • 2,410
1 vote
2 answers
792 views

How to learn noSQL databases and how to know when SQL or noSQL is better [closed]

I want learn about NoSQL and when is better to use SQL or NoSQL. I know that this question depends on the case, but I'm asking for a good documentation on NoSQL, and some explanation of when is better ...
user avatar
28 votes
6 answers
3k views

Machine learning techniques for estimating users' age based on Facebook sites they like

I have a database from my Facebook application and I am trying to use machine learning to estimate users' age based on what Facebook sites they like. There are three crucial characteristics of my ...
Wojciech Walczak's user avatar
15 votes
2 answers
4k views

Is there any APIs for crawling abstract of paper?

If I have a very long list of paper names, how could I get abstract of these papers from internet or any database? The paper names are like "Assessment of Utility in Web Mining for the Domain of ...
Alex Gao's user avatar
  • 253
13 votes
1 answer
268 views

When a relational database has better performance than a no relational

When a relational database, like MySQL, has better performance than a no relational, like MongoDB? I saw a question on Quora other day, about why Quora still uses MySQL as their backend, and that ...
Filipe Ferminiano's user avatar

15 30 50 per page