We changed our privacy policy. Read more.

Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
92
votes
12answers
18k views

How big is big data?

Lots of people use the term big data in a rather commercial way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good ...
51
votes
9answers
9k views

Is the R language suitable for Big Data

R has many libraries which are aimed at Data Analysis (e.g. JAGS, BUGS, ARULES etc..), and is mentioned in popular textbooks such as: J.Krusche, Doing Bayesian Data Analysis; B.Lantz, "Machine ...
51
votes
9answers
25k views

How to deal with version control of large amounts of (binary) data

I am a PhD student of Geophysics and work with large amounts of image data (hundreds of GB, tens of thousands of files). I know svn and ...
45
votes
12answers
52k views

Data Science in C (or C++)

I'm an R language programmer. I'm also in the group of people who are considered Data Scientists but who come from academic disciplines other than CS. This works ...
43
votes
5answers
92k views

Opening a 20GB file for analysis with pandas

I am currently trying to open a file with pandas and python for machine learning purposes it would be ideal for me to have them all in a DataFrame. Now The file is 18GB large and my RAM is 32 GB but I ...
37
votes
9answers
21k views

Do I need to learn Hadoop to be a Data Scientist?

An aspiring data scientist here. I don't know anything about Hadoop, but as I have been reading about Data Science and Big Data, I see a lot of talk about Hadoop. Is it absolutely necessary to learn ...
34
votes
6answers
30k views

How to do SVD and PCA with big data?

I have a large set of data (about 8GB). I would like to use machine learning to analyze it. So, I think that I should use SVD then PCA to reduce the data dimensionality for efficiency. However, MATLAB ...
26
votes
3answers
43k views

Data Science Project Ideas [closed]

I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion. I have just started with Data Science and ...
25
votes
5answers
15k views

Improve the speed of t-sne implementation in python for huge data

I would like to do dimensionality reduction on nearly 1 million vectors each with 200 dimensions(doc2vec). I am using TSNE ...
19
votes
3answers
8k views

Uses of NoSQL database in data science

How can NoSQL databases like MongoDB be used for data analysis? What are the features in them that can make data analysis faster and powerful?
17
votes
2answers
411 views

Use liblinear on big data for semantic analysis

I use Libsvm to train data and predict classification on semantic analysis problem. But it has a performance issue on large-scale data, because semantic analysis concerns n-dimension problem. Last ...
14
votes
8answers
16k views

Is Python suitable for big data

I read in this post Is the R language suitable for Big Data that big data constitutes 5TB, and while it does a good job of providing information about the ...
14
votes
4answers
3k views

Big data case study or use case example

I have read lot of blogs\article on how different type of industries are using Big Data Analytic. But most of these article fails to mention What kinda data these companies used. What was the size of ...
14
votes
4answers
2k views

Looking for example infrastructure stacks/workflows/pipelines

I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
14
votes
3answers
707 views

When are p-values deceptive?

What are the data conditions that we should watch out for, where p-values may not be the best way of deciding statistical significance? Are there specific problem types that fall into this category?
13
votes
4answers
4k views

Can we take of benefit of using transfer learning while training a word2vec models?

I am looking to find a pre-trained weights of an already trained models like Google News data etc. I found it hard to train a new model with enough amount (10 GB etc) of data for myself. So, I want to ...
13
votes
2answers
2k views

Is FPGrowth still considered "state of the art" in frequent pattern mining?

As far as I know the development of algorithms to solve the Frequent Pattern Mining (FPM) problem, the road of improvements have some main checkpoints. Firstly, the Apriori algorithm was proposed in ...
13
votes
1answer
226 views

When a relational database has better performance than a no relational

When a relational database, like MySQL, has better performance than a no relational, like MongoDB? I saw a question on Quora other day, about why Quora still uses MySQL as their backend, and that ...
12
votes
7answers
1k views

What is an 'old name' of data scientist?

Terms like 'data science' and 'data scientist' are increasingly used these days. Many companies are hiring 'data scientist'. But I don't think it's a completely new job. Data have existed from the ...
12
votes
3answers
482 views

How does a query into a huge database return with negligible latency?

For example, when searching something in Google, results return nigh-instantly. I understand that Google sorts and indexes pages with algorithms etc., but I imagine it infeasible for the results of ...
12
votes
2answers
7k views

Tradeoffs between Storm and Hadoop (MapReduce)

Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing ...
12
votes
2answers
4k views

Machine Learning Best Practices for Big Dataset

I am about to graduate from my Master and had learnt about machine learning as well as performed research projects with it. I wonder about the best practices in the industry when performing machine ...
12
votes
2answers
2k views

Preference Matching Algorithm

There's this side project I'm working on where I need to structure a solution to the following problem. I have two groups of people (clients). Group A intends to ...
11
votes
4answers
335 views

Working with HPC clusters

In my university, we have an HPC computing cluster. I use the cluster to train classifiers and so on. So, usually, to send a job to the cluster, (e.g. python scikit-learn script), I need to write a ...
10
votes
4answers
214 views

Why is it hard to grant efficiency while using libraries?

Any small database processing can be easily tackled by Python/Perl/... scripts, that uses libraries and/or even utilities from the language itself. However, when it comes to performance, people tend ...
10
votes
3answers
334 views

How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension?

Is there a known general table of statistical techniques that explain how they scale with sample size and dimension? For example, a friend of mine told me the other day that the computation time of ...
10
votes
1answer
31k views

Difference between interpolate() and fillna() in pandas

Since interpolate and fillna method does the same work of filling na values. What is the basic difference between the two. What is the significance of having these two different methods?? Can anyone ...
10
votes
3answers
7k views

Which is faster: PostgreSQL vs MongoDB on large JSON datasets?

I have a large dataset with 9m JSON objects at ~300 bytes each. They are posts from a link aggregator: basically links (a URL, title and author id) and comments (text and author ID) + metadata. They ...
10
votes
2answers
2k views

Scalable Outlier/Anomaly Detection

I am trying to setup a big data infrastructure using Hadoop, Hive, Elastic Search (amongst others), and I would like to run some algorithms over certain datasets. I would like the algorithms ...
10
votes
3answers
143 views

Handling a regularly increasing feature set

I'm working on a fraud detection system. In this field, new frauds appear regularly, so that new features have to be added to the model on ongoing basis. I wonder what is the best way to handle it (...
10
votes
1answer
2k views

Avoid reloading DataFrame between different python kernels

Is there a way of keeping a variable (large table / data frame) in memory and share it across multiple ipython notebooks? I'd be looking for something, which is conceptually similar to MATLAB's ...
9
votes
3answers
5k views

Human activity recognition using smartphone data set problem

I'm new to this community and hopefully my question will well fit in here. As part of my undergraduate data analytics course I have choose to do the project on human activity recognition using ...
8
votes
1answer
546 views

Original Meaning of "Intelligence" in "Business Intelligence"

What does the term "Intelligence" originally stand for in "Business Intelligence" ? Does it mean as used in "Artificial Intelligence" or as used in "Intelligence Agency" ? In other words, does "...
8
votes
2answers
837 views

Understanding how distributed PCA works

As part of big data analysis project, I'm working on, I need to perform PCA on some data, using cloud computing system. In my case, I'm using Amazon EMR for the job and Spark in particular. Leaving ...
8
votes
3answers
151 views

How to compare experiments run over different infrastructures

I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
8
votes
2answers
369 views

Filtering spam from retrieved data

I once heard that filtering spam by using blacklists is not a good approach, since some user searching for entries in your dataset may be looking for particular information from the sources blocked. ...
8
votes
1answer
71 views

Cascaded Error in Apache Storm

Going through the presentation and material of Summingbird by Twitter, one of the reasons that is mentioned for using Storm and Hadoop clusters together in Summingbird is that processing through Storm ...
7
votes
2answers
9k views

How to deal with large training data?

Currently, I use image files and transform them into a *.npy file(saved as a numpy array) as training data. At present this training data set is nearly 3GB. Now I have more image files, so the ...
7
votes
1answer
546 views

Can one build linear models on "chunks" of the data set, if one can't build them on the entire data set?

Can one build linear models on "chunks" of the data set, if one can't build them on the entire data set? Particularly, I still have over 88k variables (features) left and one cannot do much with them ...
7
votes
2answers
3k views

Lambda Architecture - How to implement the Merge Layer / Query Layer

I am reading up about lambda architecture. It makes sense. we have queue based data ingestion. we have an in-memory store for data which is very new and we have HDFS for old data. So we have our ...
7
votes
2answers
115 views

Computational aspects are typically ignored by statisticians

In the introductory chapter of "Process Mining: Data Science in Action" (2016 - Van der Aalst, pag 11) the author says that : Although data science can be seen as a continuation of statistics, the ...
7
votes
3answers
9k views

How will ADA Boost be used for solving regression problems?

I have an idea of how ADABOOST will be used for classification but I want to get the idea of how to re-weight and thus use ADABOOST in case of regression problems.
6
votes
3answers
1k views

Classifying transactions as malicious

I have a big data set of fake transactions for a company. Each row contains the username, credit card number, time, device used, and amount of money in the transaction. I need to classify each ...
6
votes
1answer
538 views

Is Data Science just a trend or is a long term concept? [closed]

I see a lot of courses in Data Science emerging in the last 2 years. Even big universities like Stanford and Columbia offers MS specifically in Data Science. But as long as I see, it looks like data ...
6
votes
2answers
450 views

What's an efficient way to compare and group millions of store names?

I'm a total amateur as far as data science goes, and I'm trying to figure out a way to do some string comparison on a large dataset. I've a Google BigQuery table storing merchant transactions, but ...
6
votes
1answer
749 views

How to detect overfitting of a stock screener

The project I am working on allows users to create Stock Screeners based on both technical and fundamental criteria. Stock Screeners are then "backtested" by simulating the results of applying in ...
6
votes
3answers
460 views

Git for Deep Learning - what are the best tools for versioning/tracking machine learning experiments?

I'm looking for a tool to track the results of several experiments/iterations in machine learning. Training a model can take days/weeks, so it's essential to track its performance and be able to ...
6
votes
3answers
2k views

Classifier and Technique to use for large number of categories

I am designing a scikit learn classifier for a sequence labelling task which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year. I have ...
6
votes
2answers
7k views

Looking for a strong Phd Topic in Predictive Analytics in the context of Big Data

I'm going to start a Computer Science phd this year and for that I need a research topic. I am interested in Predictive Analytics in the context of Big Data. I am interested by the area of Education (...
6
votes
1answer
175 views

Which Big Data technology stack is most suitable for processing tweets, extracting/expanding URLs and pushing (only) new links into 3rd party system?

(Note: Pulled this question from the list of questions in Area51, but believe the question is self explanatory. That said, believe I get the general intent of the question, and as a result likely able ...

1
2 3 4 5
9