Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
26
votes
3answers
43k views

Data Science Project Ideas [closed]

I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion. I have just started with Data Science and ...
53
votes
9answers
9k views

Is the R language suitable for Big Data

R has many libraries which are aimed at Data Analysis (e.g. JAGS, BUGS, ARULES etc..), and is mentioned in popular textbooks such as: J.Krusche, Doing Bayesian Data Analysis; B.Lantz, "Machine ...
34
votes
6answers
31k views

How to do SVD and PCA with big data?

I have a large set of data (about 8GB). I would like to use machine learning to analyze it. So, I think that I should use SVD then PCA to reduce the data dimensionality for efficiency. However, MATLAB ...
92
votes
12answers
18k views

How big is big data?

Lots of people use the term big data in a rather commercial way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good ...
6
votes
3answers
1k views

Classifying transactions as malicious

I have a big data set of fake transactions for a company. Each row contains the username, credit card number, time, device used, and amount of money in the transaction. I need to classify each ...
12
votes
2answers
2k views

Preference Matching Algorithm

There's this side project I'm working on where I need to structure a solution to the following problem. I have two groups of people (clients). Group A intends to ...
6
votes
1answer
761 views

How to detect overfitting of a stock screener

The project I am working on allows users to create Stock Screeners based on both technical and fundamental criteria. Stock Screeners are then "backtested" by simulating the results of applying in ...
3
votes
1answer
261 views

Comparing one small dataset with a big dataset for similar records

I create a varying small dataset (dataset: X) with 500 records in each query. Everytime I need to compare the dataset with a bigger one (dataset: A) (15 milion records) to find similar (or semi-...
25
votes
5answers
16k views

Improve the speed of t-sne implementation in python for huge data

I would like to do dimensionality reduction on nearly 1 million vectors each with 200 dimensions(doc2vec). I am using TSNE ...
14
votes
3answers
720 views

When are p-values deceptive?

What are the data conditions that we should watch out for, where p-values may not be the best way of deciding statistical significance? Are there specific problem types that fall into this category?
5
votes
3answers
2k views

Need help with LDA for selecting features

I am currently selecting features of products by using LDA to group 6000 keywords of product into topics. Here is the sample of my dataset after being organized into list of keywords for each product ...
4
votes
3answers
2k views

Learning resources for data science to win political campaigns?

Does anyone know, where I can learn about applying data science to win a political campaign? I know the Obama campaign had 12 data scientists in 2008 and 165 data scientists in 2012. In 2012, they ran ...
9
votes
3answers
5k views

Human activity recognition using smartphone data set problem

I'm new to this community and hopefully my question will well fit in here. As part of my undergraduate data analytics course I have choose to do the project on human activity recognition using ...
2
votes
3answers
3k views

Handling large imbalanced data set

I have an imbalanced data set consisting of some 10's of millions text strings, each with thousands of features created by uni- and bigrams, and additionally I have also the string length and entropy ...
6
votes
1answer
542 views

Is Data Science just a trend or is a long term concept? [closed]

I see a lot of courses in Data Science emerging in the last 2 years. Even big universities like Stanford and Columbia offers MS specifically in Data Science. But as long as I see, it looks like data ...
3
votes
1answer
3k views

Text Mining on Large Dataset

I have a large data set(460 Mb) which has a column - Log with 386551 rows. I wish to use clustering and N-Gram approach to form word cloud. My code is as follows: ...
0
votes
3answers
139 views

Seeking Career guidance [closed]

I am a recent graduate.Currently I work in a late stage startup (product company) as a Software Engineer where my responsibilities are to interact with customers, understand their environment and ...
6
votes
1answer
357 views

Database options for JSON storage, queried with Apache Drill

I am planning to set up a JSON storage system. It will store tens of millions of JSON records, all in the same format. I'd like to be able to query the data using Apache Drill. It looks like there is ...
4
votes
2answers
816 views

Anomaly detection in multiple parameters

I am a newbie to data science with a typical problem. I have a data set with metric1, metric2 and metric3. All these metrics are interdependent on each other. I want to detect anomalies in metric3. ...
4
votes
2answers
82 views

Pig script code error?

While running the below pig script I am getting an error in line4: If it is GROUP then I am getting error. If I change from 'GROUP' TO 'group' in line4, then the script is running. What is the ...
2
votes
2answers
128 views

How to calculate the elapsed time of a flag status per day?

I'd like to figure out the elapsed time between flag status changes. Simplified example: a person can only be sad or happy. I'd like to know how long each mood was active until it changed. I'm ...
1
vote
1answer
68 views

How to train a model using a daunting huge training dataset

I have an extremely huge dataset and I'm wondering me how could be the right way to set an experiment to use this data to train a model. I understand that I can use data-reduction to, for instance, ...
-4
votes
2answers
440 views

What is most likely the bare minimum knowledge one has to have to become data scientist? [closed]

I am a Python developer but I want to become a data scientist. My Question: At its core what is the bare minimum I need to have to make this transition? I know it cannot simply be that I need to ...