Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
1
vote
0answers
3k views

Not enough replicas available for query at consistency all (2 required but only 1 alive)

I have a Cassandra cluster of 2 nodes and a keyspace "test" with a replication factor of 2. The consistency level is All. I am running a spark job which dumps data in Cassandra database. So, the ...
1
vote
0answers
383 views

Sensitivity analysis in outlier explanation

I am trying to find the outlier explanation using the sensitivity analysis. Let’s consider that my dataset contains 19 different input values and 1 output value (So overall 20 different columns are ...
0
votes
1answer
241 views

How to build recommendation system that also takes time as a feature

Let's say a User was Interested in football, so we were recommending him posts about football after few days or weeks he starts watching Baseball. How'd we go about building a recommender system that ...
2
votes
1answer
159 views

How can I calculate Kernel matrix K for clustering based Kernel Principal Component Analysis?

In practice, a large data set leads to a large K, and storing K may become a problem. One way to deal with this is to perform clustering on the dataset, and populate the kernel with the means of those ...
1
vote
0answers
65 views

Feedback Analysis

We are trying to rank the products on the basis of score, score is calculated by analyzing the good or bad feedback about the product. After a lot of research and analysis we came up with this formula ...
0
votes
1answer
5k views

What is the difference between observation and variable?

I have a matrix with size m×n that is built from n number of individuals for person identification. So, n is the number of person and m is the number of feature's value for the person. It makes me ...
2
votes
0answers
72 views

Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?

I am using a three node system: Master node: $64$ GB RAM 2 slave node: $32$ GB RAM I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function ...
2
votes
1answer
171 views

Stratified Sampling Variable Choice

I am trying to do stratified sampling in R to sample from my data and one of the parameters is group, which takes variable names to sample from keeping same initial distribution of the data set. Is ...
2
votes
1answer
58 views

Are there deduplication algorithms that do not work on a metric space?

Recently I got interested in the process of data cleansing and specifically in record linkage. Thus far I read about deterministic and probabilistic approaches to deduplicate data sets and to some ...
1
vote
2answers
81 views

What is the easiest way to scale a data science project based on scikit stack?

This is an issue for all Data Scientists who have worked with this stack: python scikit-learn scipy-stats matplotlib etc. We are looking for ways to have a project already implemented in the ...
0
votes
1answer
136 views

Spliting training data into multiple variables using R

So right now I am trying to create multiple variables with training data, and in the process I have reached an error Error in eval(predvars, data, env) : object '1.band1' not found which is a ...
4
votes
1answer
91 views

Is MLlib compulsory to work with distributed data?

I am new to the field of machine learning and recently learned the basics and working out various algorithms in python using libraries such as pandas, numpy, matplotlib, scikitlearn , etc. ...
3
votes
2answers
3k views

Is Java or Python a better choice for an application involving data intensive algorithms employing natural language processing?

I am a professional java developer with 9 years of experience. Never worked with Python but, I do not have any restraints learning it. Still I need clarity on a couple of aspects. I hope if anyone ...
2
votes
1answer
781 views

What is Extreme Learning Machine? Why tuning of weights is not required?

The wikipedia says: Extreme learning machines are feedforward neural network for classification, regression, clustering, sparse approximation, compression and feature learning with a single ...
1
vote
1answer
521 views

R: reading big data files in R

In R, there are libraries which help speed up the process of reading huge data files. Examples of these libraries include sqldf and ff. What would be the disadvantage of using these packages to read ...
0
votes
2answers
34 views

R: I need to understand the mechanism behind reading a subset of a file in R (e.g., via sqldf) or other data-science-centered programming language

By reading only a subset of the records of a file into R (or Python), one can speed up performance tremendously. I am curious about the mechanism behind this kind of file input. To read only a small ...
-1
votes
1answer
292 views

How to visualise very large graphs with 250M nodes and 500M+ edges? [closed]

I have graph data which has 250 million nodes and over 500 million edges. My files are roughly 230GB is size, this includes the node and edge files. I am having major difficulties visualising the ...
0
votes
1answer
60 views

How much data warrants building a pipeline?

If I'm doing simple aggregation dashboards, what's the minimum amount of data which justifies building a full blown data processing pipeline? Is it reasonable to build a complicated pipeline with ...
-3
votes
1answer
2k views

Which coding language to use for very large datasets? [closed]

I have a very large panel data set (contains around 50M observations, size around 3G). I would to run an algorithm on it. The algorithm basically just loops over observations. Ideally, I would like to ...
1
vote
0answers
867 views

XGBoost: predict on only valuable features

I have a somewhat strange case that I can't find an answer to anywhere. It is really only applicable if you have a legitimately large data set. Let me describe. XGBoost returns feature importance for ...
2
votes
2answers
2k views

Handling large imbalanced data set

I have an imbalanced data set consisting of some 10's of millions text strings, each with thousands of features created by uni- and bigrams, and additionally I have also the string length and entropy ...
1
vote
0answers
86 views

Data brokers for market-related data, how to choose, where to look for?

Situation I am doing a data science project for my client. Due to NDA I can't disclose anything to the nature of the project, but let's imagine that they manufacture and sell slow cookers. I faced ...
1
vote
1answer
527 views

Running SQL-like queries over large schemaless JSON dataset in the cloud?

I've got about 5 million JSON files, about 50GB in total. They do not have a consistent schema (they're broadly the same format, but some have extra extension fields, some have missing fields, etc - ...
1
vote
1answer
6k views

Plot RDD data using a pyspark dataframe from csv file [closed]

I have just started working with pyspark on very large csv file. I am using Spark version 2.1.0. I want to read data from a .csv file and load it into a spark dataframe and then after filtering ...
1
vote
1answer
36 views

Techniques to clean the topics

I have set of topics as follows. "web based", "web-based" -> with surplus symbols "technology","technologies" -> with singular and plural "learned", "learnt", "learning" ->suffix stripping Can you ...
1
vote
1answer
245 views

What could be a right datascience approach of doing fuzzy string matching in large amount of short text data?

I have two lists to compare. List 1 contains strings which represent the strings in List2. Sometimes, they are direct matches, ...
1
vote
1answer
310 views

Yelp Dataset Archives?

The Yelp Dataset Challenge (https://www.yelp.com/dataset_challenge) releases data for a handful of cities each year. I'd like to analyze some cities from past years. Is this data archived anywhere?
5
votes
1answer
862 views

Array of categorical variables vs one-hot encoding

I have some JSON data to be transformed to a machine-learning friendly format. Every object in my data, which will eventually become an instance in my dataset, has the exact same fields (in this case, ...
2
votes
2answers
673 views

Text mining of big data using R Server

I have a big table (150 Million rows and ~ 70 columns). In three of the columns in the table I have text input (3-20 words/column), which I need to use for a classification algorithm. For smaller ...
-3
votes
1answer
105 views

Are there any programmers who might find programmable heuristics useful? [closed]

I would like to know if my recent work, which is essentially free to implement, has data applications. More obvious applications include: Guessing questions. Logic / objectivity. Psychic prediction. ...
-1
votes
1answer
75 views

What are the differences between IBM BlueMix and IBM Data Science Experience?

This may seem like a silly question, but as I am going through the documentation for both services it is difficult to disentangle what each does, specifically. From what I've gathered BlueMix is ...
8
votes
2answers
818 views

Understanding how distributed PCA works

As part of big data analysis project, I'm working on, I need to perform PCA on some data, using cloud computing system. In my case, I'm using Amazon EMR for the job and Spark in particular. Leaving ...
1
vote
1answer
133 views

what ETL technique should i use for text documents using Hadoop?

I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
-1
votes
2answers
52 views

what is the proper tool to analyse data and find trends in my case? [closed]

I have a database of hundreds of thousands of loans. I want to find some trends in user behavior, not just some information regarding all loans together but I want to find something similar for ...
0
votes
1answer
79 views

City pollution data for data mining project [closed]

I want to do my MSc Project on Data Mining. And I want to work on pollution in cities. Are there accessible and useable pollution data (any type of pollution. e.g. air, water or forest pollution) in ...
1
vote
2answers
373 views

Can R + Hadoop overcome R's memory constraints in any case?

I am an R user and I am interested in learning/understanding how Hadoop actually works. For this I previously read about Hadoop but was not able to find a satisfactory answer for my question. Can R + ...
2
votes
3answers
7k views

Working with large datasets python

My data file is of 4 GB (json), I need to apply everything on this dataset (from applying clustering/ML algorithm to wrangling it) like the way people do with pandas/scikit. But it is difficult to ...
1
vote
3answers
1k views

How to improve an existing machine learning classifier in python?

I have a big dataset (1million x 50) to which I want to predict a particular class. I have thought of segregating the dataset in batches of 20k. And then train a classifier (lets say random forest or ...
3
votes
1answer
243 views

Deploying models on bigdata platforms like Hadoop and Spark

Where exactly bigdata platforms fit in to a data science/ machine learning projects ? Say I have a large dataset for a binary classification problem - cats and dogs. Now I need to create a model for ...
2
votes
1answer
1k views

Ingestion of periodic REST API Calls into Hadoop [closed]

We are planning to use REST API calls to ingest data from an endpoint and store the data to HDFS. The REST calls are done in a periodic fashion (daily or maybe hourly). I've already done Twitter ...
2
votes
1answer
431 views

Dataset: language audio clips and country labels

I'm looking for a dataset containing audio clips of different languages from around the world, including a label indicating which country the clip comes from. Does such a dataset exist? Or is there ...
1
vote
1answer
79 views

Uploading huge dataset

I have few questions: Is there a website to upload huge research dataset (over 100GB) for free? Which type of compression (rar, zip ... etc) is good for jpeg images? In case of dataset of 120GB. what ...
4
votes
0answers
95 views

Fixed-radius range search in non-Euclidean space

I'm trying to find an indexing data structure most suitable for my metric space: set of IP network related data (IP addresses, ports, TCP flags, ...), distance function is continuous, non-Euclidean ...
2
votes
1answer
215 views

K-means clustering on big data stored on multiple nodes on HDFS

I have a very basic question about how the k-means clustering algorithm works when the data is stored in HDFS. Does the training take place on the subset of data on each node for a fixed value of K( ...
10
votes
1answer
2k views

Avoid reloading DataFrame between different python kernels

Is there a way of keeping a variable (large table / data frame) in memory and share it across multiple ipython notebooks? I'd be looking for something, which is conceptually similar to MATLAB's ...
2
votes
1answer
408 views

Creating a pie chart with historical motion in Rstudio

So, I created several pie charts showing voting percentages for republicans and democrats, from 2008 to 2016 with plotly package in Rstudio. What I would like to do, to make it sleek, is create a menu ...
2
votes
3answers
2k views

Instead of one-hot encoding, can I store the same information in one column using a single value?

Rather than creating 15 additional columns full of sparse binary data, could I: 1) use the first 15 prime numbers as indexes for the 15 categories 2) store data by multiplying the prime numbers of ...
0
votes
1answer
157 views

How to develop a good model for predicting electricity usage? [closed]

My goal is to come up with a set of research questions for which to drive my self study into my data. I have Smart Meter data that is taken every minute, on my home's power usage. How could I ...
1
vote
0answers
63 views

What tools to manage multiple machine learning projects in python do you use?

I will soon deal with multiple projects in python. Some of them have to run regularly (many times a day), they can take some time (many days) and they use/produce some data coming from and going to ...
2
votes
1answer
3k views

how can I visualize the bag-of-words in a scatterplot

I have a bag-of-words consisting of about 60000 features. Each feature represents a diminsion. I want to represent this bag-of-words in a reduced 2D space. How do I do it? I have see an example here, ...

1
3 4
5
6 7
9