Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
2
votes
2answers
352 views

How much data to use for feature selection?

Working on my master's thesis, this is a problem I'm unable to find good resources about. I'm working with data of 18 participants, who are either active or passive. Each participant is then ...
2
votes
1answer
190 views

How to predict next year's gross revenue given this year's data?

I have the following dataset (.csv format) which contains: 100 columns: $\textbf{Period}$ (in years, e.g. 2017, 2018, ..., 2028), $\textbf{Gross Revenue}$, $\textbf{Region}$ (e.g. APAC, NEMEA, etc), ...
2
votes
1answer
55 views

does storing file in hdfs parallelize it for Spark?

For Spark's RDD operations, data must be in shape of RDD or be parallelized using: ParallelizedData = sc.parallelize(data) My question is that if I store data in ...
2
votes
1answer
809 views

What is Extreme Learning Machine? Why tuning of weights is not required?

The wikipedia says: Extreme learning machines are feedforward neural network for classification, regression, clustering, sparse approximation, compression and feature learning with a single ...
2
votes
1answer
1k views

Ingestion of periodic REST API Calls into Hadoop [closed]

We are planning to use REST API calls to ingest data from an endpoint and store the data to HDFS. The REST calls are done in a periodic fashion (daily or maybe hourly). I've already done Twitter ...
2
votes
1answer
3k views

how can I visualize the bag-of-words in a scatterplot

I have a bag-of-words consisting of about 60000 features. Each feature represents a diminsion. I want to represent this bag-of-words in a reduced 2D space. How do I do it? I have see an example here, ...
2
votes
1answer
206 views

Is datawarehouse considered as datalake in big data environnment?

Suppose I have a datawarehouse (DWH) and now I would like to add many other bigdata sources of information most of them are not structured. I still keep the DWH with no architectural change. The only ...
2
votes
3answers
2k views

Finding outliers in multiple dimensions

I'm working on a dataset which isn't normally distributed. The dataset contains three dimensions like cost, discount and profit. I'm trying to find possible outliers in all these dimensions. I used ...
2
votes
3answers
4k views

Examples of the Three V's of Big Data? [closed]

What are some examples of the Three V's of Big Data? The three V's stand for: volume, velocity, variety. Reference: Three V's of Big Data, provided by Norwegian University of Science and Technology. ...
2
votes
1answer
1k views

Optimizing Weka for large data sets

First of all, I hope I'm in the right StackExchange here. If not, apologies! I'm currently working with huge amounts of feature-value vectors. There are millions of these vectors (up to 20 million ...
2
votes
1answer
138 views

Can data analytics be a basis for artificial intelligence?

I'm very passionate about how computers can be made able to think intelligently and independently (in our favour, of course!). I'm currently studying Bachelors science of information technology at UTS ...
2
votes
1answer
521 views

How does skewed data affect deep neural networks?

I'm playing around with deep neural networks for a regression problem. The dataset I have is skewed right and for a linear regression model, I would typically perform a log transform. Should I be ...
2
votes
1answer
31 views

When training NN, how do data loaders work on large datasets?

How do you normally organize large data-sets for easy loading when training Neural Networks? I have a largeish data-set which cannot fit into memory, it consists of 200,000 samples, with 10k samples ...
2
votes
1answer
85 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
2
votes
2answers
2k views

How to build a symptom checker and medical diagnose chat bot

I want to build a chatbot which can diagnose an illness depending upon the symptoms which are given to it. I want to show you an example scenario of how it should work; Application - A , User - U ...
2
votes
1answer
992 views

product out of stock prediction

Is it possible to predict the products which are going to be out of stock in next coming days/weeks.? If Yes, what could be the possible data points that I should collect? How is this problem ...
2
votes
1answer
20 views

Yarn service parameter for pseudodistributed [closed]

I'm trying to set up Hadoop in the pseudodistributed form. So I edited the suggested xml files. In the file yarn-site.xml, I had to insert: ...
2
votes
1answer
160 views

How can I calculate Kernel matrix K for clustering based Kernel Principal Component Analysis?

In practice, a large data set leads to a large K, and storing K may become a problem. One way to deal with this is to perform clustering on the dataset, and populate the kernel with the means of those ...
2
votes
1answer
186 views

Stratified Sampling Variable Choice

I am trying to do stratified sampling in R to sample from my data and one of the parameters is group, which takes variable names to sample from keeping same initial distribution of the data set. Is ...
2
votes
1answer
59 views

Are there deduplication algorithms that do not work on a metric space?

Recently I got interested in the process of data cleansing and specifically in record linkage. Thus far I read about deterministic and probabilistic approaches to deduplicate data sets and to some ...
2
votes
2answers
683 views

Text mining of big data using R Server

I have a big table (150 Million rows and ~ 70 columns). In three of the columns in the table I have text input (3-20 words/column), which I need to use for a classification algorithm. For smaller ...
2
votes
1answer
409 views

Creating a pie chart with historical motion in Rstudio

So, I created several pie charts showing voting percentages for republicans and democrats, from 2008 to 2016 with plotly package in Rstudio. What I would like to do, to make it sleek, is create a menu ...
2
votes
3answers
688 views

Clustering big data by reducing data accuracy?

I have 1 million rows with 20 attributes to do hierarchical clustering. When I want to build a distance matrix on this data by dist() in R, it says that it needs 5 ...
2
votes
3answers
101 views

How to perform Cleaning of a very large set of addresses

I have a dataset where I have a large number of addresses. The problem lies in the fact that many addresses although they are same but haven't been noted down in the same manner. So I want to find ...
2
votes
2answers
237 views

How to go about obtaining data

I am a researcher working on traffic flow prediction. For my project I would need to obtain historical traffic flow raw data. Google Maps has a traffic flow overlay which is pretty much the kind of ...
2
votes
2answers
2k views

What models are used to get the Next Best Action to convert a physician from a non writer to a write?

Assume that a sales rep has actions A, B, C, D.... X that he can perform to influence a physician to write his drug I want to predict what the next best action a sales rep should take to have the ...
2
votes
1answer
872 views

What model should I use to find a common pattern for a specific user group based on the other dimensions?

I have a big .CSV database of 25k users with various attributes of the last user's activity and events during the past 6 weeks This is an example of the data: ...
2
votes
1answer
263 views

Writing custom data analysis program

I have a number of large datasets (10GBs) each with data fetched from a NoSQL database that I have remotely downloaded on my desktop. I would like to write a Python program to run some custom data ...
2
votes
1answer
1k views

Clustering and confusion matrix

Here is the problem The initial four cluster partition {c1, c2, c3, c4} for the text collection is provided by this link text vs cluster. Assuming that the ground-truth partition is given by ...
2
votes
2answers
148 views

Privacy through fake data?

With companies and governments hungry for all the data about people, I was wondering if it was possible to gain some privacy by drowning relevant information in a sea of random data. For example a ...
2
votes
1answer
136 views

Getting GitHub repository information by different criteria

New to the Data Science forum, and first poster here! This may be kind of a specific question (hopefully not too much so), but one I'd imagine others might be interested in. I'm looking for a way to ...
2
votes
3answers
2k views

Cannot make user directory on a new CDH5 installation (Hadoop)

I download and installed CDH 5 package succesfully on a single linux node in pseudo-distributed Mode on my CentOS 6.5 Starting Hadoop and verifying it is Working Properly as in this link I ...
2
votes
1answer
28 views

Data Art/ Data Visualization Art/ Information Art

Few days ago, I learned about data art/ data visualization art/ information art. I think I have interest in it. I want to see how I can use my data science skills in this area. However, I don't know ...
2
votes
0answers
105 views

XGBoost incremental training for big datasets

I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred). I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
2
votes
0answers
52 views

Clustering large set of images

I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
2
votes
0answers
33 views

What is the difference between Pachyderm and Git?

I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that: It holds all your data in a central accessible ...
2
votes
1answer
45 views

Size of datasets over years

I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years. I just found the following statistics: The poll of KDnuggets that actually shows that over ...
2
votes
0answers
68 views

Spark: How to run PCA parallelized? Only one thread used

I use pySpark and set my configuration like following: ...
2
votes
1answer
15k views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
2
votes
0answers
53 views

How to deal with large datasets? [closed]

I have some experience with data science but I wanted some insight on how to deal with a very large dataset. I understand simply downloading it to your computer is not plausible so where do you even ...
2
votes
0answers
363 views

Algorithm or JS graph drawing library that can generate a graph of 100,000+ nodes and edges while minimizing edge crossings

I'm trying to plot a directed graph of $2^{16}$ nodes and $2^{16}$ edges (but not simply a cycle). Ultimately, I need to be able to share an interactive graph (zooming, panning, labels). Mathematica ...
2
votes
0answers
758 views

Keras' fit_generator() is not calling my generator

When I call Keras' fit_generator(), passing in a custom generator class I created, I see "Epoch 1/1" in the spew and that's all. It hangs right there, and the ...
2
votes
0answers
30 views

Dimension of the manifold on which my data sits

Suppose that I have data points, in the form of vectors with binary entries. We create a metric space, or Vietoris-Rips complex, using the Hamming distance between the data points. I would like to ...
2
votes
1answer
63 views

How to fake data based on the condition and weight

I'm trying to fake data for the coffee shop. I've two features age and menu. Menu includes various type of drinks such as coffee [latte, espresso, mocca, etc], tea [milktea, lemontea], milk [freshmilk,...
2
votes
3answers
72 views

How to fill missing enteries in column A, and add respective corresponding enteries to column-B with value of previous cell

I am facing an issue with an excel file. I have an excel sheet with 2 columns Column A : Time Increment with per second Column B : A particular value of a machine sensor The problem i am facing ...
2
votes
0answers
1k views

Alternative to Apache Spark? [closed]

I have been looking for a comprehensive alternative to Apache Spark for Big Data analytics/machine learning and couldn't find one. The ones which I have come across are: Apache Flink Google DataFlow ...
2
votes
1answer
65 views

Work experience in Data Science: Couple hours with A-level student [closed]

Our company has accepted an A-level student (18-year old, just before uni) for some work experience. The student will be with me for a couple of hours. I do data analysis and I'm quite busy at the ...
2
votes
0answers
57 views

Can Google really bring back billions of results in a blink of an eye (almost) [closed]

I was always fascinated by Google's search ability, a great achievement by Google and other search engine providers also, but more so a collective human talent and ability that makes me appreciate our ...
2
votes
0answers
52 views

Split large dataset for predictive modeling using rsparkling -sparklyr

I am trying to build machine learning models (GBM, RF, Staking) on top of a dataset that is about 3G in size on my local computer. However, I only have 4G memory (only 2G are available). My question ...
2
votes
0answers
75 views

Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?

I am using a three node system: Master node: $64$ GB RAM 2 slave node: $32$ GB RAM I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function ...

1 2 3
4
5
10