Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
1
vote
1answer
22 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
1
vote
0answers
29 views

How to deal with large datasets?

I have some experience with data science but I wanted some insight on how to deal with a very large dataset. I understand simply downloading it to your computer is not plausible so where do you even ...
0
votes
0answers
14 views

Difference between Big Data and a Data Warehouse?

I would like to know what is the exact difference between Big Data and a Data Warehouse. I know that Big Data is the technology to receive and work with really large amounts of data and a data ...
0
votes
0answers
20 views

Running LSTM model on a big data sample using pyspark

I was wondering how does one run an LSTM model on a big dataframe in pyspark. Ideally, one wants to run the model parallelly on different nodes of a spark cluster. But how does one do that?
0
votes
0answers
11 views

Free historical news data base?

I spent some time looking for news databases and APIs. I can find a lot of paid ones. The data google provides when searching for news is perfect for what I am after (something with a deep history, ...
0
votes
0answers
8 views

Relational Model Vs Aggregate-oriented Model

Can you explain to me why the relational model is not good to deal with aggregates? At the contrary, in which cases we can prefer a relational model to an aggregate-oriented model ? Thanks
0
votes
0answers
13 views

Who is the biggest competitor to Google BigQuery for big data processing

I have terabytes of data, and I would like to decide on a cloud solution provider for the big data storage and processing. I am not sure which one I should chose. I am aware of few technologies out ...
0
votes
0answers
12 views

How to get a sample/summarised data from a table for data analysis?

I am currently working on EHR data analysis. I have a measurement table which has information lab test information of the patients. However the record count is huge. These are table count in my EHR ...
0
votes
0answers
10 views

Most reliable source for Literacy data

I'm searching for the most reliable source for granular American literacy population data. I am conducting an analysis, reporting, and data visualization of American literacy. To do this, I am ...
0
votes
2answers
74 views

Question about Similarity vs Dissimilarity Matrix

Right now, I'm working on a coming up with a similarity vs dissimilarity matrix for a set of data points for a clustering algorithm. My question is, if I want to use one of the many clustering ...
0
votes
2answers
29 views

Question About Coming Up With Own Function for Distance Matrix (For Clustering)

Right now, I am currently working on implementing a clustering algorithm with millions data entries with regards to game users for a mobile game. A lot of the features I plan on using are unique to ...
0
votes
2answers
48 views

Data visualisation for big data sets

Working on some voluminous data sets, I have been interested in methods for dimension reduction and plotting. I have stumbled upon this novel technique : UMAP (https://arxiv.org/pdf/1802.03426.pdf), ...
0
votes
0answers
32 views

How to read binary video format in HDFS using pyspark?

We want to work on detection of pedestrians in a video using ML algorithm.But the video is in the HDFS file system.How can we read the binary video file format from HDFS using spark? Please help me ...
1
vote
0answers
37 views

Big dataset for multi-class classification can't be dasked and split, normal one can't be handled

I have a huge dataframe (550MB), the lending club one available here, and I have to predict the class of the grades. The dask dataframe is : ...
1
vote
2answers
49 views

Wanting to overfit model to say if data has been seen before

I would like to train a scalable model that has as an input row of a database and has an output of either 1 or 0 depending upon whether it has seen this entry of the database before or not. The ...
1
vote
1answer
59 views

How to reduce dimensionality of 3.2B categorical features?

Background: This means a dataset of 7,000 samples and 3.2B columns, which I would have to read into distributed Spark memory somehow. Obviously I want to reduce the number of columns that gets fed ...
0
votes
0answers
10 views

How to prepare for machine learning?

I'm working on a project that has weather data and information related to agriculture. Our goal is, in the long term, start identifying patterns and better understand how can we help the users to ...
1
vote
0answers
9 views

Test independence based on Kernel Density Estimation

I am working on a problem where I have a dataset of $X$ is dataset with $(X, Y, T, K)$ four attributes, I'd like to test if $P(X, Y, T)P(K) = P(X, Y, T, K)$, that is if $X, Y, T$ is independent of $K$....
0
votes
1answer
41 views

GPS generate street [closed]

I am working on GPS tracking with a huge data from vehicle. Dataset have: vehicleId, speed, orientation(0-360), coordinate (x,y) and timestamp. Can you recommend me how to clean data and model to ...
0
votes
0answers
17 views

Sampling user IDs from a large data set for retention analysis

Let's say I have a data set with user identities and their respective orders, of the form: ...
1
vote
0answers
45 views

Most efficient algorithm for sequential pattern mining on a dataset with large amount of items in each transaction?

To provide some context, I am trying to do frequent pattern mining on a dataset of system error logs from servers. I organized it into transactions based on the thread ID, which results in some very ...
1
vote
0answers
13 views

How to create tensors in spark?

I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
1
vote
0answers
27 views

Mapreduce jobs not working in hive

I was trying to execute a query select name, count(*) from amazon where review != NULL group by name ; Number of reduce tasks not specified. Estimated ...
0
votes
0answers
10 views

Implementing Data Catalog from scratch

I have a requirement of ingesting huge amount of user specific data and then searching on it. Trying to be low cost solution, I am leaning towards creating Kafka based ingestion pipelines and ...
0
votes
1answer
31 views

Is a new datapoint similar to my reference set?

I have a large reference data set and everyday I get new data points. I would like to know how can I test if those new data points are similar to the reference set? Thanks in advance. Maria
0
votes
1answer
14 views

Data driven project for a self-catering accommodation business

I want to apologise in advance for my ignorance but I'm hitting my head in a wall here as I'm not sure how to proceed with my project and I've got no experience in that field even though I'm doing a ...
1
vote
0answers
32 views

Process and clean a huge .tar file with millions of small XML files

I have one huge .tar file (~700GB) which includes millions of small XML files (<1MB each). The dataset has a lot of unnecessary information that I don't need in ...
0
votes
0answers
9 views

What statistical test would I use to test that both data sources are matching

Suppose I am only working with two datasets, Data Source 1 with columns Customer ID and ...
0
votes
0answers
15 views

How to do numpy matmul broadcasting between two numpy tensors?

I have the Pauli matrices which are (2x2) and complex ...
0
votes
0answers
18 views

How to aggregate large data

I have a quite large data (frame) with 65 physical quantities and each with different time stamps. Some are gathered in intervals of several hours and some in milliseconds. Hence, the data frame ...
0
votes
0answers
9 views

Is veracity in big data theory a good thing to use big data techniques in organization?

I am new to big data field. I know the basic 3V's of it, but I am not understanding the newly added one, which is Veracity. Is it a good thing, which led to taking great analysis from inaccurate data, ...
0
votes
0answers
27 views

Python Machine Learning to analyse test-case lab data and identify new test cases from real-world random data

I am currently working on a project where I have predefined test-cases I need to run/test and then analyse the data. My aim is to be able to identify system failures (against pre-defined criteria). ...
0
votes
0answers
17 views

How to generate artificial datasets in MATLAB based on behavior of real datasets?

To understand: Let's consider a scenario. I have real datasets of 2 buildings showing different behaviors of occupants. Now to expand the analysis for multiple buildings, I want to generate simulated ...
1
vote
0answers
20 views

Is there a way to use a pom.xml file to update spark configuration?

I am trying to update my spark configuration to solve some dependency problems. This pom.xml file seems to be useful for this purpose. I am using a spark docker image. ...
4
votes
1answer
221 views

XGBoost Huge Dataset ~1TB

Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints. Is ...
0
votes
0answers
11 views

Algorithm/approach to predict multiple questions in a survey

I have I set of 6k surveys about personal tastes (40 columns with answer number) and another set of 50M cookies about web activities (columns are all continuous). Some of these 50M can be connected to ...
1
vote
1answer
25 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
0
votes
0answers
8 views

How to reduce complexity of inference stage in recommender systems?

Given a large set of customers and a large set of items, how to make predictions given a model like this one: https://arxiv.org/pdf/1606.07792.pdf As stated in the article: "Since there are over a ...
1
vote
2answers
44 views

Classification task - how to handle large data file? [closed]

I'm planning to construct a classification model for predicting New York taxi trip fare. The CSV-datafile for this is very large, containing 112 234 626 rows (ca 10 GB). I have managed to download ...
0
votes
1answer
21 views

Parsing and storing a large amount of HTML data

I have a data chunk (~30k) in which I have htmls pages and pngs saved in a folder for websites. These folders are titled based on some randomly generated hashes. My supervisor wants me to crunch ...
2
votes
2answers
106 views

How to calculate the elapsed time of a flag status per day?

I'd like to figure out the elapsed time between flag status changes. Simplified example: a person can only be sad or happy. I'd like to know how long each mood was active until it changed. I'm ...
0
votes
0answers
205 views

Why modin.pandas takes more wall time than pandas?

All data scientist know that to scale the pandas to large dataset, we use modin.pandas instead of pandas. But when I tried to load a 500 MB csv dataset in my jupyter notebook in Ubuntu 16.04 to ...
0
votes
1answer
37 views

Plot misclassfied values in matplotlib

I have two lists. One is target and another is predicted values in a binary classification. The target is always zero. And I have to plot the results of the prediction against target in such a way, ...
1
vote
3answers
56 views

What is the correct procedure when “joining” data takes ~6 hours?

I am dealing with bike-share data. I have 2 DataFrames: trips_df (subset shown), total entries = ...
0
votes
0answers
293 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
1
vote
1answer
31 views

xgboostclassifier prediction error after saving the model and restoring it

I have trained a xgboost model and during training, the prediction works fine. But if I stop the script and start a restoring script to restore and predict, then for the same test dataset I get every ...
0
votes
0answers
21 views

Difference between retraining on different parts of the data and training initially on larger data set

I have a large data set that doesn't fit in memory and would have to use something like Keras's model.fit_generator if I would like to train the model on all of the ...
0
votes
1answer
14 views

what does “blocks” mean in the context of HDFS ecosystem? is it the same concept in the context of a single hard disk drive?

According to Hortonworks HDFS has demonstrated production scalability of up to 200 PB of storage and a single cluster of 4500 servers, supporting close to a billion files and blocks. what does ...
0
votes
0answers
92 views

Apache Nifi out of memory

I've set up a data pipeline using Apache nifi. It processes 3GB scale json data. When I try to flatten these Json files it always outputs: ...
2
votes
1answer
1k views

Training Keras model with multiple CSV files

I'm currently trying to train a Keras model on several large CSV files. I can fit one in memory, but not all combined. From my point of view, there are several ways to deal with this problem. I could ...