Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
0
votes
1answer
20 views

Time series analysis

I'm developing a prediction model that can predict the number of patients that will be admitted at a particular time in a hospital. My dataset has details like admission date, admission time and ...
-1
votes
0answers
8 views

Display direction of GPS by gmplot in python

I am working with gmplot to show many gps points. I got stuck with how to display direction of GPS point. Please Let me how to do it. Thank you
0
votes
2answers
35 views

Question about Similarity vs Dissimilarity Matrix

Right now, I'm working on a coming up with a similarity vs dissimilarity matrix for a set of data points for a clustering algorithm. My question is, if I want to use one of the many clustering ...
0
votes
2answers
28 views

Question About Coming Up With Own Function for Distance Matrix (For Clustering)

Right now, I am currently working on implementing a clustering algorithm with millions data entries with regards to game users for a mobile game. A lot of the features I plan on using are unique to ...
0
votes
2answers
42 views

Data visualisation for big data sets

Working on some voluminous data sets, I have been interested in methods for dimension reduction and plotting. I have stumbled upon this novel technique : UMAP (https://arxiv.org/pdf/1802.03426.pdf), ...
0
votes
0answers
31 views

How to read binary video format in HDFS using pyspark?

We want to work on detection of pedestrians in a video using ML algorithm.But the video is in the HDFS file system.How can we read the binary video file format from HDFS using spark? Please help me ...
1
vote
0answers
20 views

Big dataset for multi-class classification can't be dasked and split, normal one can't be handled

I have a huge dataframe (550MB), the lending club one available here, and I have to predict the class of the grades. The dask dataframe is : ...
1
vote
2answers
48 views

Wanting to overfit model to say if data has been seen before

I would like to train a scalable model that has as an input row of a database and has an output of either 1 or 0 depending upon whether it has seen this entry of the database before or not. The ...
1
vote
1answer
53 views

How to reduce dimensionality of 3.2B categorical features?

Background: This means a dataset of 7,000 samples and 3.2B columns, which I would have to read into distributed Spark memory somehow. Obviously I want to reduce the number of columns that gets fed ...
0
votes
0answers
8 views

How to prepare for machine learning?

I'm working on a project that has weather data and information related to agriculture. Our goal is, in the long term, start identifying patterns and better understand how can we help the users to ...
1
vote
0answers
8 views

Test independence based on Kernel Density Estimation

I am working on a problem where I have a dataset of $X$ is dataset with $(X, Y, T, K)$ four attributes, I'd like to test if $P(X, Y, T)P(K) = P(X, Y, T, K)$, that is if $X, Y, T$ is independent of $K$....
0
votes
1answer
38 views

GPS generate street [closed]

I am working on GPS tracking with a huge data from vehicle. Dataset have: vehicleId, speed, orientation(0-360), coordinate (x,y) and timestamp. Can you recommend me how to clean data and model to ...
0
votes
0answers
17 views

Sampling user IDs from a large data set for retention analysis

Let's say I have a data set with user identities and their respective orders, of the form: ...
1
vote
0answers
39 views

Most efficient algorithm for sequential pattern mining on a dataset with large amount of items in each transaction?

To provide some context, I am trying to do frequent pattern mining on a dataset of system error logs from servers. I organized it into transactions based on the thread ID, which results in some very ...
1
vote
0answers
11 views

How to create tensors in spark?

I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
1
vote
0answers
21 views

Mapreduce jobs not working in hive

I was trying to execute a query select name, count(*) from amazon where review != NULL group by name ; Number of reduce tasks not specified. Estimated ...
0
votes
0answers
8 views

Implementing Data Catalog from scratch

I have a requirement of ingesting huge amount of user specific data and then searching on it. Trying to be low cost solution, I am leaning towards creating Kafka based ingestion pipelines and ...
0
votes
1answer
29 views

Is a new datapoint similar to my reference set?

I have a large reference data set and everyday I get new data points. I would like to know how can I test if those new data points are similar to the reference set? Thanks in advance. Maria
0
votes
1answer
14 views

Data driven project for a self-catering accommodation business

I want to apologise in advance for my ignorance but I'm hitting my head in a wall here as I'm not sure how to proceed with my project and I've got no experience in that field even though I'm doing a ...
1
vote
0answers
32 views

Process and clean a huge .tar file with millions of small XML files

I have one huge .tar file (~700GB) which includes millions of small XML files (<1MB each). The dataset has a lot of unnecessary information that I don't need in ...
0
votes
0answers
9 views

What statistical test would I use to test that both data sources are matching

Suppose I am only working with two datasets, Data Source 1 with columns Customer ID and ...
0
votes
0answers
12 views

How to do numpy matmul broadcasting between two numpy tensors?

I have the Pauli matrices which are (2x2) and complex ...
0
votes
0answers
18 views

How to aggregate large data

I have a quite large data (frame) with 65 physical quantities and each with different time stamps. Some are gathered in intervals of several hours and some in milliseconds. Hence, the data frame ...
0
votes
0answers
9 views

Is veracity in big data theory a good thing to use big data techniques in organization?

I am new to big data field. I know the basic 3V's of it, but I am not understanding the newly added one, which is Veracity. Is it a good thing, which led to taking great analysis from inaccurate data, ...
0
votes
0answers
27 views

Python Machine Learning to analyse test-case lab data and identify new test cases from real-world random data

I am currently working on a project where I have predefined test-cases I need to run/test and then analyse the data. My aim is to be able to identify system failures (against pre-defined criteria). ...
0
votes
0answers
16 views

How to generate artificial datasets in MATLAB based on behavior of real datasets?

To understand: Let's consider a scenario. I have real datasets of 2 buildings showing different behaviors of occupants. Now to expand the analysis for multiple buildings, I want to generate simulated ...
1
vote
0answers
20 views

Is there a way to use a pom.xml file to update spark configuration?

I am trying to update my spark configuration to solve some dependency problems. This pom.xml file seems to be useful for this purpose. I am using a spark docker image. ...
4
votes
1answer
161 views

XGBoost Huge Dataset ~1TB

Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints. Is ...
0
votes
0answers
11 views

Algorithm/approach to predict multiple questions in a survey

I have I set of 6k surveys about personal tastes (40 columns with answer number) and another set of 50M cookies about web activities (columns are all continuous). Some of these 50M can be connected to ...
1
vote
1answer
22 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
0
votes
0answers
7 views

How to reduce complexity of inference stage in recommender systems?

Given a large set of customers and a large set of items, how to make predictions given a model like this one: https://arxiv.org/pdf/1606.07792.pdf As stated in the article: "Since there are over a ...
1
vote
2answers
42 views

Classification task - how to handle large data file? [closed]

I'm planning to construct a classification model for predicting New York taxi trip fare. The CSV-datafile for this is very large, containing 112 234 626 rows (ca 10 GB). I have managed to download ...
0
votes
1answer
17 views

Parsing and storing a large amount of HTML data

I have a data chunk (~30k) in which I have htmls pages and pngs saved in a folder for websites. These folders are titled based on some randomly generated hashes. My supervisor wants me to crunch ...
2
votes
2answers
105 views

How to calculate the elapsed time of a flag status per day?

I'd like to figure out the elapsed time between flag status changes. Simplified example: a person can only be sad or happy. I'd like to know how long each mood was active until it changed. I'm ...
0
votes
0answers
129 views

Why modin.pandas takes more wall time than pandas?

All data scientist know that to scale the pandas to large dataset, we use modin.pandas instead of pandas. But when I tried to load a 500 MB csv dataset in my jupyter notebook in Ubuntu 16.04 to ...
0
votes
1answer
36 views

Plot misclassfied values in matplotlib

I have two lists. One is target and another is predicted values in a binary classification. The target is always zero. And I have to plot the results of the prediction against target in such a way, ...
1
vote
3answers
55 views

What is the correct procedure when “joining” data takes ~6 hours?

I am dealing with bike-share data. I have 2 DataFrames: trips_df (subset shown), total entries = ...
0
votes
0answers
237 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
1
vote
1answer
31 views

xgboostclassifier prediction error after saving the model and restoring it

I have trained a xgboost model and during training, the prediction works fine. But if I stop the script and start a restoring script to restore and predict, then for the same test dataset I get every ...
0
votes
0answers
20 views

Difference between retraining on different parts of the data and training initially on larger data set

I have a large data set that doesn't fit in memory and would have to use something like Keras's model.fit_generator if I would like to train the model on all of the ...
0
votes
1answer
14 views

what does “blocks” mean in the context of HDFS ecosystem? is it the same concept in the context of a single hard disk drive?

According to Hortonworks HDFS has demonstrated production scalability of up to 200 PB of storage and a single cluster of 4500 servers, supporting close to a billion files and blocks. what does ...
0
votes
0answers
69 views

Apache Nifi out of memory

I've set up a data pipeline using Apache nifi. It processes 3GB scale json data. When I try to flatten these Json files it always outputs: ...
2
votes
1answer
773 views

Training Keras model with multiple CSV files

I'm currently trying to train a Keras model on several large CSV files. I can fit one in memory, but not all combined. From my point of view, there are several ways to deal with this problem. I could ...
3
votes
1answer
25 views

Reducing search iteration over millions of data

The problem goes like this, with a story. You have an application that contains a search field. When you search some input, there's an auto-complete component that pops up showing up similar results ...
0
votes
0answers
10 views

How to predict sentences using numeric features in a DL model?

I know of Deep Learning and ML models which takes numeric features as inputs, and are able to predict or classify the target, which works well when doing things like binary or multi class ...
0
votes
1answer
43 views

What is data lake?

I am familiar with the concept of Big data but how data lake is different from Big data.or is it derived from big data.Please explain.
0
votes
0answers
9 views

How to load data from posgres to bigquery

Our Postgres tables are too huge and the columns are structured in a way that it has become difficult to query the tables. So we have started using BigQuery to help us query into huge tables. We ...
4
votes
1answer
73 views

How can one quickly look up people from a large database?

Vocabulary Face detection: Finding all faces in an image. Face representation: The simplest way to represent a face is as an image (pixels / color values). This is not very space efficient and likely ...
0
votes
0answers
152 views

Using Keras.utils.Sequence on out of memory dataset

I'm trying to implement a sub-class of keras.util.sequence so that i can load data faster into the function fit_generator. Currently I have a multi-variate time series (multiple features) file which ...
0
votes
0answers
11 views

Hints on my EMR Spark Cluster

I am new to Spark, in particular I am tying to find out which are the best recommended properties to set to work on a 1 master 2 slaves cluster with 10GB memory each. Do you know which are a good set ...