Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

1
vote
0answers
16 views

Is there a way to use a pom.xml file to update spark configuration?

I am trying to update my spark configuration to solve some dependency problems. This pom.xml file seems to be useful for this purpose. I am using a spark docker image. ...
3
votes
0answers
40 views

XGBoost Huge Dataset ~1TB

Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints. Is ...
0
votes
0answers
10 views

Algorithm/approach to predict multiple questions in a survey

I have I set of 6k surveys about personal tastes (40 columns with answer number) and another set of 50M cookies about web activities (columns are all continuous). Some of these 50M can be connected to ...
1
vote
1answer
18 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
0
votes
0answers
7 views

How to reduce complexity of inference stage in recommender systems?

Given a large set of customers and a large set of items, how to make predictions given a model like this one: https://arxiv.org/pdf/1606.07792.pdf As stated in the article: "Since there are over a ...
1
vote
2answers
35 views

Classification task - how to handle large data file? [closed]

I'm planning to construct a classification model for predicting New York taxi trip fare. The CSV-datafile for this is very large, containing 112 234 626 rows (ca 10 GB). I have managed to download ...
0
votes
1answer
12 views

Parsing and storing a large amount of HTML data

I have a data chunk (~30k) in which I have htmls pages and pngs saved in a folder for websites. These folders are titled based on some randomly generated hashes. My supervisor wants me to crunch ...
2
votes
2answers
99 views

How to calculate the elapsed time of a flag status per day?

I'd like to figure out the elapsed time between flag status changes. Simplified example: a person can only be sad or happy. I'd like to know how long each mood was active until it changed. I'm ...
0
votes
0answers
40 views

Why modin.pandas takes more wall time than pandas?

All data scientist know that to scale the pandas to large dataset, we use modin.pandas instead of pandas. But when I tried to load a 500 MB csv dataset in my jupyter notebook in Ubuntu 16.04 to ...
0
votes
1answer
27 views

Plot misclassfied values in matplotlib

I have two lists. One is target and another is predicted values in a binary classification. The target is always zero. And I have to plot the results of the prediction against target in such a way, ...
1
vote
3answers
44 views

What is the correct procedure when “joining” data takes ~6 hours?

I am dealing with bike-share data. I have 2 DataFrames: trips_df (subset shown), total entries = ...
0
votes
0answers
16 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
1
vote
1answer
13 views

xgboostclassifier prediction error after saving the model and restoring it

I have trained a xgboost model and during training, the prediction works fine. But if I stop the script and start a restoring script to restore and predict, then for the same test dataset I get every ...
0
votes
0answers
15 views

Difference between retraining on different parts of the data and training initially on larger data set

I have a large data set that doesn't fit in memory and would have to use something like Keras's model.fit_generator if I would like to train the model on all of the ...
0
votes
1answer
14 views

what does “blocks” mean in the context of HDFS ecosystem? is it the same concept in the context of a single hard disk drive?

According to Hortonworks HDFS has demonstrated production scalability of up to 200 PB of storage and a single cluster of 4500 servers, supporting close to a billion files and blocks. what does ...
0
votes
0answers
17 views

Apache Nifi out of memory

I've set up a data pipeline using Apache nifi. It processes 3GB scale json data. When I try to flatten these Json files it always outputs: ...
2
votes
1answer
148 views

Training Keras model with multiple CSV files

I'm currently trying to train a Keras model on several large CSV files. I can fit one in memory, but not all combined. From my point of view, there are several ways to deal with this problem. I could ...
3
votes
1answer
25 views

Reducing search iteration over millions of data

The problem goes like this, with a story. You have an application that contains a search field. When you search some input, there's an auto-complete component that pops up showing up similar results ...
0
votes
0answers
9 views

How to predict sentences using numeric features in a DL model?

I know of Deep Learning and ML models which takes numeric features as inputs, and are able to predict or classify the target, which works well when doing things like binary or multi class ...
0
votes
1answer
35 views

What is data lake?

I am familiar with the concept of Big data but how data lake is different from Big data.or is it derived from big data.Please explain.
0
votes
0answers
6 views

How to load data from posgres to bigquery

Our Postgres tables are too huge and the columns are structured in a way that it has become difficult to query the tables. So we have started using BigQuery to help us query into huge tables. We ...
4
votes
1answer
69 views

How can one quickly look up people from a large database?

Vocabulary Face detection: Finding all faces in an image. Face representation: The simplest way to represent a face is as an image (pixels / color values). This is not very space efficient and likely ...
0
votes
0answers
63 views

Using Keras.utils.Sequence on out of memory dataset

I'm trying to implement a sub-class of keras.util.sequence so that i can load data faster into the function fit_generator. Currently I have a multi-variate time series (multiple features) file which ...
0
votes
0answers
5 views

Hints on my EMR Spark Cluster

I am new to Spark, in particular I am tying to find out which are the best recommended properties to set to work on a 1 master 2 slaves cluster with 10GB memory each. Do you know which are a good set ...
0
votes
0answers
6 views

Create a local subset copy of a cloud based MS SQL database

I have a large MS SQL database hosted on Azure. I want to develop an analysis pipeline for the data. I'm developing this pipeline locally and it's slowing down my development. I think if I had a local ...
0
votes
0answers
87 views

How to build a symptom checker and medical diagnose chat bot

I want to build a chatbot which can diagnose an illness depending upon the symptoms which are given to it. I want to show you an example scenario of how it should work; Application - A , User - U ...
0
votes
1answer
17 views

Sampling Big Data for Predictive Analytics in Python

In practice, how does one go about sampling a from big data set (eg. +/- 50 million distinct observations) to perform ML using Python? Most non-parametric models (e.g., SVM, ensemble models) start to ...
1
vote
1answer
36 views

What are the ways to partition a large file that does not fit into memory so it can later be fed as training data?

Is there any other way to partition a large file that does not fit into memory so it can be fed as training data other than using spark? or hadoop?
1
vote
0answers
15 views

Group data without losing information

Context Imagine that I have a dataset about sending messages. Each row as user_id, a batch_id, a ...
1
vote
0answers
37 views

Advice on dealing with very large datasets - HDF5, Python

I've recently started working on an application for visualization of really big datasets. While reading online it became apparent that most people use HDF5 for storing big, multi-dimensional datasets ...
2
votes
0answers
37 views

Algorithm or JS graph drawing library that can generate a graph of 100,000+ nodes and edges while minimizing edge crossings

I'm trying to plot a directed graph of $2^{16}$ nodes and $2^{16}$ edges (but not simply a cycle). Ultimately, I need to be able to share an interactive graph (zooming, panning, labels). Mathematica ...
0
votes
0answers
16 views

Data Mining in Practice: Analysing Real Estate

I need to analyse a data set of around 30,000 properties listed or sold over 2 years The main aim is to: Minimize risk and boost profit investments in a short-term market and a long-term strategy. ...
0
votes
0answers
56 views

How to extract errors from log file in scala?

"A log file needs to be processed to extract exceptions/errors.the final output will be structured data having attributes like date,time and exception name.the output will be saved ín a managed hive ...
1
vote
0answers
20 views

Probabilistic guarantees for SQL query execution

We are working on a project that tries to accelerate Spark SQL using dedicated hardware. Our approach works much better (in terms of the resulting latency) if we allow query processing that is only ...
0
votes
1answer
37 views

How to favour a particular class during classification using XGBoost?

I am using a simple XGBoost model to classify 2 classes (0 and 1) in a binary context. In case of the original data, the 0 is the majority class and 1 the minority class. The thing which is happening ...
1
vote
0answers
19 views

Big data online class [closed]

Being an IT engineer, I am exploring the Big Data specialization and wishing to earn a certificate in that field to help me get a new job later. I've been looking around and found these two ...
1
vote
1answer
143 views

Using SMOTE for Synthetic Data generation to improve performance on unbalanced data

I presently have a dataset with 21392 samples, of which, 16948 belong to the majority class (class A) and the remaining 4444 belong to the minority class (class B). I am presently using SMOTE (...
0
votes
1answer
77 views

Neural Network Sensordata as Input

I have a dataset consisting of sensor recordings about human movement. There are 22 classes of different movement like sitting or walking and 19 sensor values. Each recording of a movement has about ...
3
votes
1answer
28 views

How to find the values that never matched in a full join

Hello I have done this full join query: ...
2
votes
2answers
93 views

Where does the “deep learning needs big data” rule come from

When reading about deep learning I often come across the rule that deep learning is only effective when you have large amounts of data at your disposal. These statements are generally accompanied by a ...
1
vote
0answers
93 views

Simulate Biased dataset Python [closed]

I am trying simulate data from a normal distribution but bias the sample by excluding all negative values and values divisible by 5 . To demonstrate the effect of bias . I will probably calculate ...
2
votes
0answers
228 views

Spark DataFrame Limit Function Takes Too Much Time to Show

...
2
votes
0answers
310 views

Keras' fit_generator() is not calling my generator

When I call Keras' fit_generator(), passing in a custom generator class I created, I see "Epoch 1/1" in the spew and that's all. It hangs right there, and the ...
0
votes
1answer
37 views

SAP HANA or Hadoop? [closed]

this is a question regarding a career choice. I am a fresher and I recently joined an MNC in Data Engineering team. There I was offered training in either Hadoop or SAP HANA. I am in doubt as to which ...
0
votes
3answers
239 views

Kmeans large dataset

we are currently performing a K-MEANS under scikit-learn on a data set containing 236027 observations with 6 variables in double format (64 bits). According to our calculations, the complexity of the ...
1
vote
1answer
419 views

Merging dataframes in Pandas is taking a surprisingly long time

I'm trying to merge a list of time series dataframes (could be over 100) using Pandas. The largest file has a size of $\approx$ 50 MB. The number of rows and columns vary (for instance, one file could ...
0
votes
0answers
16 views

How to efficiently store large amounts of speech to text data

I am currently facing the problem of having to deal with a lot of data (a few hundred GB and growing) for an automated speech recognition (or speech to text) task. Most datasets I am using are coming ...
1
vote
0answers
23 views

PyMC3: how to efficiently regress on many variables?

I am sorry ahead of time if this seems like a basic question, but I had difficulty finding resources online addressing this. In PyMC3, when building a basic model of a few variables, it is easy to ...
0
votes
0answers
11 views

How to locate some definite machine printed fields on a common format receipt?

We have thousands of receipts with a common format. The format of them looks like this: https://postimg.cc/hhc1K5xy , or this: https://postimg.cc/BtmH4QJb ,especially very similar to this: https://...
0
votes
0answers
7 views

Need suggestion on picking fast aggregation framework or solution

To state the problem, this is what I have in the MariaDB/MySQL. I have the follow table structure, scores ...