Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

Sampling user IDs from a large data set for retention analysis

Let's say I have a data set with user identities and their respective orders, of the form: ...
1
vote
0answers
28 views

Most efficient algorithm for sequential pattern mining on a dataset with large amount of items in each transaction?

To provide some context, I am trying to do frequent pattern mining on a dataset of system error logs from servers. I organized it into transactions based on the thread ID, which results in some very ...
0
votes
0answers
7 views

KeyError in python [migrated]

I am trying to count the number of times each string in my list is repeated but it keeps giving me KeyError: ...
1
vote
0answers
7 views

How to create tensors in spark?

I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
1
vote
0answers
17 views

Mapreduce jobs not working in hive

I was trying to execute a query select name, count(*) from amazon where review != NULL group by name ; Number of reduce tasks not specified. Estimated ...
0
votes
0answers
7 views

Implementing Data Catalog from scratch

I have a requirement of ingesting huge amount of user specific data and then searching on it. Trying to be low cost solution, I am leaning towards creating Kafka based ingestion pipelines and ...
0
votes
1answer
28 views

Is a new datapoint similar to my reference set?

I have a large reference data set and everyday I get new data points. I would like to know how can I test if those new data points are similar to the reference set? Thanks in advance. Maria
0
votes
1answer
14 views

Data driven project for a self-catering accommodation business

I want to apologise in advance for my ignorance but I'm hitting my head in a wall here as I'm not sure how to proceed with my project and I've got no experience in that field even though I'm doing a ...
1
vote
0answers
32 views

Process and clean a huge .tar file with millions of small XML files

I have one huge .tar file (~700GB) which includes millions of small XML files (<1MB each). The dataset has a lot of unnecessary information that I don't need in ...
0
votes
0answers
8 views

What statistical test would I use to test that both data sources are matching

Suppose I am only working with two datasets, Data Source 1 with columns Customer ID and ...
0
votes
0answers
12 views

How to do numpy matmul broadcasting between two numpy tensors?

I have the Pauli matrices which are (2x2) and complex ...
0
votes
0answers
14 views

How to aggregate large data

I have a quite large data (frame) with 65 physical quantities and each with different time stamps. Some are gathered in intervals of several hours and some in milliseconds. Hence, the data frame ...
0
votes
0answers
6 views

Is veracity in big data theory a good thing to use big data techniques in organization?

I am new to big data field. I know the basic 3V's of it, but I am not understanding the newly added one, which is Veracity. Is it a good thing, which led to taking great analysis from inaccurate data, ...
0
votes
0answers
26 views

Python Machine Learning to analyse test-case lab data and identify new test cases from real-world random data

I am currently working on a project where I have predefined test-cases I need to run/test and then analyse the data. My aim is to be able to identify system failures (against pre-defined criteria). ...
0
votes
0answers
9 views

How to generate artificial datasets in MATLAB based on behavior of real datasets?

To understand: Let's consider a scenario. I have real datasets of 2 buildings showing different behaviors of occupants. Now to expand the analysis for multiple buildings, I want to generate simulated ...
1
vote
0answers
18 views

Is there a way to use a pom.xml file to update spark configuration?

I am trying to update my spark configuration to solve some dependency problems. This pom.xml file seems to be useful for this purpose. I am using a spark docker image. ...
4
votes
1answer
107 views

XGBoost Huge Dataset ~1TB

Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints. Is ...
0
votes
0answers
10 views

Algorithm/approach to predict multiple questions in a survey

I have I set of 6k surveys about personal tastes (40 columns with answer number) and another set of 50M cookies about web activities (columns are all continuous). Some of these 50M can be connected to ...
1
vote
1answer
21 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
0
votes
0answers
7 views

How to reduce complexity of inference stage in recommender systems?

Given a large set of customers and a large set of items, how to make predictions given a model like this one: https://arxiv.org/pdf/1606.07792.pdf As stated in the article: "Since there are over a ...
1
vote
2answers
40 views

Classification task - how to handle large data file? [closed]

I'm planning to construct a classification model for predicting New York taxi trip fare. The CSV-datafile for this is very large, containing 112 234 626 rows (ca 10 GB). I have managed to download ...
0
votes
1answer
14 views

Parsing and storing a large amount of HTML data

I have a data chunk (~30k) in which I have htmls pages and pngs saved in a folder for websites. These folders are titled based on some randomly generated hashes. My supervisor wants me to crunch ...
2
votes
2answers
101 views

How to calculate the elapsed time of a flag status per day?

I'd like to figure out the elapsed time between flag status changes. Simplified example: a person can only be sad or happy. I'd like to know how long each mood was active until it changed. I'm ...
0
votes
0answers
79 views

Why modin.pandas takes more wall time than pandas?

All data scientist know that to scale the pandas to large dataset, we use modin.pandas instead of pandas. But when I tried to load a 500 MB csv dataset in my jupyter notebook in Ubuntu 16.04 to ...
0
votes
1answer
36 views

Plot misclassfied values in matplotlib

I have two lists. One is target and another is predicted values in a binary classification. The target is always zero. And I have to plot the results of the prediction against target in such a way, ...
1
vote
3answers
49 views

What is the correct procedure when “joining” data takes ~6 hours?

I am dealing with bike-share data. I have 2 DataFrames: trips_df (subset shown), total entries = ...
0
votes
0answers
122 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
1
vote
1answer
25 views

xgboostclassifier prediction error after saving the model and restoring it

I have trained a xgboost model and during training, the prediction works fine. But if I stop the script and start a restoring script to restore and predict, then for the same test dataset I get every ...
0
votes
0answers
20 views

Difference between retraining on different parts of the data and training initially on larger data set

I have a large data set that doesn't fit in memory and would have to use something like Keras's model.fit_generator if I would like to train the model on all of the ...
0
votes
1answer
14 views

what does “blocks” mean in the context of HDFS ecosystem? is it the same concept in the context of a single hard disk drive?

According to Hortonworks HDFS has demonstrated production scalability of up to 200 PB of storage and a single cluster of 4500 servers, supporting close to a billion files and blocks. what does ...
0
votes
0answers
36 views

Apache Nifi out of memory

I've set up a data pipeline using Apache nifi. It processes 3GB scale json data. When I try to flatten these Json files it always outputs: ...
2
votes
1answer
487 views

Training Keras model with multiple CSV files

I'm currently trying to train a Keras model on several large CSV files. I can fit one in memory, but not all combined. From my point of view, there are several ways to deal with this problem. I could ...
3
votes
1answer
25 views

Reducing search iteration over millions of data

The problem goes like this, with a story. You have an application that contains a search field. When you search some input, there's an auto-complete component that pops up showing up similar results ...
0
votes
0answers
10 views

How to predict sentences using numeric features in a DL model?

I know of Deep Learning and ML models which takes numeric features as inputs, and are able to predict or classify the target, which works well when doing things like binary or multi class ...
0
votes
1answer
38 views

What is data lake?

I am familiar with the concept of Big data but how data lake is different from Big data.or is it derived from big data.Please explain.
0
votes
0answers
7 views

How to load data from posgres to bigquery

Our Postgres tables are too huge and the columns are structured in a way that it has become difficult to query the tables. So we have started using BigQuery to help us query into huge tables. We ...
4
votes
1answer
71 views

How can one quickly look up people from a large database?

Vocabulary Face detection: Finding all faces in an image. Face representation: The simplest way to represent a face is as an image (pixels / color values). This is not very space efficient and likely ...
0
votes
0answers
128 views

Using Keras.utils.Sequence on out of memory dataset

I'm trying to implement a sub-class of keras.util.sequence so that i can load data faster into the function fit_generator. Currently I have a multi-variate time series (multiple features) file which ...
0
votes
0answers
10 views

Hints on my EMR Spark Cluster

I am new to Spark, in particular I am tying to find out which are the best recommended properties to set to work on a 1 master 2 slaves cluster with 10GB memory each. Do you know which are a good set ...
0
votes
0answers
6 views

Create a local subset copy of a cloud based MS SQL database

I have a large MS SQL database hosted on Azure. I want to develop an analysis pipeline for the data. I'm developing this pipeline locally and it's slowing down my development. I think if I had a local ...
0
votes
0answers
224 views

How to build a symptom checker and medical diagnose chat bot

I want to build a chatbot which can diagnose an illness depending upon the symptoms which are given to it. I want to show you an example scenario of how it should work; Application - A , User - U ...
0
votes
1answer
23 views

Sampling Big Data for Predictive Analytics in Python

In practice, how does one go about sampling a from big data set (eg. +/- 50 million distinct observations) to perform ML using Python? Most non-parametric models (e.g., SVM, ensemble models) start to ...
1
vote
1answer
38 views

What are the ways to partition a large file that does not fit into memory so it can later be fed as training data?

Is there any other way to partition a large file that does not fit into memory so it can be fed as training data other than using spark? or hadoop?
1
vote
0answers
15 views

Group data without losing information

Context Imagine that I have a dataset about sending messages. Each row as user_id, a batch_id, a ...
1
vote
0answers
63 views

Advice on dealing with very large datasets - HDF5, Python

I've recently started working on an application for visualization of really big datasets. While reading online it became apparent that most people use HDF5 for storing big, multi-dimensional datasets ...
2
votes
0answers
51 views

Algorithm or JS graph drawing library that can generate a graph of 100,000+ nodes and edges while minimizing edge crossings

I'm trying to plot a directed graph of $2^{16}$ nodes and $2^{16}$ edges (but not simply a cycle). Ultimately, I need to be able to share an interactive graph (zooming, panning, labels). Mathematica ...
0
votes
0answers
16 views

Data Mining in Practice: Analysing Real Estate

I need to analyse a data set of around 30,000 properties listed or sold over 2 years The main aim is to: Minimize risk and boost profit investments in a short-term market and a long-term strategy. ...
0
votes
0answers
69 views

How to extract errors from log file in scala?

"A log file needs to be processed to extract exceptions/errors.the final output will be structured data having attributes like date,time and exception name.the output will be saved ín a managed hive ...
1
vote
0answers
20 views

Probabilistic guarantees for SQL query execution

We are working on a project that tries to accelerate Spark SQL using dedicated hardware. Our approach works much better (in terms of the resulting latency) if we allow query processing that is only ...
0
votes
1answer
64 views

How to favour a particular class during classification using XGBoost?

I am using a simple XGBoost model to classify 2 classes (0 and 1) in a binary context. In case of the original data, the 0 is the majority class and 1 the minority class. The thing which is happening ...