Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

What is the best programming paradigm for Data Engineering: Object Oriented, Functional, Procedural or a combination of them?

How would you structure a typical Data Engineering pipeline in terms of Programming paradigm? would you keep the traditional procedural approach or use OOP with functional elements? where is a good ...
1
vote
3answers
40 views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
0
votes
0answers
51 views

Access global variable from UDF (User Defined Function) in python in spark

I am trying to alter a global variable from inside a pyspark.sql.functions.udf function in python. But, the change in not getting reflected in the global variable. ...
0
votes
0answers
39 views

What is the learning path in Big Data for a self-taught person? [closed]

I am currently learning the Python language and I am quite interested in developing myself in the Big Data field as a self-taught. Currently I do not have the possibility of going to university so I ...
1
vote
1answer
39 views

ML modeling a data with big amount of rows

I want to do ML modeling such XGboost, KNN, and similar models on data with 9 numerical features and more than 25 million rows and the size of data is almost 2.5 Gig and I prefer to use all the data ...
0
votes
1answer
22 views

How to choose feedforward architecture for few number of features but very large instance?

Assume I have 1 million of data instance and each instance contains 100 feature. For each instance, I also have a lable. The ...
0
votes
0answers
27 views

Examples of Open source huge business intelligence datasets

I've observed that many of the datasets available for traditional ML and data science algorithms seem to be in the order of MB. I assumed these may be because earlier computers were not that ...
2
votes
0answers
30 views

XGBoost incremental training for big datasets

I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred). I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
0
votes
0answers
12 views

What are the execution service and the event service in Netflix big data

From the following article by the netflix engineering team: The core architecture of the big data platform at Netflix involves three key services. These are the execution service (Genie), the ...
0
votes
1answer
82 views

How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?

I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in ...
0
votes
1answer
34 views

Subsampling the “right” amout of data to train an ML model

I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
0
votes
0answers
11 views

Scalable Infrence Server for Object Detection

I have created a Django service (nginx + Gunicorn) for object detection models. For my case i have 50+ models with resnet 50 based back bone. Server Machine Specification: 16 CPU 64 GB Ram I have ...
0
votes
0answers
22 views

What should be the architecture design for the dataset to be used for a machine learning API

I have aroud 30GB of market dataset in csv format which is downloaded onto my server computer everyday. Now I want to use this dataset to do some computations and provide some analysis. The user will ...
1
vote
0answers
31 views

How do you perform basic statistical analysis on a Binary Dataset?

I have never worked with a Binary dataset (1 and 0) (True or False) so I'm unsure what kind of statistical tests I should run to draw up simple conclusions. I'm doing a data science project and the ...
6
votes
1answer
108 views

What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)?

I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are: AWS instance with massive RAM and train CPU (slow, but ...
0
votes
0answers
25 views

Working with mySQL and pytorch Dataset

I am working with high frequency time indexed data. We have 2 types of data each with about 5 columns. For each type of data we have 2500 streams coming in and being updated every 1ms-100ms (the ...
0
votes
0answers
37 views

Extracting possible relationships and insights from a large dataset

I have a dataset with about 26'000 variables and 33'000 continuous observations. My task is to extract any interesting insights/possible relationships between various variables. I am looking for any ...
0
votes
0answers
10 views

How to Combine Oracle and Snowflake/Redshift data on the fly

Have a Schema.Table_Name on Oracle. Exactly similar table is on Snowflake with same metadata(Schema.Table_Name). How can I combine data from two on the fly without moving Oracle data to Redshift. Most ...
1
vote
1answer
19 views

Where can I find a dataset that contains criminal case sentencing data? [closed]

I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as: Type of crime Defendant Age/Sex/Race Plea Verdict Sentence Is there a dataset ...
0
votes
0answers
31 views

In Spark, what's the role of each of Cluster manager and Spark Context?

What's the role of a cluster manager and the role spark context (Driver program) in terms of managing worker nodes. How do they communicate with each other (and with the worker nodes) and when they do ...
0
votes
0answers
10 views

Trigger job in SAP BW from SAC

Is there a way to have the following occur: When a job is finished on the SAC (SAP Analytics Cloud) side, a job on the SAP BW side should be triggered. Is that at all possible? I have found this which ...
0
votes
0answers
20 views

Difference between MapReduce, Dryag and Pregel in terms of efficiency

What is the difference between the three parallel processing paradigms: MapReduce, Dryag, and Pregel in terms of efficiency for big data analytics applications? Thank you.
-2
votes
1answer
16 views

What data to collect to create useful insights?

I am creating a Social Media app and I want to know what type of data shall I collect to create useful insights for Business accounts or Content Creators? For example if I want to create an insight ...
1
vote
1answer
39 views

How to train a model using a daunting huge training dataset

I have an extremely huge dataset and I'm wondering me how could be the right way to set an experiment to use this data to train a model. I understand that I can use data-reduction to, for instance, ...
0
votes
0answers
34 views

How to Integrate/map Population Data with/over Sample Data

Problem Definition: Our organization is conducting different type of surveys and Census in our country. The basic difference between Census and Survey is that, the target of Census is the Complete ...
2
votes
0answers
52 views

Clustering large set of images

I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
2
votes
0answers
30 views

What is the difference between Pachyderm and Git?

I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that: It holds all your data in a central accessible ...
3
votes
1answer
122 views

Why do machine learning engineers insist on training with more data than validation set?

Among my colleagues I have noticed a curious insistence on training with, say, 70% or 80% of data and validating on the remainder. The reason it is curious to me is the lack of any theoretical ...
0
votes
0answers
30 views

What is the azure solution for storing and querying Netcdf data on the fly?

We currently run a live web application which utilises standard azure mounted file storage to store 2tb worth of metocean data in netcdf format. On here we run python queries directly on the data e.g. ...
0
votes
1answer
33 views

How to grade an interaction that a user had with a post with an AI based on big data?

Context I'm creating a social network. The thing is, I don't want to order posts by likes, or something like that, I'm using an AI (lightfm in ...
1
vote
1answer
21 views

How do data types influence hardware (CPU / GPU / TPU) performance?

I am currently dealing with a relatively big data set, for which I have some memory usage concerns. I am dealing with most of the different data types : floats, integers, Booleans, characters strings ...
0
votes
0answers
21 views

Is there a data structure that works similarly to a (binary) tree?

Im currently working with some data, which is part of a very large data set (height, about 6000 elements per timestep - timestep every minute over days at a time). I need to differentiate between ...
0
votes
0answers
17 views

How are big models like GPT-3 developed?

For models trained with a moderate amount of data, I would expect a the development workflow to include extensive hyper-parameter optimization through extensive retraining. But how is this handled for ...
0
votes
0answers
16 views

Ideas on handling large quantities of “waste” data

Hello and thank you for taking time out of your day to help me. I am currently working on developing a machine vision application for production monitoring. The application handles images, about 20,...
-1
votes
1answer
363 views

How to store efficiently very large sparse 3D matrices

To train a CNN, I have stacked arrays of images over observations [observations x width x length]. The dataset is very sparse ($95\%$). What would be an efficient ...
0
votes
0answers
38 views

Create User Table from Google Firebase Analytics and Cloud Firestore Collections

I am exporting my snowplow collections collected on Firebase using the extention Export Collections to BigQuery firestore-bigquery-export v0.1.10. Whenever a document is created, updated, imported, or ...
2
votes
1answer
43 views

Size of datasets over years

I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years. I just found the following statistics: The poll of KDnuggets that actually shows that over ...
2
votes
1answer
376 views

How does skewed data affect deep neural networks?

I'm playing around with deep neural networks for a regression problem. The dataset I have is skewed right and for a linear regression model, I would typically perform a log transform. Should I be ...
1
vote
1answer
141 views

Public dataset for news articles with their associated categories for multilabel data classification

I am wondering if there are any public datasets of news, like New York Times (NYT) or similar to various news categories such as politics, entertainment, lifestyle, general news, sports etc. I want to ...
1
vote
0answers
12 views

How much information is produced in our Society? [closed]

I was reading an interesting article about the growth of data in our society: https://rss.onlinelibrary.wiley.com/doi/epdf/10.1111/j.1740-9713.2012.00584.x Does any of you know if there exists a more ...
0
votes
0answers
16 views

Automatic EDA for BIG Data

I have data set of cicids2017 which in total has 2830743 rows and 79 columns. Since my local jupyter was crashing a lot, SO I am using Google Colab. But Google Colab is also crashing if some big ...
4
votes
1answer
129 views

What is the main difference between Hadoop and Spark? [closed]

I recently read the following about Hadoop vs. Spark: Insist upon in-memory columnar data querying. This was the killer-feature that let Apache Spark run in seconds the queries that would take Hadoop ...
0
votes
0answers
18 views

Looking for the proper algorithm to compress many lowres images of nearby locations

I have an optimization problem that I'm looking for the right algorithm to solve. What I have: A large set of low-res 360 images that were taken on a regular grid within a certain area. each of these ...
-1
votes
1answer
39 views

How can I predict the true label for data with incomplete features based on the model learned by data with complete features? [closed]

for example, the model was learned by training data with complete features (f1,f2,f3,f4,f5,f6) but, I wonder the model can test data with incomplete features (f1,f2,f3) to attach the true label to ...
1
vote
1answer
1k views

How to determine sample rate of a time series dataset?

I have a dataset of magnetometer sensor readings which looks like: ...
1
vote
0answers
205 views

How do I efficiently load data from disk during training of deep learning models in pytorch?

I'm trying to train a deep learning model without loading the entire dataset into memory. My main question is, what's the best way of doing this? It seems like HDF5 is a common method that people ...
2
votes
0answers
61 views

Spark: How to run PCA parallelized? Only one thread used

I use pySpark and set my configuration like following: ...
1
vote
0answers
25 views

What is important for Pharmaceutical companies to answer with Big Data Analysis?

I am a data scientist, and I have some biological background (genetics). I have been asked to give a talk for our customers from pharmaceutical industry. I should show them how they benefit from Big ...
1
vote
2answers
59 views

Maximum number of images to train a convolutional neural network

I just wondered if there is a technical limit on the number of images to train a neural network. I want to work with extremely high numbers of images, around 1,000,000 to 10,000,000 images. Is there ...
0
votes
1answer
62 views

Training models from sklearn using tf.distribute.MirroredStrategy

I want to distribute the training of a simple model, such as a support vector classifier like sklearn.svm.SVC() across some or all CPUs and GPUs on a single device. I have never utilized a GPU before ...

1
2 3 4 5
9