Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

How to Calculate degrees in un-directed network and plot it? | Using python COLAB [duplicate]

I have loaded txt file (com-dblp.ungraph.txt) which includes DBLP collaboration network to a COLAB notebook. I found 317080 nodes & 1049866 edges in the network by reading the txt manually. Trying ...
user avatar
0 votes
0 answers
19 views

How to Calculate values inside DBLP collaboration un-directed network? | Using python

(EDITED- thanks) I have loaded txt file (com-dblp.ungraph.txt) which includes network to a COLAB notebook. I want to calculate describe vals of the net in the COLAB- how can I do it? Number of nodes ...
user avatar
0 votes
0 answers
51 views

Is it possible to implement an rdd version of a for loop having map and reduce using pyspark?

I need to test an algorithm that computes a function on a dataframe where in each execution I drop a column and computes the function. This is a example in python pyspark but without using rdd: ...
user avatar
  • 1
2 votes
1 answer
22 views

The single CSV created by combining a large number of CSV files is too large to process. What options do I have?

The dataset I am currently working on has more than 100 csv files, with each of size more than 250MB. These are files containing time series data captured from different locations and all the files ...
user avatar
0 votes
0 answers
13 views

Efficient way to compare one record to millions of rows

We have a production table that contains a bucket of customer data. A customer could be the same customer/person at location A and at location B. They are different by how the name is spelled, address ...
user avatar
1 vote
2 answers
54 views

Dimensionality reduction for millions of features

I have a dataset with 10 million observations and 1 million sparse features. I would like to build a binary classifier for predicting a particular feature of interest. My main problem is how to deal ...
user avatar
  • 11
1 vote
0 answers
17 views

Repeating values caught with a binary classifier

If my machine is broken, it starts to repeat certain channels. Thing is if there are no out-liars, it is difficult to tell it's broken as we would expect all data points to be around the same value. I ...
user avatar
  • 11
3 votes
1 answer
60 views

How do I normalize json data into pandas (Covid-19 data) [closed]

I am trying to import all up-to-date datasets in JSON format on the covid-19 pandemic into a pandas dataframe. I believe it should be possible by using ...
user avatar
2 votes
1 answer
37 views

Best way to preprocess data

I need to create a machine learning model to predict if a structure is an hotel or an apartment. I have a dataset structured as well: ...
user avatar
  • 43
1 vote
0 answers
120 views

Huge matrix multiplication with PySpark

I am currently struggling with a problem. I have been googling around and cannot find an answer. Also I am kinda new to PySpark and Spark maybe that's why I am struggling. Let's say I have a huge $m \...
user avatar
1 vote
1 answer
120 views

How to get dummy variables from "first name"

I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage. Since the datasets are so big (...
user avatar
0 votes
0 answers
10 views

Data Pipelines - Apache Spark - Updating Another Data Store or Input to Queue

What is the correct way to think about or approach something like Spark breaking down a 160GB file into smaller manageable parts, or single records and then doing something like updating records in ...
user avatar
  • 101
5 votes
4 answers
799 views

Best platform to work with when having millions of rows in dataframe

I have table with around 20 features and millions of observations (rows). I need to create model base on this table, however, as it is huge, training models like random forest or XGB takes forever. I'...
user avatar
  • 349
2 votes
1 answer
46 views

Data Art/ Data Visualization Art/ Information Art

Few days ago, I learned about data art/ data visualization art/ information art. I think I have interest in it. I want to see how I can use my data science skills in this area. However, I don't know ...
user avatar
  • 171
1 vote
0 answers
43 views

Word count with map reduce

Suppose we use an input file that contains the following lyrics from a famous song: We’re up all night till the sun We’re up all night to get some The input pairs for the Map phase will be the ...
user avatar
1 vote
2 answers
143 views

Where can I find study materials? [closed]

Can anyone recommend me some material (books, blogs, youtube channels, ...) to study statistics, Machine Learning and in general Data Science topics? Thanks
user avatar
  • 459
0 votes
1 answer
50 views

How do I create a dataset from many CSV files that is too large for RAM

I have been handed about 40 GB of CSV files that I need to turn into a database. The files are arranged in a file structure that uses location in that file structure to create a relationship between ...
user avatar
0 votes
0 answers
35 views

Find a column by name in a row in scala spark

I have a Seq[Row].Each row is an Array of Struct.Struct has four fields: a,b,c and d all of which are String.The data in a particular row is something like this: [{"a":"ahahk",&...
user avatar
1 vote
0 answers
199 views

Best way to find nearest neighbor distance for large datasets

I am a grad student doing research using generative machine learning with pytorch, and I have generated a set of points. I would like to check how similar these new points are to the points I used in ...
user avatar
0 votes
2 answers
408 views

Fastest way to replace a value in a pandas DataFrame?

I am loading in 1.5m images with 80,000 classes (or I will have to when I eventually train) into a Keras generator and am using a pandas dataframe to do so. The problem is, with so many images, my ...
user avatar
0 votes
1 answer
116 views

How to do EDA on large datasets

I have a table in Postgres with ~5million records. When I load the dataset using pandas to perform EDA, I run out of memory. ...
user avatar
  • 1
0 votes
0 answers
90 views

KMeans using Mapreduce in Python

I wrote a mapreduce code in python which works locally i.e., cat test_mapper |python mapper.py sort the result, and ...
user avatar
  • 101
0 votes
0 answers
13 views

suggestion needed for big data development

I am trying to find out what is state of the art with database, python, and big data. My starting point began with a SQL server, and multiprocessing pandas, and dask. Imagine I need to maintain a ...
user avatar
  • 111
1 vote
3 answers
309 views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
user avatar
  • 11
0 votes
1 answer
48 views

ML modeling a data with big amount of rows

I want to do ML modeling such XGboost, KNN, and similar models on data with 9 numerical features and more than 25 million rows and the size of data is almost 2.5 Gig and I prefer to use all the data ...
user avatar
0 votes
1 answer
22 views

How to choose feedforward architecture for few number of features but very large instance?

Assume I have 1 million of data instance and each instance contains 100 feature. For each instance, I also have a lable. The ...
user avatar
  • 299
0 votes
0 answers
34 views

Examples of Open source huge business intelligence datasets

I've observed that many of the datasets available for traditional ML and data science algorithms seem to be in the order of MB. I assumed these may be because earlier computers were not that ...
user avatar
1 vote
0 answers
310 views

XGBoost incremental training for big datasets

I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred). I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
user avatar
0 votes
1 answer
400 views

How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?

I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in ...
user avatar
0 votes
2 answers
137 views

Subsampling the “right” amout of data to train an ML model

I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
user avatar
  • 1
1 vote
0 answers
44 views

How do you perform basic statistical analysis on a Binary Dataset?

I have never worked with a Binary dataset (1 and 0) (True or False) so I'm unsure what kind of statistical tests I should run to draw up simple conclusions. I'm doing a data science project and the ...
user avatar
  • 11
5 votes
2 answers
195 views

What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)?

I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are: AWS instance with massive RAM and train CPU (slow, but ...
user avatar
  • 181
1 vote
1 answer
21 views

Where can I find a dataset that contains criminal case sentencing data? [closed]

I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as: Type of crime Defendant Age/Sex/Race Plea Verdict Sentence Is there a dataset ...
user avatar
  • 111
-2 votes
1 answer
19 views

What data to collect to create useful insights?

I am creating a Social Media app and I want to know what type of data shall I collect to create useful insights for Business accounts or Content Creators? For example if I want to create an insight ...
user avatar
1 vote
1 answer
81 views

How to train a model using a daunting huge training dataset

I have an extremely huge dataset and I'm wondering me how could be the right way to set an experiment to use this data to train a model. I understand that I can use data-reduction to, for instance, ...
user avatar
  • 113
0 votes
0 answers
39 views

How to Integrate/map Population Data with/over Sample Data

Problem Definition: Our organization is conducting different type of surveys and Census in our country. The basic difference between Census and Survey is that, the target of Census is the Complete ...
user avatar
2 votes
0 answers
59 views

Clustering large set of images

I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
user avatar
2 votes
1 answer
61 views

What is the difference between Pachyderm and Git?

I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that: It holds all your data in a central accessible ...
user avatar
3 votes
1 answer
130 views

Why do machine learning engineers insist on training with more data than validation set?

Among my colleagues I have noticed a curious insistence on training with, say, 70% or 80% of data and validating on the remainder. The reason it is curious to me is the lack of any theoretical ...
user avatar
0 votes
1 answer
34 views

How to grade an interaction that a user had with a post with an AI based on big data?

Context I'm creating a social network. The thing is, I don't want to order posts by likes, or something like that, I'm using an AI (lightfm in ...
user avatar
  • 117
1 vote
1 answer
39 views

How do data types influence hardware (CPU / GPU / TPU) performance?

I am currently dealing with a relatively big data set, for which I have some memory usage concerns. I am dealing with most of the different data types : floats, integers, Booleans, characters strings ...
user avatar
  • 2,135
0 votes
0 answers
16 views

Ideas on handling large quantities of "waste" data

Hello and thank you for taking time out of your day to help me. I am currently working on developing a machine vision application for production monitoring. The application handles images, about 20,...
user avatar
0 votes
1 answer
1k views

How to store efficiently very large sparse 3D matrices

To train a CNN, I have stacked arrays of images over observations [observations x width x length]. The dataset is very sparse ($95\%$). What would be an efficient ...
user avatar
  • 1,688
2 votes
1 answer
73 views

Size of datasets over years

I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years. I just found the following statistics: The poll of KDnuggets that actually shows that over ...
user avatar
  • 133
2 votes
1 answer
1k views

How does skewed data affect deep neural networks?

I'm playing around with deep neural networks for a regression problem. The dataset I have is skewed right and for a linear regression model, I would typically perform a log transform. Should I be ...
user avatar
1 vote
1 answer
236 views

Public dataset for news articles with their associated categories for multilabel data classification

I am wondering if there are any public datasets of news, like The New York Times (NYT) or similar to various news categories such as politics, entertainment, lifestyle, general news, sports, etc. I ...
user avatar
1 vote
0 answers
13 views

How much information is produced in our Society? [closed]

I was reading an interesting article about the growth of data in our society: https://rss.onlinelibrary.wiley.com/doi/epdf/10.1111/j.1740-9713.2012.00584.x Does any of you know if there exists a more ...
user avatar
  • 111
3 votes
1 answer
232 views

What is the main difference between Hadoop and Spark? [closed]

I recently read the following about Hadoop vs. Spark: Insist upon in-memory columnar data querying. This was the killer-feature that let Apache Spark run in seconds the queries that would take Hadoop ...
user avatar
-1 votes
1 answer
77 views

How can I predict the true label for data with incomplete features based on the model learned by data with complete features? [closed]

for example, the model was learned by training data with complete features (f1,f2,f3,f4,f5,f6) but, I wonder the model can test data with incomplete features (f1,f2,f3) to attach the true label to ...
user avatar
2 votes
1 answer
3k views

How to determine sample rate of a time series dataset?

I have a dataset of magnetometer sensor readings which looks like: ...
user avatar
  • 143

1
2 3 4 5
10