Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
0 votes
0 answers
4 views

How to use BI and Big Data in different industries?

This article describes how real estate uses big data to track clients Can you give more examples on how to use BI and Big Data for other industries or business?
0 votes
0 answers
14 views

understanding quadratic form in proof of positive definiteness of laplacian matrix

Consider the proof at page 2 found here: https://people.orie.cornell.edu/dpw/orie6334/Fall2016/lecture7.pdf I cant wrap my head around the 2nd and third line: \begin{align} &= \sum_{i \in V}x(i)^2 ...
0 votes
0 answers
245 views

How to dynamically add columns in Dataframe Spark by using a config.ini file parameters

I am learning to use Pyspark and at work I have been assigned a task that I can not solve for now. There is a config.ini file where there are a series of ...
0 votes
0 answers
12 views

How to reference an intermediary object in scala?

I am using spark/scala to do some transformations on a spark dataframe. I have the following code: ...
  • 11
0 votes
1 answer
26 views

Can I update the source of Data found in a Data Lake or Data Blob

Is it possible to update the source of data found in a Data Lake or Data Blob? What about while using HDInsight or Azure Databricks?
  • 101
0 votes
1 answer
33 views

Gensim: create a dictionary from a large corpus without loading it in RAM?

The topic modelling library Gensim offers the ability to stream a large document instead of storing it in memory. Streaming is possible for the stage of converting the corpus to BOW, but the ...
  • 24.5k
0 votes
2 answers
48 views

What kinds of ML models should I use when the outcome variable does not vary with time but only vary across individuals and groups?

I am trying to predict individuals’ income in 2018 using 18 years worth of data for people who were born in 1978,1979, and 1980 on many variables such as family income, location, family members’ ...
0 votes
0 answers
19 views

Error in system(cmd, input = filelist, intern = TRUE) : 'zip' not found

I am new to programming, I just learned R. This is the error I'm dealing with Error in system(cmd, input = filelist, intern = TRUE) : 'zip' not found. This is my code and I am trying to save this file ...
0 votes
0 answers
102 views

Dimensionality Reduction of Categorical Variables in Spark

First off, I hope this question hasn't been asked already. I've found questions regarding the use of PCA vs. MCA in the reduction of categorical variables, but I've yet to see a solution as it ...
1 vote
1 answer
31 views

What is the name of my problem - distribution of counts of elements having certain attribute

I have the following problem: There is a large set of records. Each record in the set has an attribute. For some values of the attribute, there is only one record, for other values there are many ...
  • 111
2 votes
3 answers
78 views

Is big data a fallacy if most phenomena can be mostly described by few variables?

Is big data a fallacy if most phenomena can be mostly described by few variables? This has confused me. Surely there are big data sets, but there are also cases when the set of significant or ...
  • 416
0 votes
0 answers
48 views

Hadoop creates two input splits (hence two maps) for a very small file : 14 bytes

I am working on a single node cluster, and i am running a wordcount job of a very small file, the file contains roughly 10 words, but the job creates two input splits which means two map jobs, i need ...
0 votes
2 answers
44 views

How to manage large datasets (approx 95GB)

I was planning some data analysis on a dataset I'll be using for some projects. The dataset in question is ZINC20. Now, I don't need the whole thing so I was going to write some functions that would ...
  • 1
1 vote
0 answers
75 views

Best Practice to handle large data in HDBSCAN cluserting using tfidf

I am now about to implement HDBSCAN on a very large set of data, a few million sentences, using tfidf. However, as we know ...
  • 421
0 votes
0 answers
24 views

RCPP error when installing rmr2 package

i am trying to use RHADOOP packages, when i get to the rmr2 installation it gives this error: ...
0 votes
0 answers
16 views

Big data storage for PowerBI reporting

Not really a question I guess. More like asking for advice. I’m trying to store our Azure cloud cost data in a centralized location so that our PowerBI can query the data faster rather than calling ...
0 votes
0 answers
12 views

Behavioral Segmentation Clustering Mixed Large Data - with or without categorical variables

This question may be closed due to being too broad but I feel as though this is the best place to ask my question. At the moment, I am dealing with some customer data and am looking on segmenting them ...
1 vote
1 answer
31 views

Do best hyperparameters remain constant when data size is scaled?

Basically what the title is. The problem I currently have is that my dataset consists of 2.8 billion rows, and I have it as a Pyspark data frame. I want to use some library such as FLAML for finding ...
0 votes
0 answers
21 views

How to Calculate degrees in un-directed network and plot it? | Using python COLAB [duplicate]

I have loaded txt file (com-dblp.ungraph.txt) which includes DBLP collaboration network to a COLAB notebook. I found 317080 nodes & 1049866 edges in the network by reading the txt manually. Trying ...
0 votes
0 answers
316 views

Is it possible to implement an rdd version of a for loop having map and reduce using pyspark?

I need to test an algorithm that computes a function on a dataframe where in each execution I drop a column and computes the function. This is a example in python pyspark but without using rdd: ...
  • 1
2 votes
1 answer
276 views

The single CSV created by combining a large number of CSV files is too large to process. What options do I have?

The dataset I am currently working on has more than 100 csv files, with each of size more than 250MB. These are files containing time series data captured from different locations and all the files ...
0 votes
0 answers
57 views

Efficient way to compare one record to millions of rows

We have a production table that contains a bucket of customer data. A customer could be the same customer/person at location A and at location B. They are different by how the name is spelled, address ...
1 vote
2 answers
107 views

Dimensionality reduction for millions of features

I have a dataset with 10 million observations and 1 million sparse features. I would like to build a binary classifier for predicting a particular feature of interest. My main problem is how to deal ...
  • 11
1 vote
0 answers
22 views

Repeating values caught with a binary classifier

If my machine is broken, it starts to repeat certain channels. Thing is if there are no out-liars, it is difficult to tell it's broken as we would expect all data points to be around the same value. I ...
  • 11
2 votes
1 answer
843 views

How do I normalize json data into pandas (Covid-19 data) [closed]

I am trying to import all up-to-date datasets in JSON format on the covid-19 pandemic into a pandas dataframe. I believe it should be possible by using ...
2 votes
1 answer
48 views

Best way to preprocess data

I need to create a machine learning model to predict if a structure is an hotel or an apartment. I have a dataset structured as well: ...
  • 53
1 vote
1 answer
219 views

How to get dummy variables from "first name"

I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage. Since the datasets are so big (...
5 votes
4 answers
2k views

Best platform to work with when having millions of rows in dataframe

I have table with around 20 features and millions of observations (rows). I need to create model base on this table, however, as it is huge, training models like random forest or XGB takes forever. I'...
  • 349
2 votes
1 answer
58 views

Data Art/ Data Visualization Art/ Information Art

Few days ago, I learned about data art/ data visualization art/ information art. I think I have interest in it. I want to see how I can use my data science skills in this area. However, I don't know ...
  • 171
1 vote
0 answers
95 views

Word count with map reduce

Suppose we use an input file that contains the following lyrics from a famous song: We’re up all night till the sun We’re up all night to get some The input pairs for the Map phase will be the ...
1 vote
2 answers
151 views

Where can I find study materials? [closed]

Can anyone recommend me some material (books, blogs, youtube channels, ...) to study statistics, Machine Learning and in general Data Science topics? Thanks
  • 471
1 vote
1 answer
104 views

How do I create a dataset from many CSV files that is too large for RAM

I have been handed about 40 GB of CSV files that I need to turn into a database. The files are arranged in a file structure that uses location in that file structure to create a relationship between ...
1 vote
0 answers
467 views

Best way to find nearest neighbor distance for large datasets

I am a grad student doing research using generative machine learning with pytorch, and I have generated a set of points. I would like to check how similar these new points are to the points I used in ...
0 votes
2 answers
708 views

Fastest way to replace a value in a pandas DataFrame?

I am loading in 1.5m images with 80,000 classes (or I will have to when I eventually train) into a Keras generator and am using a pandas dataframe to do so. The problem is, with so many images, my ...
0 votes
1 answer
221 views

How to do EDA on large datasets

I have a table in Postgres with ~5million records. When I load the dataset using pandas to perform EDA, I run out of memory. ...
  • 1
1 vote
3 answers
563 views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
  • 11
0 votes
1 answer
62 views

ML modeling a data with big amount of rows

I want to do ML modeling such XGboost, KNN, and similar models on data with 9 numerical features and more than 25 million rows and the size of data is almost 2.5 Gig and I prefer to use all the data ...
0 votes
1 answer
23 views

How to choose feedforward architecture for few number of features but very large instance?

Assume I have 1 million of data instance and each instance contains 100 feature. For each instance, I also have a lable. The ...
  • 309
0 votes
0 answers
35 views

Examples of Open source huge business intelligence datasets

I've observed that many of the datasets available for traditional ML and data science algorithms seem to be in the order of MB. I assumed these may be because earlier computers were not that ...
1 vote
0 answers
691 views

XGBoost incremental training for big datasets

I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred). I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
0 votes
1 answer
648 views

How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?

I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in ...
0 votes
2 answers
370 views

Subsampling the “right” amout of data to train an ML model

I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
  • 1
1 vote
0 answers
51 views

How do you perform basic statistical analysis on a Binary Dataset?

I have never worked with a Binary dataset (1 and 0) (True or False) so I'm unsure what kind of statistical tests I should run to draw up simple conclusions. I'm doing a data science project and the ...
  • 11
5 votes
2 answers
358 views

What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)?

I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are: AWS instance with massive RAM and train CPU (slow, but ...
  • 181
1 vote
1 answer
22 views

Where can I find a dataset that contains criminal case sentencing data? [closed]

I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as: Type of crime Defendant Age/Sex/Race Plea Verdict Sentence Is there a dataset ...
  • 111
-2 votes
1 answer
19 views

What data to collect to create useful insights?

I am creating a Social Media app and I want to know what type of data shall I collect to create useful insights for Business accounts or Content Creators? For example if I want to create an insight ...
1 vote
1 answer
118 views

How to train a model using a daunting huge training dataset

I have an extremely huge dataset and I'm wondering me how could be the right way to set an experiment to use this data to train a model. I understand that I can use data-reduction to, for instance, ...
  • 113
0 votes
0 answers
44 views

How to Integrate/map Population Data with/over Sample Data

Problem Definition: Our organization is conducting different type of surveys and Census in our country. The basic difference between Census and Survey is that, the target of Census is the Complete ...
2 votes
0 answers
65 views

Clustering large set of images

I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
2 votes
1 answer
85 views

What is the difference between Pachyderm and Git?

I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that: It holds all your data in a central accessible ...

1
2 3 4 5
10