Questions tagged [bigdata]
Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.
462
questions
0
votes
0
answers
4
views
How to use BI and Big Data in different industries?
This article describes how real estate uses big data to track clients
Can you give more examples on how to use BI and Big Data for other industries or business?
0
votes
0
answers
14
views
understanding quadratic form in proof of positive definiteness of laplacian matrix
Consider the proof at page 2 found here: https://people.orie.cornell.edu/dpw/orie6334/Fall2016/lecture7.pdf
I cant wrap my head around the 2nd and third line:
\begin{align}
&= \sum_{i \in V}x(i)^2 ...
0
votes
0
answers
245
views
How to dynamically add columns in Dataframe Spark by using a config.ini file parameters
I am learning to use Pyspark and at work I have been assigned a task that I can not solve for now.
There is a config.ini file where there are a series of ...
- 101
0
votes
0
answers
12
views
How to reference an intermediary object in scala?
I am using spark/scala to do some transformations on a spark dataframe. I have the following code:
...
- 11
0
votes
1
answer
26
views
Can I update the source of Data found in a Data Lake or Data Blob
Is it possible to update the source of data found in a Data Lake or Data Blob? What about while using HDInsight or Azure Databricks?
- 101
0
votes
1
answer
33
views
Gensim: create a dictionary from a large corpus without loading it in RAM?
The topic modelling library Gensim offers the ability to stream a large document instead of storing it in memory.
Streaming is possible for the stage of converting the corpus to BOW, but the ...
- 24.5k
0
votes
2
answers
48
views
What kinds of ML models should I use when the outcome variable does not vary with time but only vary across individuals and groups?
I am trying to predict individuals’ income in 2018 using 18 years worth of data for people who were born in 1978,1979, and 1980 on many variables such as family income, location, family members’ ...
0
votes
0
answers
19
views
Error in system(cmd, input = filelist, intern = TRUE) : 'zip' not found
I am new to programming, I just learned R. This is the error I'm dealing with Error in system(cmd, input = filelist, intern = TRUE) : 'zip' not found.
This is my code and I am trying to save this file ...
0
votes
0
answers
102
views
Dimensionality Reduction of Categorical Variables in Spark
First off, I hope this question hasn't been asked already. I've found questions regarding the use of PCA vs. MCA in the reduction of categorical variables, but I've yet to see a solution as it ...
- 1
1
vote
1
answer
31
views
What is the name of my problem - distribution of counts of elements having certain attribute
I have the following problem:
There is a large set of records. Each record in the set has an attribute. For some values of the attribute, there is only one record, for other values there are many ...
- 111
2
votes
3
answers
78
views
Is big data a fallacy if most phenomena can be mostly described by few variables?
Is big data a fallacy if most phenomena can be mostly described by few variables?
This has confused me. Surely there are big data sets, but there are also cases when the set of significant or ...
- 416
0
votes
0
answers
48
views
Hadoop creates two input splits (hence two maps) for a very small file : 14 bytes
I am working on a single node cluster, and i am running a wordcount job of a very small file, the file contains roughly 10 words, but the job creates two input splits which means two map jobs, i need ...
- 19
0
votes
2
answers
44
views
How to manage large datasets (approx 95GB)
I was planning some data analysis on a dataset I'll be using for some projects. The dataset in question is ZINC20. Now, I don't need the whole thing so I was going to write some functions that would ...
- 1
1
vote
0
answers
75
views
Best Practice to handle large data in HDBSCAN cluserting using tfidf
I am now about to implement HDBSCAN on a very large set of data, a few million sentences, using tfidf. However, as we know ...
- 421
0
votes
0
answers
24
views
RCPP error when installing rmr2 package
i am trying to use RHADOOP packages, when i get to the rmr2 installation it gives this error:
...
0
votes
0
answers
16
views
Big data storage for PowerBI reporting
Not really a question I guess. More like asking for advice. I’m trying to store our Azure cloud cost data in a centralized location so that our PowerBI can query the data faster rather than calling ...
- 101
0
votes
0
answers
12
views
Behavioral Segmentation Clustering Mixed Large Data - with or without categorical variables
This question may be closed due to being too broad but I feel as though this is the best place to ask my question.
At the moment, I am dealing with some customer data and am looking on segmenting them ...
1
vote
1
answer
31
views
Do best hyperparameters remain constant when data size is scaled?
Basically what the title is. The problem I currently have is that my dataset consists of 2.8 billion rows, and I have it as a Pyspark data frame.
I want to use some library such as FLAML for finding ...
- 121
0
votes
0
answers
21
views
How to Calculate degrees in un-directed network and plot it? | Using python COLAB [duplicate]
I have loaded txt file (com-dblp.ungraph.txt) which includes DBLP collaboration network to a COLAB notebook. I found 317080 nodes & 1049866 edges in the network by reading the txt manually.
Trying ...
0
votes
0
answers
316
views
Is it possible to implement an rdd version of a for loop having map and reduce using pyspark?
I need to test an algorithm that computes a function on a dataframe where in each execution I drop a column and computes the function. This is a example in python pyspark but without using rdd:
...
- 1
2
votes
1
answer
276
views
The single CSV created by combining a large number of CSV files is too large to process. What options do I have?
The dataset I am currently working on has more than 100 csv files, with each of size more than 250MB. These are files containing time series data captured from different locations and all the files ...
- 21
0
votes
0
answers
57
views
Efficient way to compare one record to millions of rows
We have a production table that contains a bucket of customer data. A customer could be the same customer/person at location A and at location B. They are different by how the name is spelled, address ...
1
vote
2
answers
107
views
Dimensionality reduction for millions of features
I have a dataset with 10 million observations and 1 million sparse features.
I would like to build a binary classifier for predicting a particular feature of interest.
My main problem is how to deal ...
- 11
1
vote
0
answers
22
views
Repeating values caught with a binary classifier
If my machine is broken, it starts to repeat certain channels. Thing is if there are no out-liars, it is difficult to tell it's broken as we would expect all data points to be around the same value. I ...
- 11
2
votes
1
answer
843
views
How do I normalize json data into pandas (Covid-19 data) [closed]
I am trying to import all up-to-date datasets in JSON format on the covid-19 pandemic into a pandas dataframe. I believe it should be possible by using ...
- 21
2
votes
1
answer
48
views
Best way to preprocess data
I need to create a machine learning model to predict if a structure is an hotel or an apartment. I have a dataset structured as well:
...
- 53
1
vote
1
answer
219
views
How to get dummy variables from "first name"
I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage.
Since the datasets are so big (...
- 111
5
votes
4
answers
2k
views
Best platform to work with when having millions of rows in dataframe
I have table with around 20 features and millions of observations (rows).
I need to create model base on this table, however, as it is huge, training models like random forest or XGB takes forever. I'...
- 349
2
votes
1
answer
58
views
Data Art/ Data Visualization Art/ Information Art
Few days ago, I learned about data art/ data visualization art/ information art. I think I have interest in it. I want to see how I can use my data science skills in this area. However, I don't know ...
- 171
1
vote
0
answers
95
views
Word count with map reduce
Suppose we use an input file that contains the following lyrics from a famous song:
We’re up all night till the sun
We’re up all night to get some
The input pairs for the Map phase will be the ...
- 111
1
vote
2
answers
151
views
Where can I find study materials? [closed]
Can anyone recommend me some material (books, blogs, youtube channels, ...) to study statistics, Machine Learning and in general Data Science topics?
Thanks
- 471
1
vote
1
answer
104
views
How do I create a dataset from many CSV files that is too large for RAM
I have been handed about 40 GB of CSV files that I need to turn into a database. The files are arranged in a file structure that uses location in that file structure to create a relationship between ...
- 111
1
vote
0
answers
467
views
Best way to find nearest neighbor distance for large datasets
I am a grad student doing research using generative machine learning with pytorch, and I have generated a set of points. I would like to check how similar these new points are to the points I used in ...
0
votes
2
answers
708
views
Fastest way to replace a value in a pandas DataFrame?
I am loading in 1.5m images with 80,000 classes (or I will have to when I eventually train) into a Keras generator and am using a pandas dataframe to do so. The problem is, with so many images, my ...
- 451
0
votes
1
answer
221
views
How to do EDA on large datasets
I have a table in Postgres with ~5million records. When I load the dataset using pandas to perform EDA, I run out of memory.
...
- 1
1
vote
3
answers
563
views
Efficiently modify a large csv file in Pandas
I have a csv file and would like to do the following modification on it:
...
- 11
0
votes
1
answer
62
views
ML modeling a data with big amount of rows
I want to do ML modeling such XGboost, KNN, and similar models on data with 9 numerical features and more than 25 million rows and the size of data is almost 2.5 Gig and I prefer to use all the data ...
0
votes
1
answer
23
views
How to choose feedforward architecture for few number of features but very large instance?
Assume I have 1 million of data instance and each instance contains 100 feature. For each instance, I also have a lable.
The ...
- 309
0
votes
0
answers
35
views
Examples of Open source huge business intelligence datasets
I've observed that many of the datasets available for traditional ML and data science algorithms seem to be in the order of MB. I assumed these may be because earlier computers were not that ...
1
vote
0
answers
691
views
XGBoost incremental training for big datasets
I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred).
I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
0
votes
1
answer
648
views
How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?
I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in ...
0
votes
2
answers
370
views
Subsampling the “right” amout of data to train an ML model
I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
- 1
1
vote
0
answers
51
views
How do you perform basic statistical analysis on a Binary Dataset?
I have never worked with a Binary dataset (1 and 0) (True or False) so I'm unsure what kind of statistical tests I should run to draw up simple conclusions.
I'm doing a data science project and the ...
- 11
5
votes
2
answers
358
views
What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)?
I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are:
AWS instance with massive RAM and train CPU (slow, but ...
- 181
1
vote
1
answer
22
views
Where can I find a dataset that contains criminal case sentencing data? [closed]
I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as:
Type of crime
Defendant Age/Sex/Race
Plea
Verdict
Sentence
Is there a dataset ...
- 111
-2
votes
1
answer
19
views
What data to collect to create useful insights?
I am creating a Social Media app and I want to know what type of data shall I collect to create useful insights for Business accounts or Content Creators? For example if I want to create an insight ...
1
vote
1
answer
118
views
How to train a model using a daunting huge training dataset
I have an extremely huge dataset and I'm wondering me how could be the right way to set an experiment to use this data to train a model.
I understand that I can use data-reduction to, for instance, ...
- 113
0
votes
0
answers
44
views
How to Integrate/map Population Data with/over Sample Data
Problem Definition:
Our organization is conducting different type of surveys and Census in our country. The basic difference between Census and Survey is that, the target of Census is the Complete ...
2
votes
0
answers
65
views
Clustering large set of images
I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
- 21
2
votes
1
answer
85
views
What is the difference between Pachyderm and Git?
I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that:
It holds all your data in a central accessible ...
- 436