Questions tagged [bigdata]
Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.
459
questions
0
votes
0
answers
19
views
statistical analysis of two data sets have different number of sampling
I have two datasets from my statistical fault analysis: one with 400 iterations and another with 500 iterations. Both datasets have the same maximum and minimum values, and some data points are ...
- 1
2
votes
2
answers
66
views
How to deal with high data volumes? (Tools, techniques, concepts, etc.)
I have some doubts about how to deal with high volumes of data. I'm currently working in the data analysis/data science field, so I've had the chance to perform calculations, manipulate data, and ...
- 21
0
votes
0
answers
12
views
Key value stores
I am writing an essay & I couldn't find a clear answer, I am new to this concept. Do key-value stores have specific attributes? What are they? I need a source for further understanding if possible....
0
votes
0
answers
15
views
understanding how apache spark functionality overlaps with other popular sql query engine and big data frameworks
i m really interested in seeing how functionality of diff big data tools overlaps between apache spark, apache drill, cloudera impala, trino, presto db.
i m confused about the difference btw"...
- 111
1
vote
0
answers
20
views
How to find own way in data field? [closed]
I need help finding my path in the data field. I just completed my MSc in Big Data at university. During my studies, I gained approximately Junior level Python and SQL programming skills, worked with ...
- 11
0
votes
2
answers
30
views
Data imputation for heavily missing features
I am currently working on the dataset IEEE-CIS Fraud Detection, provided via Kaggle, with around 350 features, with around 600k instances. However, some features are missing large amounts of values, ...
0
votes
0
answers
4
views
How to use BI and Big Data in different industries?
This article describes how real estate uses big data to track clients
Can you give more examples on how to use BI and Big Data for other industries or business?
0
votes
0
answers
15
views
understanding quadratic form in proof of positive definiteness of laplacian matrix
Consider the proof at page 2 found here: https://people.orie.cornell.edu/dpw/orie6334/Fall2016/lecture7.pdf
I cant wrap my head around the 2nd and third line:
\begin{align}
&= \sum_{i \in V}x(i)^2 ...
0
votes
0
answers
575
views
How to dynamically add columns in Dataframe Spark by using a config.ini file parameters
I am learning to use Pyspark and at work I have been assigned a task that I can not solve for now.
There is a config.ini file where there are a series of ...
- 101
0
votes
0
answers
23
views
How to reference an intermediary object in scala?
I am using spark/scala to do some transformations on a spark dataframe. I have the following code:
...
- 11
0
votes
1
answer
32
views
Can I update the source of Data found in a Data Lake or Data Blob
Is it possible to update the source of data found in a Data Lake or Data Blob? What about while using HDInsight or Azure Databricks?
- 101
0
votes
1
answer
86
views
Gensim: create a dictionary from a large corpus without loading it in RAM?
The topic modelling library Gensim offers the ability to stream a large document instead of storing it in memory.
Streaming is possible for the stage of converting the corpus to BOW, but the ...
- 25k
0
votes
2
answers
53
views
What kinds of ML models should I use when the outcome variable does not vary with time but only vary across individuals and groups?
I am trying to predict individuals’ income in 2018 using 18 years worth of data for people who were born in 1978,1979, and 1980 on many variables such as family income, location, family members’ ...
1
vote
1
answer
33
views
What is the name of my problem - distribution of counts of elements having certain attribute
I have the following problem:
There is a large set of records. Each record in the set has an attribute. For some values of the attribute, there is only one record, for other values there are many ...
- 111
2
votes
3
answers
83
views
Is big data a fallacy if most phenomena can be mostly described by few variables?
Is big data a fallacy if most phenomena can be mostly described by few variables?
This has confused me. Surely there are big data sets, but there are also cases when the set of significant or ...
- 416
0
votes
2
answers
52
views
How to manage large datasets (approx 95GB)
I was planning some data analysis on a dataset I'll be using for some projects. The dataset in question is ZINC20. Now, I don't need the whole thing so I was going to write some functions that would ...
- 1
1
vote
0
answers
252
views
Best Practice to handle large data in HDBSCAN cluserting using tfidf
I am now about to implement HDBSCAN on a very large set of data, a few million sentences, using tfidf. However, as we know ...
- 420
1
vote
1
answer
40
views
Do best hyperparameters remain constant when data size is scaled?
Basically what the title is. The problem I currently have is that my dataset consists of 2.8 billion rows, and I have it as a Pyspark data frame.
I want to use some library such as FLAML for finding ...
- 121
2
votes
1
answer
474
views
The single CSV created by combining a large number of CSV files is too large to process. What options do I have?
The dataset I am currently working on has more than 100 csv files, with each of size more than 250MB. These are files containing time series data captured from different locations and all the files ...
- 21
1
vote
2
answers
250
views
Dimensionality reduction for millions of features
I have a dataset with 10 million observations and 1 million sparse features.
I would like to build a binary classifier for predicting a particular feature of interest.
My main problem is how to deal ...
- 11
1
vote
0
answers
24
views
Repeating values caught with a binary classifier
If my machine is broken, it starts to repeat certain channels. Thing is if there are no out-liars, it is difficult to tell it's broken as we would expect all data points to be around the same value. I ...
- 11
2
votes
1
answer
1k
views
How do I normalize json data into pandas (Covid-19 data) [closed]
I am trying to import all up-to-date datasets in JSON format on the covid-19 pandemic into a pandas dataframe. I believe it should be possible by using ...
- 21
2
votes
1
answer
61
views
Best way to preprocess data
I need to create a machine learning model to predict if a structure is an hotel or an apartment. I have a dataset structured as well:
...
- 53
1
vote
1
answer
273
views
How to get dummy variables from "first name"
I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage.
Since the datasets are so big (...
- 111
5
votes
4
answers
3k
views
Best platform to work with when having millions of rows in dataframe
I have table with around 20 features and millions of observations (rows).
I need to create model base on this table, however, as it is huge, training models like random forest or XGB takes forever. I'...
- 349
2
votes
1
answer
70
views
Data Art/ Data Visualization Art/ Information Art
Few days ago, I learned about data art/ data visualization art/ information art. I think I have interest in it. I want to see how I can use my data science skills in this area. However, I don't know ...
- 171
1
vote
0
answers
111
views
Word count with map reduce
Suppose we use an input file that contains the following lyrics from a famous song:
We’re up all night till the sun
We’re up all night to get some
The input pairs for the Map phase will be the ...
- 151
1
vote
2
answers
156
views
Where can I find study materials? [closed]
Can anyone recommend me some material (books, blogs, youtube channels, ...) to study statistics, Machine Learning and in general Data Science topics?
Thanks
- 481
1
vote
1
answer
173
views
How do I create a dataset from many CSV files that is too large for RAM
I have been handed about 40 GB of CSV files that I need to turn into a database. The files are arranged in a file structure that uses location in that file structure to create a relationship between ...
- 111
1
vote
0
answers
661
views
Best way to find nearest neighbor distance for large datasets
I am a grad student doing research using generative machine learning with pytorch, and I have generated a set of points. I would like to check how similar these new points are to the points I used in ...
0
votes
2
answers
911
views
Fastest way to replace a value in a pandas DataFrame?
I am loading in 1.5m images with 80,000 classes (or I will have to when I eventually train) into a Keras generator and am using a pandas dataframe to do so. The problem is, with so many images, my ...
- 451
0
votes
1
answer
371
views
How to do EDA on large datasets
I have a table in Postgres with ~5million records. When I load the dataset using pandas to perform EDA, I run out of memory.
...
- 1
1
vote
3
answers
984
views
Efficiently modify a large csv file in Pandas
I have a csv file and would like to do the following modification on it:
...
- 11
0
votes
1
answer
96
views
ML modeling a data with big amount of rows
I want to do ML modeling such XGboost, KNN, and similar models on data with 9 numerical features and more than 25 million rows and the size of data is almost 2.5 Gig and I prefer to use all the data ...
0
votes
1
answer
26
views
How to choose feedforward architecture for few number of features but very large instance?
Assume I have 1 million of data instance and each instance contains 100 feature. For each instance, I also have a lable.
The ...
- 319
0
votes
0
answers
37
views
Examples of Open source huge business intelligence datasets
I've observed that many of the datasets available for traditional ML and data science algorithms seem to be in the order of MB. I assumed these may be because earlier computers were not that ...
2
votes
0
answers
1k
views
XGBoost incremental training for big datasets
I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred).
I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
0
votes
1
answer
799
views
How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?
I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in ...
1
vote
2
answers
611
views
Subsampling the “right” amout of data to train an ML model
I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
- 11
1
vote
0
answers
62
views
How do you perform basic statistical analysis on a Binary Dataset?
I have never worked with a Binary dataset (1 and 0) (True or False) so I'm unsure what kind of statistical tests I should run to draw up simple conclusions.
I'm doing a data science project and the ...
- 11
6
votes
2
answers
663
views
What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)?
I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are:
AWS instance with massive RAM and train CPU (slow, but ...
- 211
1
vote
1
answer
36
views
Where can I find a dataset that contains criminal case sentencing data? [closed]
I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as:
Type of crime
Defendant Age/Sex/Race
Plea
Verdict
Sentence
Is there a dataset ...
- 111
-2
votes
1
answer
21
views
What data to collect to create useful insights?
I am creating a Social Media app and I want to know what type of data shall I collect to create useful insights for Business accounts or Content Creators? For example if I want to create an insight ...
1
vote
1
answer
208
views
How to train a model using a daunting huge training dataset
I have an extremely huge dataset and I'm wondering me how could be the right way to set an experiment to use this data to train a model.
I understand that I can use data-reduction to, for instance, ...
- 113
0
votes
0
answers
50
views
How to Integrate/map Population Data with/over Sample Data
Problem Definition:
Our organization is conducting different type of surveys and Census in our country. The basic difference between Census and Survey is that, the target of Census is the Complete ...
3
votes
0
answers
102
views
Clustering large set of images
I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
- 31
2
votes
1
answer
104
views
What is the difference between Pachyderm and Git?
I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that:
It holds all your data in a central accessible ...
- 516
3
votes
1
answer
142
views
Why do machine learning engineers insist on training with more data than validation set?
Among my colleagues I have noticed a curious insistence on training with, say, 70% or 80% of data and validating on the remainder. The reason it is curious to me is the lack of any theoretical ...
- 200
0
votes
1
answer
35
views
How to grade an interaction that a user had with a post with an AI based on big data?
Context
I'm creating a social network.
The thing is, I don't want to order posts by likes, or something like that, I'm using an AI (lightfm in ...
- 117
1
vote
1
answer
91
views
How do data types influence hardware (CPU / GPU / TPU) performance?
I am currently dealing with a relatively big data set, for which I have some memory usage concerns. I am dealing with most of the different data types : floats, integers, Booleans, characters strings ...
- 2,567