Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
0 votes
0 answers
24 views

Fuzzy Name Matching with Machine Learning. Input data encoding

I have a huge amount of data in my dataset: Last name, first name, date of birth of Indian residents and I need to match them for similarity. The matching is fuzzy, the data looks like this (names are ...
ккк ккк's user avatar
0 votes
0 answers
82 views

How to do feature selection correctly in xgboost for time series forecasting after obtaining a good predictive model?

I have a very large dataset (~7 million rows) for which I have extracted ~500 features during feature engineering phase. I have trained an XGBoost which has a fairly good predictive capability (based ...
guestar's user avatar
0 votes
0 answers
24 views

statistical analysis of two data sets have different number of sampling

I have two datasets from my statistical fault analysis: one with 400 iterations and another with 500 iterations. Both datasets have the same maximum and minimum values, and some data points are ...
Fadi's user avatar
  • 1
2 votes
2 answers
139 views

How to deal with high data volumes? (Tools, techniques, concepts, etc.)

I have some doubts about how to deal with high volumes of data. I'm currently working in the data analysis/data science field, so I've had the chance to perform calculations, manipulate data, and ...
tms's user avatar
  • 21
0 votes
0 answers
13 views

Key value stores

I am writing an essay & I couldn't find a clear answer, I am new to this concept. Do key-value stores have specific attributes? What are they? I need a source for further understanding if possible....
Kay zhrani's user avatar
0 votes
0 answers
16 views

understanding how apache spark functionality overlaps with other popular sql query engine and big data frameworks

i m really interested in seeing how functionality of diff big data tools overlaps between apache spark, apache drill, cloudera impala, trino, presto db. i m confused about the difference btw"...
wando's user avatar
  • 111
1 vote
0 answers
20 views

How to find own way in data field? [closed]

I need help finding my path in the data field. I just completed my MSc in Big Data at university. During my studies, I gained approximately Junior level Python and SQL programming skills, worked with ...
Gleb's user avatar
  • 11
0 votes
2 answers
35 views

Data imputation for heavily missing features

I am currently working on the dataset IEEE-CIS Fraud Detection, provided via Kaggle, with around 350 features, with around 600k instances. However, some features are missing large amounts of values, ...
Hai Nguyen's user avatar
0 votes
1 answer
36 views

Can I update the source of Data found in a Data Lake or Data Blob

Is it possible to update the source of data found in a Data Lake or Data Blob? What about while using HDInsight or Azure Databricks?
JF0001's user avatar
  • 101
0 votes
1 answer
118 views

Gensim: create a dictionary from a large corpus without loading it in RAM?

The topic modelling library Gensim offers the ability to stream a large document instead of storing it in memory. Streaming is possible for the stage of converting the corpus to BOW, but the ...
Erwan's user avatar
  • 25.5k
0 votes
2 answers
54 views

What kinds of ML models should I use when the outcome variable does not vary with time but only vary across individuals and groups?

I am trying to predict individuals’ income in 2018 using 18 years worth of data for people who were born in 1978,1979, and 1980 on many variables such as family income, location, family members’ ...
Aman Desai's user avatar
1 vote
1 answer
33 views

What is the name of my problem - distribution of counts of elements having certain attribute

I have the following problem: There is a large set of records. Each record in the set has an attribute. For some values of the attribute, there is only one record, for other values there are many ...
danatel's user avatar
  • 111
2 votes
3 answers
87 views

Is big data a fallacy if most phenomena can be mostly described by few variables?

Is big data a fallacy if most phenomena can be mostly described by few variables? This has confused me. Surely there are big data sets, but there are also cases when the set of significant or ...
mavavilj's user avatar
  • 416
0 votes
2 answers
58 views

How to manage large datasets (approx 95GB)

I was planning some data analysis on a dataset I'll be using for some projects. The dataset in question is ZINC20. Now, I don't need the whole thing so I was going to write some functions that would ...
apvn's user avatar
  • 1
1 vote
0 answers
384 views

Best Practice to handle large data in HDBSCAN cluserting using tfidf

I am now about to implement HDBSCAN on a very large set of data, a few million sentences, using tfidf. However, as we know ...
Jazz's user avatar
  • 420
1 vote
1 answer
44 views

Do best hyperparameters remain constant when data size is scaled?

Basically what the title is. The problem I currently have is that my dataset consists of 2.8 billion rows, and I have it as a Pyspark data frame. I want to use some library such as FLAML for finding ...
Vishal Balaji's user avatar
2 votes
1 answer
618 views

The single CSV created by combining a large number of CSV files is too large to process. What options do I have?

The dataset I am currently working on has more than 100 csv files, with each of size more than 250MB. These are files containing time series data captured from different locations and all the files ...
odd_wolf's user avatar
1 vote
2 answers
327 views

Dimensionality reduction for millions of features

I have a dataset with 10 million observations and 1 million sparse features. I would like to build a binary classifier for predicting a particular feature of interest. My main problem is how to deal ...
j k's user avatar
  • 11
1 vote
0 answers
26 views

Repeating values caught with a binary classifier

If my machine is broken, it starts to repeat certain channels. Thing is if there are no out-liars, it is difficult to tell it's broken as we would expect all data points to be around the same value. I ...
gotenks's user avatar
  • 11
2 votes
1 answer
2k views

How do I normalize json data into pandas (Covid-19 data) [closed]

I am trying to import all up-to-date datasets in JSON format on the covid-19 pandemic into a pandas dataframe. I believe it should be possible by using ...
jontheil's user avatar
2 votes
1 answer
63 views

Best way to preprocess data

I need to create a machine learning model to predict if a structure is an hotel or an apartment. I have a dataset structured as well: ...
Fabio's user avatar
  • 53
1 vote
1 answer
287 views

How to get dummy variables from "first name"

I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage. Since the datasets are so big (...
Raha Moosavi's user avatar
5 votes
4 answers
3k views

Best platform to work with when having millions of rows in dataframe

I have table with around 20 features and millions of observations (rows). I need to create model base on this table, however, as it is huge, training models like random forest or XGB takes forever. I'...
Reut's user avatar
  • 349
2 votes
1 answer
82 views

Data Art/ Data Visualization Art/ Information Art

Few days ago, I learned about data art/ data visualization art/ information art. I think I have interest in it. I want to see how I can use my data science skills in this area. However, I don't know ...
Mr Prof's user avatar
  • 171
1 vote
0 answers
115 views

Word count with map reduce

Suppose we use an input file that contains the following lyrics from a famous song: We’re up all night till the sun We’re up all night to get some The input pairs for the Map phase will be the ...
def __init__'s user avatar
1 vote
2 answers
159 views

Where can I find study materials? [closed]

Can anyone recommend me some material (books, blogs, youtube channels, ...) to study statistics, Machine Learning and in general Data Science topics? Thanks
Inuraghe's user avatar
  • 481
1 vote
1 answer
222 views

How do I create a dataset from many CSV files that is too large for RAM

I have been handed about 40 GB of CSV files that I need to turn into a database. The files are arranged in a file structure that uses location in that file structure to create a relationship between ...
Finncent Price's user avatar
1 vote
0 answers
738 views

Best way to find nearest neighbor distance for large datasets

I am a grad student doing research using generative machine learning with pytorch, and I have generated a set of points. I would like to check how similar these new points are to the points I used in ...
Amateur Coding Bird's user avatar
0 votes
2 answers
1k views

Fastest way to replace a value in a pandas DataFrame?

I am loading in 1.5m images with 80,000 classes (or I will have to when I eventually train) into a Keras generator and am using a pandas dataframe to do so. The problem is, with so many images, my ...
Finn Williams's user avatar
0 votes
1 answer
430 views

How to do EDA on large datasets

I have a table in Postgres with ~5million records. When I load the dataset using pandas to perform EDA, I run out of memory. ...
MikeB's user avatar
  • 1
1 vote
3 answers
1k views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
Jost A.'s user avatar
  • 11
0 votes
1 answer
120 views

ML modeling a data with big amount of rows

I want to do ML modeling such XGboost, KNN, and similar models on data with 9 numerical features and more than 25 million rows and the size of data is almost 2.5 Gig and I prefer to use all the data ...
Amin Zadenoori's user avatar
0 votes
1 answer
26 views

How to choose feedforward architecture for few number of features but very large instance?

Assume I have 1 million of data instance and each instance contains 100 feature. For each instance, I also have a lable. The ...
jason's user avatar
  • 329
0 votes
0 answers
39 views

Examples of Open source huge business intelligence datasets

I've observed that many of the datasets available for traditional ML and data science algorithms seem to be in the order of MB. I assumed these may be because earlier computers were not that ...
Harsh Kumar's user avatar
2 votes
0 answers
1k views

XGBoost incremental training for big datasets

I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred). I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
Mattia Surricchio's user avatar
0 votes
1 answer
887 views

How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?

I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in ...
Sebastian's user avatar
1 vote
2 answers
800 views

Subsampling the “right” amout of data to train an ML model

I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
giz's user avatar
  • 11
1 vote
0 answers
72 views

How do you perform basic statistical analysis on a Binary Dataset?

I have never worked with a Binary dataset (1 and 0) (True or False) so I'm unsure what kind of statistical tests I should run to draw up simple conclusions. I'm doing a data science project and the ...
Devin's user avatar
  • 11
7 votes
2 answers
806 views

What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)?

I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are: AWS instance with massive RAM and train CPU (slow, but ...
lara_toff's user avatar
  • 221
1 vote
1 answer
38 views

Where can I find a dataset that contains criminal case sentencing data? [closed]

I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as: Type of crime Defendant Age/Sex/Race Plea Verdict Sentence Is there a dataset ...
N00b101's user avatar
  • 111
-2 votes
1 answer
21 views

What data to collect to create useful insights?

I am creating a Social Media app and I want to know what type of data shall I collect to create useful insights for Business accounts or Content Creators? For example if I want to create an insight ...
Rehmat Singh Gill's user avatar
1 vote
1 answer
366 views

How to train a model using a daunting huge training dataset

I have an extremely huge dataset and I'm wondering me how could be the right way to set an experiment to use this data to train a model. I understand that I can use data-reduction to, for instance, ...
Duloren's user avatar
  • 113
0 votes
0 answers
53 views

How to Integrate/map Population Data with/over Sample Data

Problem Definition: Our organization is conducting different type of surveys and Census in our country. The basic difference between Census and Survey is that, the target of Census is the Complete ...
Asif Khawaja's user avatar
3 votes
0 answers
134 views

Clustering large set of images

I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
Overloop's user avatar
2 votes
1 answer
111 views

What is the difference between Pachyderm and Git?

I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that: It holds all your data in a central accessible ...
Lerner Zhang's user avatar
3 votes
1 answer
148 views

Why do machine learning engineers insist on training with more data than validation set?

Among my colleagues I have noticed a curious insistence on training with, say, 70% or 80% of data and validating on the remainder. The reason it is curious to me is the lack of any theoretical ...
tdMJN6B2JtUe's user avatar
0 votes
1 answer
35 views

How to grade an interaction that a user had with a post with an AI based on big data?

Context I'm creating a social network. The thing is, I don't want to order posts by likes, or something like that, I'm using an AI (lightfm in ...
johannb75's user avatar
  • 117
2 votes
1 answer
139 views

How do data types influence hardware (CPU / GPU / TPU) performance?

I am currently dealing with a relatively big data set, for which I have some memory usage concerns. I am dealing with most of the different data types : floats, integers, Booleans, characters strings ...
Lucas Morin's user avatar
  • 2,244
0 votes
0 answers
16 views

Ideas on handling large quantities of "waste" data

Hello and thank you for taking time out of your day to help me. I am currently working on developing a machine vision application for production monitoring. The application handles images, about 20,...
Ηλίας Κωνσταντινίδης's user avatar
2 votes
1 answer
3k views

How to store efficiently very large sparse 3D matrices

To train a CNN, I have stacked arrays of images over observations [observations x width x length]. The dataset is very sparse ($95\%$). What would be an efficient ...
hH1sG0n3's user avatar
  • 2,058

1
2 3 4 5
10