Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
3
votes
6answers
2k views

Sentiment Analysis of Movie Reviews using Python

I am currently doing sentiment analysis using Python. Here I am taking all the reviews from movie dataset and using Naive Bayes algorithm to predict whether the review is positive or negative. From ...
1
vote
1answer
67 views

How to fake data based on the condition and weight

I'm trying to fake data for the coffee shop. I've two features age and menu. Menu includes various type of drinks such as coffee [latte, espresso, mocca, etc], tea [milktea, lemontea], milk [freshmilk,...
0
votes
4answers
273 views

What is Data Lake?

I am familiar with the concept of "Big Data" but how does "Data Lake" differ from Big Data? Is it derived from Big Data? Please explain.
1
vote
0answers
11 views

Repeating values caught with a binary classifier

If my machine is broken, it starts to repeat certain channels. Thing is if there are no out-liars, it is difficult to tell it's broken as we would expect all data points to be around the same value. I ...
0
votes
2answers
160 views

huge doubt on anomaly detection

from the naked eye itself, we can tell in the region 5161 the network usage is high so that is the anomaly in my case, then why do we want to apply k-means and other machine learning algorithms to ...
5
votes
2answers
152 views

What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)?

I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are: AWS instance with massive RAM and train CPU (slow, but ...
6
votes
4answers
2k views

XGBoost Huge Dataset ~1TB

Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints. Is ...
3
votes
1answer
39 views

How do I normalize json data into pandas (Covid-19 data) [closed]

I am trying to import all up-to-date datasets in JSON format on the covid-19 pandemic into a pandas dataframe. I believe it should be possible by using ...
0
votes
2answers
707 views

Question about Similarity vs Dissimilarity Matrix

Right now, I'm working on coming up with a similarity vs dissimilarity matrix for a set of data points for a clustering algorithm. My question is if I want to use one of the many clustering algorithms ...
1
vote
1answer
439 views

Sensitivity analysis in outlier explanation

I am trying to find the outlier explanation using the sensitivity analysis. Let’s consider that my dataset contains 19 different input values and 1 output value (So overall 20 different columns are ...
2
votes
3answers
78 views

How to fill missing enteries in column A, and add respective corresponding enteries to column-B with value of previous cell

I am facing an issue with an excel file. I have an excel sheet with 2 columns Column A : Time Increment with per second Column B : A particular value of a machine sensor The problem i am facing ...
1
vote
3answers
134 views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
0
votes
1answer
62 views

How to run Spark python code in Jupyter Notebook via command prompt

I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below After that I want to run this in CMD ...
0
votes
1answer
149 views

Spliting training data into multiple variables using R

So right now I am trying to create multiple variables with training data, and in the process I have reached an error Error in eval(predvars, data, env) : object '1.band1' not found which is a ...
0
votes
1answer
509 views

Neural Network Sensordata as Input

I have a dataset consisting of sensor recordings about human movement. There are 22 classes of different movement like sitting or walking and 19 sensor values. Each recording of a movement has about ...
0
votes
1answer
470 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
3
votes
2answers
481 views

Where does the "deep learning needs big data" rule come from

When reading about deep learning I often come across the rule that deep learning is only effective when you have large amounts of data at your disposal. These statements are generally accompanied by a ...
2
votes
1answer
36 views

Data Art/ Data Visualization Art/ Information Art

Few days ago, I learned about data art/ data visualization art/ information art. I think I have interest in it. I want to see how I can use my data science skills in this area. However, I don't know ...
2
votes
1answer
25 views

Best way to preprocess data

I need to create a machine learning model to predict if a structure is an hotel or an apartment. I have a dataset structured as well: ...
1
vote
0answers
58 views

Huge matrix multiplication with PySpark

I am currently struggling with a problem. I have been googling around and cannot find an answer. Also I am kinda new to PySpark and Spark maybe that's why I am struggling. Let's say I have a huge $m \...
0
votes
1answer
42 views

ML modeling a data with big amount of rows

I want to do ML modeling such XGboost, KNN, and similar models on data with 9 numerical features and more than 25 million rows and the size of data is almost 2.5 Gig and I prefer to use all the data ...
1
vote
0answers
11 views

How to get dummy variables from "first name"

I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage. Since the datasets are so big (...
2
votes
1answer
51 views

Size of datasets over years

I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years. I just found the following statistics: The poll of KDnuggets that actually shows that over ...
0
votes
0answers
7 views

Data Pipelines - Apache Spark - Updating Another Data Store or Input to Queue

What is the correct way to think about or approach something like Spark breaking down a 160GB file into smaller manageable parts, or single records and then doing something like updating records in ...
2
votes
1answer
15k views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
3
votes
1answer
355 views

Advice on dealing with very large datasets - HDF5, Python

I've recently started working on an application for visualization of really big datasets. While reading online it became apparent that most people use HDF5 for storing big, multi-dimensional datasets ...
5
votes
4answers
580 views

Best platform to work with when having millions of rows in dataframe

I have table with around 20 features and millions of observations (rows). I need to create model base on this table, however, as it is huge, training models like random forest or XGB takes forever. I'...
1
vote
1answer
183 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
1
vote
2answers
102 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
0
votes
1answer
82 views

How do I determine the best statistical way for data transformation for standardization (like log, sq root) to remove bias between different datasets?

I'm currently working on applying data science to High Performance Computing cluster, by analyzing the log files generated and trying to see if there is a pattern that leads to a system failure(...
0
votes
1answer
56 views

How to do EDA on large datasets

I have a table in Postgres with ~5million records. When I load the dataset using pandas to perform EDA, I run out of memory. ...
2
votes
1answer
103 views

Selecting the number of hashes for minhash? Working with extremely sparse data and want more collisions

I'm attempting to use minhash to generate clusters and similarities, and I am primarily using ideas from these resources. http://www2007.org/papers/paper570.pdf https://chrisjmccormick.wordpress.com/...
1
vote
1answer
33 views

Recommender system: Give a feature more significance than another

I am trying to build a recommender system that predicts hotel prices based on a great number of features. I have a column representing the hotel rating out of 5 and another column indicating the ...
1
vote
0answers
24 views

Word count with map reduce

Suppose we use an input file that contains the following lyrics from a famous song: We’re up all night till the sun We’re up all night to get some The input pairs for the Map phase will be the ...
3
votes
1answer
66 views

anomaly detection in vehicle sensor data

I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors. It seems like there is no proper book or article that goes deeper into ...
1
vote
1answer
202 views

Public dataset for news articles with their associated categories for multilabel data classification

I am wondering if there are any public datasets of news, like The New York Times (NYT) or similar to various news categories such as politics, entertainment, lifestyle, general news, sports, etc. I ...
1
vote
2answers
139 views

Where can I find study materials? [closed]

Can anyone recommend me some material (books, blogs, youtube channels, ...) to study statistics, Machine Learning and in general Data Science topics? Thanks
0
votes
1answer
241 views

How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?

I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in ...
0
votes
1answer
64 views

Subsampling the “right” amout of data to train an ML model

I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
6
votes
1answer
100 views

Differences between big data, data warehousing, business intelligence and data science?

I know they are four different areas, but I would like to know what are the main differences between those disciplines, and how they are related to each other if some of them depend on each other, and ...
1
vote
0answers
73 views

Best way to find nearest neighbor distance for large datasets

I am a grad student doing research using generative machine learning with pytorch, and I have generated a set of points. I would like to check how similar these new points are to the points I used in ...
0
votes
1answer
23 views

How do I create a dataset from many CSV files that is too large for RAM

I have been handed about 40 GB of CSV files that I need to turn into a database. The files are arranged in a file structure that uses location in that file structure to create a relationship between ...
0
votes
0answers
11 views

Find a column by name in a row in scala spark

I have a Seq[Row].Each row is an Array of Struct.Struct has four fields: a,b,c and d all of which are String.The data in a particular row is something like this: [{"a":"ahahk",&...
2
votes
1answer
55 views

does storing file in hdfs parallelize it for Spark?

For Spark's RDD operations, data must be in shape of RDD or be parallelized using: ParallelizedData = sc.parallelize(data) My question is that if I store data in ...
2
votes
1answer
88 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
0
votes
2answers
175 views

Fastest way to replace a value in a pandas DataFrame?

I am loading in 1.5m images with 80,000 classes (or I will have to when I eventually train) into a Keras generator and am using a pandas dataframe to do so. The problem is, with so many images, my ...
2
votes
3answers
3k views

Handling large imbalanced data set

I have an imbalanced data set consisting of some 10's of millions text strings, each with thousands of features created by uni- and bigrams, and additionally I have also the string length and entropy ...
0
votes
0answers
38 views

KMeans using Mapreduce in Python

I wrote a mapreduce code in python which works locally i.e., cat test_mapper |python mapper.py sort the result, and ...
0
votes
2answers
78 views

Is there an unsupervised learning algorithm that can cluster data based on more than two dimensions?

I am just beginning to get into data science and have never posted here before, apologies if this question is worded incorrectly! I am curious if there is an unsupervised machine learning algorithm ...
0
votes
0answers
13 views

suggestion needed for big data development

I am trying to find out what is state of the art with database, python, and big data. My starting point began with a SQL server, and multiprocessing pandas, and dask. Imagine I need to maintain a ...

1
2 3 4 5
10