Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
1
vote
0answers
35 views

Best way to find nearest neighbor distance for large datasets

I am a grad student doing research using generative machine learning with pytorch, and I have generated a set of points. I would like to check how similar these new points are to the points I used in ...
1
vote
2answers
100 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
-1
votes
0answers
11 views

How would you find the average number of phone calls a day given the date, weekday, the hour, and total calls from the previous month? [closed]

How would you find the average number of phone calls a day using this type of data? I am trying to do this for the previous month. This is just part the data that is displayed here. I want to try to ...
2
votes
1answer
61 views

How to fake data based on the condition and weight

I'm trying to fake data for the coffee shop. I've two features age and menu. Menu includes various type of drinks such as coffee [latte, espresso, mocca, etc], tea [milktea, lemontea], milk [freshmilk,...
0
votes
2answers
155 views

huge doubt on anomaly detection

from the naked eye itself, we can tell in the region 5161 the network usage is high so that is the anomaly in my case, then why do we want to apply k-means and other machine learning algorithms to ...
2
votes
1answer
103 views

Selecting the number of hashes for minhash? Working with extremely sparse data and want more collisions

I'm attempting to use minhash to generate clusters and similarities, and I am primarily using ideas from these resources. http://www2007.org/papers/paper570.pdf https://chrisjmccormick.wordpress.com/...
0
votes
1answer
13 views

How do I create a dataset from many CSV files that is too large for RAM

I have been handed about 40 GB of CSV files that I need to turn into a database. The files are arranged in a file structure that uses location in that file structure to create a relationship between ...
0
votes
0answers
11 views

Find a column by name in a row in scala spark

I have a Seq[Row].Each row is an Array of Struct.Struct has four fields: a,b,c and d all of which are String.The data in a particular row is something like this: [{"a":"ahahk",&...
2
votes
1answer
55 views

does storing file in hdfs parallelize it for Spark?

For Spark's RDD operations, data must be in shape of RDD or be parallelized using: ParallelizedData = sc.parallelize(data) My question is that if I store data in ...
1
vote
1answer
405 views

Sensitivity analysis in outlier explanation

I am trying to find the outlier explanation using the sensitivity analysis. Let’s consider that my dataset contains 19 different input values and 1 output value (So overall 20 different columns are ...
2
votes
1answer
85 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
0
votes
2answers
22 views

Fastest way to replace a value in a pandas DataFrame?

I am loading in 1.5m images with 80,000 classes (or I will have to when I eventually train) into a Keras generator and am using a pandas dataframe to do so. The problem is, with so many images, my ...
1
vote
1answer
31 views

Recommender system: Give a feature more significance than another

I am trying to build a recommender system that predicts hotel prices based on a great number of features. I have a column representing the hotel rating out of 5 and another column indicating the ...
2
votes
3answers
3k views

Handling large imbalanced data set

I have an imbalanced data set consisting of some 10's of millions text strings, each with thousands of features created by uni- and bigrams, and additionally I have also the string length and entropy ...
1
vote
1answer
149 views

Public dataset for news articles with their associated categories for multilabel data classification

I am wondering if there are any public datasets of news, like The New York Times (NYT) or similar to various news categories such as politics, entertainment, lifestyle, general news, sports, etc. I ...
1
vote
3answers
60 views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
0
votes
1answer
26 views

How to do EDA on large datasets

I have a table in Postgres with ~5million records. When I load the dataset using pandas to perform EDA, I run out of memory. ...
0
votes
1answer
52 views

How to run Spark python code in Jupyter Notebook via command prompt

I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below After that I want to run this in CMD ...
0
votes
2answers
603 views

Question about Similarity vs Dissimilarity Matrix

Right now, I'm working on coming up with a similarity vs dissimilarity matrix for a set of data points for a clustering algorithm. My question is if I want to use one of the many clustering algorithms ...
0
votes
0answers
15 views

KMeans using Mapreduce in Python

I wrote a mapreduce code in python which works locally i.e., cat test_mapper |python mapper.py sort the result, and ...
0
votes
1answer
140 views

Spliting training data into multiple variables using R

So right now I am trying to create multiple variables with training data, and in the process I have reached an error Error in eval(predvars, data, env) : object '1.band1' not found which is a ...
0
votes
2answers
73 views

Is there an unsupervised learning algorithm that can cluster data based on more than two dimensions?

I am just beginning to get into data science and have never posted here before, apologies if this question is worded incorrectly! I am curious if there is an unsupervised machine learning algorithm ...
0
votes
0answers
11 views

suggestion needed for big data development

I am trying to find out what is state of the art with database, python, and big data. My starting point began with a SQL server, and multiprocessing pandas, and dask. Imagine I need to maintain a ...
0
votes
1answer
493 views

Neural Network Sensordata as Input

I have a dataset consisting of sensor recordings about human movement. There are 22 classes of different movement like sitting or walking and 19 sensor values. Each recording of a movement has about ...
0
votes
1answer
422 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
3
votes
2answers
438 views

Where does the "deep learning needs big data" rule come from

When reading about deep learning I often come across the rule that deep learning is only effective when you have large amounts of data at your disposal. These statements are generally accompanied by a ...
4
votes
2answers
2k views

Amazon S3 vs Google Drive [closed]

The majority of people use S3. However, Google Drive seems a promising alternative solution for storing large amounts of data. Are there specific reasons why one is better than the other?
1
vote
1answer
39 views

ML modeling a data with big amount of rows

I want to do ML modeling such XGboost, KNN, and similar models on data with 9 numerical features and more than 25 million rows and the size of data is almost 2.5 Gig and I prefer to use all the data ...
2
votes
1answer
43 views

Size of datasets over years

I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years. I just found the following statistics: The poll of KDnuggets that actually shows that over ...
0
votes
0answers
138 views

Access global variable from UDF (User Defined Function) in python in spark

I am trying to alter a global variable from inside a pyspark.sql.functions.udf function in python. But, the change in not getting reflected in the global variable. ...
1
vote
1answer
166 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
3
votes
1answer
346 views

Advice on dealing with very large datasets - HDF5, Python

I've recently started working on an application for visualization of really big datasets. While reading online it became apparent that most people use HDF5 for storing big, multi-dimensional datasets ...
3
votes
1answer
46 views

Merging 2 datawarehouses

Suppose I have 2 datawarehouses of two different companies that are being merged. Some of the tables are common but includes different records, while others are different. If I use big data ...
0
votes
3answers
224 views

What is Data Lake?

I am familiar with the concept of "Big Data" but how does "Data Lake" differ from Big Data? Is it derived from Big Data? Please explain.
3
votes
1answer
57 views

anomaly detection in vehicle sensor data

I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors. It seems like there is no proper book or article that goes deeper into ...
0
votes
1answer
39 views

Subsampling the “right” amout of data to train an ML model

I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
2
votes
1answer
15k views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
2
votes
3answers
72 views

How to fill missing enteries in column A, and add respective corresponding enteries to column-B with value of previous cell

I am facing an issue with an excel file. I have an excel sheet with 2 columns Column A : Time Increment with per second Column B : A particular value of a machine sensor The problem i am facing ...
0
votes
1answer
130 views

How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?

I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in ...
0
votes
1answer
22 views

How to choose feedforward architecture for few number of features but very large instance?

Assume I have 1 million of data instance and each instance contains 100 feature. For each instance, I also have a lable. The ...
0
votes
0answers
27 views

Examples of Open source huge business intelligence datasets

I've observed that many of the datasets available for traditional ML and data science algorithms seem to be in the order of MB. I assumed these may be because earlier computers were not that ...
-1
votes
1answer
462 views

How to store efficiently very large sparse 3D matrices

To train a CNN, I have stacked arrays of images over observations [observations x width x length]. The dataset is very sparse ($95\%$). What would be an efficient ...
2
votes
0answers
73 views

XGBoost incremental training for big datasets

I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred). I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
1
vote
1answer
23 views

How do data types influence hardware (CPU / GPU / TPU) performance?

I am currently dealing with a relatively big data set, for which I have some memory usage concerns. I am dealing with most of the different data types : floats, integers, Booleans, characters strings ...
0
votes
0answers
13 views

What are the execution service and the event service in Netflix big data

From the following article by the netflix engineering team: The core architecture of the big data platform at Netflix involves three key services. These are the execution service (Genie), the ...
-1
votes
1answer
75 views

What are the differences between IBM BlueMix and IBM Data Science Experience?

This may seem like a silly question, but as I am going through the documentation for both services it is difficult to disentangle what each does, specifically. From what I've gathered BlueMix is ...
3
votes
6answers
8k views

How much of a background in programming is necessary to become a data scientist? [closed]

I once knew some Java, but that was close to 10 years ago. Assuming I can learn a language to get into data analytics.... what language do you recommend?
0
votes
0answers
11 views

Scalable Infrence Server for Object Detection

I have created a Django service (nginx + Gunicorn) for object detection models. For my case i have 50+ models with resnet 50 based back bone. Server Machine Specification: 16 CPU 64 GB Ram I have ...
1
vote
0answers
128 views

Most efficient algorithm for sequential pattern mining on a dataset with large amount of items in each transaction?

To provide some context, I am trying to do frequent pattern mining on a dataset of system error logs from servers. I organized it into transactions based on the thread ID, which results in some very ...
0
votes
0answers
22 views

What should be the architecture design for the dataset to be used for a machine learning API

I have aroud 30GB of market dataset in csv format which is downloaded onto my server computer everyday. Now I want to use this dataset to do some computations and provide some analysis. The user will ...

1
2 3 4 5
10