Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
0
votes
1answer
21 views

How to determine sample rate of a time series dataset?

I have a dataset of magnetometer sensor readings which looks like: ...
0
votes
0answers
11 views

How do I efficiently load data from disk during training of deep learning models in pytorch?

I'm trying to train a deep learning model without loading the entire dataset into memory. My main question is, what's the best way of doing this? It seems like HDF5 is a common method that people ...
2
votes
0answers
19 views

Spark: How to run PCA parallelized? Only one thread used

I use pySpark and set my configuration like following: ...
1
vote
0answers
24 views

What is important for Pharmaceutical companies to answer with Big Data Analysis?

I am a data scientist, and I have some biological background (genetics). I have been asked to give a talk for our customers from pharmaceutical industry. I should show them how they benefit from Big ...
1
vote
2answers
19 views

Maximum number of images to train a convolutional neural network

I just wondered if there is a technical limit on the number of images to train a neural network. I want to work with extremely high numbers of images, around 1,000,000 to 10,000,000 images. Is there ...
0
votes
1answer
18 views

Training models from sklearn using tf.distribute.MirroredStrategy

I want to distribute the training of a simple model, such as a support vector classifier like sklearn.svm.SVC() across some or all CPUs and GPUs on a single device. I have never utilized a GPU before ...
0
votes
1answer
12 views

Suggestion of dataset

I am implementing my own deep network, but I am not so good at calculus so my network only works for binary data in the moment. I have been searching for big tabular datasets that are for binary ...
0
votes
0answers
7 views

What makes a good image dataset to compare biological NNs to ANNs?

I'm going to be comparing biological NNs to ANNs based on compounded adversarial attacks, but wish to know what makes a good image dataset to test on biological NNs? Going for image classification. I'...
2
votes
0answers
26 views

anomaly detection in vehicle sensor data

I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors. It seems like there is no proper book or article that goes deeper into ...
1
vote
1answer
15 views

How to feed key-value features (aggregated data) to LSTM?

I have the following time-series aggregated input for an LSTM-based model: ...
0
votes
0answers
3 views

Surpassing MongoCompass limit() limitations

I have a cluster containing aproximately 8 million files in json format(tweets). My problem is that when analyzing the schema, I can only visualize 1000 files, no matter what number i enter in the ...
2
votes
1answer
22 views

When training NN, how do data loaders work on large datasets?

How do you normally organize large data-sets for easy loading when training Neural Networks? I have a largeish data-set which cannot fit into memory, it consists of 200,000 samples, with 10k samples ...
0
votes
0answers
12 views

Using lmer in R Studio with a large data set

I have a very large data set (127,000 observations, each point has 22 variables). I am only interested in 7 of those variables. The two dependent variables (linear and rotational acceleration) are ...
0
votes
0answers
24 views

Evaluate new store location using customer movement data

I'm trying to build a decision support model for a brand to help in deciding where to open a new store of the brand in an urban area. The model will be focused on location observations (lat-lng, ...
0
votes
0answers
17 views

How to use data provided in EEM20 forecasting competition?

I am new to competitions. I have gone through few Kaggle competitions and most of the time, they provide train data, test data and other supplementary data. Recently, I came across a new competition (...
0
votes
1answer
19 views

Creating more than one worker nodes for local windows machine [closed]

I am using windows laptop. And I installed apache spark for my laptop. And I try to measure spark performance by changing spark components. because of that I want to create more than one worker nodes ...
0
votes
0answers
25 views

Alternatives to reshaping the data

I have a medical dataset which looks like this: ...
0
votes
0answers
21 views

SGDClassifier on Big Data

I am trying actually to train a SGDClassifier with over 4,000,000 samples of data without any positive results. X vector has 6 features and looks like : [ 2 , 4 , 56431555 , 1 , 0 , 33] Y vector has ...
0
votes
1answer
22 views

How to run Spark python code in Jupyter Notebook via command prompt

I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below After that I want to run this in CMD ...
0
votes
0answers
41 views

Dask error when reading data from a large zip file

I have a .zip folder with a lot of .csv files which I read into a Dask DataFrame like this: ...
0
votes
0answers
10 views

Basket items optimisation minimising constraints

I have a real problem (not home work) when I have to distribute an ordered list by position to respect some constraints eg. 1. 11 2. 15 3. 18 4. 18 5. 1 baskets:...
0
votes
0answers
10 views

Loading file into and out of HDFS via system call/cmd line vs using libhdfs

I am trying to implement a simple C/C++ program for the HDFS file system like word count, it takes a file from the input path puts it into HDFS (where it gets split), processed my map-reduce function ...
0
votes
0answers
10 views

Storing event data in JSON vs. rows in another table

I want to better understand the constrains and considerations I need to keep in mind when processing site analytics. For example, let's say that I have a table that contains page view data. There is a ...
0
votes
0answers
12 views

Approximate evaluation for deep learning architecture

When train on big dataset for deep learning architecture, like imagenet, it takes long time to judge whether our new neural network architecture is good, say for image classification. Is there a way ...
0
votes
0answers
15 views

Trouble understanding how I could use multivariate time series to predict when an error will occur?

First off, I have very limited knowledge statistics-wise and am more of a coder. I was thrown into a large scale project and could use some guidance. I have a large multivariate time series dataset ...
0
votes
2answers
47 views

Is there an unsupervised learning algorithm that can cluster data based on more than two dimensions?

I am just beginning to get into data science and have never posted here before, apologies if this question is worded incorrectly! I am curious if there is an unsupervised machine learning algorithm ...
1
vote
0answers
34 views

need scalable test-to-control matching method for Amazon cloud environment

I'm working on a project to port some of my employer's processes from our local Unix servers to the Amazon cloud. One process matches records from a test group to records from a control group using ...
1
vote
0answers
17 views

NearestNeighbors testing

I have used nbrs = NearestNeighbors(metric= 'cosine', algorithm='brute').fit(items_features) distances, indices = nbrs.kneighbors(item_features) to find some ...
1
vote
2answers
73 views

Multiprocessing or Parallel Computing Python Code

I am working on bioinformatics big datasets; training set and optimisation taking huge time to execute. I check and found that training and optimisation compiling on one core of cpu and because of ...
3
votes
1answer
134 views

Comparing one small dataset with a big dataset for similar records

I create a varying small dataset (dataset: X) with 500 records in each query. Everytime I need to compare the dataset with a bigger one (dataset: A) (15 milion records) to find similar (or semi-...
0
votes
0answers
15 views

Project structure ML time-series forecasting

I'm about to start with a ML learning time-series forecasting project which includes large-quantities of (2-million) time-series stored in JSON-files. The first step will involve feature engineering....
0
votes
0answers
33 views

Machine Learning or Big Data?

I am a new MSc applied math graduate, with some programming skills in Java and Python. My focus during my master's was differential equations and numerical analysis. Once upon a time, I learned other ...
-1
votes
1answer
15 views

Searching chronological data

I have daily data of more than 10 years in Excel so I want to select data of individual days from all the years. So, how can I do it? Suggestion either in R or python is highly appreciated because it ...
0
votes
0answers
15 views

HDFS on docker containers - is it possible?

we are using Hadoop cluster based on HDP version - 2.6.5 with ambari platform we want to know if we can use the following HDFS components on docker containers as: journal nodes ZK fail controller ...
1
vote
0answers
22 views

Advantage of Centralized training in deep learning over distributed training

I know distributed training is the ultimate solution for scalability and solving resource constraints and sometimes distributed training outperforms centralized training in terms of accuracy. But I'm ...
1
vote
1answer
24 views

How to compute modulo of a hash?

Let's say that I have a set of users in my database, that have GUIDs as their IDs. I use xxhash to generate fixed-length hashes for each value, so that I can then ...
2
votes
1answer
45 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
0
votes
0answers
57 views

Fastest frequent itemset library in R or python

I am trying to map a problem with frequent itemset mining. I have only 60 transactions in total but most of them are big. Like the smallest transaction have 30 items and the largest transaction has ...
3
votes
2answers
295 views

How do tech-companies employ Random Forest on large data sets?

The algorithm takes quite a long time to train on large data sets with a moderate number of parameters: https://stats.stackexchange.com/questions/37370/random-forest-computing-time-in-r https://...
0
votes
0answers
17 views

Where is the start point if I want to use artificial intelligence on NASA space industry?

It sounds like dump question but as a beginner, working/learning AI and want to understand what artificial intelligence in the NASA space industry is doing. I want to work on this field but I don't ...
0
votes
0answers
32 views

R language - text.tree()'s pretty parameter

running the below R code to perform analysis on RoomOccupancy dataset ...
1
vote
1answer
3k views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
-1
votes
1answer
52 views

Can a machine learning model be trained on Call Detail Record(CDR) Data to predict user's daily locations?

I have a CDR data for two months and my goal is to extract daily or frequent locations(cell towers) of the user along with the departure and arrival time on those locations. The spatial resolution of ...
1
vote
2answers
115 views

Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0?

Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2....
1
vote
1answer
16 views

NoSQL Comparison - Is this part of my Job?

For the more experienced Data Scientists here, i was asked to perform a case study on how Redis / HBase etc performs, compared to each other.How does data science play a role into this? Note that ...
-1
votes
1answer
25 views

Reading and Processing twitter network data

I have this big data collected from here. So what I would like to asks are How to perform file I/O with the file? From the download like it was mentioned that it used .tsv format, but after ...
2
votes
1answer
44 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
2
votes
0answers
46 views

How to deal with large datasets?

I have some experience with data science but I wanted some insight on how to deal with a very large dataset. I understand simply downloading it to your computer is not plausible so where do you even ...
0
votes
0answers
28 views

Difference between Big Data and a Data Warehouse?

I would like to know what is the exact difference between Big Data and a Data Warehouse. I know that Big Data is the technology to receive and work with really large amounts of data and a data ...
0
votes
0answers
57 views

Running LSTM model on a big data sample using pyspark

I was wondering how does one run an LSTM model on a big dataframe in pyspark. Ideally, one wants to run the model parallelly on different nodes of a spark cluster. But how does one do that?

1
2 3 4 5
9