Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

What are the execution service and the event service in Netflix big data

From the following article by the netflix engineering team: The core architecture of the big data platform at Netflix involves three key services. These are the execution service (Genie), the ...
0
votes
1answer
164 views

How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?

I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in ...
0
votes
1answer
44 views

Subsampling the “right” amout of data to train an ML model

I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
0
votes
0answers
11 views

Scalable Infrence Server for Object Detection

I have created a Django service (nginx + Gunicorn) for object detection models. For my case i have 50+ models with resnet 50 based back bone. Server Machine Specification: 16 CPU 64 GB Ram I have ...
0
votes
0answers
22 views

What should be the architecture design for the dataset to be used for a machine learning API

I have aroud 30GB of market dataset in csv format which is downloaded onto my server computer everyday. Now I want to use this dataset to do some computations and provide some analysis. The user will ...
0
votes
0answers
45 views

Working with mySQL and pytorch Dataset

I am working with high frequency time indexed data. We have 2 types of data each with about 5 columns. For each type of data we have 2500 streams coming in and being updated every 1ms-100ms (the ...
0
votes
0answers
37 views

Extracting possible relationships and insights from a large dataset

I have a dataset with about 26'000 variables and 33'000 continuous observations. My task is to extract any interesting insights/possible relationships between various variables. I am looking for any ...
0
votes
0answers
10 views

How to Combine Oracle and Snowflake/Redshift data on the fly

Have a Schema.Table_Name on Oracle. Exactly similar table is on Snowflake with same metadata(Schema.Table_Name). How can I combine data from two on the fly without moving Oracle data to Redshift. Most ...
0
votes
0answers
33 views

In Spark, what's the role of each of Cluster manager and Spark Context?

What's the role of a cluster manager and the role spark context (Driver program) in terms of managing worker nodes. How do they communicate with each other (and with the worker nodes) and when they do ...
0
votes
0answers
15 views

Trigger job in SAP BW from SAC

Is there a way to have the following occur: When a job is finished on the SAC (SAP Analytics Cloud) side, a job on the SAP BW side should be triggered. Is that at all possible? I have found this which ...
0
votes
0answers
25 views

Difference between MapReduce, Dryag and Pregel in terms of efficiency

What is the difference between the three parallel processing paradigms: MapReduce, Dryag, and Pregel in terms of efficiency for big data analytics applications? Thank you.
0
votes
0answers
38 views

How to Integrate/map Population Data with/over Sample Data

Problem Definition: Our organization is conducting different type of surveys and Census in our country. The basic difference between Census and Survey is that, the target of Census is the Complete ...
0
votes
0answers
39 views

What is the azure solution for storing and querying Netcdf data on the fly?

We currently run a live web application which utilises standard azure mounted file storage to store 2tb worth of metocean data in netcdf format. On here we run python queries directly on the data e.g. ...
0
votes
0answers
21 views

Is there a data structure that works similarly to a (binary) tree?

Im currently working with some data, which is part of a very large data set (height, about 6000 elements per timestep - timestep every minute over days at a time). I need to differentiate between ...
0
votes
0answers
18 views

How are big models like GPT-3 developed?

For models trained with a moderate amount of data, I would expect a the development workflow to include extensive hyper-parameter optimization through extensive retraining. But how is this handled for ...
0
votes
0answers
16 views

Ideas on handling large quantities of "waste" data

Hello and thank you for taking time out of your day to help me. I am currently working on developing a machine vision application for production monitoring. The application handles images, about 20,...
0
votes
0answers
42 views

Create User Table from Google Firebase Analytics and Cloud Firestore Collections

I am exporting my snowplow collections collected on Firebase using the extention Export Collections to BigQuery firestore-bigquery-export v0.1.10. Whenever a document is created, updated, imported, or ...
0
votes
1answer
74 views

Training models from sklearn using tf.distribute.MirroredStrategy

I want to distribute the training of a simple model, such as a support vector classifier like sklearn.svm.SVC() across some or all CPUs and GPUs on a single device. I have never utilized a GPU before ...
0
votes
1answer
19 views

Suggestion of dataset

I am implementing my own deep network, but I am not so good at calculus so my network only works for binary data in the moment. I have been searching for big tabular datasets that are for binary ...
0
votes
0answers
31 views

Alternatives to reshaping the data

I have a medical dataset which looks like this: ...
0
votes
1answer
54 views

How to run Spark python code in Jupyter Notebook via command prompt

I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below After that I want to run this in CMD ...
0
votes
2answers
631 views

Question about Similarity vs Dissimilarity Matrix

Right now, I'm working on coming up with a similarity vs dissimilarity matrix for a set of data points for a clustering algorithm. My question is if I want to use one of the many clustering algorithms ...
0
votes
1answer
86 views

GPS generate street [closed]

I am working on GPS tracking with a huge data from vehicle. Dataset have: vehicleId, speed, orientation(0-360), coordinate (x,y) and timestamp. Can you recommend me how to clean data and model to ...
0
votes
1answer
16 views

Data driven project for a self-catering accommodation business

I want to apologise in advance for my ignorance but I'm hitting my head in a wall here as I'm not sure how to proceed with my project and I've got no experience in that field even though I'm doing a ...
0
votes
0answers
36 views

Python Machine Learning to analyse test-case lab data and identify new test cases from real-world random data

I am currently working on a project where I have predefined test-cases I need to run/test and then analyse the data. My aim is to be able to identify system failures (against pre-defined criteria). ...
0
votes
0answers
526 views

Why modin.pandas takes more wall time than pandas?

All data scientist know that to scale the pandas to large dataset, we use modin.pandas instead of pandas. But when I tried to load a 500 MB csv dataset in my jupyter notebook in Ubuntu 16.04 to ...
0
votes
1answer
496 views

Neural Network Sensordata as Input

I have a dataset consisting of sensor recordings about human movement. There are 22 classes of different movement like sitting or walking and 19 sensor values. Each recording of a movement has about ...
0
votes
2answers
299 views

Disadvantages of hyperparameter tuning on a random sample of dataset

I often work with very large datasets where it would be impractical to check all relevant combinations of hyperparameters when constructing a machine learning model. I'm considering randomly sampling ...
0
votes
1answer
28 views

Loading Data into DW: Direct Insert from PreProcessing vs PreProcess and then loading from CSV files

I have a preprocessing script (google cloud function) that generates files (stored in Google Drive). I want to load those files in my DW (Big Query). What are the pros and cons of: 1) Running the ...
0
votes
1answer
433 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
2answers
155 views

huge doubt on anomaly detection

from the naked eye itself, we can tell in the region 5161 the network usage is high so that is the anomaly in my case, then why do we want to apply k-means and other machine learning algorithms to ...
0
votes
1answer
142 views

Spliting training data into multiple variables using R

So right now I am trying to create multiple variables with training data, and in the process I have reached an error Error in eval(predvars, data, env) : object '1.band1' not found which is a ...
0
votes
1answer
60 views

How much data warrants building a pipeline?

If I'm doing simple aggregation dashboards, what's the minimum amount of data which justifies building a full blown data processing pipeline? Is it reasonable to build a complicated pipeline with ...
0
votes
3answers
233 views

Need to prepare the data to Link Analysis project?

I've a dataset with following schema: ...
0
votes
1answer
161 views

What are the applications of Solr in Big Data? [closed]

Why is solr used in Big Data? What is it's purpose?
0
votes
1answer
53 views

SAP HANA or Hadoop? [closed]

this is a question regarding a career choice. I am a fresher and I recently joined an MNC in Data Engineering team. There I was offered training in either Hadoop or SAP HANA. I am in doubt as to which ...
-1
votes
1answer
47 views

How do I infer user activities from historical location data and other relevant data? [closed]

Suppose you have the historical location data of users, you also know the locations of shops on the street, can you infer what a user is doing (e.g. taking a tube, being on a business trip, traveling, ...
-1
votes
1answer
44 views

How can I predict the true label for data with incomplete features based on the model learned by data with complete features? [closed]

for example, the model was learned by training data with complete features (f1,f2,f3,f4,f5,f6) but, I wonder the model can test data with incomplete features (f1,f2,f3) to attach the true label to ...
-1
votes
1answer
303 views

How to visualise very large graphs with 250M nodes and 500M+ edges? [closed]

I have graph data which has 250 million nodes and over 500 million edges. My files are roughly 230GB is size, this includes the node and edge files. I am having major difficulties visualising the ...
-1
votes
1answer
75 views

What are the differences between IBM BlueMix and IBM Data Science Experience?

This may seem like a silly question, but as I am going through the documentation for both services it is difficult to disentangle what each does, specifically. From what I've gathered BlueMix is ...
-1
votes
2answers
52 views

what is the proper tool to analyse data and find trends in my case? [closed]

I have a database of hundreds of thousands of loans. I want to find some trends in user behavior, not just some information regarding all loans together but I want to find something similar for ...
-1
votes
1answer
541 views

How to store efficiently very large sparse 3D matrices

To train a CNN, I have stacked arrays of images over observations [observations x width x length]. The dataset is very sparse ($95\%$). What would be an efficient ...
-1
votes
1answer
111 views

Can a machine learning model be trained on Call Detail Record(CDR) Data to predict user's daily locations?

I have a CDR data for two months and my goal is to extract daily or frequent locations(cell towers) of the user along with the departure and arrival time on those locations. The spatial resolution of ...
-1
votes
1answer
28 views

Reading and Processing twitter network data

I have this big data collected from here. So what I would like to asks are How to perform file I/O with the file? From the download like it was mentioned that it used .tsv format, but after ...
-1
votes
1answer
90 views

Connecting Twitter API to a Big Data Environment?

I am trying to get live twitter data and store it in a Hadoop environment. Currently, I am learning how to push the data into a Cloudera environment and process it. My questions are: Is Cloudera ...
-2
votes
1answer
44 views

How to improve when starting out in data science [closed]

A little bit of background info,I recently started a new job as a data analyst where I have to use data to provide insights on the IT operations and devops process of my company. I’m fresh out of ...
-2
votes
2answers
3k views

Master thesis topics [closed]

I am looking for a thesis to complete my master, I am interested in Predictive Analytics in marketing, HR, management or financial subject, using Data Mining Application. I have found a very ...
-2
votes
1answer
119 views

How much data space is used by all scientific articles?

I was wondering if there is any research or study made to calculate the volume of space is used by all scientific articles. It could be in pdf, txt, compressed, or any other format. Is there even a ...
-2
votes
1answer
17 views

What data to collect to create useful insights?

I am creating a Social Media app and I want to know what type of data shall I collect to create useful insights for Business accounts or Content Creators? For example if I want to create an insight ...
-3
votes
1answer
109 views

Are there any programmers who might find programmable heuristics useful? [closed]

I would like to know if my recent work, which is essentially free to implement, has data applications. More obvious applications include: Guessing questions. Logic / objectivity. Psychic prediction. ...

1
6 7 8
9
10