Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
3
votes
1answer
334 views

Advice on dealing with very large datasets - HDF5, Python

I've recently started working on an application for visualization of really big datasets. While reading online it became apparent that most people use HDF5 for storing big, multi-dimensional datasets ...
0
votes
0answers
253 views

Does Call Detail Record(CDR) data record different cell-site/tower on incoming and outgoing call/text?

I am working on CDR data to find important locations of user. The data I have does not have coordinates so I will be using cell-site location as user's location. But after doing some working on data I ...
3
votes
1answer
44 views

Merging 2 datawarehouses

Suppose I have 2 datawarehouses of two different companies that are being merged. Some of the tables are common but includes different records, while others are different. If I use big data ...
0
votes
3answers
208 views

What is Data Lake?

I am familiar with the concept of "Big Data" but how does "Data Lake" differ from Big Data? Is it derived from Big Data? Please explain.
-2
votes
0answers
8 views

What is more useful for a Machine Learning/ Data science engineer, apache hadoop or apache hive? [closed]

Hi I am learning machine learning on my own and part of the job of machine learning engineer is going to be dealing with huge amount of data. I know SQL but it has it's limitations. So decided to ...
1
vote
1answer
38 views

Size of datasets over years

I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years. I just found the following statistics: The poll of KDnuggets that actually shows that over ...
0
votes
1answer
36 views

ML modeling a data with big amount of rows

I want to do ML modeling such XGboost, KNN, and similar models on data with 9 numerical features and more than 25 million rows and the size of data is almost 2.5 Gig and I prefer to use all the data ...
3
votes
1answer
57 views

anomaly detection in vehicle sensor data

I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors. It seems like there is no proper book or article that goes deeper into ...
0
votes
0answers
20 views

What is the learning path in Big Data for a self-taught person?

I am currently learning the Python language and I am quite interested in developing myself in the Big Data field as a self-taught. Currently I do not have the possibility of going to university so I ...
0
votes
1answer
30 views

Subsampling the “right” amout of data to train an ML model

I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process ...
2
votes
1answer
15k views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
2
votes
3answers
72 views

How to fill missing enteries in column A, and add respective corresponding enteries to column-B with value of previous cell

I am facing an issue with an excel file. I have an excel sheet with 2 columns Column A : Time Increment with per second Column B : A particular value of a machine sensor The problem i am facing ...
0
votes
1answer
43 views

How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?

I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in ...
0
votes
1answer
21 views

How to choose feedforward architecture for few number of features but very large instance?

Assume I have 1 million of data instance and each instance contains 100 feature. For each instance, I also have a lable. The ...
1
vote
1answer
98 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
0
votes
0answers
26 views

Examples of Open source huge business intelligence datasets

I've observed that many of the datasets available for traditional ML and data science algorithms seem to be in the order of MB. I assumed these may be because earlier computers were not that ...
-1
votes
1answer
263 views

How to store efficiently very large sparse 3D matrices

To train a CNN, I have stacked arrays of images over observations [observations x width x length]. The dataset is very sparse ($95\%$). What would be an efficient ...
0
votes
2answers
151 views

huge doubt on anomaly detection

from the naked eye itself, we can tell in the region 5161 the network usage is high so that is the anomaly in my case, then why do we want to apply k-means and other machine learning algorithms to ...
2
votes
0answers
19 views

XGBoost incremental training for big datasets

I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred). I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
2
votes
1answer
56 views

How to fake data based on the condition and weight

I'm trying to fake data for the coffee shop. I've two features age and menu. Menu includes various type of drinks such as coffee [latte, espresso, mocca, etc], tea [milktea, lemontea], milk [freshmilk,...
1
vote
1answer
21 views

How do data types influence hardware (CPU / GPU / TPU) performance?

I am currently dealing with a relatively big data set, for which I have some memory usage concerns. I am dealing with most of the different data types : floats, integers, Booleans, characters strings ...
0
votes
0answers
11 views

What are the execution service and the event service in Netflix big data

From the following article by the netflix engineering team: The core architecture of the big data platform at Netflix involves three key services. These are the execution service (Genie), the ...
2
votes
1answer
78 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
-1
votes
1answer
75 views

What are the differences between IBM BlueMix and IBM Data Science Experience?

This may seem like a silly question, but as I am going through the documentation for both services it is difficult to disentangle what each does, specifically. From what I've gathered BlueMix is ...
3
votes
6answers
8k views

How much of a background in programming is necessary to become a data scientist? [closed]

I once knew some Java, but that was close to 10 years ago. Assuming I can learn a language to get into data analytics.... what language do you recommend?
0
votes
1answer
46 views

How to run Spark python code in Jupyter Notebook via command prompt

I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below After that I want to run this in CMD ...
0
votes
0answers
10 views

Scalable Infrence Server for Object Detection

I have created a Django service (nginx + Gunicorn) for object detection models. For my case i have 50+ models with resnet 50 based back bone. Server Machine Specification: 16 CPU 64 GB Ram I have ...
1
vote
1answer
128 views

Public dataset for news articles with their associated categories for multilabel data classification

I am wondering if there are any public datasets of news, like New York Times (NYT) or similar to various news categories such as politics, entertainment, lifestyle, general news, sports etc. I want to ...
1
vote
0answers
124 views

Most efficient algorithm for sequential pattern mining on a dataset with large amount of items in each transaction?

To provide some context, I am trying to do frequent pattern mining on a dataset of system error logs from servers. I organized it into transactions based on the thread ID, which results in some very ...
0
votes
0answers
21 views

What should be the architecture design for the dataset to be used for a machine learning API

I have aroud 30GB of market dataset in csv format which is downloaded onto my server computer everyday. Now I want to use this dataset to do some computations and provide some analysis. The user will ...
0
votes
2answers
66 views

Is there an unsupervised learning algorithm that can cluster data based on more than two dimensions?

I am just beginning to get into data science and have never posted here before, apologies if this question is worded incorrectly! I am curious if there is an unsupervised machine learning algorithm ...
0
votes
1answer
136 views

Spliting training data into multiple variables using R

So right now I am trying to create multiple variables with training data, and in the process I have reached an error Error in eval(predvars, data, env) : object '1.band1' not found which is a ...
1
vote
0answers
28 views

How do you perform basic statistical analysis on a Binary Dataset?

I have never worked with a Binary dataset (1 and 0) (True or False) so I'm unsure what kind of statistical tests I should run to draw up simple conclusions. I'm doing a data science project and the ...
0
votes
1answer
474 views

Neural Network Sensordata as Input

I have a dataset consisting of sensor recordings about human movement. There are 22 classes of different movement like sitting or walking and 19 sensor values. Each recording of a movement has about ...
0
votes
0answers
37 views

Extracting possible relationships and insights from a large dataset

I have a dataset with about 26'000 variables and 33'000 continuous observations. My task is to extract any interesting insights/possible relationships between various variables. I am looking for any ...
6
votes
1answer
103 views

What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)?

I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are: AWS instance with massive RAM and train CPU (slow, but ...
0
votes
0answers
22 views

Working with mySQL and pytorch Dataset

I am working with high frequency time indexed data. We have 2 types of data each with about 5 columns. For each type of data we have 2500 streams coming in and being updated every 1ms-100ms (the ...
-1
votes
1answer
30 views

How can I predict the true label for data with incomplete features based on the model learned by data with complete features? [closed]

for example, the model was learned by training data with complete features (f1,f2,f3,f4,f5,f6) but, I wonder the model can test data with incomplete features (f1,f2,f3) to attach the true label to ...
2
votes
0answers
30 views

What is the difference between Pachyderm and Git?

I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that: It holds all your data in a central accessible ...
0
votes
0answers
9 views

How to Combine Oracle and Snowflake/Redshift data on the fly

Have a Schema.Table_Name on Oracle. Exactly similar table is on Snowflake with same metadata(Schema.Table_Name). How can I combine data from two on the fly without moving Oracle data to Redshift. Most ...
0
votes
1answer
375 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
1
vote
1answer
19 views

Where can I find a dataset that contains criminal case sentencing data? [closed]

I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as: Type of crime Defendant Age/Sex/Race Plea Verdict Sentence Is there a dataset ...
0
votes
1answer
52 views

SAP HANA or Hadoop? [closed]

this is a question regarding a career choice. I am a fresher and I recently joined an MNC in Data Engineering team. There I was offered training in either Hadoop or SAP HANA. I am in doubt as to which ...
3
votes
2answers
420 views

Where does the “deep learning needs big data” rule come from

When reading about deep learning I often come across the rule that deep learning is only effective when you have large amounts of data at your disposal. These statements are generally accompanied by a ...
1
vote
2answers
2k views

How to build a symptom checker and medical diagnose chat bot

I want to build a chatbot which can diagnose an illness depending upon the symptoms which are given to it. I want to show you an example scenario of how it should work; Application - A , User - U ...
0
votes
2answers
564 views

Question about Similarity vs Dissimilarity Matrix

Right now, I'm working on a coming up with a similarity vs dissimilarity matrix for a set of data points for a clustering algorithm. My question is, if I want to use one of the many clustering ...
0
votes
1answer
51 views

Software for automated database processing

I faced a problem which I'd like to solve w/o any programming. And looking for a software to do this. I have a dataset, for example: (brand-id, brand-name, product-class-name;) ...
3
votes
2answers
5k views

Training Keras model with multiple CSV files

I'm currently trying to train a Keras model on several large CSV files. I can fit one in memory, but not all combined. From my point of view, there are several ways to deal with this problem. I could ...
0
votes
0answers
31 views

In Spark, what's the role of each of Cluster manager and Spark Context?

What's the role of a cluster manager and the role spark context (Driver program) in terms of managing worker nodes. How do they communicate with each other (and with the worker nodes) and when they do ...
1
vote
1answer
154 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...

1
2 3 4 5
9