Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

hive 2.1.0 installation on windows windows with derby [migrated]

i am using Hadoop-3.1.0 and trying to install hive- 2.1.0 with derby - 10.12.1.1. the hive command is giving me the following error. I am new to Hadoop and it would be a great help to resolve this. ...
0
votes
0answers
8 views

Loading file into and out of HDFS via system call/cmd line vs using libhdfs

I am trying to implement a simple C/C++ program for the HDFS file system like word count, it takes a file from the input path puts it into HDFS (where it gets split), processed my map-reduce function ...
0
votes
0answers
14 views

What is the structure of the data that can be stored in a Hadoop Distributed File System blocks? [closed]

I have learned various theorical informations about big data and HADOOP. It is known that block in HDFS is the smallest unit of data in file system. However, I do not undrstand how the data are ...
0
votes
0answers
8 views

Storing event data in JSON vs. rows in another table

I want to better understand the constrains and considerations I need to keep in mind when processing site analytics. For example, let's say that I have a table that contains page view data. There is a ...
0
votes
0answers
11 views

Approximate evaluation for deep learning architecture

When train on big dataset for deep learning architecture, like imagenet, it takes long time to judge whether our new neural network architecture is good, say for image classification. Is there a way ...
0
votes
0answers
12 views

Trouble understanding how I could use multivariate time series to predict when an error will occur?

First off, I have very limited knowledge statistics-wise and am more of a coder. I was thrown into a large scale project and could use some guidance. I have a large multivariate time series dataset ...
0
votes
2answers
34 views

Is there an unsupervised learning algorithm that can cluster data based on more than two dimensions?

I am just beginning to get into data science and have never posted here before, apologies if this question is worded incorrectly! I am curious if there is an unsupervised machine learning algorithm ...
1
vote
0answers
33 views

need scalable test-to-control matching method for Amazon cloud environment

I'm working on a project to port some of my employer's processes from our local Unix servers to the Amazon cloud. One process matches records from a test group to records from a control group using ...
1
vote
0answers
14 views

NearestNeighbors testing

I have used nbrs = NearestNeighbors(metric= 'cosine', algorithm='brute').fit(items_features) distances, indices = nbrs.kneighbors(item_features) to find some ...
1
vote
2answers
34 views

Multiprocessing or Parallel Computing Python Code

I am working on bioinformatics big datasets; training set and optimisation taking huge time to execute. I check and found that training and optimisation compiling on one core of cpu and because of ...
3
votes
1answer
123 views

Comparing one small dataset with a big dataset for similar records

I create a varying small dataset (dataset: X) with 500 records in each query. Everytime I need to compare the dataset with a bigger one (dataset: A) (15 milion records) to find similar (or semi-...
0
votes
0answers
11 views

Project structure ML time-series forecasting

I'm about to start with a ML learning time-series forecasting project which includes large-quantities of (2-million) time-series stored in JSON-files. The first step will involve feature engineering....
0
votes
0answers
27 views

Machine Learning or Big Data?

I am a new MSc applied math graduate, with some programming skills in Java and Python. My focus during my master's was differential equations and numerical analysis. Once upon a time, I learned other ...
-1
votes
1answer
14 views

Searching chronological data

I have daily data of more than 10 years in Excel so I want to select data of individual days from all the years. So, how can I do it? Suggestion either in R or python is highly appreciated because it ...
0
votes
0answers
14 views

HDFS on docker containers - is it possible?

we are using Hadoop cluster based on HDP version - 2.6.5 with ambari platform we want to know if we can use the following HDFS components on docker containers as: journal nodes ZK fail controller ...
1
vote
0answers
22 views

Advantage of Centralized training in deep learning over distributed training

I know distributed training is the ultimate solution for scalability and solving resource constraints and sometimes distributed training outperforms centralized training in terms of accuracy. But I'm ...
1
vote
1answer
22 views

How to compute modulo of a hash?

Let's say that I have a set of users in my database, that have GUIDs as their IDs. I use xxhash to generate fixed-length hashes for each value, so that I can then ...
2
votes
1answer
41 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
0
votes
0answers
28 views

Fastest frequent itemset library in R or python

I am trying to map a problem with frequent itemset mining. I have only 60 transactions in total but most of them are big. Like the smallest transaction have 30 items and the largest transaction has ...
3
votes
2answers
292 views

How do tech-companies employ Random Forest on large data sets?

The algorithm takes quite a long time to train on large data sets with a moderate number of parameters: https://stats.stackexchange.com/questions/37370/random-forest-computing-time-in-r https://...
0
votes
0answers
15 views

Where is the start point if I want to use artificial intelligence on NASA space industry?

It sounds like dump question but as a beginner, working/learning AI and want to understand what artificial intelligence in the NASA space industry is doing. I want to work on this field but I don't ...
0
votes
0answers
21 views

R language - text.tree()'s pretty parameter

running the below R code to perform analysis on RoomOccupancy dataset ...
1
vote
1answer
281 views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
-1
votes
1answer
39 views

Can a machine learning model be trained on Call Detail Record(CDR) Data to predict user's daily locations?

I have a CDR data for two months and my goal is to extract daily or frequent locations(cell towers) of the user along with the departure and arrival time on those locations. The spatial resolution of ...
1
vote
2answers
74 views

Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0?

Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2....
1
vote
1answer
16 views

NoSQL Comparison - Is this part of my Job?

For the more experienced Data Scientists here, i was asked to perform a case study on how Redis / HBase etc performs, compared to each other.How does data science play a role into this? Note that ...
-1
votes
1answer
23 views

Reading and Processing twitter network data

I have this big data collected from here. So what I would like to asks are How to perform file I/O with the file? From the download like it was mentioned that it used .tsv format, but after ...
2
votes
1answer
39 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
1
vote
0answers
43 views

How to deal with large datasets?

I have some experience with data science but I wanted some insight on how to deal with a very large dataset. I understand simply downloading it to your computer is not plausible so where do you even ...
0
votes
0answers
15 views

Difference between Big Data and a Data Warehouse?

I would like to know what is the exact difference between Big Data and a Data Warehouse. I know that Big Data is the technology to receive and work with really large amounts of data and a data ...
0
votes
0answers
45 views

Running LSTM model on a big data sample using pyspark

I was wondering how does one run an LSTM model on a big dataframe in pyspark. Ideally, one wants to run the model parallelly on different nodes of a spark cluster. But how does one do that?
0
votes
0answers
17 views

Free historical news data base?

I spent some time looking for news databases and APIs. I can find a lot of paid ones. The data google provides when searching for news is perfect for what I am after (something with a deep history, ...
0
votes
0answers
20 views

Relational Model Vs Aggregate-oriented Model

Can you explain to me why the relational model is not good to deal with aggregates? At the contrary, in which cases we can prefer a relational model to an aggregate-oriented model ? Thanks
0
votes
0answers
15 views

Who is the biggest competitor to Google BigQuery for big data processing

I have terabytes of data, and I would like to decide on a cloud solution provider for the big data storage and processing. I am not sure which one I should chose. I am aware of few technologies out ...
0
votes
0answers
13 views

Most reliable source for Literacy data

I'm searching for the most reliable source for granular American literacy population data. I am conducting an analysis, reporting, and data visualization of American literacy. To do this, I am ...
0
votes
2answers
258 views

Question about Similarity vs Dissimilarity Matrix

Right now, I'm working on a coming up with a similarity vs dissimilarity matrix for a set of data points for a clustering algorithm. My question is, if I want to use one of the many clustering ...
0
votes
2answers
34 views

Question About Coming Up With Own Function for Distance Matrix (For Clustering)

Right now, I am currently working on implementing a clustering algorithm with millions data entries with regards to game users for a mobile game. A lot of the features I plan on using are unique to ...
0
votes
2answers
56 views

Data visualisation for big data sets

Working on some voluminous data sets, I have been interested in methods for dimension reduction and plotting. I have stumbled upon this novel technique : UMAP (https://arxiv.org/pdf/1802.03426.pdf), ...
0
votes
0answers
36 views

How to read binary video format in HDFS using pyspark?

We want to work on detection of pedestrians in a video using ML algorithm.But the video is in the HDFS file system.How can we read the binary video file format from HDFS using spark? Please help me ...
1
vote
0answers
118 views

Big dataset for multi-class classification can't be dasked and split, normal one can't be handled

I have a huge dataframe (550MB), the lending club one available here, and I have to predict the class of the grades. The dask dataframe is : ...
1
vote
2answers
51 views

Wanting to overfit model to say if data has been seen before

I would like to train a scalable model that has as an input row of a database and has an output of either 1 or 0 depending upon whether it has seen this entry of the database before or not. The ...
1
vote
1answer
95 views

How to reduce dimensionality of 3.2B categorical features?

Background: This means a dataset of 7,000 samples and 3.2B columns, which I would have to read into distributed Spark memory somehow. Obviously I want to reduce the number of columns that gets fed ...
0
votes
0answers
10 views

How to prepare for machine learning?

I'm working on a project that has weather data and information related to agriculture. Our goal is, in the long term, start identifying patterns and better understand how can we help the users to ...
1
vote
0answers
11 views

Test independence based on Kernel Density Estimation

I am working on a problem where I have a dataset of $X$ is dataset with $(X, Y, T, K)$ four attributes, I'd like to test if $P(X, Y, T)P(K) = P(X, Y, T, K)$, that is if $X, Y, T$ is independent of $K$....
0
votes
1answer
65 views

GPS generate street [closed]

I am working on GPS tracking with a huge data from vehicle. Dataset have: vehicleId, speed, orientation(0-360), coordinate (x,y) and timestamp. Can you recommend me how to clean data and model to ...
0
votes
0answers
20 views

Sampling user IDs from a large data set for retention analysis

Let's say I have a data set with user identities and their respective orders, of the form: ...
1
vote
0answers
64 views

Most efficient algorithm for sequential pattern mining on a dataset with large amount of items in each transaction?

To provide some context, I am trying to do frequent pattern mining on a dataset of system error logs from servers. I organized it into transactions based on the thread ID, which results in some very ...
1
vote
0answers
29 views

How to create tensors in spark?

I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
1
vote
0answers
42 views

Mapreduce jobs not working in hive

I was trying to execute a query select name, count(*) from amazon where review != NULL group by name ; Number of reduce tasks not specified. Estimated ...
0
votes
0answers
10 views

Implementing Data Catalog from scratch

I have a requirement of ingesting huge amount of user specific data and then searching on it. Trying to be low cost solution, I am leaning towards creating Kafka based ingestion pipelines and ...

1
2 3 4 5
9