Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
-1
votes
1answer
34 views

Opinion on recommander systems [closed]

I write this message to ask your opinions about some questions : First, some context : I want to build a project of recommander system for a many shop, what I have ? All the history data about sell, ...
-1
votes
1answer
14 views

Searching chronological data

I have daily data of more than 10 years in Excel so I want to select data of individual days from all the years. So, how can I do it? Suggestion either in R or python is highly appreciated because it ...
-1
votes
0answers
10 views

Easiest way to Run computationally intensive task(a)

I have a large dataset stored on my laptop and I will need to run a few scripts on it (O(n^2) computations). Dataset is around 2gb; it's a csv file containing a few million rows of data. This would ...
0
votes
0answers
14 views

HDFS on docker containers - is it possible?

we are using Hadoop cluster based on HDP version - 2.6.5 with ambari platform we want to know if we can use the following HDFS components on docker containers as: journal nodes ZK fail controller ...
-1
votes
0answers
15 views

Course on Data Science in Agriculture

Is there anyone know any on-campus Data Science short course/training (~10 days) mainly focus on Agriculture? As I am new in data science I want to attend a short course. Please suggest me if you know ...
1
vote
0answers
21 views

Advantage of Centralized training in deep learning over distributed training

I know distributed training is the ultimate solution for scalability and solving resource constraints and sometimes distributed training outperforms centralized training in terms of accuracy. But I'm ...
1
vote
1answer
19 views

How to compute modulo of a hash?

Let's say that I have a set of users in my database, that have GUIDs as their IDs. I use xxhash to generate fixed-length hashes for each value, so that I can then ...
2
votes
1answer
40 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
0
votes
0answers
17 views

Fastest frequent itemset library in R or python

I am trying to map a problem with frequent itemset mining. I have only 60 transactions in total but most of them are big. Like the smallest transaction have 30 items and the largest transaction has ...
3
votes
2answers
290 views

How do tech-companies employ Random Forest on large data sets?

The algorithm takes quite a long time to train on large data sets with a moderate number of parameters: https://stats.stackexchange.com/questions/37370/random-forest-computing-time-in-r https://...
0
votes
0answers
15 views

Where is the start point if I want to use artificial intelligence on NASA space industry?

It sounds like dump question but as a beginner, working/learning AI and want to understand what artificial intelligence in the NASA space industry is doing. I want to work on this field but I don't ...
0
votes
0answers
16 views

R language - text.tree()'s pretty parameter

running the below R code to perform analysis on RoomOccupancy dataset ...
1
vote
1answer
102 views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
-1
votes
1answer
25 views

Can a machine learning model be trained on Call Detail Record(CDR) Data to predict user's daily locations?

I have a CDR data for two months and my goal is to extract daily or frequent locations(cell towers) of the user along with the departure and arrival time on those locations. The spatial resolution of ...
1
vote
2answers
61 views

Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0?

Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2....
1
vote
1answer
16 views

NoSQL Comparison - Is this part of my Job?

For the more experienced Data Scientists here, i was asked to perform a case study on how Redis / HBase etc performs, compared to each other.How does data science play a role into this? Note that ...
-1
votes
1answer
23 views

Reading and Processing twitter network data

I have this big data collected from here. So what I would like to asks are How to perform file I/O with the file? From the download like it was mentioned that it used .tsv format, but after ...
2
votes
1answer
35 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
1
vote
0answers
42 views

How to deal with large datasets?

I have some experience with data science but I wanted some insight on how to deal with a very large dataset. I understand simply downloading it to your computer is not plausible so where do you even ...
0
votes
0answers
15 views

Difference between Big Data and a Data Warehouse?

I would like to know what is the exact difference between Big Data and a Data Warehouse. I know that Big Data is the technology to receive and work with really large amounts of data and a data ...
0
votes
0answers
39 views

Running LSTM model on a big data sample using pyspark

I was wondering how does one run an LSTM model on a big dataframe in pyspark. Ideally, one wants to run the model parallelly on different nodes of a spark cluster. But how does one do that?
0
votes
0answers
16 views

Free historical news data base?

I spent some time looking for news databases and APIs. I can find a lot of paid ones. The data google provides when searching for news is perfect for what I am after (something with a deep history, ...
0
votes
0answers
18 views

Relational Model Vs Aggregate-oriented Model

Can you explain to me why the relational model is not good to deal with aggregates? At the contrary, in which cases we can prefer a relational model to an aggregate-oriented model ? Thanks
0
votes
0answers
14 views

Who is the biggest competitor to Google BigQuery for big data processing

I have terabytes of data, and I would like to decide on a cloud solution provider for the big data storage and processing. I am not sure which one I should chose. I am aware of few technologies out ...
0
votes
0answers
11 views

Most reliable source for Literacy data

I'm searching for the most reliable source for granular American literacy population data. I am conducting an analysis, reporting, and data visualization of American literacy. To do this, I am ...
0
votes
2answers
197 views

Question about Similarity vs Dissimilarity Matrix

Right now, I'm working on a coming up with a similarity vs dissimilarity matrix for a set of data points for a clustering algorithm. My question is, if I want to use one of the many clustering ...
0
votes
2answers
33 views

Question About Coming Up With Own Function for Distance Matrix (For Clustering)

Right now, I am currently working on implementing a clustering algorithm with millions data entries with regards to game users for a mobile game. A lot of the features I plan on using are unique to ...
0
votes
2answers
55 views

Data visualisation for big data sets

Working on some voluminous data sets, I have been interested in methods for dimension reduction and plotting. I have stumbled upon this novel technique : UMAP (https://arxiv.org/pdf/1802.03426.pdf), ...
0
votes
0answers
32 views

How to read binary video format in HDFS using pyspark?

We want to work on detection of pedestrians in a video using ML algorithm.But the video is in the HDFS file system.How can we read the binary video file format from HDFS using spark? Please help me ...
1
vote
0answers
96 views

Big dataset for multi-class classification can't be dasked and split, normal one can't be handled

I have a huge dataframe (550MB), the lending club one available here, and I have to predict the class of the grades. The dask dataframe is : ...
1
vote
2answers
51 views

Wanting to overfit model to say if data has been seen before

I would like to train a scalable model that has as an input row of a database and has an output of either 1 or 0 depending upon whether it has seen this entry of the database before or not. The ...
1
vote
1answer
83 views

How to reduce dimensionality of 3.2B categorical features?

Background: This means a dataset of 7,000 samples and 3.2B columns, which I would have to read into distributed Spark memory somehow. Obviously I want to reduce the number of columns that gets fed ...
0
votes
0answers
10 views

How to prepare for machine learning?

I'm working on a project that has weather data and information related to agriculture. Our goal is, in the long term, start identifying patterns and better understand how can we help the users to ...
1
vote
0answers
11 views

Test independence based on Kernel Density Estimation

I am working on a problem where I have a dataset of $X$ is dataset with $(X, Y, T, K)$ four attributes, I'd like to test if $P(X, Y, T)P(K) = P(X, Y, T, K)$, that is if $X, Y, T$ is independent of $K$....
0
votes
1answer
57 views

GPS generate street [closed]

I am working on GPS tracking with a huge data from vehicle. Dataset have: vehicleId, speed, orientation(0-360), coordinate (x,y) and timestamp. Can you recommend me how to clean data and model to ...
0
votes
0answers
19 views

Sampling user IDs from a large data set for retention analysis

Let's say I have a data set with user identities and their respective orders, of the form: ...
1
vote
0answers
58 views

Most efficient algorithm for sequential pattern mining on a dataset with large amount of items in each transaction?

To provide some context, I am trying to do frequent pattern mining on a dataset of system error logs from servers. I organized it into transactions based on the thread ID, which results in some very ...
1
vote
0answers
22 views

How to create tensors in spark?

I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
1
vote
0answers
37 views

Mapreduce jobs not working in hive

I was trying to execute a query select name, count(*) from amazon where review != NULL group by name ; Number of reduce tasks not specified. Estimated ...
0
votes
0answers
10 views

Implementing Data Catalog from scratch

I have a requirement of ingesting huge amount of user specific data and then searching on it. Trying to be low cost solution, I am leaning towards creating Kafka based ingestion pipelines and ...
0
votes
1answer
31 views

Is a new datapoint similar to my reference set?

I have a large reference data set and everyday I get new data points. I would like to know how can I test if those new data points are similar to the reference set? Thanks in advance. Maria
0
votes
1answer
14 views

Data driven project for a self-catering accommodation business

I want to apologise in advance for my ignorance but I'm hitting my head in a wall here as I'm not sure how to proceed with my project and I've got no experience in that field even though I'm doing a ...
1
vote
0answers
32 views

Process and clean a huge .tar file with millions of small XML files

I have one huge .tar file (~700GB) which includes millions of small XML files (<1MB each). The dataset has a lot of unnecessary information that I don't need in ...
0
votes
0answers
11 views

What statistical test would I use to test that both data sources are matching

Suppose I am only working with two datasets, Data Source 1 with columns Customer ID and ...
0
votes
0answers
31 views

How to do numpy matmul broadcasting between two numpy tensors?

I have the Pauli matrices which are (2x2) and complex ...
0
votes
0answers
24 views

How to aggregate large data

I have a quite large data (frame) with 65 physical quantities and each with different time stamps. Some are gathered in intervals of several hours and some in milliseconds. Hence, the data frame ...
0
votes
0answers
9 views

Is veracity in big data theory a good thing to use big data techniques in organization?

I am new to big data field. I know the basic 3V's of it, but I am not understanding the newly added one, which is Veracity. Is it a good thing, which led to taking great analysis from inaccurate data, ...
0
votes
0answers
28 views

Python Machine Learning to analyse test-case lab data and identify new test cases from real-world random data

I am currently working on a project where I have predefined test-cases I need to run/test and then analyse the data. My aim is to be able to identify system failures (against pre-defined criteria). ...
0
votes
0answers
19 views

How to generate artificial datasets in MATLAB based on behavior of real datasets?

To understand: Let's consider a scenario. I have real datasets of 2 buildings showing different behaviors of occupants. Now to expand the analysis for multiple buildings, I want to generate simulated ...
1
vote
0answers
25 views

Is there a way to use a pom.xml file to update spark configuration?

I am trying to update my spark configuration to solve some dependency problems. This pom.xml file seems to be useful for this purpose. I am using a spark docker image. ...

1 2 3 4 5 9