Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

115 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
0answers
97 views

Fixed-radius range search in non-Euclidean space

I'm trying to find an indexing data structure most suitable for my metric space: set of IP network related data (IP addresses, ports, TCP flags, ...), distance function is continuous, non-Euclidean ...
3
votes
1answer
66 views

anomaly detection in vehicle sensor data

I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors. It seems like there is no proper book or article that goes deeper into ...
3
votes
1answer
356 views

Advice on dealing with very large datasets - HDF5, Python

I've recently started working on an application for visualization of really big datasets. While reading online it became apparent that most people use HDF5 for storing big, multi-dimensional datasets ...
3
votes
2answers
483 views

Where does the "deep learning needs big data" rule come from

When reading about deep learning I often come across the rule that deep learning is only effective when you have large amounts of data at your disposal. These statements are generally accompanied by a ...
3
votes
0answers
504 views

What is behind "A. Grothendieck scheme theory" in Mondobrain?

Mondobrain proposes a "big data" technology with: a new generation of algorithms based on A. Grothendieck scheme theory (Field Medal) that extract knowledge and rules from data without any ...
2
votes
1answer
36 views

Data Art/ Data Visualization Art/ Information Art

Few days ago, I learned about data art/ data visualization art/ information art. I think I have interest in it. I want to see how I can use my data science skills in this area. However, I don't know ...
2
votes
0answers
53 views

Clustering large set of images

I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
2
votes
0answers
44 views

What is the difference between Pachyderm and Git?

I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that: It holds all your data in a central accessible ...
2
votes
1answer
53 views

Size of datasets over years

I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years. I just found the following statistics: The poll of KDnuggets that actually shows that over ...
2
votes
1answer
84 views

Spark: How to run PCA parallelized? Only one thread used

I use pySpark and set my configuration like following: ...
2
votes
1answer
15k views

How to create a new dataframe using the another dataframe

I have created and worked on a DataFrame for a project. It looks like the following: Critics Items Ratings a...........1..........5 b...........2..........3 b...........3..........2 c...........
2
votes
0answers
381 views

Algorithm or JS graph drawing library that can generate a graph of 100,000+ nodes and edges while minimizing edge crossings

I'm trying to plot a directed graph of $2^{16}$ nodes and $2^{16}$ edges (but not simply a cycle). Ultimately, I need to be able to share an interactive graph (zooming, panning, labels). Mathematica ...
2
votes
0answers
769 views

Keras' fit_generator() is not calling my generator

When I call Keras' fit_generator(), passing in a custom generator class I created, I see "Epoch 1/1" in the spew and that's all. It hangs right there, and the ...
2
votes
0answers
30 views

Dimension of the manifold on which my data sits

Suppose that I have data points, in the form of vectors with binary entries. We create a metric space, or Vietoris-Rips complex, using the Hamming distance between the data points. I would like to ...
2
votes
3answers
78 views

How to fill missing enteries in column A, and add respective corresponding enteries to column-B with value of previous cell

I am facing an issue with an excel file. I have an excel sheet with 2 columns Column A : Time Increment with per second Column B : A particular value of a machine sensor The problem i am facing ...
2
votes
0answers
52 views

Split large dataset for predictive modeling using rsparkling -sparklyr

I am trying to build machine learning models (GBM, RF, Staking) on top of a dataset that is about 3G in size on my local computer. However, I only have 4G memory (only 2G are available). My question ...
2
votes
0answers
75 views

Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?

I am using a three node system: Master node: $64$ GB RAM 2 slave node: $32$ GB RAM I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function ...
2
votes
0answers
85 views

What is the scientific term/keyword for "big data time series"?

There are business articles from 2014 and 2015 that the time series analysis of the big data is the next big thing. However Google returns almost no scientific articles under the query "big data time ...
2
votes
0answers
111 views

Interpreting statistical formulas using python (mu + sigma * data)

I am working with data using python. I tried to plot a histogram using mathplotlib. I found a few examples on how to analyze data to get more accurate graphs. So ...
2
votes
0answers
83 views

Core components of data literacy for working professionals

I am a social scientist invited to give a 90-minute seminar on data literacy for working professionals in the U.S. government. I've struggled to develop a concise definition of data literacy, but ...
2
votes
1answer
103 views

Selecting the number of hashes for minhash? Working with extremely sparse data and want more collisions

I'm attempting to use minhash to generate clusters and similarities, and I am primarily using ideas from these resources. http://www2007.org/papers/paper570.pdf https://chrisjmccormick.wordpress.com/...
2
votes
0answers
159 views

Combining parameters for Douglas-Peucker Simplification

NOTE : I'm not sure if this is the right forum for this question. if not, please advice. Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect ...
2
votes
0answers
130 views

Probability tree in SPSS/ R/ others?

Similar questions have been asked in this context but this one seems a bit different. [Aim] We would like to find out what the probability is of a person purchasing a 4USD column D (see below) price ...
2
votes
0answers
234 views

Time Complexity notation in Big Data platforms

I am redesigning some of the classical algorithms for Hadoop/MapReduce framework. I was wondering if there any established approach for denoting Big(O) kind of expressions to measure time complexity? ...
2
votes
0answers
1k views

How append works in hdfs? Where the newly created instance of file is placed?

I'm a Newbie to Hadoop! By web search and after going through hadoop guides, it is clear that hdfs doesn't allow us to edit the file but to append some data to existing file, it will be creating a ...
2
votes
0answers
121 views

scikit-learn OMP mem error

I tried to use OMP algorithm available in scikit-learn. My net datasize which includes both target signal and dictionary ~ 1G. However when I ran the code, it exited with mem-error. The machine has ...
1
vote
0answers
12 views

Repeating values caught with a binary classifier

If my machine is broken, it starts to repeat certain channels. Thing is if there are no out-liars, it is difficult to tell it's broken as we would expect all data points to be around the same value. I ...
1
vote
0answers
60 views

Huge matrix multiplication with PySpark

I am currently struggling with a problem. I have been googling around and cannot find an answer. Also I am kinda new to PySpark and Spark maybe that's why I am struggling. Let's say I have a huge $m \...
1
vote
0answers
11 views

How to get dummy variables from "first name"

I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage. Since the datasets are so big (...
1
vote
0answers
24 views

Word count with map reduce

Suppose we use an input file that contains the following lyrics from a famous song: We’re up all night till the sun We’re up all night to get some The input pairs for the Map phase will be the ...
1
vote
0answers
76 views

Best way to find nearest neighbor distance for large datasets

I am a grad student doing research using generative machine learning with pytorch, and I have generated a set of points. I would like to check how similar these new points are to the points I used in ...
1
vote
3answers
142 views

Efficiently modify a large csv file in Pandas

I have a csv file and would like to do the following modification on it: ...
1
vote
0answers
172 views

XGBoost incremental training for big datasets

I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred). I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
1
vote
0answers
40 views

How do you perform basic statistical analysis on a Binary Dataset?

I have never worked with a Binary dataset (1 and 0) (True or False) so I'm unsure what kind of statistical tests I should run to draw up simple conclusions. I'm doing a data science project and the ...
1
vote
1answer
204 views

Public dataset for news articles with their associated categories for multilabel data classification

I am wondering if there are any public datasets of news, like The New York Times (NYT) or similar to various news categories such as politics, entertainment, lifestyle, general news, sports, etc. I ...
1
vote
0answers
296 views

How do I efficiently load data from disk during training of deep learning models in pytorch?

I'm trying to train a deep learning model without loading the entire dataset into memory. My main question is, what's the best way of doing this? It seems like HDF5 is a common method that people ...
1
vote
0answers
27 views

What is important for Pharmaceutical companies to answer with Big Data Analysis?

I am a data scientist, and I have some biological background (genetics). I have been asked to give a talk for our customers from pharmaceutical industry. I should show them how they benefit from Big ...
1
vote
0answers
379 views

Dask error when reading data from a large zip file

I have a .zip folder with a lot of .csv files which I read into a Dask DataFrame like this: ...
1
vote
0answers
12 views

Basket items optimisation minimising constraints

I have a real problem (not home work) when I have to distribute an ordered list by position to respect some constraints eg. 1. 11 2. 15 3. 18 4. 18 5. 1 baskets:...
1
vote
0answers
31 views

Loading file into and out of HDFS via system call/cmd line vs using libhdfs

I am trying to implement a simple C/C++ program for the HDFS file system like word count, it takes a file from the input path puts it into HDFS (where it gets split), processed my map-reduce function ...
1
vote
0answers
40 views

Scalable test-to-control matching method needed for Amazon cloud environment

I'm working on a project to port some of my employer's processes from our local Unix servers to the Amazon cloud. One process matches records from a test group to records from a control group using ...
1
vote
0answers
22 views

NearestNeighbors testing

I have used nbrs = NearestNeighbors(metric= 'cosine', algorithm='brute').fit(items_features) distances, indices = nbrs.kneighbors(item_features) to find some ...
1
vote
0answers
58 views

How to read binary video format in HDFS using pyspark?

We want to work on detection of pedestrians in a video using ML algorithm.But the video is in the HDFS file system.How can we read the binary video file format from HDFS using spark? Please help me ...
1
vote
0answers
315 views

Big dataset for multi-class classification can't be dasked and split, normal one can't be handled

I have a huge dataframe (550MB), the lending club one available here, and I have to predict the class of the grades. The dask dataframe is : ...
1
vote
0answers
24 views

Test independence based on Kernel Density Estimation

I am working on a problem where I have a dataset of $X$ is dataset with $(X, Y, T, K)$ four attributes, I'd like to test if $P(X, Y, T)P(K) = P(X, Y, T, K)$, that is if $X, Y, T$ is independent of $K$....
1
vote
0answers
132 views

Most efficient algorithm for sequential pattern mining on a dataset with large amount of items in each transaction?

To provide some context, I am trying to do frequent pattern mining on a dataset of system error logs from servers. I organized it into transactions based on the thread ID, which results in some very ...
1
vote
0answers
142 views

How to create tensors in spark?

I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
1
vote
0answers
248 views

Mapreduce jobs not working in hive

I was trying to execute a hive query: select name, count(*) from amazon where review != NULL group by name ; Number of reduce tasks not specified. Estimated ...
1
vote
0answers
51 views

Process and clean a huge .tar file with millions of small XML files

I have one huge .tar file (~700GB) which includes millions of small XML files (<1MB each). The dataset has a lot of unnecessary information that I don't need in ...
1
vote
0answers
27 views

Is there a way to use a pom.xml file to update spark configuration?

I am trying to update my spark configuration to solve some dependency problems. This pom.xml file seems to be useful for this purpose. I am using a spark docker image. ...