Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

0
votes
1answer
14 views

What kind of storage media should I use for large data storage [on hold]

In my hands there is a data of about 60k × 1k. This data is stored in the txt file. Because the data is large, I would like to use the hard disk capacity efficiently for what kind of data file or ...
1
vote
0answers
11 views

Advice on dealing with very large datasets - HDF5, Python

I've recently started working on an application for visualization of really big datasets. While reading online it became apparent that most people use HDF5 for storing big, multi-dimensional datasets ...
1
vote
0answers
8 views

Algorithm or JS graph drawing library that can generate a graph of 100,000+ nodes and edges while minimizing edge crossings

I'm trying to plot a directed graph of $2^{16}$ nodes and $2^{16}$ edges (but not simply a cycle). Ultimately, I need to be able to share an interactive graph (zooming, panning, labels). Mathematica ...
0
votes
0answers
12 views

Data Mining in Practice: Analysing Real Estate

I need to analyse a data set of around 30,000 properties listed or sold over 2 years The main aim is to: Minimize risk and boost profit investments in a short-term market and a long-term strategy. ...
0
votes
0answers
31 views

How to extract errors from log file in scala?

"A log file needs to be processed to extract exceptions/errors.the final output will be structured data having attributes like date,time and exception name.the output will be saved ín a managed hive ...
1
vote
0answers
18 views

Probabilistic guarantees for SQL query execution

We are working on a project that tries to accelerate Spark SQL using dedicated hardware. Our approach works much better (in terms of the resulting latency) if we allow query processing that is only ...
0
votes
1answer
19 views

How to favour a particular class during classification using XGBoost?

I am using a simple XGBoost model to classify 2 classes (0 and 1) in a binary context. In case of the original data, the 0 is the majority class and 1 the minority class. The thing which is happening ...
1
vote
0answers
16 views

Big data online class [closed]

Being an IT engineer, I am exploring the Big Data specialization and wishing to earn a certificate in that field to help me get a new job later. I've been looking around and found these two ...
1
vote
1answer
35 views

Using SMOTE for Synthetic Data generation to improve performance on unbalanced data

I presently have a dataset with 21392 samples, of which, 16948 belong to the majority class (class A) and the remaining 4444 belong to the minority class (class B). I am presently using SMOTE (...
0
votes
1answer
27 views

Neural Network Sensordata as Input

I have a dataset consisting of sensor recordings about human movement. There are 22 classes of different movement like sitting or walking and 19 sensor values. Each recording of a movement has about ...
0
votes
0answers
12 views

For those who use Dask in a single machine, what is your hardware specifications?

I want to use dask for a big dataset, if considering 2 000 000 lines is big. I did try and that is kinda slow for me. What is sure is I do not master the library so it is possible to miss something in ...
3
votes
1answer
22 views

How to find the values that never matched in a full join

Hello I have done this full join query: ...
2
votes
2answers
51 views

Where does the “deep learning needs big data” rule come from

When reading about deep learning I often come across the rule that deep learning is only effective when you have large amounts of data at your disposal. These statements are generally accompanied by a ...
1
vote
0answers
44 views

Simulate Biased dataset Python [closed]

I am trying simulate data from a normal distribution but bias the sample by excluding all negative values and values divisible by 5 . To demonstrate the effect of bias . I will probably calculate ...
2
votes
0answers
45 views
2
votes
0answers
100 views

Keras' fit_generator() is not calling my generator

When I call Keras' fit_generator(), passing in a custom generator class I created, I see "Epoch 1/1" in the spew and that's all. It hangs right there, and the ...
0
votes
1answer
28 views

SAP HANA or Hadoop? [closed]

this is a question regarding a career choice. I am a fresher and I recently joined an MNC in Data Engineering team. There I was offered training in either Hadoop or SAP HANA. I am in doubt as to which ...
0
votes
3answers
47 views

Kmeans large dataset

we are currently performing a K-MEANS under scikit-learn on a data set containing 236027 observations with 6 variables in double format (64 bits). According to our calculations, the complexity of the ...
1
vote
1answer
88 views

Merging dataframes in Pandas is taking a surprisingly long time

I'm trying to merge a list of time series dataframes (could be over 100) using Pandas. The largest file has a size of $\approx$ 50 MB. The number of rows and columns vary (for instance, one file could ...
0
votes
0answers
10 views

How to efficiently store large amounts of speech to text data

I am currently facing the problem of having to deal with a lot of data (a few hundred GB and growing) for an automated speech recognition (or speech to text) task. Most datasets I am using are coming ...
1
vote
0answers
18 views

PyMC3: how to efficiently regress on many variables?

I am sorry ahead of time if this seems like a basic question, but I had difficulty finding resources online addressing this. In PyMC3, when building a basic model of a few variables, it is easy to ...
0
votes
0answers
8 views

How to locate some definite machine printed fields on a common format receipt?

We have thousands of receipts with a common format. The format of them looks like this: https://postimg.cc/hhc1K5xy , or this: https://postimg.cc/BtmH4QJb ,especially very similar to this: https://...
0
votes
0answers
7 views

Need suggestion on picking fast aggregation framework or solution

To state the problem, this is what I have in the MariaDB/MySQL. I have the follow table structure, scores ...
0
votes
2answers
45 views

Disadvantages of hyperparameter tuning on a random sample of dataset

I often work with very large datasets where it would be impractical to check all relevant combinations of hyperparameters when constructing a machine learning model. I'm considering randomly sampling ...
1
vote
0answers
197 views

“10-year-challenge” data for age algorithms? [closed]

Both on FB and IG, I see people posting themselves before 10y and now. I have no idea how this challenge started. Could it be a way to collect a colossal amount of data, that could be used to train ...
2
votes
2answers
69 views

How much data to use for feature selection?

Working on my master's thesis, this is a problem I'm unable to find good resources about. I'm working with data of 18 participants, who are either active or passive. Each participant is then ...
0
votes
0answers
8 views

data/AI related graduate school majors good for job hunting

I'm looking for some good graduate school programs/majors related to artificial intelligence/data science that are good regarding job hunting.
-1
votes
0answers
18 views

Loading Data into DW: Direct Insert from PreProcessing vs PreProcess and then loading from CSV files

I have a preprocessing script (google cloud function) that generates files (stored in Google Drive). I want to load those files in my DW (Big Query). What are the pros and cons of: 1) Running the ...
0
votes
0answers
24 views

Is there a big data solution to analyse hydroacoustic data?

I'm looking for tools to work on hydroacoustic data, to which I found 3 big challenges : First, data size: recording from hydroacoustic sonars is big - tens or hundreds of TB can be produced in no ...
0
votes
1answer
31 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
1answer
49 views

Will I be able to fetch a Data Analyst job if I learn everything in this image?

India Asking such a question because I am interested learn everything myself and I am confused about what to read. And I am interested by reading books, will that suffice? or should I compulsorily ...
-1
votes
0answers
16 views

Which is the best way to generate synthetic data for data science development

I have a data pipeline in place for streaming and batch data, which collects data from various sources and land it into hadoop's hdfs storage. I want to generate synthetic data with the same schema of ...
1
vote
1answer
37 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
0
votes
1answer
23 views

algorithm to auto-download articles from the internet

I have an issued in my homeworks and I thinked if there is an rxisted algorithm or if can i create new one that takes key words like "germany" and "polution" and parses in google scholar. It parses ...
2
votes
0answers
24 views

Dimension of the manifold on which my data sits

Suppose that I have data points, in the form of vectors with binary entries. We create a metric space, or Vietoris-Rips complex, using the Hamming distance between the data points. I would like to ...
0
votes
1answer
26 views

Software for automated database processing

I faced a problem which I'd like to solve w/o any programming. And looking for a software to do this. I have a dataset, for example: (brand-id, brand-name, product-class-name;) ...
1
vote
2answers
20 views

Continuous values in a dataset

In a data analysis continuous data needs to be discretised, however in a computer it's impossible to have values $\in \mathbb{R}$, so what does this mean?
0
votes
1answer
87 views

Target feature in training set or not?

If I analyse a random forest in python with scikit I do: ...
1
vote
0answers
19 views

How to fake data based on the condition and weight

I'm trying to fake data for the coffee shop. I've two features age and menu. Menu includes various type of drinks such as coffee [latte, espresso, mocca, etc], tea [milktea, lemontea], milk [freshmilk,...
5
votes
1answer
50 views

Differences between big data, data warehousing, business intelligence and data science?

I know they are four different áreas, but I would like to know what are the main differences between those disciplines, and how are them related to each other, if some of them depends of another, and ...
1
vote
2answers
76 views

huge doubt on anomaly detection

from the naked eye itself, we can tell in the region 5161 the network usage is high so that is the anomaly in my case, then why do we want to apply k-means and other machine learning algorithms to ...
2
votes
2answers
51 views

How to fill missing enteries in column A, and add respective corresponding enteries to column-B with value of previous cell

I am facing an issue with an excel file. I have an excel sheet with 2 columns Column A : Time Increment with per second Column B : A particular value of a machine sensor The problem i am facing ...
1
vote
0answers
14 views

Branching decision tree exit node and visitation duration classfication

I have a dataset where every record in each data cluster corresponds to a step in a branched tree (from an IVR system). I am still learning some of the more machine learning concepts of data science ...
4
votes
2answers
113 views

Pandas vs Linux Datascience [closed]

I joined a data science learning community in my college and we are using linux terminal commands and awk commands to practice gathering some information from big datasets stored in csv files. About ...
1
vote
0answers
19 views

Given Youtube Videos, is there a face recognition technology to detect faces in all new videos?

Given Youtube Videos, is there a face recognition technology to detect faces in all new videos and categorize them in a database for future use? My research showed that especially rate limits were a ...
3
votes
1answer
62 views

Why imbalanced data-set will bias the prediction model towards the more common class?

As we know, an imbalanced data-set has a disadvantage of training a model for deep learning. However, I don't know how to explain it with mathematics?
2
votes
2answers
138 views

Navigating the jungle of choices for scalable ML deployment

I have prototyped a machine learning (ML) model on my local machine and would like to scale it to both train and serve on much larger datasets than could be feasible on a single machine. (The model ...
1
vote
0answers
215 views

Alternative to Apache Spark? [closed]

I have been looking for a comprehensive alternative to Apache Spark for Big Data analytics/machine learning and couldn't find one. The ones which I have come across are: Apache Flink Google DataFlow ...
1
vote
1answer
147 views

How to upload a saved file from google colab to a site such as kaggle or github? [closed]

We are able to load a file from a site such as kaggle and git hub to google collab, we apply the below code to download from other sites into google colab: ...
1
vote
0answers
54 views

Counting similarity between two GIT commits

I am writing as part of my master thesis work an analysis of developers' work. I am looking for a new parameters to be consumed in Topological Data Analysis process. I would like to compare a ...