Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

1
vote
0answers
4 views

Merging dataframes in Pandas is taking a surprisingly long time

I'm trying to merge a list of time series dataframes (could be over 100) using Pandas. The largest file has a size of $\approx$ 50 MB. The number of rows and columns vary (for instance, one file could ...
0
votes
0answers
6 views

How to efficiently store large amounts of speech to text data

I am currently facing the problem of having to deal with a lot of data (a few hundred GB and growing) for an automated speech recognition (or speech to text) task. Most datasets I am using are coming ...
0
votes
0answers
10 views

Info regarding unis in United States [on hold]

I really need some inputs here. I am an electrical engineer from India and really interested in the maths behind data science and want to come to US for my MS. I have self taught myself the UG ...
1
vote
0answers
15 views

PyMC3: how to efficiently regress on many variables?

I am sorry ahead of time if this seems like a basic question, but I had difficulty finding resources online addressing this. In PyMC3, when building a basic model of a few variables, it is easy to ...
0
votes
0answers
6 views

How to locate some definite machine printed fields on a common format receipt?

We have thousands of receipts with a common format. The format of them looks like this: https://postimg.cc/hhc1K5xy , or this: https://postimg.cc/BtmH4QJb ,especially very similar to this: https://...
0
votes
0answers
6 views

Need suggestion on picking fast aggregation framework or solution

To state the problem, this is what I have in the MariaDB/MySQL. I have the follow table structure, scores ...
0
votes
2answers
35 views

Disadvantages of hyperparameter tuning on a random sample of dataset

I often work with very large datasets where it would be impractical to check all relevant combinations of hyperparameters when constructing a machine learning model. I'm considering randomly sampling ...
1
vote
0answers
105 views

“10-year-challenge” data for age algorithms? [closed]

Both on FB and IG, I see people posting themselves before 10y and now. I have no idea how this challenge started. Could it be a way to collect a colossal amount of data, that could be used to train ...
2
votes
2answers
59 views

How much data to use for feature selection?

Working on my master's thesis, this is a problem I'm unable to find good resources about. I'm working with data of 18 participants, who are either active or passive. Each participant is then ...
0
votes
0answers
7 views

data/AI related graduate school majors good for job hunting

I'm looking for some good graduate school programs/majors related to artificial intelligence/data science that are good regarding job hunting.
0
votes
0answers
15 views

Loading Data into DW: Direct Insert from PreProcessing vs PreProcess and then loading from CSV files

I have a preprocessing script (google cloud function) that generates files (stored in Google Drive). I want to load those files in my DW (Big Query). What are the pros and cons of: 1) Running the ...
0
votes
0answers
22 views

Is there a big data solution to analyse hydroacoustic data?

I'm looking for tools to work on hydroacoustic data, to which I found 3 big challenges : First, data size: recording from hydroacoustic sonars is big - tens or hundreds of TB can be produced in no ...
0
votes
0answers
12 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
1answer
42 views

Will I be able to fetch a Data Analyst job if I learn everything in this image?

India Asking such a question because I am interested learn everything myself and I am confused about what to read. And I am interested by reading books, will that suffice? or should I compulsorily ...
0
votes
0answers
9 views

Which is the best way to generate synthetic data for data science development

I have a data pipeline in place for streaming and batch data, which collects data from various sources and land it into hadoop's hdfs storage. I want to generate synthetic data with the same schema of ...
1
vote
1answer
25 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
0
votes
1answer
22 views

algorithm to auto-download articles from the internet

I have an issued in my homeworks and I thinked if there is an rxisted algorithm or if can i create new one that takes key words like "germany" and "polution" and parses in google scholar. It parses ...
2
votes
0answers
22 views

Dimension of the manifold on which my data sits

Suppose that I have data points, in the form of vectors with binary entries. We create a metric space, or Vietoris-Rips complex, using the Hamming distance between the data points. I would like to ...
-1
votes
0answers
21 views

flatten nested list of tuples

i need some small help here. Test data: ...
-1
votes
0answers
26 views

arXiv API: parsing the full paper and load data in ElasticSearch

I want to make some analysis with ElasticSearch and Kibana using the arXiv API. As far as I've seen the arXiv API allows to extract almost everything about an article, like the date, authors, ...
0
votes
1answer
25 views

Software for automated database processing

I faced a problem which I'd like to solve w/o any programming. And looking for a software to do this. I have a dataset, for example: (brand-id, brand-name, product-class-name;) ...
0
votes
0answers
14 views

How to feed huge datasets into Keras Models at a super fast rate?

I'm trying to find an efficient way to train a NN on a huge dataset. To understand the volume of the problem, I have 100x400mb csv files, which take up 20+GB in memory when loaded in pandas. Now the ...
1
vote
2answers
18 views

Continuous values in a dataset

In a data analysis continuous data needs to be discretised, however in a computer it's impossible to have values $\in \mathbb{R}$, so what does this mean?
0
votes
1answer
46 views

Target feature in training set or not?

If I analyse a random forest in python with scikit I do: ...
-1
votes
0answers
22 views

MondrianEvaluationException on a date dimension with more than 1000 entries

I'm having trouble handling large date ranges in a OLAP report, the error I get is the following: mondrian.olap.fun.MondrianEvaluationException: Aggregation is not supported over to list with ...
1
vote
0answers
19 views

How to fake data based on the condition and weight

I'm trying to fake data for the coffee shop. I've two features age and menu. Menu includes various type of drinks such as coffee [latte, espresso, mocca, etc], tea [milktea, lemontea], milk [freshmilk,...
3
votes
0answers
29 views

Differences between big data, data warehousing, business intelligence and data science?

I know they are four different áreas, but I would like to know what are the main differences between those disciplines, and how are them related to each other, if some of them depends of another, and ...
1
vote
2answers
70 views

huge doubt on anomaly detection

from the naked eye itself, we can tell in the region 5161 the network usage is high so that is the anomaly in my case, then why do we want to apply k-means and other machine learning algorithms to ...
2
votes
2answers
50 views

How to fill missing enteries in column A, and add respective corresponding enteries to column-B with value of previous cell

I am facing an issue with an excel file. I have an excel sheet with 2 columns Column A : Time Increment with per second Column B : A particular value of a machine sensor The problem i am facing ...
1
vote
0answers
14 views

Branching decision tree exit node and visitation duration classfication

I have a dataset where every record in each data cluster corresponds to a step in a branched tree (from an IVR system). I am still learning some of the more machine learning concepts of data science ...
4
votes
2answers
88 views

Pandas vs Linux Datascience [closed]

I joined a data science learning community in my college and we are using linux terminal commands and awk commands to practice gathering some information from big datasets stored in csv files. About ...
1
vote
0answers
17 views

Given Youtube Videos, is there a face recognition technology to detect faces in all new videos?

Given Youtube Videos, is there a face recognition technology to detect faces in all new videos and categorize them in a database for future use? My research showed that especially rate limits were a ...
3
votes
1answer
48 views

Why imbalanced data-set will bias the prediction model towards the more common class?

As we know, an imbalanced data-set has a disadvantage of training a model for deep learning. However, I don't know how to explain it with mathematics?
1
vote
1answer
90 views

Navigating the jungle of choices for scalable ML deployment

I have prototyped a machine learning (ML) model on my local machine and would like to scale it to both train and serve on much larger datasets than could be feasible on a single machine. (The model ...
1
vote
0answers
125 views

Alternative to Apache Spark? [closed]

I have been looking for a comprehensive alternative to Apache Spark for Big Data analytics/machine learning and couldn't find one. The ones which I have come across are: Apache Flink Google DataFlow ...
1
vote
1answer
108 views

How to upload a saved file from google colab to a site such as kaggle or github? [closed]

We are able to load a file from a site such as kaggle and git hub to google collab, we apply the below code to download from other sites into google colab: ...
1
vote
0answers
51 views

Counting similarity between two GIT commits

I am writing as part of my master thesis work an analysis of developers' work. I am looking for a new parameters to be consumed in Topological Data Analysis process. I would like to compare a ...
0
votes
0answers
15 views

Looking for large scale annotated multivariate time-series!

I'm looking for labeled large-scale multivariate time-series, such that for each sample the whole sequence is labeled. I especially look for multi-class datasets, and interested in both cases of big ...
1
vote
0answers
9 views

Grouping prefix and term queries

Consider the following context: You have a website that is offering a search function, which produces a list of matching articles. The search has prefix functionality: if you are looking for "...
0
votes
0answers
13 views

How to calculate time between 2 actions on different tables, also considering some of start of stop actions are missing?

I'm collecting data from a production line and there are sensors at the beginning and the end. Very simple, start sensor and finish sensor send timestamps whenever they detect a product on them. ...
1
vote
0answers
15 views

Seminal papers about analysis and application based on publicly released data [closed]

The US government releases a lot of varied data here. Data science and data analytics have greatly matured over the last decade varying from Cambridge Analytica on one extreme to Google/Amazon Ads ...
0
votes
1answer
53 views

Running a python script with heavy memory requirements [closed]

I need to run a python script where I am loading several(~15) large modules including keras and load several (approx 20) LSTM models and large amount of data for computation. These script has several ...
5
votes
1answer
71 views

Git for Deep Learning - what are the best tools for versioning/tracking machine learning experiments?

I'm looking for a tool to track the results of several experiments/iterations in machine learning. Training a model can take days/weeks, so it's essential to track its performance and be able to ...
0
votes
0answers
57 views

What really does a Data Scientist? [duplicate]

I'm asking that, because after going to a presentation I understood, that Data Scientist is practically someone who only makes calculations and statistics, neither program or very little. Can someone ...
0
votes
0answers
51 views

How to measure halo effect of a campaign?

Let's say that I have two campaigns, that they both advertise the same product. There are two options: Either these campaigns run at the same (option A), or these ...
6
votes
2answers
111 views

Computational aspects are typically ignored by statisticians

In the introductory chapter of "Process Mining: Data Science in Action" (2016 - Van der Aalst, pag 11) the author says that : Although data science can be seen as a continuation of statistics, the ...
0
votes
1answer
100 views

How to find 4th point equidistant from 3 other points

For example, if a=100 , b =200, c=300, then what should be the 4th point equidistant to all the 3 points ie a,b,c. Similarly, is there is any solution for nth point equidistant from n-1 points. The ...
0
votes
0answers
15 views

Preprocessing during training and scoring of LSH

I am using LSH algorithm given in Spark. The dataset I am using has both numeric and categorical columns. Hence before training ...
2
votes
1answer
63 views

Work experience in Data Science: Couple hours with A-level student [closed]

Our company has accepted an A-level student (18-year old, just before uni) for some work experience. The student will be with me for a couple of hours. I do data analysis and I'm quite busy at the ...
1
vote
0answers
18 views

If an NMT dataset is artificially enlarged by splitting sequences up, should it still train for the same number of epochs?

I have a neural machine translation dataset that looks something like this (letters represent words in an input sentence): ...