Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

0
votes
1answer
11 views

What is one good way to handle big data when having a Scala mathematical simulation with data visualization in R?

I have a mathematical simulation written in scala (random numbers, small calculations, lots of going through collections and producing a lot of data). Currently I produce some csv files as output. ...
-1
votes
0answers
23 views

Chain Rule and Gradient [on hold]

I want to compute the gradient using the chain rule. But I do not know and I am hoping someone can explain how can I use the chain rule to compute the gradient?
0
votes
0answers
12 views

flatten nested list of tuples

i need some small help here. Test data: ...
0
votes
0answers
19 views

arXiv API: parsing the full paper and load data in ElasticSearch

I want to make some analysis with ElasticSearch and Kibana using the arXiv API. As far as I've seen the arXiv API allows to extract almost everything about an article, like the date, authors, ...
0
votes
1answer
16 views

Software for automated database processing

I faced a problem which I'd like to solve w/o any programming. And looking for a software to do this. I have a dataset, for example: (brand-id, brand-name, product-class-name;) ...
0
votes
0answers
11 views

How to feed huge datasets into Keras Models at a super fast rate?

I'm trying to find an efficient way to train a NN on a huge dataset. To understand the volume of the problem, I have 100x400mb csv files, which take up 20+GB in memory when loaded in pandas. Now the ...
0
votes
1answer
9 views

Continuous values in a dataset

In a data analysis continuous data needs to be discretized. However in a computer it's impossible to have values $\in \mathbb{R}$, so what this mean ? Thanks
0
votes
1answer
26 views

Target feature in training set or not?

If I analyse a random forest in python with scikit I do: ...
0
votes
0answers
20 views

MondrianEvaluationException on a date dimension with more than 1000 entries

I'm having trouble handling large date ranges in a OLAP report, the error I get is the following: mondrian.olap.fun.MondrianEvaluationException: Aggregation is not supported over to list with ...
1
vote
0answers
18 views

How to fake data based on the condition and weight

I'm trying to fake data for the coffee shop. I've two features age and menu. Menu includes various type of drinks such as coffee [latte, espresso, mocca, etc], tea [milktea, lemontea], milk [freshmilk,...
2
votes
0answers
25 views

Differences between big data, data warehousing, business intelligence and data science?

I know they are four different áreas, but I would like to know what are the main differences between those disciplines, and how are them related to each other, if some of them depends of another, and ...
1
vote
2answers
57 views

huge doubt on anomaly detection

from the naked eye itself, we can tell in the region 5161 the network usage is high so that is the anomaly in my case, then why do we want to apply k-means and other machine learning algorithms to ...
2
votes
2answers
43 views

How to fill missing enteries in column A, and add respective corresponding enteries to column-B with value of previous cell

I am facing an issue with an excel file. I have an excel sheet with 2 columns Column A : Time Increment with per second Column B : A particular value of a machine sensor The problem i am facing ...
1
vote
0answers
13 views

Branching decision tree exit node and visitation duration classfication

I have a dataset where every record in each data cluster corresponds to a step in a branched tree (from an IVR system). I am still learning some of the more machine learning concepts of data science ...
4
votes
2answers
65 views

Pandas vs Linux Datascience [closed]

I joined a data science learning community in my college and we are using linux terminal commands and awk commands to practice gathering some information from big datasets stored in csv files. About ...
1
vote
0answers
13 views

Given Youtube Videos, is there a face recognition technology to detect faces in all new videos?

Given Youtube Videos, is there a face recognition technology to detect faces in all new videos and categorize them in a database for future use? My research showed that especially rate limits were a ...
2
votes
1answer
39 views

Why imbalanced data-set will bias the prediction model towards the more common class?

As we know, an imbalanced data-set has a disadvantage of training a model for deep learning. However, I don't know how to explain it with mathematics?
1
vote
1answer
72 views

Navigating the jungle of choices for scalable ML deployment

I have prototyped a machine learning (ML) model on my local machine and would like to scale it to both train and serve on much larger datasets than could be feasible on a single machine. (The model ...
0
votes
0answers
43 views

Mathematical Representation of a High Dimensional Data Set

I have a data set which is composed of 240 observations/signals and every observation/signal have 101 features/attributes. The data set is given in below Image: This is just a single Matrix, y shows ...
1
vote
0answers
52 views

Alternative to Apache Spark?

I have been looking for a comprehensive alternative to Apache Spark for Big Data analytics/machine learning and couldn't find one. The ones which I have come across are: Apache Flink Google DataFlow ...
1
vote
1answer
81 views

How to upload a saved file from google colab to a site such as kaggle or github? [closed]

We are able to load a file from a site such as kaggle and git hub to google collab, we apply the below code to download from other sites into google colab: ...
1
vote
0answers
42 views

Counting similarity between two GIT commits

I am writing as part of my master thesis work an analysis of developers' work. I am looking for a new parameters to be consumed in Topological Data Analysis process. I would like to compare a ...
0
votes
0answers
14 views

Looking for large scale annotated multivariate time-series!

I'm looking for labeled large-scale multivariate time-series, such that for each sample the whole sequence is labeled. I especially look for multi-class datasets, and interested in both cases of big ...
1
vote
0answers
8 views

Grouping prefix and term queries

Consider the following context: You have a website that is offering a search function, which produces a list of matching articles. The search has prefix functionality: if you are looking for "...
0
votes
0answers
13 views

How to calculate time between 2 actions on different tables, also considering some of start of stop actions are missing?

I'm collecting data from a production line and there are sensors at the beginning and the end. Very simple, start sensor and finish sensor send timestamps whenever they detect a product on them. ...
1
vote
0answers
13 views

Seminal papers about analysis and application based on publicly released data [closed]

The US government releases a lot of varied data here. Data science and data analytics have greatly matured over the last decade varying from Cambridge Analytica on one extreme to Google/Amazon Ads ...
0
votes
1answer
41 views

Running a python script with heavy memory requirements [closed]

I need to run a python script where I am loading several(~15) large modules including keras and load several (approx 20) LSTM models and large amount of data for computation. These script has several ...
3
votes
1answer
49 views

Git for Deep Learning - what are the best tools for versioning/tracking machine learning experiments?

I'm looking for a tool to track the results of several experiments/iterations in machine learning. Training a model can take days/weeks, so it's essential to track its performance and be able to ...
0
votes
0answers
57 views

What really does a Data Scientist? [duplicate]

I'm asking that, because after going to a presentation I understood, that Data Scientist is practically someone who only makes calculations and statistics, neither program or very little. Can someone ...
0
votes
0answers
36 views

How to measure halo effect of a campaign?

Let's say that I have two campaigns, that they both advertise the same product. There are two options: Either these campaigns run at the same (option A), or these ...
6
votes
2answers
111 views

Computational aspects are typically ignored by statisticians

In the introductory chapter of "Process Mining: Data Science in Action" (2016 - Van der Aalst, pag 11) the author says that : Although data science can be seen as a continuation of statistics, the ...
0
votes
1answer
29 views

How to find 4th point equidistant from 3 other points

For example, if a=100 , b =200, c=300, then what should be the 4th point equidistant to all the 3 points ie a,b,c. Similarly, is there is any solution for nth point equidistant from n-1 points. The ...
0
votes
0answers
11 views

Preprocessing during training and scoring of LSH

I am using LSH algorithm given in Spark. The dataset I am using has both numeric and categorical columns. Hence before training ...
2
votes
1answer
62 views

Work experience in Data Science: Couple hours with A-level student [closed]

Our company has accepted an A-level student (18-year old, just before uni) for some work experience. The student will be with me for a couple of hours. I do data analysis and I'm quite busy at the ...
1
vote
0answers
17 views

If an NMT dataset is artificially enlarged by splitting sequences up, should it still train for the same number of epochs?

I have a neural machine translation dataset that looks something like this (letters represent words in an input sentence): ...
2
votes
0answers
44 views

Can Google really bring back billions of results in a blink of an eye (almost) [closed]

I was always fascinated by Google's search ability, a great achievement by Google and other search engine providers also, but more so a collective human talent and ability that makes me appreciate our ...
0
votes
1answer
18 views

Metric to combine average with number of accurance

I was analysing a dataset which has got two main columns which I can name like: category:String amount: Int So I wanted to know the average of amount for each of the category. So I did something ...
1
vote
0answers
13 views

Need help in choosing units. Data processing or big data management [closed]

I'm having dilemma of choosing between the following 2 units for my next semester. As I can only do 1 unit out of these 2, which one is more useful in terms of usefulness and industry application? 1. ...
0
votes
1answer
196 views

Pre-processing irregular, high frequency time-series data in python

...posted originally in StackOverflow (might be better suited here) Small Picture: I am working on pre-processing irregular, high frequency time-series data. In one second, I can have multiple data ...
3
votes
1answer
58 views

How to get Big Data Sets?

It sounds like dump question but as a beginner, I'm really getting confuse. For my academic thesis, I choose a conference paper on Big Data in Healthcare field. Now, problem is to get the Data sets. ...
0
votes
1answer
124 views

How to compare fast the value with the ones of neighbours in a 3D array

Question: I want to compare every item in a 3D array with its first neighborhoods. It's really slow when I have a 500x500x500 (e.g. with only values of 0, 1, 2) ndarray. I post the principle lines ...
0
votes
2answers
48 views

Which tool should I use for combining this large dataset? [closed]

This is my first foray into data science and I've hit a snag before even getting to the analysis. I have 40 CSV files; each one contains 2 columns - a time and value column. I would like to ...
0
votes
0answers
7 views

How to infer racial voting trend along the lines of city types and geographical states?

My country, Malaysia, recently concluded the 14th General Election. The country has 10 states, and each states consist of parliamentary seats that that can categorized into urban, semi-urban and rural ...
0
votes
0answers
20 views

MetaCost Algorithm

I'm studying Machine Learning and Big Data. I'm stuck with the MetaCost operator. Can you explain me in layman terms what the MetaCost algorithm does ?
-1
votes
1answer
106 views

Reshape features in rows intro columns python?

My features are arranged in rows, Can anyone tell me please how can I transform them into columns with theirs corresponding times and values?? I mean each features has a set of corresponding times and ...
1
vote
1answer
34 views

How to predict next year's gross revenue given this year's data?

I have the following dataset (.csv format) which contains: 100 columns: $\textbf{Period}$ (in years, e.g. 2017, 2018, ..., 2028), $\textbf{Gross Revenue}$, $\textbf{Region}$ (e.g. APAC, NEMEA, etc), ...
3
votes
0answers
140 views

Communication between name node, data node and client in hadoop by analysing Packet Capturing

if I say that in hadoop cluster system... A client has 10GB data to send to name node. After analysing the "packet capturing" actually the name node doesn't divide the data for data node, ...
-2
votes
1answer
38 views

How to improve when starting out in data science [closed]

A little bit of background info,I recently started a new job as a data analyst where I have to use data to provide insights on the IT operations and devops process of my company. I’m fresh out of ...
7
votes
1answer
300 views

Can one build linear models on “chunks” of the data set, if one can't build them on the entire data set?

Can one build linear models on "chunks" of the data set, if one can't build them on the entire data set? Particularly, I still have over 88k variables (features) left and one cannot do much with them ...
0
votes
0answers
18 views

Methods for maximizing algorithm “intensity”, when one knows available memory?

So I have a problem that I'm doing analysis on 800 obs and 100000 variables. This data is too large to input all the features on some algos such as http://scikitlearn.org/stable/modules/generated/...