Questions tagged [bigdata]

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

Filter by
Sorted by
Tagged with
1
vote
0answers
12 views

Grouping prefix and term queries

Consider the following context: You have a website that is offering a search function, which produces a list of matching articles. The search has prefix functionality: if you are looking for "...
1
vote
0answers
17 views

Seminal papers about analysis and application based on publicly released data [closed]

The US government releases a lot of varied data here. Data science and data analytics have greatly matured over the last decade varying from Cambridge Analytica on one extreme to Google/Amazon Ads ...
0
votes
1answer
345 views

Running a python script with heavy memory requirements [closed]

I need to run a python script where I am loading several(~15) large modules including keras and load several (approx 20) LSTM models and large amount of data for computation. These script has several ...
6
votes
3answers
443 views

Git for Deep Learning - what are the best tools for versioning/tracking machine learning experiments?

I'm looking for a tool to track the results of several experiments/iterations in machine learning. Training a model can take days/weeks, so it's essential to track its performance and be able to ...
7
votes
2answers
115 views

Computational aspects are typically ignored by statisticians

In the introductory chapter of "Process Mining: Data Science in Action" (2016 - Van der Aalst, pag 11) the author says that : Although data science can be seen as a continuation of statistics, the ...
0
votes
1answer
8k views

How to find 4th point equidistant from 3 other points

For example, if a=100 , b =200, c=300, then what should be the 4th point equidistant to all the 3 points ie a,b,c. Similarly, is there is any solution for nth point equidistant from n-1 points. The ...
2
votes
1answer
65 views

Work experience in Data Science: Couple hours with A-level student [closed]

Our company has accepted an A-level student (18-year old, just before uni) for some work experience. The student will be with me for a couple of hours. I do data analysis and I'm quite busy at the ...
1
vote
0answers
29 views

If an NMT dataset is artificially enlarged by splitting sequences up, should it still train for the same number of epochs?

I have a neural machine translation dataset that looks something like this (letters represent words in an input sentence): ...
2
votes
0answers
55 views

Can Google really bring back billions of results in a blink of an eye (almost) [closed]

I was always fascinated by Google's search ability, a great achievement by Google and other search engine providers also, but more so a collective human talent and ability that makes me appreciate our ...
0
votes
1answer
22 views

Metric to combine average with number of accurance

I was analysing a dataset which has got two main columns which I can name like: category:String amount: Int So I wanted to know the average of amount for each of the category. So I did something ...
1
vote
0answers
16 views

Need help in choosing units. Data processing or big data management [closed]

I'm having dilemma of choosing between the following 2 units for my next semester. As I can only do 1 unit out of these 2, which one is more useful in terms of usefulness and industry application? 1. ...
0
votes
1answer
1k views

Pre-processing irregular, high frequency time-series data in python

...posted originally in StackOverflow (might be better suited here) Small Picture: I am working on pre-processing irregular, high frequency time-series data. In one second, I can have multiple data ...
4
votes
1answer
94 views

How to get Big Data Sets?

It sounds like dump question but as a beginner, I'm really getting confuse. For my academic thesis, I choose a conference paper on Big Data in Healthcare field. Now, problem is to get the Data sets. I ...
0
votes
1answer
1k views

How to compare fast the value with the ones of neighbours in a 3D array

Question: I want to compare every item in a 3D array with its first neighborhoods. It's really slow when I have a 500x500x500 (e.g. with only values of 0, 1, 2) ndarray. I post the principle lines ...
1
vote
2answers
120 views

Which tool should I use for combining this large dataset? [closed]

This is my first foray into data science and I've hit a snag before even getting to the analysis. I have 40 CSV files; each one contains 2 columns - a time and value column. I would like to ...
0
votes
1answer
590 views

Reshape features in rows intro columns python?

My features are arranged in rows, Can anyone tell me please how can I transform them into columns with theirs corresponding times and values?? I mean each features has a set of corresponding times and ...
2
votes
1answer
186 views

How to predict next year's gross revenue given this year's data?

I have the following dataset (.csv format) which contains: 100 columns: $\textbf{Period}$ (in years, e.g. 2017, 2018, ..., 2028), $\textbf{Gross Revenue}$, $\textbf{Region}$ (e.g. APAC, NEMEA, etc), ...
3
votes
0answers
477 views

Communication between name node, data node and client in hadoop by analysing Packet Capturing [closed]

if I say that in hadoop cluster system... A client has 10GB data to send to name node. After analysing the "packet capturing" actually the name node doesn't divide the data for data node, ...
-2
votes
1answer
44 views

How to improve when starting out in data science [closed]

A little bit of background info,I recently started a new job as a data analyst where I have to use data to provide insights on the IT operations and devops process of my company. I’m fresh out of ...
7
votes
1answer
527 views

Can one build linear models on “chunks” of the data set, if one can't build them on the entire data set?

Can one build linear models on "chunks" of the data set, if one can't build them on the entire data set? Particularly, I still have over 88k variables (features) left and one cannot do much with them ...
2
votes
1answer
282 views

Question about K-Fold Cross Validation

In a machine learning procedure, suppose we've chosen k=10 for the "K-Fold Cross Validation". After we do the k steps of "K-Fold Cross Validation", how do we choose the final model for the classifier ?...
1
vote
0answers
269 views

Need lesser memory consumption than doing sklearn.decomposition.TruncatedSVD on 99999 variables

Need lesser memory consumption than doing sklearn.decomposition.TruncatedSVD on 99999 variables. So I know that I can use TruncatedSVD to do the dimensionality reduction. However, my problem is about ...
1
vote
1answer
244 views

What to infer from a given scatter plot?

I am quite new to data science and I have been given the task to find relationship between two entities for a given data.This is the scatter plot graph I made It is a time series data.Here x axis is ...
2
votes
1answer
52 views

does storing file in hdfs parallelize it for Spark?

For Spark's RDD operations, data must be in shape of RDD or be parallelized using: ParallelizedData = sc.parallelize(data) My question is that if I store data in ...
2
votes
0answers
52 views

Split large dataset for predictive modeling using rsparkling -sparklyr

I am trying to build machine learning models (GBM, RF, Staking) on top of a dataset that is about 3G in size on my local computer. However, I only have 4G memory (only 2G are available). My question ...
1
vote
1answer
930 views

product out of stock prediction

Is it possible to predict the products which are going to be out of stock in next coming days/weeks.? If Yes, what could be the possible data points that I should collect? How is this problem ...
2
votes
1answer
1k views

Euclidean vs. cosine similarity

I have a text dataset which I vectorize using a tfidf technique and now in order to make a cluster analysis I am measuring distances between these vector representations. I have found that a common ...
5
votes
2answers
1k views

SGDClassifier fit and partial_fit functions

I wanted to know what is the correct way to train the SGDClassier model on new data observations? Should I use the fit function or the ...
2
votes
2answers
87 views

twitter data analysis? [closed]

I am involved in twitter analysis data. I want to find trending topics in tweets with some hashtags, like #finance or #technology. I have a hugh data set of tweets and now I need to analyze them. I ...
1
vote
1answer
106 views

What is Big Data? [closed]

I want to know what is Big Data? Can I have practical example. How big data can be? I need numbers where big data term is applicable. If you can provide link for case study with actual numbers, with ...
1
vote
0answers
23 views

Recommender system: Give a feature more significance than another

I am trying to build a recommender system that predicts hotel prices based on a great number of features. I have a column representing the hotel rating out of 5 and another column indicating the ...
1
vote
2answers
33 views

Requirements collecting [closed]

How would gather requirements for a data science project? How do you convince stakeholders that whatever you develop is going to benefit them? How to build trust to make use of data science for the ...
1
vote
0answers
21 views

Which are the Best Big Data Services companies in 2018 [closed]

are there any companies with a proven track record to help in big data services or big data consulting services.
1
vote
1answer
128 views

Applying machine learning algorithms to subset of attributes in dataframe

I have this huge mixed data set consisting of both numerical and categorical attributes which upon OneHotEncoding results into a data set with very high dimensionality. Is it wise to apply machine ...
1
vote
0answers
27 views

Cloud computing with country-specific region for Switzerland

Are there any cloud computing services that allow for processing and storing data exclusively in Switzerland? Are there any that have machine-learning-specific functionality? Most do not seem to have ...
1
vote
1answer
63 views

Training a Convnet on 300GB data

I have a large training set of ~300GB (which is a subset of an even larger dataset ~15TB). I am trying train a Convnet with Keras (Tensorflow backend) to do something similar to semantic ...
1
vote
1answer
1k views

Difference between Blocking and Clustering?

What is the difference between blocking and clustering? As far as I know clustering originates from the Machine Learning branch and refers to methods of grouping data points together in a "We and ...
5
votes
1answer
134 views

Dealing with population instability

I am working on using machine learning to correctly predict a binary classification using an input dataset that I receive about once a month. The idea is that I train, test and validate the ...
0
votes
1answer
27 views

Looking for advice: data transfer

I'm requesting data from a government body, and they asked me what format I want to receive the data in. This will be a table of about 400,000 rows and about 10 columns. My options are: "comma or tab ...
42
votes
5answers
89k views

Opening a 20GB file for analysis with pandas

I am currently trying to open a file with pandas and python for machine learning purposes it would be ideal for me to have them all in a DataFrame. Now The file is 18GB large and my RAM is 32 GB but I ...
3
votes
1answer
193 views

Big Data - Data Warehouse Solutions?

I have a dozen of databases that stores different data, and each of them are 100TBs in size. All of the data is stored in AWS services such as RDS, Aurora and Dynamo. Many times I find myself need to ...
1
vote
0answers
98 views

Tasks of a Yarn Process in Hadoop

I am new to big data and Apache Hadoop YARN. This picture show how Yarn runs applications: A client requests a Yarn Application. So, in a certain datanode a Container is created and the process is ...
1
vote
0answers
31 views

Hdfs Data Balance on Cluster

Im new to Big Data, so please just dont blast me. Can you explain me why HDFS works best when the file blocks are evenly spread across the cluster ?
1
vote
0answers
293 views

Hadoop - checksum while reading file from client [closed]

In hadoop , if i want to read a file from client , the class DFSInputStream enters in action. About this, my textbook says The DFSInputStream also verifies checksums for the data transferred to it ...
2
votes
1answer
20 views

Yarn service parameter for pseudodistributed [closed]

I'm trying to set up Hadoop in the pseudodistributed form. So I edited the suggested xml files. In the file yarn-site.xml, I had to insert: ...
3
votes
3answers
629 views

Is it possible to train the neural networks for programming

I want to train my neural networks for the task of programming in a different language. So, I want to know whether is it possible or not to train a neural net for programming?
3
votes
1answer
3k views

How to make R or Python as fast as SAS for ODBC Oracle queries? [closed]

I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS. I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The ...
1
vote
1answer
45 views

Hadoop and input informations divided in splits [closed]

Hadoop divides the input to a MapReduce job into fixed-size pieces called input splits, or just splits. Hadoop creates one map task for each split, which runs the user-defined map function for ...
10
votes
1answer
30k views

Difference between interpolate() and fillna() in pandas

Since interpolate and fillna method does the same work of filling na values. What is the basic difference between the two. What is the significance of having these two different methods?? Can anyone ...
1
vote
0answers
414 views

Spark fitting into Data Science Paradigm for timeseries data

I have seen Dataframe as new API on Spark2 instead of RDD.So I have following few question about the utility of Spark in terms of time series data. Is forecasting still limited to memory available in ...

1 2 3
4
5
9