Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [apache-spark]

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.

0
votes
0answers
61 views

Standard Scaler drops accuracy significantly in Scala Spark

I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly. Without the accuracy is ~90% (on training ...
0
votes
0answers
9 views

Spark ML Internals

I am new to distributed machine learning. I have been browsing for good articles on distributed ML and internals of Spark ML. I could not get one which clearly explains the internals. Following are ...
1
vote
1answer
33 views

Is there any point in learning Hadoop in 2018?

I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing ...
1
vote
0answers
10 views

Is there a built-in pyspark function for the paradigm union(filter, take)? [closed]

Essentially, filtering by key, taking a specific number (based on key) from the DataFrame, and unioning these results for a number of values. For example, ...
1
vote
1answer
25 views

XGBoost most important features appear in multiple trees multiple times

I am fitting xgboost model (scala-spark) to my dataset of transactions. I have about 2 millions of transactions in my training set which is highly unbalanced with a ratio of positive/negative<0.001 ...
1
vote
1answer
27 views

Best Data Science & Machine Learning Conferences 2019? [closed]

What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are ...
1
vote
2answers
14 views

Lightweight execution of Spark MLLib models

I have some training data which I am using to build a Spark MLLib model which is in a Hive database. I am using simple linear regression models and the PySpark API. I have code set up to train this ...
0
votes
1answer
13 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
1
vote
0answers
83 views

cosine similarity between items (purchase data) and normalisation

I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. ...
-1
votes
0answers
38 views

Unary Classification Spark MLlib

Checking to see if anyone has insight or implementations within a Spark environment for unary classification. For instance, ML implementations such as scikit-learn have one-class SVM (https://scikit-...
0
votes
1answer
128 views

Accumulators in Spark (PySpark) without global variables?

BACKGROUND Consider the following textbook example which uses accumulators to add vectors. ...
0
votes
1answer
42 views

Why am I getting the same value for the accuracy and recall when using spark's mllib's MulticlassClassificationEvaluator?

So, I am having a play around with some tree based algorithms from Spark's mllib. The code I have is here; ...
0
votes
1answer
157 views

PySpark: java.io.EOFException

System: 1 name node, 4 cores, 16 GB RAM 1 master node, 4 cores, 16 GB RAM 6 data nodes, 4 cores, 16 GB RAM each 6 worker nodes, 4 cores, 16 GB RAM each around 5 Terabytes of storage space The data ...
0
votes
1answer
326 views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
0
votes
1answer
872 views

Replacing null with average in pyspark

I have a problem during upsampling operation in PySpark. My dataframe is: ...
1
vote
1answer
90 views

Navigating the jungle of choices for scalable ML deployment

I have prototyped a machine learning (ML) model on my local machine and would like to scale it to both train and serve on much larger datasets than could be feasible on a single machine. (The model ...
0
votes
1answer
308 views

Plotting in PySpark?

I have been searching for methods to plot in PySpark. I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are: through method ...
1
vote
0answers
39 views

Deep Learning in Spark Clusters vs on GPUs?

What would be a better way to perform deep learning on any given task, either using Spark Clusters or by taking advantage of GPU's multiple cores. Which would give less training time and be better ...
1
vote
0answers
125 views

Alternative to Apache Spark? [closed]

I have been looking for a comprehensive alternative to Apache Spark for Big Data analytics/machine learning and couldn't find one. The ones which I have come across are: Apache Flink Google DataFlow ...
0
votes
0answers
25 views

Why Spark needs specialized DL libraries?

Why are 3rd party libraries needed to implement Deep Learning on Apache Spark? Such as this. Why can't I use Keras/TF/other DL libraries directly on Spark? Regards
0
votes
0answers
18 views

Apache Spark Deep Learning

I understand that Apache Spark has a few DL libraries. But my question is, a GPU also performs distributed computing through its cores. Why/When should I be using Apache Spark when I can use GPUs ...
0
votes
2answers
54 views

Apache Spark Understanding

I have read a lot about Apache Spark. About RDD, DataFrames etc etc. But I haven't come across a simple explanation as to why and where should we be using Apache Spark? If my primary goal is machine/...
0
votes
0answers
29 views

Error while accessing prediction results on Zeppelin

I trained a RandomForestClassifier(rfModel) on Zeppelin and trying to test it using raw data. I am running into error when I try to access the predicted dataframe using show()/print functions. Any ...
1
vote
0answers
27 views

Unable to open application master UI in spark1.6.1 in cluster mode

Hadoop cluster has 8 nodes with high availability of resource manager. Active ResourceManager is in node 3. and Standby ResourceManager in node 2. when i submit the application in cluster mode. the ...
1
vote
0answers
191 views

Frequent itemset vs. frequent sequence mining algorithms on variables with an inherent order

I've been using Pyspark's PrefixSpan algorithm to mine sequences of variables with an inherent order (bus stops). I know that PrefixSpan takes into consideration the order of the items it is fed, ...
2
votes
1answer
183 views

Why does spark.ml.feautures.Word2Vec vectorize sentences instead of single words?

In the process of understanding how Word2Vec in Spark differs from gensim one, I got very confused by the example presented in the Spark docs (reference link: https://spark.apache.org/docs/2.2.0/ml-...
0
votes
1answer
703 views

How to apply StandardScaler and OneHotEncoder simultaneously in Spark Machine learning?

I try to create a machine learning model, linear regression, to predict a price of a diamonds. All examples that I found online do not have a step with scaling of data, using MinMaxScaler or ...
0
votes
0answers
44 views

Streaming Recommender System using Spark

I have trained a recommender model using ALS from spark.mlib. Now I want to consider the incoming ratings to my model. My data is in form of spark's DataFrame. The idea is to use spark streaming, so I ...
1
vote
0answers
93 views

Interpreting rawPrediction from Spark ML LinearSVC

I am using Spark ML's LinearSVC in a binary classification model. The transform method creates two columns, prediction and ...
0
votes
0answers
401 views

Data Cleaning Framework/Library for Scala Spark?

I have crawled data from websites and Data is in Json format having different columns. I selected a column e.g (raw_text) but this column has very irrelevent things,I want to clean it using Apache ...
0
votes
2answers
170 views

Exponential runtime on distance matrix

I have an exploratory script running a Databricks notebook that performs a simple arithmetic function (Pythagorean theorem) on all possible pairwise combinations of a list of pairs of floats (akin to ...
3
votes
1answer
2k views
1
vote
0answers
30 views

Modelling if condition of multiple estimators in a pipeline

How to correctly model a if condition to choose estimator/predictor(linear regression, gbt) to be used in scikit/spark-ml in a single pipeline. ...
1
vote
1answer
39 views

parquet format: advise on log content

I'm using a python script to log IO of a grid job. the log is formatted like this: ...
2
votes
1answer
31 views

PySpark Filter shows only 1 row

I have a 12 GB .csv file and I am trying to filter rows based on some value in a column under "AppName" Doing this in Pyspark: ...
0
votes
0answers
883 views

How to do cosine similarity between two dataFrame efficiently

I have one DataFrame (d1) with (index, features) and a second one (d2) with the same columns....
2
votes
1answer
31 views

does storing file in hdfs parallelize it for Spark?

For Spark's RDD operations, data must be in shape of RDD or be parallelized using: ParallelizedData = sc.parallelize(data) My question is that if I store data in ...
0
votes
0answers
8 views

Is selection Mesos vs Spark formally possible for scaled TF/IDF batches?

Data: lots of small to medium sized text files as corpus for TF/IDF. Is that formally possible to differentiate in terms of effort and scalability whether you would go for Apache Spark, or Mesos or ...
4
votes
2answers
594 views

What are the alternatives to Python + Spark (pyspark)?

I like Python, and I like Spark, but they don't fit very well together. In particular, it is very hard to use python functions in spark (have to create JVM binding for function in python) it is ...
2
votes
1answer
1k views

Run python script using spark-submit on windows 7

I'm trying to submit a python script using spark-submit on windows 7. I'm using spark 2.3.0 built for hadoop 2.7 . The python script is supposed to read data from a csv file and plot graphs. is it ...
0
votes
0answers
79 views

How Can I use Naive Bayes after applying LDA in Spark?

The goal of my project is to classify with the help of LDA topic model. First, I need to apply LDA and then apply a Naive Bayes or any other classification algorithm on LDA topics to classify ...
0
votes
0answers
129 views

Binomial regression and Logistic regression returns different models on Spark

I have 2 models compared on Spark: ...
3
votes
1answer
296 views

What are the tools to speed up the running time of machine learning algorithms? [closed]

I know Spark as the fastest tools for Data processing, but not sure if it would be useful to speed up the running time of ML algorithms. For example, my ML models to be built needs about 24h when I ...
3
votes
1answer
4k views

PySpark dataframe repartition

What happens when we do repartition on a PySpark dataframe based on the column. For example dataframe.repartition('id') Does this moves the data with the similar ...
0
votes
1answer
109 views

Retrieve user features in real time from UserId for prediction

Let's say I'm building an app like Uber and I want to predict the user's most likely destination based on the user's past history, current latitude/longitude, and time/date. Here is the proposed ...
0
votes
0answers
117 views

Item Similarity with Spark MLLib

I currently have a log file of activity that I am trying to run an item-similarity analysis. Example data: ...
2
votes
1answer
202 views

Best practice for developing using Spark [closed]

I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ ...
0
votes
0answers
50 views

spark.dynamicAllocation + setting the spark parameters according to ambari cluster

we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,, ...
0
votes
1answer
522 views

How to Predict/Forecast street's Traffic based on previous values?

I have a dataset which has the following 5 columns: date, hour, day_of_week, street_id, counts My dataset has information about the number of cars that each ...
0
votes
1answer
144 views

How to Access a Text File from IBM DSX Scala-Spark Notebook [closed]

I have recently faced problem in accessing text files that I have uploaded from my local computer to the IBM DSX Cloud Object Storage so that I can use it in my Spark-Scala notebooks. It appeared to ...