Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [apache-spark]

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.

1
vote
2answers
19 views

Record Linkage problem

I am building Matching Alogoritm using ML.Project is to match Internal customer data with external customer data.Features are names,address,city,state and zip. We create pairs between data sets and ...
0
votes
0answers
3 views

PCAModel and PipelineModel, how to get explainedVariance

in pyspark PCAModel contains explainedVariance() method , but once you use Pipeline and specify PCA as a "stage", you will get a PipelineModel as an output and this one does not contain ...
0
votes
0answers
9 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
1
vote
0answers
15 views

Spark creating dat file - header/trailer issue

Although this is a DS community but I saw spark tag so asking query here. 0 I've a specific scenario of creating a file extract (dat/delimited) in ...
0
votes
0answers
8 views

scala spark FP-Growth no results displayed

I have implemented the FP-growth algorithm and it works fine with this sample data: r z h k p z y x w v u t s s x o n r x z y m t s q e z x z y r q t p when I ...
0
votes
0answers
5 views

Hints on my EMR Spark Cluster

I am new to Spark, in particular I am tying to find out which are the best recommended properties to set to work on a 1 master 2 slaves cluster with 10GB memory each. Do you know which are a good set ...
1
vote
0answers
17 views

In Spark, is it a must not to have executors running in Master node when running in cluster mode?

I was running my Spark application on AWS EMR, I launched a master node of m5.xlarge its CPU and RAM usages are both very low. So master node is run for Yarn and main driver to collect calculated data ...
1
vote
0answers
19 views

How do I recommend items to out of training users based on its recent views?

I used Spark's ALS implementation of matrix factorization (Collaborative Filtering for Implicit Feedback) to train user and item embeddings. Since we have a lot of users in system, I had to sample ...
1
vote
0answers
18 views

Spark: how to process tree aggregation and statistic

I have a description of file system in form of a csv: ...
1
vote
0answers
20 views

Probabilistic guarantees for SQL query execution

We are working on a project that tries to accelerate Spark SQL using dedicated hardware. Our approach works much better (in terms of the resulting latency) if we allow query processing that is only ...
2
votes
0answers
19 views

Obtain learning curve of Gradient Boosted Tree model in PySpark

Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a ...
0
votes
2answers
28 views

Apache Spark alternatives for local compute

I am creating a relatively large hobby project in Scala that needs a few ml algorithms for text classification into topics. My dataset is not huge, it is < 500,000 items with dimensionality 5 (2 ...
2
votes
0answers
139 views
0
votes
0answers
46 views

What is the difference between PySpark's featuresCol, labelCol, predictionCol, and probabilityCol?

I am attempting to train a random forest classifier (pyspark.ml.classification.RandomForestClassifier) on a large dataset (~70gb). However, I am not sure what to ...
0
votes
0answers
47 views

Is plotting gain of XGBoost trees useful?

I am working on a XGBoost model for fraud detection (2 class classification) using XGBoost v0.7 on Spark. I am looking at different aspects of the model to find important features but also to ...
1
vote
1answer
114 views

Standard Scaler drops accuracy significantly in Scala Spark

I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly. Without the accuracy is ~90% (on training ...
2
votes
1answer
68 views

Is there any point in learning Hadoop in 2018?

I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing ...
1
vote
0answers
12 views

Is there a built-in pyspark function for the paradigm union(filter, take)? [closed]

Essentially, filtering by key, taking a specific number (based on key) from the DataFrame, and unioning these results for a number of values. For example, ...
1
vote
1answer
68 views

XGBoost most important features appear in multiple trees multiple times

I am fitting xgboost model (scala-spark) to my dataset of transactions. I have about 2 millions of transactions in my training set which is highly unbalanced with a ratio of positive/negative<0.001 ...
1
vote
1answer
45 views

Best Data Science & Machine Learning Conferences 2019? [closed]

What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are ...
1
vote
2answers
38 views

Lightweight execution of Spark MLLib models

I have some training data which I am using to build a Spark MLLib model which is in a Hive database. I am using simple linear regression models and the PySpark API. I have code set up to train this ...
0
votes
1answer
30 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
2
votes
1answer
233 views

cosine similarity between items (purchase data) and normalisation

I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. ...
0
votes
1answer
372 views

Accumulators in Spark (PySpark) without global variables?

BACKGROUND Consider the following textbook example which uses accumulators to add vectors. ...
0
votes
1answer
120 views

Why am I getting the same value for the accuracy and recall when using spark's mllib's MulticlassClassificationEvaluator?

So, I am having a play around with some tree based algorithms from Spark's mllib. The code I have is here; ...
0
votes
1answer
806 views

PySpark: java.io.EOFException

System: 1 name node, 4 cores, 16 GB RAM 1 master node, 4 cores, 16 GB RAM 6 data nodes, 4 cores, 16 GB RAM each 6 worker nodes, 4 cores, 16 GB RAM each around 5 Terabytes of storage space The data ...
0
votes
1answer
1k views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
0
votes
1answer
2k views

Replacing null with average in pyspark

I have a problem during upsampling operation in PySpark. My dataframe is: ...
2
votes
2answers
168 views

Navigating the jungle of choices for scalable ML deployment

I have prototyped a machine learning (ML) model on my local machine and would like to scale it to both train and serve on much larger datasets than could be feasible on a single machine. (The model ...
0
votes
1answer
2k views

Plotting in PySpark?

I have been searching for methods to plot in PySpark. I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are: through method ...
1
vote
0answers
49 views

Deep Learning in Spark Clusters vs on GPUs?

What would be a better way to perform deep learning on any given task, either using Spark Clusters or by taking advantage of GPU's multiple cores. Which would give less training time and be better ...
2
votes
0answers
298 views

Alternative to Apache Spark? [closed]

I have been looking for a comprehensive alternative to Apache Spark for Big Data analytics/machine learning and couldn't find one. The ones which I have come across are: Apache Flink Google DataFlow ...
0
votes
2answers
69 views

Apache Spark Understanding

I have read a lot about Apache Spark. About RDD, DataFrames etc etc. But I haven't come across a simple explanation as to why and where should we be using Apache Spark? If my primary goal is machine/...
1
vote
0answers
34 views

Unable to open application master UI in spark1.6.1 in cluster mode

Hadoop cluster has 8 nodes with high availability of resource manager. Active ResourceManager is in node 3. and Standby ResourceManager in node 2. when i submit the application in cluster mode. the ...
1
vote
0answers
352 views

Frequent itemset vs. frequent sequence mining algorithms on variables with an inherent order

I've been using Pyspark's PrefixSpan algorithm to mine sequences of variables with an inherent order (bus stops). I know that PrefixSpan takes into consideration the order of the items it is fed, ...
3
votes
1answer
522 views

Why does spark.ml.feautures.Word2Vec vectorize sentences instead of single words?

In the process of understanding how Word2Vec in Spark differs from gensim one, I got very confused by the example presented in the Spark docs (reference link: https://spark.apache.org/docs/2.2.0/ml-...
0
votes
1answer
1k views

How to apply StandardScaler and OneHotEncoder simultaneously in Spark Machine learning?

I try to create a machine learning model, linear regression, to predict a price of a diamonds. All examples that I found online do not have a step with scaling of data, using MinMaxScaler or ...
1
vote
0answers
221 views

Interpreting rawPrediction from Spark ML LinearSVC

I am using Spark ML's LinearSVC in a binary classification model. The transform method creates two columns, prediction and ...
0
votes
2answers
322 views

Exponential runtime on distance matrix

I have an exploratory script running a Databricks notebook that performs a simple arithmetic function (Pythagorean theorem) on all possible pairwise combinations of a list of pairs of floats (akin to ...
3
votes
1answer
3k views
1
vote
0answers
47 views

Modelling if condition of multiple estimators in a pipeline

How to correctly model a if condition to choose estimator/predictor(linear regression, gbt) to be used in scikit/spark-ml in a single pipeline. ...
1
vote
1answer
49 views

parquet format: advise on log content

I'm using a python script to log IO of a grid job. the log is formatted like this: ...
2
votes
1answer
37 views

PySpark Filter shows only 1 row

I have a 12 GB .csv file and I am trying to filter rows based on some value in a column under "AppName" Doing this in Pyspark: ...
2
votes
1answer
39 views

does storing file in hdfs parallelize it for Spark?

For Spark's RDD operations, data must be in shape of RDD or be parallelized using: ParallelizedData = sc.parallelize(data) My question is that if I store data in ...
4
votes
2answers
930 views

What are the alternatives to Python + Spark (pyspark)?

I like Python, and I like Spark, but they don't fit very well together. In particular, it is very hard to use python functions in spark (have to create JVM binding for function in python) it is ...
2
votes
1answer
2k views

Run python script using spark-submit on windows 7

I'm trying to submit a python script using spark-submit on windows 7. I'm using spark 2.3.0 built for hadoop 2.7 . The python script is supposed to read data from a csv file and plot graphs. is it ...
3
votes
1answer
394 views

What are the tools to speed up the running time of machine learning algorithms? [closed]

I know Spark as the fastest tools for Data processing, but not sure if it would be useful to speed up the running time of ML algorithms. For example, my ML models to be built needs about 24h when I ...
6
votes
1answer
6k views

PySpark dataframe repartition

What happens when we do repartition on a PySpark dataframe based on the column. For example dataframe.repartition('id') Does this moves the data with the similar ...
0
votes
1answer
131 views

Retrieve user features in real time from UserId for prediction

Let's say I'm building an app like Uber and I want to predict the user's most likely destination based on the user's past history, current latitude/longitude, and time/date. Here is the proposed ...
1
vote
0answers
163 views

Item Similarity with Spark MLLib

I currently have a log file of activity that I am trying to run an item-similarity analysis. Example data: ...