Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [apache-spark]

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.

0
votes
0answers
5 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
1
vote
0answers
10 views

cosine similarity between items (purchase data) and normalisation

I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. ...
0
votes
0answers
3 views

Spark for feature extraction from audio files

I have a python script that uses multiproc to do feature extraction on a list of audio files in a directory, where each process opens a audio(wav) file from a local directory into a numpy array and ...
0
votes
0answers
22 views

Unary Classification Spark MLlib

Checking to see if anyone has insight or implementations within a Spark environment for unary classification. For instance, ML implementations such as scikit-learn have one-class SVM (https://scikit-...
0
votes
1answer
49 views

Accumulators in Spark (PySpark) without global variables?

BACKGROUND Consider the following textbook example which uses accumulators to add vectors. ...
0
votes
1answer
28 views

Why am I getting the same value for the accuracy and recall when using spark's mllib's MulticlassClassificationEvaluator?

So, I am having a play around with some tree based algorithms from Spark's mllib. The code I have is here; ...
0
votes
1answer
39 views

PySpark: java.io.EOFException

System: 1 name node, 4 cores, 16 GB RAM 1 master node, 4 cores, 16 GB RAM 6 data nodes, 4 cores, 16 GB RAM each 6 worker nodes, 4 cores, 16 GB RAM each around 5 Terabytes of storage space The data ...
0
votes
0answers
16 views

CSV to Parquet Conversion

Can we combining CSV format and Parquet format ?If yes then in which scenario we covert the csv format to parquet format ?
0
votes
1answer
129 views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
0
votes
1answer
311 views

Replacing null with average in pyspark

I have a problem during upsampling operation in PySpark. My dataframe is: ...
1
vote
1answer
72 views

Navigating the jungle of choices for scalable ML deployment

I have prototyped a machine learning (ML) model on my local machine and would like to scale it to both train and serve on much larger datasets than could be feasible on a single machine. (The model ...
0
votes
1answer
159 views

Plotting in PySpark?

I have been searching for methods to plot in PySpark. I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are: through method ...
1
vote
0answers
33 views

Deep Learning in Spark Clusters vs on GPUs?

What would be a better way to perform deep learning on any given task, either using Spark Clusters or by taking advantage of GPU's multiple cores. Which would give less training time and be better ...
1
vote
0answers
52 views

Alternative to Apache Spark?

I have been looking for a comprehensive alternative to Apache Spark for Big Data analytics/machine learning and couldn't find one. The ones which I have come across are: Apache Flink Google DataFlow ...
0
votes
0answers
24 views

Why Spark needs specialized DL libraries?

Why are 3rd party libraries needed to implement Deep Learning on Apache Spark? Such as this. Why can't I use Keras/TF/other DL libraries directly on Spark? Regards
0
votes
0answers
15 views

Apache Spark Deep Learning

I understand that Apache Spark has a few DL libraries. But my question is, a GPU also performs distributed computing through its cores. Why/When should I be using Apache Spark when I can use GPUs ...
0
votes
1answer
27 views

Apache Spark Understanding

I have read a lot about Apache Spark. About RDD, DataFrames etc etc. But I haven't come across a simple explanation as to why and where should we be using Apache Spark? If my primary goal is machine/...
0
votes
0answers
18 views

Error while accessing prediction results on Zeppelin

I trained a RandomForestClassifier(rfModel) on Zeppelin and trying to test it using raw data. I am running into error when I try to access the predicted dataframe using show()/print functions. Any ...
1
vote
0answers
16 views

Unable to open application master UI in spark1.6.1 in cluster mode

Hadoop cluster has 8 nodes with high availability of resource manager. Active ResourceManager is in node 3. and Standby ResourceManager in node 2. when i submit the application in cluster mode. the ...
1
vote
0answers
137 views

Frequent itemset vs. frequent sequence mining algorithms on variables with an inherent order

I've been using Pyspark's PrefixSpan algorithm to mine sequences of variables with an inherent order (bus stops). I know that PrefixSpan takes into consideration the order of the items it is fed, ...
2
votes
1answer
69 views

Why does spark.ml.feautures.Word2Vec vectorize sentences instead of single words?

In the process of understanding how Word2Vec in Spark differs from gensim one, I got very confused by the example presented in the Spark docs (reference link: https://spark.apache.org/docs/2.2.0/ml-...
0
votes
1answer
354 views

How to apply StandardScaler and OneHotEncoder simultaneously in Spark Machine learning?

I try to create a machine learning model, linear regression, to predict a price of a diamonds. All examples that I found online do not have a step with scaling of data, using MinMaxScaler or ...
0
votes
0answers
38 views

Streaming Recommender System using Spark

I have trained a recommender model using ALS from spark.mlib. Now I want to consider the incoming ratings to my model. My data is in form of spark's DataFrame. The idea is to use spark streaming, so I ...
1
vote
0answers
57 views

Interpreting rawPrediction from Spark ML LinearSVC

I am using Spark ML's LinearSVC in a binary classification model. The transform method creates two columns, prediction and ...
0
votes
0answers
319 views

Data Cleaning Framework/Library for Scala Spark?

I have crawled data from websites and Data is in Json format having different columns. I selected a column e.g (raw_text) but this column has very irrelevent things,I want to clean it using Apache ...
0
votes
2answers
94 views

Exponential runtime on distance matrix

I have an exploratory script running a Databricks notebook that performs a simple arithmetic function (Pythagorean theorem) on all possible pairwise combinations of a list of pairs of floats (akin to ...
3
votes
1answer
1k views
1
vote
0answers
24 views

Modelling if condition of multiple estimators in a pipeline

How to correctly model a if condition to choose estimator/predictor(linear regression, gbt) to be used in scikit/spark-ml in a single pipeline. ...
1
vote
1answer
30 views

parquet format: advise on log content

I'm using a python script to log IO of a grid job. the log is formatted like this: ...
2
votes
1answer
28 views

PySpark Filter shows only 1 row

I have a 12 GB .csv file and I am trying to filter rows based on some value in a column under "AppName" Doing this in Pyspark: ...
0
votes
0answers
696 views

How to do cosine similarity between two dataFrame efficiently

I have one DataFrame (d1) with (index, features) and a second one (d2) with the same columns....
2
votes
1answer
28 views

does storing file in hdfs parallelize it for Spark?

For Spark's RDD operations, data must be in shape of RDD or be parallelized using: ParallelizedData = sc.parallelize(data) My question is that if I store data in ...
0
votes
0answers
8 views

Is selection Mesos vs Spark formally possible for scaled TF/IDF batches?

Data: lots of small to medium sized text files as corpus for TF/IDF. Is that formally possible to differentiate in terms of effort and scalability whether you would go for Apache Spark, or Mesos or ...
3
votes
2answers
421 views

What are the alternatives to Python + Spark (pyspark)?

I like Python, and I like Spark, but they don't fit very well together. In particular, it is very hard to use python functions in spark (have to create JVM binding for function in python) it is ...
2
votes
1answer
963 views

Run python script using spark-submit on windows 7

I'm trying to submit a python script using spark-submit on windows 7. I'm using spark 2.3.0 built for hadoop 2.7 . The python script is supposed to read data from a csv file and plot graphs. is it ...
0
votes
0answers
64 views

How Can I use Naive Bayes after applying LDA in Spark?

The goal of my project is to classify with the help of LDA topic model. First, I need to apply LDA and then apply a Naive Bayes or any other classification algorithm on LDA topics to classify ...
0
votes
0answers
106 views
3
votes
1answer
238 views

What are the tools to speed up the running time of machine learning algorithms? [closed]

I know Spark as the fastest tools for Data processing, but not sure if it would be useful to speed up the running time of ML algorithms. For example, my ML models to be built needs about 24h when I ...
3
votes
1answer
3k views

PySpark dataframe repartition

What happens when we do repartition on a PySpark dataframe based on the column. For example dataframe.repartition('id') Does this moves the data with the similar ...
0
votes
1answer
82 views

Retrieve user features in real time from UserId for prediction

Let's say I'm building an app like Uber and I want to predict the user's most likely destination based on the user's past history, current latitude/longitude, and time/date. Here is the proposed ...
0
votes
0answers
100 views

Item Similarity with Spark MLLib

I currently have a log file of activity that I am trying to run an item-similarity analysis. Example data: ...
1
vote
1answer
176 views

Best practice for developing using Spark [closed]

I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ ...
0
votes
0answers
45 views

spark.dynamicAllocation + setting the spark parameters according to ambari cluster

we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,, ...
0
votes
1answer
391 views

How to Predict/Forecast street's Traffic based on previous values?

I have a dataset which has the following 5 columns: date, hour, day_of_week, street_id, counts My dataset has information about the number of cars that each ...
0
votes
1answer
134 views

How to Access a Text File from IBM DSX Scala-Spark Notebook [closed]

I have recently faced problem in accessing text files that I have uploaded from my local computer to the IBM DSX Cloud Object Storage so that I can use it in my Spark-Scala notebooks. It appeared to ...
0
votes
2answers
56 views

KMeans for crime patterns

I'm trying to find the patterns in the crime records I have in a database. I thought clustering would be a way to do it. This is my (cooked up) dataset: ...
2
votes
1answer
299 views

Use cases for graph algorithms and graph data structures in finance and banking

I work in a bank and most data is in tabular format in relational databases. I have been reading about graph algorithms (page rank), graph libraries (spark graphx) and graph databases (neo4j). I would ...
1
vote
0answers
80 views

Is spark als item feature comparable between several runs

I am using spark als.train() to build my user-items recommendation system. The problem is I want to cover more item feature. So, I need to input 7 days user action data. But the als train become slow ...
5
votes
1answer
473 views

Saving Large Spark ML Pipeline to HDFS

I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...
0
votes
1answer
45 views

Which one of these tasks will benefit the most from SPARK?

My company processes Data (I am only an intern). We primarily use Hadoop. We're starting to deploy spark in production. Currently we have have two jobs, we will choose just one to begin with spark. ...