Questions tagged [apache-spark]

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.

Filter by
Sorted by
Tagged with
1
vote
0answers
22 views

cannot access hive from spark [closed]

I am trying to install a hadoop + spark + hive cluster. I am using hadoop 3.1.2, spark 2.4.5 (scala 2.11 prebuilt with user-provided hadoop) and hive 2.3.3 (also tried 3.1.2 with the exact same ...
0
votes
1answer
19 views

Spark/Databricks: GPU does not appear to be utilized for ML regression (cross-validation, prediction) notebook

I have created and attached a notebook to a GPU-enabled Databricks cluster (6.4 ML (includes Apache Spark 2.4.5, GPU, Scala 2.11), EC2 type: p2.xlarge). I have started running the notebook that ...
0
votes
0answers
13 views

how to configure and access data on spark

friend this week i am very busy on installation and configuration of spark envirnoment i did the installation and I begin basic operations like creating folder and adding text file to the folder,...
0
votes
1answer
18 views

Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
2
votes
1answer
45 views

BERT in production

I've created a BERT model. What are the ways to do the deployment of this model? Is it possible to use it with Spark, Hadoop or Docker?
0
votes
0answers
26 views

What differences and similarities are between “dataflow engines” and “stream processing systems”?

In book Data Intensive Applications, Ch 10 Batch Processing says Batch processing systems (offline systems) A batch processing system takes a large amount of input data, runs a job to ...
1
vote
1answer
49 views

Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?

I would like to implement a model based on some cleaned and prepared data set. I already have a bit of experience with PySpark, but from a data scientist's perspective it can be cumbersome to work ...
0
votes
0answers
7 views

Explain the relations between SPARK, Oozie, machine learning model

Background: Now I have a machine learning model written by Python. For example, the model has been saved as "model.h5". I have input data every hour. I want to use the model to test/predict these ...
1
vote
1answer
38 views

Scalable way to calculate betweenness centrality for a graph in spark

I have a use-case to calculate betweenness centrality of nodes. I have tried graphx with spark-betweenness but it is a very long running job.Has anyone successfully calculated betweenness centrality ...
0
votes
1answer
44 views

How to apply K-Medoids in PySpark?

the pyspark ml library does not provide any clustering methods for K-Medoids. So my question is, how can one apply K-Medoids in a pyspark context?
2
votes
0answers
19 views

Can i access data on one spark cluster from another?

What if I want to sample a table over hive on Spark-cluster-1, but I'm logged in on Spark-cluster-2? ...
0
votes
0answers
10 views

Can I optimize complex queries mixing Hive and classic RDMS in the same query?

Can I improve the performance of a Spark software using a relational database (RDBMS like PostgreSQL) for small datasets, and Hive only for the really big tables? (secondary/detailed questions) How ...
1
vote
0answers
13 views

How to push tasks to remote Hadoop/Spark cluster

I have access to a Hadoop cluster provided by a third party. Currently, I use Spark on this cluster by logging in to a Zeppelin ...
2
votes
1answer
69 views

Calculating Rank Ordering Error Metric for implicit recommendation

I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula: $...
0
votes
1answer
1k views

How to select multiple columns in a RDD with Spark (pySpark)?

Lets say I have a RDD that has comma delimited data. Each comma delimited value represents the amount of hours slept in the day of a week. So for i.e. [8,7,6,7,8,8,5] How can I manipulate the RDD so ...
0
votes
0answers
32 views

Regression Task - Spark, PyTorch, TensorFlow or scikit

I know it's a broad question, sorry for that, but I'm still testing the waters with machine learning. I have a typical regression task (predict target numbers with the help of features x,y,z) and a ...
0
votes
0answers
9 views

(Py)Spark infrastructure for use in Java/C# website

Lets say I have experience creating ML models in Python, and have decided on training my models on Spark using Pyspark. This will form part of an ML infrastructure for a website with a Java or C# ...
0
votes
0answers
45 views

Running LSTM model on a big data sample using pyspark

I was wondering how does one run an LSTM model on a big dataframe in pyspark. Ideally, one wants to run the model parallelly on different nodes of a spark cluster. But how does one do that?
1
vote
0answers
36 views

Label Propagation for weighted graph

I am working on "community detection" on a big network graph and I've been using the Label Propagation Algorithm from GraphFrames (Spark package). However, my graph is weighted and I was wondering ...
2
votes
2answers
80 views

Random forest vs majority voting

I'm using spark with scala to implement majority voting of decision trees and random forest (both are configured in the same way - same depth, the same amount of base classifiers etc.). Dataset is ...
1
vote
0answers
20 views

Is there a Spark library for computing canonical correlation?

I can't seem to find a library for computing canonical correlation in Spark. Canonical correlation analysis (CCA) is common in other data science-focused systems. This exists in R's base and in ...
0
votes
0answers
30 views

Setting up Zeppelin to work with a Spark Cluster

I have made a spark cluster containing 2 workers and 1 master. I followed the following link to set up the spark clusters. After successfully setting up the spark clusters I wanted to connect it to ...
0
votes
0answers
17 views

Is Logistic Regression with SGD Optimization supported in the latest version of Spark ML?

LogisticRegressionWithSGD is deprecated from Spark Mllib. Is there a way to run Logistic Regression using SGD optimization under the hood, instead of LBFGS?
0
votes
1answer
32 views

Regarding join in pyspark

Given a rdd in pyspark, $A=[(0, 1), (1, 2), (1, 3), (3, 0)]$ I am trying to find self join of A such that the second element of previous pair is equal to first element of next pair i.e. $...
1
vote
1answer
32 views

How to deal with evaluations in a multi step ML pipeline

Apache Spark provides us the pipeline to construct a ML pipeline. Moreover, we can evaluate the regressors in the pipeline, now here is the point I dont understand: how to apply to testing data. Ok, ...
1
vote
0answers
30 views

How to create tensors in spark?

I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
1
vote
0answers
25 views

Is there a way to use a pom.xml file to update spark configuration?

I am trying to update my spark configuration to solve some dependency problems. This pom.xml file seems to be useful for this purpose. I am using a spark docker image. ...
3
votes
0answers
212 views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this dataframe (df) ...
5
votes
1answer
2k views

How to implement LSTM with Spark?

I would like to build an LSTM network for text classification with PySpark, but I don't find any library or function about it. ...
1
vote
2answers
111 views

Record Linkage problem

I am building Matching Alogoritm using ML.Project is to match Internal customer data with external customer data.Features are names,address,city,state and zip. We create pairs between data sets and ...
0
votes
0answers
5 views

PCAModel and PipelineModel, how to get explainedVariance

in pyspark PCAModel contains explainedVariance() method , but once you use Pipeline and specify PCA as a "stage", you will get a PipelineModel as an output and this one does not contain ...
0
votes
0answers
374 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
1
vote
0answers
123 views

Spark creating dat file - header/trailer issue

Although this is a DS community but I saw spark tag so asking query here. 0 I've a specific scenario of creating a file extract (dat/delimited) in ...
0
votes
0answers
97 views

scala spark FP-Growth no results displayed

I have implemented the FP-growth algorithm and it works fine with this sample data: r z h k p z y x w v u t s s x o n r x z y m t s q e z x z y r q t p when I ...
0
votes
0answers
16 views

Hints on my EMR Spark Cluster

I am new to Spark, in particular I am tying to find out which are the best recommended properties to set to work on a 1 master 2 slaves cluster with 10GB memory each. Do you know which are a good set ...
1
vote
0answers
27 views

In Spark, is it a must not to have executors running in Master node when running in cluster mode?

I was running my Spark application on AWS EMR, I launched a master node of m5.xlarge its CPU and RAM usages are both very low. So master node is run for Yarn and main driver to collect calculated data ...
1
vote
0answers
21 views

How do I recommend items to out of training users based on its recent views?

I used Spark's ALS implementation of matrix factorization (Collaborative Filtering for Implicit Feedback) to train user and item embeddings. Since we have a lot of users in system, I had to sample ...
1
vote
0answers
36 views

Spark: how to process tree aggregation and statistic

I have a description of file system in form of a csv: ...
1
vote
0answers
24 views

Probabilistic guarantees for SQL query execution

We are working on a project that tries to accelerate Spark SQL using dedicated hardware. Our approach works much better (in terms of the resulting latency) if we allow query processing that is only ...
2
votes
0answers
54 views

Obtain learning curve of Gradient Boosted Tree model in PySpark

Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a ...
0
votes
2answers
270 views

Apache Spark alternatives for local compute

I am creating a relatively large hobby project in Scala that needs a few ml algorithms for text classification into topics. My dataset is not huge, it is < 500,000 items with dimensionality 5 (2 ...
3
votes
1answer
3k views

Spark DataFrame Limit Function Takes Too Much Time to Show

...
2
votes
0answers
252 views

What is the difference between PySpark's featuresCol, labelCol, predictionCol, and probabilityCol?

I am attempting to train a random forest classifier (pyspark.ml.classification.RandomForestClassifier) on a large dataset (~70gb). However, I am not sure what to ...
1
vote
1answer
237 views

Standard Scaler drops accuracy significantly in Scala Spark

I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly. Without the accuracy is ~90% (on training ...
2
votes
1answer
225 views

Is there any point in learning Hadoop in 2018?

I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing ...
1
vote
0answers
15 views

Is there a built-in pyspark function for the paradigm union(filter, take)? [closed]

Essentially, filtering by key, taking a specific number (based on key) from the DataFrame, and unioning these results for a number of values. For example, ...
2
votes
1answer
125 views

XGBoost most important features appear in multiple trees multiple times

I am fitting xgboost model (scala-spark) to my dataset of transactions. I have about 2 millions of transactions in my training set which is highly unbalanced with a ratio of positive/negative<0.001 ...
1
vote
1answer
54 views

Best Data Science & Machine Learning Conferences 2019? [closed]

What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are ...
1
vote
2answers
80 views

Lightweight execution of Spark MLLib models

I have some training data which I am using to build a Spark MLLib model which is in a Hive database. I am using simple linear regression models and the PySpark API. I have code set up to train this ...
0
votes
1answer
42 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...

1
2 3 4 5