Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [apache-spark]

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.

0
votes
0answers
12 views

How to store dataframe result into text file?

These process I have done on spark shell. Result is stored in "Result" . Now I want to store that result in text file. ( I am using terminal ) Please provide detail steps as I am new to spark.
1
vote
0answers
18 views

Probabilistic guarantees for SQL query execution

We are working on a project that tries to accelerate Spark SQL using dedicated hardware. Our approach works much better (in terms of the resulting latency) if we allow query processing that is only ...
2
votes
0answers
18 views

Obtain learning curve of Gradient Boosted Tree model in PySpark

Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a ...
0
votes
2answers
23 views

Apache Spark alternatives for local compute

I am creating a relatively large hobby project in Scala that needs a few ml algorithms for text classification into topics. My dataset is not huge, it is < 500,000 items with dimensionality 5 (2 ...
2
votes
0answers
41 views
0
votes
0answers
20 views

What is the difference between PySpark's featuresCol, labelCol, predictionCol, and probabilityCol?

I am attempting to train a random forest classifier (pyspark.ml.classification.RandomForestClassifier) on a large dataset (~70gb). However, I am not sure what to ...
0
votes
0answers
34 views

Is plotting gain of XGBoost trees useful?

I am working on a XGBoost model for fraud detection (2 class classification) using XGBoost v0.7 on Spark. I am looking at different aspects of the model to find important features but also to ...
1
vote
1answer
84 views

Standard Scaler drops accuracy significantly in Scala Spark

I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly. Without the accuracy is ~90% (on training ...
2
votes
1answer
51 views

Is there any point in learning Hadoop in 2018?

I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing ...
1
vote
0answers
11 views

Is there a built-in pyspark function for the paradigm union(filter, take)? [closed]

Essentially, filtering by key, taking a specific number (based on key) from the DataFrame, and unioning these results for a number of values. For example, ...
1
vote
1answer
54 views

XGBoost most important features appear in multiple trees multiple times

I am fitting xgboost model (scala-spark) to my dataset of transactions. I have about 2 millions of transactions in my training set which is highly unbalanced with a ratio of positive/negative<0.001 ...
1
vote
1answer
35 views

Best Data Science & Machine Learning Conferences 2019? [closed]

What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are ...
1
vote
2answers
24 views

Lightweight execution of Spark MLLib models

I have some training data which I am using to build a Spark MLLib model which is in a Hive database. I am using simple linear regression models and the PySpark API. I have code set up to train this ...
0
votes
1answer
20 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
1
vote
0answers
165 views

cosine similarity between items (purchase data) and normalisation

I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. ...
0
votes
1answer
263 views

Accumulators in Spark (PySpark) without global variables?

BACKGROUND Consider the following textbook example which uses accumulators to add vectors. ...
0
votes
1answer
64 views

Why am I getting the same value for the accuracy and recall when using spark's mllib's MulticlassClassificationEvaluator?

So, I am having a play around with some tree based algorithms from Spark's mllib. The code I have is here; ...
0
votes
1answer
432 views

PySpark: java.io.EOFException

System: 1 name node, 4 cores, 16 GB RAM 1 master node, 4 cores, 16 GB RAM 6 data nodes, 4 cores, 16 GB RAM each 6 worker nodes, 4 cores, 16 GB RAM each around 5 Terabytes of storage space The data ...
0
votes
1answer
654 views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
0
votes
1answer
1k views

Replacing null with average in pyspark

I have a problem during upsampling operation in PySpark. My dataframe is: ...
2
votes
2answers
135 views

Navigating the jungle of choices for scalable ML deployment

I have prototyped a machine learning (ML) model on my local machine and would like to scale it to both train and serve on much larger datasets than could be feasible on a single machine. (The model ...
0
votes
1answer
746 views

Plotting in PySpark?

I have been searching for methods to plot in PySpark. I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are: through method ...
1
vote
0answers
44 views

Deep Learning in Spark Clusters vs on GPUs?

What would be a better way to perform deep learning on any given task, either using Spark Clusters or by taking advantage of GPU's multiple cores. Which would give less training time and be better ...
1
vote
0answers
213 views

Alternative to Apache Spark? [closed]

I have been looking for a comprehensive alternative to Apache Spark for Big Data analytics/machine learning and couldn't find one. The ones which I have come across are: Apache Flink Google DataFlow ...
0
votes
2answers
62 views

Apache Spark Understanding

I have read a lot about Apache Spark. About RDD, DataFrames etc etc. But I haven't come across a simple explanation as to why and where should we be using Apache Spark? If my primary goal is machine/...
1
vote
0answers
29 views

Unable to open application master UI in spark1.6.1 in cluster mode

Hadoop cluster has 8 nodes with high availability of resource manager. Active ResourceManager is in node 3. and Standby ResourceManager in node 2. when i submit the application in cluster mode. the ...
1
vote
0answers
267 views

Frequent itemset vs. frequent sequence mining algorithms on variables with an inherent order

I've been using Pyspark's PrefixSpan algorithm to mine sequences of variables with an inherent order (bus stops). I know that PrefixSpan takes into consideration the order of the items it is fed, ...
2
votes
1answer
332 views

Why does spark.ml.feautures.Word2Vec vectorize sentences instead of single words?

In the process of understanding how Word2Vec in Spark differs from gensim one, I got very confused by the example presented in the Spark docs (reference link: https://spark.apache.org/docs/2.2.0/ml-...
0
votes
1answer
989 views

How to apply StandardScaler and OneHotEncoder simultaneously in Spark Machine learning?

I try to create a machine learning model, linear regression, to predict a price of a diamonds. All examples that I found online do not have a step with scaling of data, using MinMaxScaler or ...
1
vote
0answers
135 views

Interpreting rawPrediction from Spark ML LinearSVC

I am using Spark ML's LinearSVC in a binary classification model. The transform method creates two columns, prediction and ...
0
votes
2answers
232 views

Exponential runtime on distance matrix

I have an exploratory script running a Databricks notebook that performs a simple arithmetic function (Pythagorean theorem) on all possible pairwise combinations of a list of pairs of floats (akin to ...
3
votes
1answer
2k views
1
vote
0answers
43 views

Modelling if condition of multiple estimators in a pipeline

How to correctly model a if condition to choose estimator/predictor(linear regression, gbt) to be used in scikit/spark-ml in a single pipeline. ...
1
vote
1answer
42 views

parquet format: advise on log content

I'm using a python script to log IO of a grid job. the log is formatted like this: ...
2
votes
1answer
33 views

PySpark Filter shows only 1 row

I have a 12 GB .csv file and I am trying to filter rows based on some value in a column under "AppName" Doing this in Pyspark: ...
2
votes
1answer
36 views

does storing file in hdfs parallelize it for Spark?

For Spark's RDD operations, data must be in shape of RDD or be parallelized using: ParallelizedData = sc.parallelize(data) My question is that if I store data in ...
0
votes
0answers
9 views

Is selection Mesos vs Spark formally possible for scaled TF/IDF batches?

Data: lots of small to medium sized text files as corpus for TF/IDF. Is that formally possible to differentiate in terms of effort and scalability whether you would go for Apache Spark, or Mesos or ...
4
votes
2answers
752 views

What are the alternatives to Python + Spark (pyspark)?

I like Python, and I like Spark, but they don't fit very well together. In particular, it is very hard to use python functions in spark (have to create JVM binding for function in python) it is ...
2
votes
1answer
2k views

Run python script using spark-submit on windows 7

I'm trying to submit a python script using spark-submit on windows 7. I'm using spark 2.3.0 built for hadoop 2.7 . The python script is supposed to read data from a csv file and plot graphs. is it ...
3
votes
1answer
360 views

What are the tools to speed up the running time of machine learning algorithms? [closed]

I know Spark as the fastest tools for Data processing, but not sure if it would be useful to speed up the running time of ML algorithms. For example, my ML models to be built needs about 24h when I ...
4
votes
1answer
5k views

PySpark dataframe repartition

What happens when we do repartition on a PySpark dataframe based on the column. For example dataframe.repartition('id') Does this moves the data with the similar ...
0
votes
1answer
119 views

Retrieve user features in real time from UserId for prediction

Let's say I'm building an app like Uber and I want to predict the user's most likely destination based on the user's past history, current latitude/longitude, and time/date. Here is the proposed ...
1
vote
0answers
129 views

Item Similarity with Spark MLLib

I currently have a log file of activity that I am trying to run an item-similarity analysis. Example data: ...
2
votes
1answer
234 views

Best practice for developing using Spark [closed]

I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ ...
0
votes
1answer
595 views

How to Predict/Forecast street's Traffic based on previous values?

I have a dataset which has the following 5 columns: date, hour, day_of_week, street_id, counts My dataset has information about the number of cars that each ...
0
votes
1answer
155 views

How to Access a Text File from IBM DSX Scala-Spark Notebook [closed]

I have recently faced problem in accessing text files that I have uploaded from my local computer to the IBM DSX Cloud Object Storage so that I can use it in my Spark-Scala notebooks. It appeared to ...
0
votes
2answers
83 views

KMeans for crime patterns

I'm trying to find the patterns in the crime records I have in a database. I thought clustering would be a way to do it. This is my (cooked up) dataset: ...
2
votes
1answer
428 views

Use cases for graph algorithms and graph data structures in finance and banking

I work in a bank and most data is in tabular format in relational databases. I have been reading about graph algorithms (page rank), graph libraries (spark graphx) and graph databases (neo4j). I would ...
1
vote
0answers
93 views

Is spark als item feature comparable between several runs

I am using spark als.train() to build my user-items recommendation system. The problem is I want to cover more item feature. So, I need to input 7 days user action data. But the als train become slow ...
5
votes
1answer
622 views

Saving Large Spark ML Pipeline to HDFS

I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...