Questions tagged [apache-spark]

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.

Filter by
Sorted by
Tagged with
2 votes
1 answer
253 views

Spark: How to run PCA parallelized? Only one thread used

I use pySpark and set my configuration like following: ...
2 votes
1 answer
129 views

How do I recommend items to out of training users based on its recent views?

I used Spark's ALS implementation of matrix factorization (Collaborative Filtering for Implicit Feedback) to train user and item embeddings. Since we have a lot of users in system, I had to sample ...
0 votes
1 answer
263 views

Storage of N-dimensional matrices (tensors) as part of machine learning pipelines

I'm an infra person working on a storage product. I've been googling quite a bit to find an answer to the following question but unable to do so. Hence, I am attemping to ask the question here. I am ...
2 votes
1 answer
503 views

Spark ALS-WR giving the same recommended items for all users

We are trying to build a recommendation system for a supermarket with diverse item types (ranging from fast-moving grocery to low-moving electronic items). Some items are purchased more frequently in ...
1 vote
2 answers
140 views

Is spark als item feature comparable between several runs

I am using spark als.train() to build my user-items recommendation system. The problem is I want to cover more item feature. So, I need to input 7 days user action data. But the als train become slow ...
2 votes
1 answer
1k views

Scalable way to calculate betweenness centrality for a graph in spark

I have a use-case to calculate betweenness centrality of nodes. I have tried graphx with spark-betweenness but it is a very long running job. Has anyone successfully calculated betweenness centrality ...
0 votes
1 answer
875 views

How to Predict/Forecast street's Traffic based on previous values?

I have a dataset which has the following 5 columns: date, hour, day_of_week, street_id, counts My dataset has information about the number of cars that each ...
1 vote
1 answer
274 views

Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
2 votes
1 answer
2k views

How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?

How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation? I can not seem to find it anywhere, even not from the author of LDA, ...
0 votes
1 answer
266 views

Group a spark dataframe by a starting event to an ending event

Given a series of events (with datetime) such as: failed, failed, passed, failed, passed, passed I want to retrieve the time from when it first "failed" to when it first "passed," ...
0 votes
1 answer
91 views

CREATE TABLE USING Oracle DATA_SOURCE

I am trying to create a table using ORACLE as a data source using spark query but getting an error. %sql CREATE TABLE TEST USING org.apache.spark.sql.jdbc OPTIONS ( url "jdbc:oracle:thin:@...
0 votes
1 answer
768 views

How to apply K-Medoids in PySpark?

the pyspark ml library does not provide any clustering methods for K-Medoids. So my question is, how can one apply K-Medoids in a pyspark context?
3 votes
1 answer
2k views

Why is Spark's LinearRegressionWithSGD very slow locally?

I have been trying to run linear regression with SGD that is found in Spark mllib for some time and I am experiencing huge performance problems. All examples that I was looking have number of ...
0 votes
1 answer
435 views

How to run Spark python code in Jupyter Notebook via command prompt

I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below After that I want to run this in CMD ...
0 votes
1 answer
23 views

Any Interface/Library that can take the Python ML code and run on spark cluster without learning PySpark?

I have been working with Python for machine learning and have a fair amount of code written in Python using libraries such as scikit-learn, pandas, and numpy. Recently, I’ve been faced with larger ...
0 votes
0 answers
21 views

Databricks Extracting features out of Big Files like Step and PDF

So i dont know if this is the right community to ask this question, but lets see. So we are using databricks in our company and we extract Features out of big binary files like PDF and STEP. We trying ...
15 votes
3 answers
107k views

How to calculate the mean of a dataframe column and find the top 10%

I am very new to Scala and Spark, and am working on some self-made exercises using baseball statistics. I am using a case class create a RDD and assign a schema to the data, and am then turning it ...
0 votes
0 answers
16 views

understanding how apache spark functionality overlaps with other popular sql query engine and big data frameworks

i m really interested in seeing how functionality of diff big data tools overlaps between apache spark, apache drill, cloudera impala, trino, presto db. i m confused about the difference btw"...
1 vote
3 answers
261 views

Retrieve user features in real time from UserId for prediction

Let's say I'm building an app like Uber and I want to predict the user's most likely destination based on the user's past history, current latitude/longitude, and time/date. Here is the proposed ...
0 votes
0 answers
14 views

Filtering dataset using PySpark

I'm trying to apply an algorithm to the FMA dataset. This dataset is available in https://github.com/mdeff/fma, and I downloaded the fma_metadata.zip. The FMA dataset consists of 106,574 music tracks ...
0 votes
1 answer
59 views

Hadoop, Spark and Cloud

It seems Hadoop, Spark, and different versions of Clouds offer facilities to store and analyze big data. There are some articles comparing Hadoop and Spark (for example, this article). There are also ...
3 votes
3 answers
2k views

Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?

I would like to implement a model based on some cleaned and prepared data set. I already have a bit of experience with PySpark, but from a data scientist's perspective it can be cumbersome to work ...
0 votes
1 answer
245 views

IllegalArgumentException at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source) when training an ALS implementation of spark in scala

I was following this tutorial trying to write a collaborative recommender system using the alternating least squares algorithm in spark. I am using the movie lens dataset which can be found here. My ...
0 votes
0 answers
26 views

Can someone suggest good books or resources on SparkR?

I'm trying to find a resource on SparkR specifically so that I can learn about the subtleties in it. But there doesn't seem to be much, other than the docmentation. I found a book called Mastering ...
1 vote
0 answers
44 views

Not able to read data from Mongodb for below schema [closed]

Am trying to read very complex json from mongoDB. Tried in multiple ways nut no luck. Sample schema below : ...
0 votes
1 answer
35 views

What is the difference between Data Modeling and Data Processing?

When discussing big data, it is sometimes mentioned that data modeling can be done by using a tool like map reduce, while data processing may be performed by apache spark. What is the difference ...
1 vote
0 answers
27 views

Working with massive data what is the right approach

let's say I have database with massive data (millions of rows) additionally Let's say 26 Million rows are entered every day I want to build a fraud model to check these 26 Million rows every day.. as ...
4 votes
2 answers
270 views

Lightweight execution of Spark MLLib models

I have some training data which I am using to build a Spark MLLib model which is in a Hive database. I am using simple linear regression models and the PySpark API. I have a code set up to train this ...
0 votes
2 answers
2k views

Scala RDD operation

I am new in scala. I have a csv file stored in hdfs. I am reading that file in scala using ...
2 votes
3 answers
2k views

Random forest vs majority voting

I'm using spark with scala to implement majority voting of decision trees and random forest (both are configured in the same way - same depth, the same amount of base classifiers etc.). Dataset is ...
1 vote
2 answers
3k views

spark ml StringIndexer vs OneHotEncoder, when to use which?

Confused as to when to use StringIndexer vs StringIndexer+OneHotEncoder. The OneHotEncoder docs say For string type input data, it is common to encode categorical features using StringIndexer first. ...
2 votes
2 answers
3k views

Accumulators in Spark (PySpark) without global variables?

BACKGROUND Consider the following textbook example which uses accumulators to add vectors. ...
1 vote
3 answers
10k views

PicklingError in pyspark (PicklingError: Can't pickle <class '__main__.Person'>: attribute lookup Person on __main__ failed)

I am unable to pickle the below class. I am using data bricks 6.5 ML (includes Apache Spark 2.4.5, Scala 2.11) ...
0 votes
0 answers
204 views

Generalized Additive Modeling Apache Spark implementation

Does Spark MLlib support Generalized Additive Modeling? How does one go about implementing GAM models in Spark? I want to implement GAM (Generalized additive model) model in Spark. Based on my ...
0 votes
0 answers
273 views

Creating table in Databricks using the table from Oracle

I am trying to create a table in Databricks using a object in Oracle Db, but getting an error ...
1 vote
1 answer
282 views

Outlier Elimination in Spark With InterQuartileRange Results in Error

I have the following function that is supposed to calculate the outlier for a given dataset. ...
1 vote
1 answer
30 views

Would it be possible/practical to build a distributed deep learning engine by tapping into ordinary PCs' unused resources?

I started thinking about this in the context of Apple's new line of desktop CPUs with dedicated neural engines. From what I hear, these chips are quite adept at solving deep learning problems (as the ...
0 votes
1 answer
40 views

How to make model not too dependent on one variable?

Let's suppose I have a generic model: Variable A | Variable B | Variable C | Variable D Variable Dis a categorical variable. ( ...
7 votes
1 answer
1k views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this data frame (df): col1 col2 col3 col4 A f 5 g D er 2e sd F ...
1 vote
0 answers
35 views

modeling binary classification data

I am new to Machine learning. While reading SparkMLLib java code, I found Binary_classification dataset. But I am not able to understand how this data is modeled and if I want to model same type of ...
2 votes
1 answer
70 views

does storing file in hdfs parallelize it for Spark?

For Spark's RDD operations, data must be in shape of RDD or be parallelized using: ParallelizedData = sc.parallelize(data) My question is that if I store data in ...
4 votes
1 answer
1k views

Has anyone succeeded in finding a good Scala/Spark kernel for Jupyter?

The ones I've tried so far Almond: Works very well for just Scala, but you have to import dependencies, and it gets tedious after a while. And unfortunately can't run when using Spark with YARN ...
2 votes
0 answers
387 views

Hashing trick for dimensionality reduction

I am building a model that uses TF-IDF NLP features in Spark Mllib. The TF-IDF HashingTF function in Mllib uses the 'hashing trick' to efficiently allocate terms to features. My question is: does the ...
5 votes
1 answer
2k views

Saving Large Spark ML Pipeline to HDFS

I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...
1 vote
0 answers
210 views

how to calculate the cosine similarity between two files?

I am using spark and scala to implement an issue. files contain phrases or sentences. I want to use domain based method to calculate the cosine similarity between tags.I convert two files into a ...
1 vote
0 answers
39 views

Why do I need to create a dataframe just to add a column? [closed]

So I worked with a bit of pandas and in that, it's easy to add a column to an existing dataframe. But in spark, a new dataframe needs to be created using 'withColumn'. I'm a complete newbie so is ...
1 vote
0 answers
429 views

How to understand Xgboost model dump

Noticed that spark xgboost does not have a API trees_to_dataframe() as that in Python API, I am trying to parse the getModelDump ...
4 votes
3 answers
12k views

Remove all columns where the entire column is null

I have a very dirty csv where there are several columns with only null values. I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
-1 votes
1 answer
285 views

anyone Explain why Spark is considered “in-memory” compared to Hadoop [closed]

I don't need different between Explain why Spark is considered “in-memory” compared to Hadoop.
15 votes
3 answers
32k views

How to convert categorical data to numerical data in Pyspark

I am using Ipython notebook to work with pyspark applications. I have a CSV file with lots of categorical columns to determine whether the income falls under or over the 50k range. I would like to ...

1
2 3 4 5