Questions tagged [apache-spark]

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

Merging and deduplicating mass amounts of data in ORC on a 24 hour interval

I have the following problem I'm looking to ideate on and looking for advice on how to think about it from a data science perspective. I'll keep the numbers small for simplicity. Day 1 I get 100 ...
0
votes
0answers
8 views

How to train effective domain specific word2vec models?

I'm training a word2vec model based on our custom data. Due to the size of data and the resources available to me, I'll use pyspark to train spark's built-in word2vec model. However, I'm curious to ...
0
votes
0answers
18 views

LightFM implementation for scala/java on spark

I am looking for hybrid recommendation libraries such as lightfm that I can use on Spark (with Scala). Any alternative? Or best would be for me to build a hybrid recommendation system on spark's mllib ...
1
vote
0answers
10 views

Apache Spark unexpected error on dataframe `show` [migrated]

I just installed Spark and set Environment Variables on Windows, and went through the following steps to define a dataframe in Spark (by defining its schema) : ...
0
votes
0answers
20 views

how to calculate the cosine similarity between two files?

I am using spark and scala to implement an issue. files contain phrases or sentences. I want to use domain based method to calculate the cosine similarity between tags.I convert two files into a ...
0
votes
0answers
30 views

In Spark, what's the role of each of Cluster manager and Spark Context?

What's the role of a cluster manager and the role spark context (Driver program) in terms of managing worker nodes. How do they communicate with each other (and with the worker nodes) and when they do ...
0
votes
0answers
10 views

How to prepare/optimise data for FP-Growth algorithm

I'm using spark mllib for FP-Growth algorithm for our ML model. Description of my issue: I have taken transactional data from our production database to mine the frequent brought items recommendation. ...
1
vote
0answers
20 views

Distributed Tensorflow VS Tensorflow on Spark

I am new to Deep Learning and Tensorflow. I learned TensorFlow can be code to run in stand-alone mode and in distributed mode. I also came across Tensorflow on Spark framework that will allow the ...
2
votes
0answers
46 views

Clustering large set of images

I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
-1
votes
1answer
37 views

Tools for a data science job [closed]

Could someone list out all the important tools and software required for a data science job? Example spark, tensorflow, SQL, tableau etc. It would be helpful if you could list the domain and then 2 ...
0
votes
0answers
12 views

Time-series data pipeline for Kafka + Spark?

I'm working on a project related to predicting syscall sequences' malice. The data, which are syscall sequences that each belongs to a program, will be stored temporary to Kafka (which I have no idea ...
3
votes
1answer
72 views

What is the main difference between Hadoop and Spark? [closed]

I recently read the following about Hadoop vs. Spark: Insist upon in-memory columnar data querying. This was the killer-feature that let Apache Spark run in seconds the queries that would take Hadoop ...
1
vote
1answer
64 views

Has anyone succeeded in finding a good Scala/Spark kernel for Jupyter?

The ones I've tried so far Almond: Works very well for just Scala, but you have to import dependencies, and it gets tedious after a while. And unfortunately can't run when using Spark with YARN ...
1
vote
0answers
219 views

Flattening of a structure field in pyspark is slow?

I am rather new to (py)spark. I am using a python UDF to compute several outputs. My idea was to pack all the values in a StructType and then flatten the structure before collecting the result. The ...
2
votes
0answers
47 views

Spark: How to run PCA parallelized? Only one thread used

I use pySpark and set my configuration like following: ...
3
votes
0answers
23 views

User defined aggregations on data of around 200GB where row order matters

I am working with "medium large" data of around 200GB. The data are long form log files, where there are several thousand logs for each "entity". The entities are actually flights ...
1
vote
0answers
25 views

What is important for Pharmaceutical companies to answer with Big Data Analysis?

I am a data scientist, and I have some biological background (genetics). I have been asked to give a talk for our customers from pharmaceutical industry. I should show them how they benefit from Big ...
0
votes
0answers
437 views

multi-label prediction with pySpark

I am new to Spark I am using pyspark to predict a multi label results. I have converted multi labels to binary So my labels will look like this ...
0
votes
0answers
216 views

PicklingError in pyspark (PicklingError: Can't pickle <class '__main__.Person'>: attribute lookup Person on __main__ failed)

I am unable to pickle the below class. I am using data bricks 6.5 ML (includes Apache Spark 2.4.5, Scala 2.11) ...
0
votes
0answers
18 views

How to define schema so that I can store sklearn object in pyspark dataframe

Using @pandas_udf(schema, PandasUDFType.GROUPED_MAP), is there any workaround to return sklearn fitted models in data frame ? Something like this - ...
1
vote
0answers
15 views

Efficient way of adding new columns to datamart without reprocessing complete pipeline

I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
0
votes
1answer
21 views

Creating more than one worker nodes for local windows machine [closed]

I am using windows laptop. And I installed apache spark for my laptop. And I try to measure spark performance by changing spark components. because of that I want to create more than one worker nodes ...
1
vote
2answers
2k views

Remove all columns where the entire column is null

I have a very dirty csv where there are several columns with only null values. I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
0
votes
1answer
42 views

How to run Spark python code in Jupyter Notebook via command prompt

I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below After that I want to run this in CMD ...
1
vote
0answers
735 views

cannot access hive from spark [closed]

I am trying to install a hadoop + spark + hive cluster. I am using hadoop 3.1.2, spark 2.4.5 (scala 2.11 prebuilt with user-provided hadoop) and hive 2.3.3 (also tried 3.1.2 with the exact same ...
0
votes
1answer
123 views

Spark/Databricks: GPU does not appear to be utilized for ML regression (cross-validation, prediction) notebook

I have created and attached a notebook to a GPU-enabled Databricks cluster (6.4 ML (includes Apache Spark 2.4.5, GPU, Scala 2.11), EC2 type: p2.xlarge). I have started running the notebook that ...
1
vote
0answers
16 views

how to configure and access data on spark

friend this week i am very busy on installation and configuration of spark envirnoment i did the installation and I begin basic operations like creating folder and adding text file to the folder,...
1
vote
1answer
114 views

Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
3
votes
1answer
221 views

BERT in production

I've created a BERT model. What are the ways to do the deployment of this model? Is it possible to use it with Spark, Hadoop or Docker?
1
vote
2answers
604 views

Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?

I would like to implement a model based on some cleaned and prepared data set. I already have a bit of experience with PySpark, but from a data scientist's perspective it can be cumbersome to work ...
2
votes
1answer
352 views

Scalable way to calculate betweenness centrality for a graph in spark

I have a use-case to calculate betweenness centrality of nodes. I have tried graphx with spark-betweenness but it is a very long running job. Has anyone successfully calculated betweenness centrality ...
0
votes
1answer
196 views

How to apply K-Medoids in PySpark?

the pyspark ml library does not provide any clustering methods for K-Medoids. So my question is, how can one apply K-Medoids in a pyspark context?
2
votes
0answers
51 views

Can i access data on one spark cluster from another? [closed]

What if I want to sample a table over hive on Spark-cluster-1, but I'm logged in on Spark-cluster-2? ...
1
vote
0answers
37 views

How to push tasks to remote Hadoop/Spark cluster

I have access to a Hadoop cluster provided by a third party. Currently, I use Spark on this cluster by logging in to a Zeppelin ...
2
votes
1answer
303 views

Calculating Rank Ordering Error Metric for implicit recommendation

I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula: $...
2
votes
1answer
5k views

How to select multiple columns in a RDD with Spark (pySpark)?

Lets say I have a RDD that has comma delimited data. Each comma delimited value represents the amount of hours slept in the day of a week. So for i.e. [8,7,6,7,8,8,5] How can I manipulate the RDD so ...
0
votes
0answers
61 views

Regression Task - Spark, PyTorch, TensorFlow or scikit

I know it's a broad question, sorry for that, but I'm still testing the waters with machine learning. I have a typical regression task (predict target numbers with the help of features x,y,z) and a ...
1
vote
0answers
83 views

Label Propagation for weighted graph

I am working on "community detection" on a big network graph and I've been using the Label Propagation Algorithm from GraphFrames (Spark package). However, my graph is weighted and I was wondering ...
2
votes
2answers
667 views

Random forest vs majority voting

I'm using spark with scala to implement majority voting of decision trees and random forest (both are configured in the same way - same depth, the same amount of base classifiers etc.). Dataset is ...
1
vote
0answers
38 views

Is there a Spark library for computing canonical correlation?

I can't seem to find a library for computing canonical correlation in Spark. Canonical correlation analysis (CCA) is common in other data science-focused systems. This exists in R's base and in ...
0
votes
1answer
49 views

Regarding join in pyspark

Given a rdd in pyspark, $A=[(0, 1), (1, 2), (1, 3), (3, 0)]$ I am trying to find self join of A such that the second element of previous pair is equal to first element of next pair i.e. $...
1
vote
1answer
85 views

How to deal with evaluations in a multi step ML pipeline

Apache Spark provides us the pipeline to construct a ML pipeline. Moreover, we can evaluate the regressors in the pipeline, now here is the point I dont understand: how to apply to testing data. Ok, ...
1
vote
0answers
87 views

How to create tensors in spark?

I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
1
vote
0answers
27 views

Is there a way to use a pom.xml file to update spark configuration?

I am trying to update my spark configuration to solve some dependency problems. This pom.xml file seems to be useful for this purpose. I am using a spark docker image. ...
5
votes
1answer
303 views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this dataframe (df) ...
5
votes
1answer
4k views

How to implement LSTM with Spark?

I would like to build an LSTM network for text classification with PySpark, but I don't find any library or function about it. ...
1
vote
2answers
201 views

Record Linkage problem

I am building Matching Alogoritm using ML.Project is to match Internal customer data with external customer data.Features are names,address,city,state and zip. We create pairs between data sets and ...
0
votes
0answers
413 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
1
vote
0answers
372 views

Spark creating dat file - header/trailer issue

Although this is a DS community but I saw spark tag so asking query here. 0 I've a specific scenario of creating a file extract (dat/delimited) in ...
1
vote
1answer
24 views

How do I recommend items to out of training users based on its recent views?

I used Spark's ALS implementation of matrix factorization (Collaborative Filtering for Implicit Feedback) to train user and item embeddings. Since we have a lot of users in system, I had to sample ...

1
2 3 4 5