2022 Developer Survey is open! Take survey.

Questions tagged [apache-spark]

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

Computing number of business days between start/end columns

I have two Dataframes facts: columns: data, start_date and end_date holidays: column: <...
user avatar
  • 101
0 votes
0 answers
12 views

Is it possible to implement an rdd version of a for loop having map and reduce using pyspark?

I need to test an algorithm that computes a function on a dataframe where in each execution I drop a column and computes the function. This is a example in python pyspark but without using rdd: ...
user avatar
  • 1
0 votes
0 answers
15 views

Group a spark dataframe by a starting event to an ending event

Given a series of events (with datetime) such as: failed, failed, passed, failed, passed, passed I want to retrieve the time from when it first "failed" to when it first "passed," ...
user avatar
  • 101
0 votes
0 answers
22 views

What data/analytics tools I need to use at my current e-commerce workplace?

I recently started a new position as a data scientist at an E-commerce company. The company is founded about 4-5 years ago and is new to many data-related areas. Specifically, I'm their first data ...
user avatar
  • 121
0 votes
1 answer
15 views

Storage of N-dimensional matrices (tensors) as part of machine learning pipelines

I'm an infra person working on a storage product. I've been googling quite a bit to find an answer to the following question but unable to do so. Hence, I am attemping to ask the question here. I am ...
user avatar
  • 101
0 votes
0 answers
36 views

Generalized Additive Modeling Apache Spark implementation

Does Spark MLlib support Generalized Additive Modeling? How does one go about implementing GAM models in Spark? I want to implement GAM (Generalized additive model) model in Spark. Based on my ...
user avatar
  • 1
0 votes
0 answers
7 views

Tips on plotting library for distributed data/compute

Does anyone know of any plotting/visualization libraries (preferably Python) that take advantage of distributed data and compute backends (Spark, DASK, Ray, or similar)? To elaborate a bit more: A ...
user avatar
0 votes
0 answers
13 views

Is model training using spark ML distributed?

Is model training using spark ML distributed? If training happens on a single machine then why use spark?
user avatar
  • 1
0 votes
1 answer
26 views

CREATE TABLE USING Oracle DATA_SOURCE

I am trying to create a table using ORACLE as a data source using spark query but getting an error. %sql CREATE TABLE TEST USING org.apache.spark.sql.jdbc OPTIONS ( url "jdbc:oracle:thin:@...
user avatar
0 votes
0 answers
58 views

Creating table in Databricks using the table from Oracle

I am trying to create a table in Databricks using a object in Oracle Db, but getting an error ...
user avatar
1 vote
1 answer
38 views

Outlier Elimination in Spark With InterQuartileRange Results in Error

I have the following function that is supposed to calculate the outlier for a given dataset. ...
user avatar
  • 137
1 vote
1 answer
25 views

Would it be possible/practical to build a distributed deep learning engine by tapping into ordinary PCs' unused resources?

I started thinking about this in the context of Apple's new line of desktop CPUs with dedicated neural engines. From what I hear, these chips are quite adept at solving deep learning problems (as the ...
user avatar
0 votes
1 answer
23 views

How to make model not too dependent on one variable?

Let's suppose I have a generic model: Variable A | Variable B | Variable C | Variable D Variable Dis a categorical variable. ( ...
user avatar
1 vote
0 answers
103 views

Huge matrix multiplication with PySpark

I am currently struggling with a problem. I have been googling around and cannot find an answer. Also I am kinda new to PySpark and Spark maybe that's why I am struggling. Let's say I have a huge $m \...
user avatar
0 votes
0 answers
10 views

Data Pipelines - Apache Spark - Updating Another Data Store or Input to Queue

What is the correct way to think about or approach something like Spark breaking down a 160GB file into smaller manageable parts, or single records and then doing something like updating records in ...
user avatar
  • 101
0 votes
0 answers
31 views

Spark Dataframe APIs vs Spark SQL

I have a relatively complex query which runs against a database and contains multiple join statements, lead/lag functions, subquery, etc. These tables are available as individual files in my object ...
user avatar
0 votes
0 answers
29 views

Find a column by name in a row in scala spark

I have a Seq[Row].Each row is an Array of Struct.Struct has four fields: a,b,c and d all of which are String.The data in a particular row is something like this: [{"a":"ahahk",&...
user avatar
0 votes
0 answers
40 views

Issue while reading a csv file through Spark

I am trying to read a csv file through Spark. However one of the columns has the data in the below format and because of comma it is being split into multiple columns. The input csv file is a comma ...
user avatar
1 vote
0 answers
99 views

Hashing trick for dimensionality reduction

I am building a model that uses TF-IDF NLP features in Spark Mllib. The TF-IDF HashingTF function in Mllib uses the 'hashing trick' to efficiently allocate terms to features. My question is: does the ...
user avatar
  • 11
0 votes
0 answers
341 views

Access global variable from UDF (User Defined Function) in python in spark

I am trying to alter a global variable from inside a pyspark.sql.functions.udf function in python. But, the change in not getting reflected in the global variable. ...
user avatar
1 vote
1 answer
342 views

spark ml StringIndexer vs OneHotEncoder, when to use which?

Confused as to when to use StringIndexer vs StringIndexer+OneHotEncoder. The OneHotEncoder docs say For string type input data, it is common to encode categorical features using StringIndexer first. ...
user avatar
1 vote
0 answers
36 views

Why do I need to create a dataframe just to add a column? [closed]

So I worked with a bit of pandas and in that, it's easy to add a column to an existing dataframe. But in spark, a new dataframe needs to be created using 'withColumn'. I'm a complete newbie so is ...
user avatar
1 vote
0 answers
134 views

How to understand Xgboost model dump

Noticed that spark xgboost does not have a API trees_to_dataframe() as that in Python API, I am trying to parse the getModelDump ...
user avatar
-1 votes
1 answer
210 views

anyone Explain why Spark is considered “in-memory” compared to Hadoop [closed]

I don't need different between Explain why Spark is considered “in-memory” compared to Hadoop.
user avatar
1 vote
0 answers
100 views

how to calculate the cosine similarity between two files?

I am using spark and scala to implement an issue. files contain phrases or sentences. I want to use domain based method to calculate the cosine similarity between tags.I convert two files into a ...
user avatar
  • 11
1 vote
0 answers
32 views

Distributed Tensorflow VS Tensorflow on Spark

I am new to Deep Learning and Tensorflow. I learned TensorFlow can be code to run in stand-alone mode and in distributed mode. I also came across Tensorflow on Spark framework that will allow the ...
user avatar
  • 159
2 votes
0 answers
57 views

Clustering large set of images

I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
user avatar
-1 votes
1 answer
46 views

Tools for a data science job [closed]

Could someone list out all the important tools and software required for a data science job? Example spark, tensorflow, SQL, tableau etc. It would be helpful if you could list the domain and then 2 ...
user avatar
  • 460
4 votes
1 answer
226 views

What is the main difference between Hadoop and Spark? [closed]

I recently read the following about Hadoop vs. Spark: Insist upon in-memory columnar data querying. This was the killer-feature that let Apache Spark run in seconds the queries that would take Hadoop ...
user avatar
4 votes
1 answer
678 views

Has anyone succeeded in finding a good Scala/Spark kernel for Jupyter?

The ones I've tried so far Almond: Works very well for just Scala, but you have to import dependencies, and it gets tedious after a while. And unfortunately can't run when using Spark with YARN ...
user avatar
2 votes
1 answer
104 views

Spark: How to run PCA parallelized? Only one thread used

I use pySpark and set my configuration like following: ...
user avatar
  • 141
3 votes
0 answers
24 views

User defined aggregations on data of around 200GB where row order matters

I am working with "medium large" data of around 200GB. The data are long form log files, where there are several thousand logs for each "entity". The entities are actually flights ...
user avatar
  • 226
1 vote
0 answers
27 views

What is important for Pharmaceutical companies to answer with Big Data Analysis?

I am a data scientist, and I have some biological background (genetics). I have been asked to give a talk for our customers from pharmaceutical industry. I should show them how they benefit from Big ...
user avatar
  • 143
0 votes
0 answers
1k views

multi-label prediction with pySpark

I am new to Spark I am using pyspark to predict a multi label results. I have converted multi labels to binary So my labels will look like this ...
user avatar
  • 497
1 vote
2 answers
2k views

PicklingError in pyspark (PicklingError: Can't pickle <class '__main__.Person'>: attribute lookup Person on __main__ failed)

I am unable to pickle the below class. I am using data bricks 6.5 ML (includes Apache Spark 2.4.5, Scala 2.11) ...
user avatar
1 vote
0 answers
17 views

Efficient way of adding new columns to datamart without reprocessing complete pipeline

I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
user avatar
  • 11
0 votes
1 answer
25 views

Creating more than one worker nodes for local windows machine [closed]

I am using windows laptop. And I installed apache spark for my laptop. And I try to measure spark performance by changing spark components. because of that I want to create more than one worker nodes ...
user avatar
1 vote
3 answers
6k views

Remove all columns where the entire column is null

I have a very dirty csv where there are several columns with only null values. I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
user avatar
  • 113
0 votes
1 answer
104 views

How to run Spark python code in Jupyter Notebook via command prompt

I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below After that I want to run this in CMD ...
user avatar
1 vote
0 answers
1k views

cannot access hive from spark [closed]

I am trying to install a hadoop + spark + hive cluster. I am using hadoop 3.1.2, spark 2.4.5 (scala 2.11 prebuilt with user-provided hadoop) and hive 2.3.3 (also tried 3.1.2 with the exact same ...
user avatar
0 votes
1 answer
268 views

Spark/Databricks: GPU does not appear to be utilized for ML regression (cross-validation, prediction) notebook

I have created and attached a notebook to a GPU-enabled Databricks cluster (6.4 ML (includes Apache Spark 2.4.5, GPU, Scala 2.11), EC2 type: p2.xlarge). I have started running the notebook that ...
user avatar
1 vote
0 answers
16 views

how to configure and access data on spark

friend this week i am very busy on installation and configuration of spark envirnoment i did the installation and I begin basic operations like creating folder and adding text file to the folder,...
user avatar
  • 11
1 vote
1 answer
191 views

Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
user avatar
3 votes
1 answer
289 views

BERT in production

I've created a BERT model. What are the ways to do the deployment of this model? Is it possible to use it with Spark, Hadoop or Docker?
user avatar
3 votes
2 answers
2k views

Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?

I would like to implement a model based on some cleaned and prepared data set. I already have a bit of experience with PySpark, but from a data scientist's perspective it can be cumbersome to work ...
user avatar
  • 165
2 votes
1 answer
618 views

Scalable way to calculate betweenness centrality for a graph in spark

I have a use-case to calculate betweenness centrality of nodes. I have tried graphx with spark-betweenness but it is a very long running job. Has anyone successfully calculated betweenness centrality ...
user avatar
  • 141
0 votes
1 answer
394 views

How to apply K-Medoids in PySpark?

the pyspark ml library does not provide any clustering methods for K-Medoids. So my question is, how can one apply K-Medoids in a pyspark context?
user avatar
  • 111
2 votes
0 answers
105 views

Can i access data on one spark cluster from another? [closed]

What if I want to sample a table over hive on Spark-cluster-1, but I'm logged in on Spark-cluster-2? ...
user avatar
  • 121
1 vote
0 answers
75 views

How to push tasks to remote Hadoop/Spark cluster

I have access to a Hadoop cluster provided by a third party. Currently, I use Spark on this cluster by logging in to a Zeppelin ...
user avatar
  • 6,872
2 votes
1 answer
549 views

Calculating Rank Ordering Error Metric for implicit recommendation

I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula: $...
user avatar
  • 1,664

1
2 3 4 5