Questions tagged [apache-spark]
Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.
248
questions
0
votes
0
answers
8
views
Filtering dataset using PySpark
I'm trying to apply an algorithm to the FMA dataset. This dataset is available in https://github.com/mdeff/fma, and I downloaded the fma_metadata.zip.
The FMA dataset consists of 106,574 music tracks ...
0
votes
1
answer
33
views
Hadoop, Spark and Cloud
It seems Hadoop, Spark, and different versions of Clouds offer facilities to store and analyze big data. There are some articles comparing Hadoop and Spark (for example, this article). There are also ...
- 1
0
votes
0
answers
395
views
How to dynamically add columns in Dataframe Spark by using a config.ini file parameters
I am learning to use Pyspark and at work I have been assigned a task that I can not solve for now.
There is a config.ini file where there are a series of ...
- 101
0
votes
0
answers
15
views
How to reference an intermediary object in scala?
I am using spark/scala to do some transformations on a spark dataframe. I have the following code:
...
- 11
0
votes
1
answer
114
views
IllegalArgumentException at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source) when training an ALS implementation of spark in scala
I was following this tutorial trying to write a collaborative recommender system using the alternating least squares algorithm in spark. I am using the movie lens dataset which can be found here.
My ...
- 11
0
votes
0
answers
13
views
Extracting variables from different properties with similar timestamps -- PySpark
I have a list of variables I need to extract. I am using the PySpark API. Almost all variables are already in the same property, but just one variable I also need was logged in another property. For ...
- 101
0
votes
0
answers
11
views
privacy algorithm implementation in Spark
I have the following code on k-Anonymity ('*' suppress values version) that I want to check its correctness. The code compiles and outputs as expected.
...
- 11
0
votes
0
answers
16
views
Can someone suggest good books or resources on SparkR?
I'm trying to find a resource on SparkR specifically so that I can learn about the subtleties in it. But there doesn't seem to be much, other than the docmentation. I found a book called Mastering ...
- 101
0
votes
0
answers
71
views
Unable to connect to data connection power BI
I am trying to connect to azure databricks but getting the following error.
I have checked the path and the spark odbc driver is present there still it is throwing error.
Details: "ODBC: ERROR [...
- 157
1
vote
0
answers
23
views
Not able to read data from Mongodb for below schema [closed]
Am trying to read very complex json from mongoDB. Tried in multiple ways nut no luck. Sample schema below :
...
- 11
0
votes
1
answer
29
views
What is the difference between Data Modeling and Data Processing?
When discussing big data, it is sometimes mentioned that data modeling can be done by using a tool like map reduce, while data processing may be performed by apache spark. What is the difference ...
0
votes
0
answers
17
views
Learning HDFS and spark online
I am looking for online resources (sandbox?) which already have Hadoop HDFS and Spark configured and I would allow me to practise and learn them (I am a beginner). My laptop does not have much space ...
0
votes
0
answers
35
views
Pyspark install failing on Mac OSX
Not able to run pyspark, spark-shell, apache-spark on my mac after attempting install.
At ...
- 111
1
vote
0
answers
21
views
Working with massive data what is the right approach
let's say I have database with massive data (millions of rows)
additionally Let's say 26 Million rows are entered every day
I want to build a fraud model to check these 26 Million rows every day..
as ...
- 35
0
votes
0
answers
271
views
Spark - Group by coulmn_1, and calculate the precentiles of coulmn_2 where the precentile to calculate is in coulmn_3
I have a data frame with 3 main columns.
coulmn_1 - the column I want to preform group by on
coulmn_2 - contains numeric values I would like to calculate percentiles on
coulmn_3 - contains a number ...
- 101
0
votes
0
answers
56
views
How do you iterate an entire query based on a condition calculated from the result of the query in Spark/pyspark?
I am writing a pyspark program to iterate an entire query many times. A mix of relational and functional spark transformations get applied to a dataframe and produce another dataframe with exactly the ...
- 1
0
votes
1
answer
133
views
Group a spark dataframe by a starting event to an ending event
Given a series of events (with datetime) such as:
failed, failed, passed, failed, passed, passed
I want to retrieve the time from when it first "failed" to when it first "passed," ...
- 101
0
votes
1
answer
126
views
Storage of N-dimensional matrices (tensors) as part of machine learning pipelines
I'm an infra person working on a storage product. I've been googling quite a bit to find an answer to the following question but unable to do so. Hence, I am attemping to ask the question here.
I am ...
- 101
0
votes
0
answers
123
views
Generalized Additive Modeling Apache Spark implementation
Does Spark MLlib support Generalized Additive Modeling? How does one go about implementing GAM models in Spark?
I want to implement GAM (Generalized additive model) model in Spark. Based on my ...
- 1
0
votes
1
answer
78
views
CREATE TABLE USING Oracle DATA_SOURCE
I am trying to create a table using ORACLE as a data source using spark query but getting an error.
%sql
CREATE TABLE TEST
USING org.apache.spark.sql.jdbc
OPTIONS (
url "jdbc:oracle:thin:@...
0
votes
0
answers
196
views
Creating table in Databricks using the table from Oracle
I am trying to create a table in Databricks using a object in Oracle Db, but getting an error
...
1
vote
1
answer
262
views
Outlier Elimination in Spark With InterQuartileRange Results in Error
I have the following function that is supposed to calculate the outlier for a given dataset.
...
- 137
1
vote
1
answer
30
views
Would it be possible/practical to build a distributed deep learning engine by tapping into ordinary PCs' unused resources?
I started thinking about this in the context of Apple's new line of desktop CPUs with dedicated neural engines. From what I hear, these chips are quite adept at solving deep learning problems (as the ...
- 363
0
votes
1
answer
28
views
How to make model not too dependent on one variable?
Let's suppose I have a generic model:
Variable A | Variable B | Variable C | Variable D
Variable Dis a categorical variable. ( ...
2
votes
0
answers
268
views
Hashing trick for dimensionality reduction
I am building a model that uses TF-IDF NLP features in Spark Mllib. The TF-IDF HashingTF function in Mllib uses the 'hashing trick' to efficiently allocate terms to features.
My question is: does the ...
- 21
1
vote
2
answers
2k
views
spark ml StringIndexer vs OneHotEncoder, when to use which?
Confused as to when to use StringIndexer vs StringIndexer+OneHotEncoder.
The OneHotEncoder docs say
For string type input data, it is common to encode categorical features using StringIndexer first.
...
1
vote
0
answers
37
views
Why do I need to create a dataframe just to add a column? [closed]
So I worked with a bit of pandas and in that, it's easy to add a column to an existing dataframe. But in spark, a new dataframe needs to be created using 'withColumn'. I'm a complete newbie so is ...
1
vote
0
answers
339
views
How to understand Xgboost model dump
Noticed that spark xgboost does not have a API trees_to_dataframe() as that in Python API, I am trying to parse the getModelDump ...
- 111
-1
votes
1
answer
242
views
anyone Explain why Spark is considered “in-memory” compared to Hadoop [closed]
I don't need different between
Explain why Spark is considered “in-memory” compared to Hadoop.
1
vote
0
answers
184
views
how to calculate the cosine similarity between two files?
I am using spark and scala to implement an issue. files contain phrases or sentences. I want to use domain based method to calculate the cosine similarity between tags.I convert two files into a ...
- 11
1
vote
0
answers
43
views
Distributed Tensorflow VS Tensorflow on Spark
I am new to Deep Learning and Tensorflow. I learned TensorFlow can be code to run in stand-alone mode and in distributed mode. I also came across Tensorflow on Spark framework that will allow the ...
- 159
2
votes
0
answers
69
views
Clustering large set of images
I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
- 21
-1
votes
1
answer
50
views
Tools for a data science job [closed]
Could someone list out all the important tools and software required for a data science job? Example spark, tensorflow, SQL, tableau etc.
It would be helpful if you could list the domain and then 2 ...
- 482
3
votes
1
answer
370
views
What is the main difference between Hadoop and Spark? [closed]
I recently read the following about Hadoop vs. Spark:
Insist upon in-memory columnar data querying. This was the killer-feature that let Apache Spark run in seconds the queries that would take Hadoop ...
- 31
4
votes
1
answer
1k
views
Has anyone succeeded in finding a good Scala/Spark kernel for Jupyter?
The ones I've tried so far
Almond: Works very well for just Scala, but you have to import dependencies, and it gets tedious after a while. And unfortunately can't run when using Spark with YARN ...
- 141
2
votes
1
answer
203
views
Spark: How to run PCA parallelized? Only one thread used
I use pySpark and set my configuration like following:
...
- 241
3
votes
0
answers
25
views
User defined aggregations on data of around 200GB where row order matters
I am working with "medium large" data of around 200GB. The data are long form log files, where there are several thousand logs for each "entity". The entities are actually flights ...
- 226
1
vote
0
answers
28
views
What is important for Pharmaceutical companies to answer with Big Data Analysis?
I am a data scientist, and I have some biological background (genetics). I have been asked to give a talk for our customers from pharmaceutical industry. I should show them how they benefit from Big ...
- 143
0
votes
0
answers
1k
views
multi-label prediction with pySpark
I am new to Spark
I am using pyspark to predict a multi label results.
I have converted multi labels to binary
So my labels will look like this
...
- 527
1
vote
3
answers
7k
views
PicklingError in pyspark (PicklingError: Can't pickle <class '__main__.Person'>: attribute lookup Person on __main__ failed)
I am unable to pickle the below class. I am using data bricks 6.5 ML (includes Apache Spark 2.4.5, Scala 2.11)
...
1
vote
0
answers
41
views
Efficient way of adding new columns to datamart without reprocessing complete pipeline
I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
- 11
0
votes
1
answer
47
views
Creating more than one worker nodes for local windows machine [closed]
I am using windows laptop. And I installed apache spark for my laptop. And I try to measure spark performance by changing spark components. because of that I want to create more than one worker nodes ...
3
votes
3
answers
10k
views
Remove all columns where the entire column is null
I have a very dirty csv where there are several columns with only null values.
I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
- 133
0
votes
1
answer
313
views
How to run Spark python code in Jupyter Notebook via command prompt
I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below
After that I want to run this in CMD ...
1
vote
0
answers
1k
views
cannot access hive from spark [closed]
I am trying to install a hadoop + spark + hive cluster.
I am using hadoop 3.1.2, spark 2.4.5 (scala 2.11 prebuilt with user-provided hadoop) and hive 2.3.3 (also tried 3.1.2 with the exact same ...
- 11
0
votes
1
answer
459
views
Spark/Databricks: GPU does not appear to be utilized for ML regression (cross-validation, prediction) notebook
I have created and attached a notebook to a GPU-enabled Databricks cluster (6.4 ML (includes Apache Spark 2.4.5, GPU, Scala 2.11), EC2 type: p2.xlarge).
I have started running the notebook that ...
- 103
1
vote
0
answers
16
views
how to configure and access data on spark
friend this week i am very busy on installation and configuration of spark envirnoment i did the installation and I begin basic operations like creating folder and adding text file to the folder,...
- 11
1
vote
1
answer
226
views
Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?
If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
3
votes
1
answer
439
views
BERT in production
I've created a BERT model. What are the ways to do the deployment of this model? Is it possible to use it with Spark, Hadoop or Docker?
- 306
3
votes
3
answers
2k
views
Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?
I would like to implement a model based on some cleaned and prepared data set. I already have a bit of experience with PySpark, but from a data scientist's perspective it can be cumbersome to work ...
- 165