Skip to main content

Questions tagged [apache-spark]

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.

63 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7 votes
0 answers
943 views

ALS in Spark: what loss function is it minimizing?

I’ve playing with the MovieLens ratings dataset under Spark’s ALS and a manual implementation of ALS and comparing results with the same hyperparameters. I’d like to know this exactly in order to make ...
anymous.asker's user avatar
3 votes
0 answers
126 views

Clustering large set of images

I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of ...
Overloop's user avatar
3 votes
0 answers
29 views

User defined aggregations on data of around 200GB where row order matters

I am working with "medium large" data of around 200GB. The data are long form log files, where there are several thousand logs for each "entity". The entities are actually flights ...
Placidia's user avatar
  • 226
3 votes
0 answers
850 views

Model params tuning

I have a pipeline of models: Training pipeline (on training data): Tokenizer -> HashingTF -> IDF -> KMeans Test pipeline (prediction on query data): <...
vvinton's user avatar
  • 81
3 votes
1 answer
2k views

Why is Spark's LinearRegressionWithSGD very slow locally?

I have been trying to run linear regression with SGD that is found in Spark mllib for some time and I am experiencing huge performance problems. All examples that I was looking have number of ...
Nikola Stojiljkovic's user avatar
2 votes
0 answers
389 views

Hashing trick for dimensionality reduction

I am building a model that uses TF-IDF NLP features in Spark Mllib. The TF-IDF HashingTF function in Mllib uses the 'hashing trick' to efficiently allocate terms to features. My question is: does the ...
John's user avatar
  • 21
2 votes
1 answer
255 views

Spark: How to run PCA parallelized? Only one thread used

I use pySpark and set my configuration like following: ...
bonfab's user avatar
  • 261
2 votes
1 answer
1k views

Scalable way to calculate betweenness centrality for a graph in spark

I have a use-case to calculate betweenness centrality of nodes. I have tried graphx with spark-betweenness but it is a very long running job. Has anyone successfully calculated betweenness centrality ...
abhati's user avatar
  • 141
2 votes
0 answers
77 views

Is there a Spark library for computing canonical correlation?

I can't seem to find a library for computing canonical correlation in Spark. Canonical correlation analysis (CCA) is common in other data science-focused systems. This exists in R's base and in ...
xiaodai's user avatar
  • 630
2 votes
0 answers
266 views

How to create tensors in spark?

I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
neverevernever's user avatar
2 votes
1 answer
131 views

How do I recommend items to out of training users based on its recent views?

I used Spark's ALS implementation of matrix factorization (Collaborative Filtering for Implicit Feedback) to train user and item embeddings. Since we have a lot of users in system, I had to sample ...
Sn.'s user avatar
  • 21
2 votes
0 answers
97 views

Obtain learning curve of Gradient Boosted Tree model in PySpark

Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a ...
HFulcher's user avatar
  • 182
2 votes
0 answers
1k views

What is the difference between PySpark's featuresCol, labelCol, predictionCol, and probabilityCol?

I am attempting to train a random forest classifier (pyspark.ml.classification.RandomForestClassifier) on a large dataset (~70gb). However, I am not sure what to ...
Joe B's user avatar
  • 322
2 votes
0 answers
337 views

Modelling if condition of multiple estimators in a pipeline

How to correctly model a if condition to choose estimator/predictor(linear regression, gbt) to be used in scikit/spark-ml in a single pipeline. ...
pratyush's user avatar
2 votes
0 answers
91 views

Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?

I am using a three node system: Master node: $64$ GB RAM 2 slave node: $32$ GB RAM I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function ...
Chandan Gautam's user avatar
2 votes
0 answers
3k views

Unable to use the Python Data Frame method "iloc" on a Data Frame created in pyspark's SQLContext

After loading a dataset as DataFrame in pyspark's SQLContext, unable to use the Python DataFrame property of 'iloc' on it. Does a DataFrame created in SQLContext of pyspark behave differently and ...
Avinash Peyyety's user avatar
2 votes
1 answer
2k views

How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?

How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation? I can not seem to find it anywhere, even not from the author of LDA, ...
blpasd's user avatar
  • 155
2 votes
1 answer
503 views

Spark ALS-WR giving the same recommended items for all users

We are trying to build a recommendation system for a supermarket with diverse item types (ranging from fast-moving grocery to low-moving electronic items). Some items are purchased more frequently in ...
samdas's user avatar
  • 21
1 vote
0 answers
27 views

Working with massive data what is the right approach

let's say I have database with massive data (millions of rows) additionally Let's say 26 Million rows are entered every day I want to build a fraud model to check these 26 Million rows every day.. as ...
MAS's user avatar
  • 35
1 vote
0 answers
432 views

How to understand Xgboost model dump

Noticed that spark xgboost does not have a API trees_to_dataframe() as that in Python API, I am trying to parse the getModelDump ...
helloworld's user avatar
1 vote
0 answers
211 views

how to calculate the cosine similarity between two files?

I am using spark and scala to implement an issue. files contain phrases or sentences. I want to use domain based method to calculate the cosine similarity between tags.I convert two files into a ...
salm's user avatar
  • 11
1 vote
0 answers
54 views

Distributed Tensorflow VS Tensorflow on Spark

I am new to Deep Learning and Tensorflow. I learned TensorFlow can be code to run in stand-alone mode and in distributed mode. I also came across Tensorflow on Spark framework that will allow the ...
Raj's user avatar
  • 159
1 vote
0 answers
31 views

What is important for Pharmaceutical companies to answer with Big Data Analysis?

I am a data scientist, and I have some biological background (genetics). I have been asked to give a talk for our customers from pharmaceutical industry. I should show them how they benefit from Big ...
Rebecca's user avatar
  • 143
1 vote
0 answers
70 views

Efficient way of adding new columns to datamart without reprocessing complete pipeline

I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
Abdul R's user avatar
  • 11
1 vote
0 answers
16 views

how to configure and access data on spark

friend this week i am very busy on installation and configuration of spark envirnoment i did the installation and I begin basic operations like creating folder and adding text file to the folder,...
haftuk's user avatar
  • 11
1 vote
1 answer
274 views

Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
Soorya Paturi's user avatar
1 vote
0 answers
127 views

How to push tasks to remote Hadoop/Spark cluster

I have access to a Hadoop cluster provided by a third party. Currently, I use Spark on this cluster by logging in to a Zeppelin ...
Peter's user avatar
  • 7,466
1 vote
0 answers
214 views

Label Propagation for weighted graph

I am working on "community detection" on a big network graph and I've been using the Label Propagation Algorithm from GraphFrames (Spark package). However, my graph is weighted and I was wondering ...
Alex's user avatar
  • 165
1 vote
0 answers
39 views

Is there a way to use a pom.xml file to update spark configuration?

I am trying to update my spark configuration to solve some dependency problems. This pom.xml file seems to be useful for this purpose. I am using a spark docker image. ...
Jay's user avatar
  • 425
1 vote
0 answers
499 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
3nomis's user avatar
  • 541
1 vote
0 answers
972 views

Spark creating dat file - header/trailer issue

Although this is a DS community but I saw spark tag so asking query here. 0 I've a specific scenario of creating a file extract (dat/delimited) in ...
knowone's user avatar
  • 111
1 vote
0 answers
75 views

Spark: how to process tree aggregation and statistic

I have a description of file system in form of a csv: ...
Kiwy's user avatar
  • 125
1 vote
0 answers
132 views

Unable to open application master UI in spark1.6.1 in cluster mode

Hadoop cluster has 8 nodes with high availability of resource manager. Active ResourceManager is in node 3. and Standby ResourceManager in node 2. when i submit the application in cluster mode. the ...
shashikumar's user avatar
1 vote
0 answers
632 views

Interpreting rawPrediction from Spark ML LinearSVC

I am using Spark ML's LinearSVC in a binary classification model. The transform method creates two columns, prediction and ...
jonwithers's user avatar
1 vote
0 answers
35 views

modeling binary classification data

I am new to Machine learning. While reading SparkMLLib java code, I found Binary_classification dataset. But I am not able to understand how this data is modeled and if I want to model same type of ...
shiv's user avatar
  • 111
1 vote
2 answers
141 views

Is spark als item feature comparable between several runs

I am using spark als.train() to build my user-items recommendation system. The problem is I want to cover more item feature. So, I need to input 7 days user action data. But the als train become slow ...
seabiscuit08's user avatar
1 vote
0 answers
426 views

Spark fitting into Data Science Paradigm for timeseries data

I have seen Dataframe as new API on Spark2 instead of RDD.So I have following few question about the utility of Spark in terms of time series data. Is forecasting still limited to memory available in ...
user160187's user avatar
1 vote
0 answers
4k views

Not enough replicas available for query at consistency all (2 required but only 1 alive)

I have a Cassandra cluster of 2 nodes and a keyspace "test" with a replication factor of 2. The consistency level is All. I am running a spark job which dumps data in Cassandra database. So, the ...
Credosam's user avatar
1 vote
0 answers
170 views

Creating a user-based recommendation engine?

I have a data set of some questions and answers that users have completed by choices. I'm trying to build a recommendation engine to find similar users based on their answers on top of this. So for ...
Amin Alaee's user avatar
1 vote
0 answers
592 views

How to do Feature Extraction using Apache Spark

I am a newbie to Machine Learning and I have to do feature extraction for a banking application to detect / predict fraudulent transactions. I have come across few articles on Feature Extraction ...
JKC's user avatar
  • 111
1 vote
0 answers
109 views

How to adapt the LBFGS algorithm to accept a different data input parameter in Spark?

I´m trying to use the LBFGS algorithm to minimize the parameters of a model used for Customer Life Time Value analysis (https://github.com/CamDavidsonPilon/lifetimes). The lifetimes lib in python uses ...
neilson's user avatar
  • 111
1 vote
0 answers
195 views

Finding lookalike for large number of users

We have a large user-base, within which we want to find lookalikes, around 25-30 Million, for the users, we have data such as show liked by the user, genre liked etc. What we need to do is ...
Nikhil Verma's user avatar
1 vote
0 answers
320 views

Is there a way to find the weights of every feature in spark ml model?

I am using spark ml classifier to create a model to predict whether a user will buy a product or not and I am using features both continuous and categorical. The model is predicting the outcome with ...
Consumpta's user avatar
1 vote
0 answers
64 views

Spark Deeplearning4j Training Problem

I am training a model in Spark using Dl4J library in Yarn-Cluster mode. When I train the model on 2 lakh data (approx 200MB) then the job succeeds but when I go to train the model with 3 lakh data (...
Credosam's user avatar
1 vote
0 answers
266 views

Scalable training/updating of many small LSTM models

My situation is that I have many thousands of devices which each have their own specific LSTM model for anomaly prediction. These devices behave wildly differently so I don't think there is any way to ...
NMR's user avatar
  • 33
1 vote
0 answers
176 views

How to multiply a "fat and short" matrix with a "tall and thin" matrix using MapReduce?

Assume that $A_{m \times n}$ and $B_{n \times k}$ are to be multiplied to get $C_{m \times k}$. Now if $n$ is too large for a single row $A_{j}$ of $A$ to fit in RAM (and similarly for columns of B) ...
wabbit's user avatar
  • 1,297
1 vote
0 answers
609 views

Spark MLlib recommendation - restaurant/ item similarity - issues/improvement

Use Case Recommend similar items (restaurants) to diner. Solution: We have used apache spark MLlib ALS algorithm. Values for lambda and rank has been obtained by iterating through all permutations ...
Shrey Shivam's user avatar
0 votes
0 answers
22 views

Databricks Extracting features out of Big Files like Step and PDF

So i dont know if this is the right community to ask this question, but lets see. So we are using databricks in our company and we extract Features out of big binary files like PDF and STEP. We trying ...
muhmuhhum's user avatar
0 votes
0 answers
16 views

understanding how apache spark functionality overlaps with other popular sql query engine and big data frameworks

i m really interested in seeing how functionality of diff big data tools overlaps between apache spark, apache drill, cloudera impala, trino, presto db. i m confused about the difference btw"...
wando's user avatar
  • 111
0 votes
1 answer
64 views

Hadoop, Spark and Cloud

It seems Hadoop, Spark, and different versions of Clouds offer facilities to store and analyze big data. There are some articles comparing Hadoop and Spark (for example, this article). There are also ...
Tara's user avatar
  • 1