Skip to main content

Questions tagged [apache-spark]

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.

Filter by
Sorted by
Tagged with
7 votes
1 answer
4k views

Why does logistic regression in Spark and R return different models for the same data?

I've compared the logistic regression models on R (glm) and on Spark (LogisticRegressionWithLBFGS) on a dataset of 390 obs. of ...
SparkUser's user avatar
  • 113
9 votes
3 answers
9k views

How to run a pyspark application in windows 8 command prompt

I have a python script written with Spark Context and I want to run it. I tried to integrate IPython with Spark, but I could not do that. So, I tried to set the spark path [ Installation folder/bin ] ...
SRS's user avatar
  • 1,065
7 votes
1 answer
1k views

SPARK, ML: Naive Bayes classifier often assigns 1 as probability prediction

Hi I am using Spark ML to optimise a Naive Bayes multi-class classifier. I have about 300 categories and I am classifying text documents. The training set is balanced enough and there is about 300 ...
Rami's user avatar
  • 594
3 votes
1 answer
2k views

Use cases for graph algorithms and graph data structures in finance and banking

I work in a bank and most data is in tabular format in relational databases. I have been reading about graph algorithms (page rank), graph libraries (spark graphx) and graph databases (neo4j). I would ...
Victor's user avatar
  • 611
3 votes
3 answers
1k views

Solution for in Time/Space Complexity challenge in Recommendation System?

I have a book Recommendation System project and have a huge data set of feature vectors. What is the best solution for in memory computation? I mean, the program should: calculate the cosine ...
Elmira Khodaee's user avatar
2 votes
1 answer
899 views

Calculating Rank Ordering Error Metric for implicit recommendation

I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula: $...
Dan Scally's user avatar
  • 1,754
1 vote
2 answers
392 views

Record Linkage problem

I am building Matching Alogoritm using ML.Project is to match Internal customer data with external customer data.Features are names,address,city,state and zip. We create pairs between data sets and ...
vishal's user avatar
  • 11
0 votes
1 answer
206 views

How to Access a Text File from IBM DSX Scala-Spark Notebook [closed]

I have recently faced problem in accessing text files that I have uploaded from my local computer to the IBM DSX Cloud Object Storage so that I can use it in my Spark-Scala notebooks. It appeared to ...
H.Rehman's user avatar