PriceHardman
  • Member for 5 years, 5 months
  • Last seen more than a month ago
  • San Francisco, CA, USA
Task not serializable Error
3 votes

Looking at another question on Stack Overflow about serialization exceptions in Spark, it says that anonymous functions serialize their containing class, and if that class contains the SparkContext -- ...

View answer
Eigenvectors and eigenvalues for natural language processing
2 votes

Latent Semantic Analysis (LSA) relies on linear-algebraic decompositions (e.g. SVD), which in turn involve eigenvectors/values (see here). Not sure if this is quite what your question is driving at, ...

View answer
When does it makes senses to use Dot-Product as similarity measure instead of Cosine?
2 votes

Borrowing from this Quora answer -- which includes some concrete examples of when you might prefer one measure over the other -- it comes down to whether you care about taking the magnitude of the ...

View answer
How to deal with a skewed data-set having all the samples almost similar?
Accepted answer
2 votes

This is the so-called "Class Imbalance Problem". Fortunately, there are a number of possible approaches. Probably easiest is going to be to use the class_weight option present in a number of scikit-...

View answer
Spark: calculating percentage from a non-Int field
1 votes

EDIT, in light of your comment and closer reading of your code Ok, so it looks like in your code you used the RDD api to count the total number of rows in the file (treating it as just a plain RDD[...

View answer