Santoshi M
  • Member for 6 years, 5 months
  • Last seen more than a month ago
  • India
Replace all numeric values in a pyspark dataframe by a constant value
8 votes

Using lit would convert all values of the column to the given value. To do it only for non-null values of dataframe, you would have to filter non-null values of each column and replace your value. ...

View answer
spark item similarity recommendation
7 votes

For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task. Item similarity approach is just about creating ...

View answer
Data set size versus data dimension, is there a rule of thumb?
Accepted answer
3 votes

In this video by Caltech prof. Yaser Abu-Mostafa, he explains the relationship between dimension of a dataset and it's size required for any learning model to work. As a general rule of thumb, size ...

View answer
Deploying models on bigdata platforms like Hadoop and Spark
2 votes

1 Since dataset is huge I can make use any disributed platform for faster computaion and model creation right? Yes, that is what distributed platforms are for. 2 Once the model is ready, then ...

View answer
Filling missing data with other than mean values
2 votes

There is a difference between data with missing values and sparse data. Missing values are generally there because of invalid input, loss or error during data collection or are created when cleaning ...

View answer
Jaccard similarity between two items
Accepted answer
1 votes

In any commerce setting, the concept of item similarity is very not straightforward. Two users usually buying same kinds of products can be considered as similar, but we cannot say the same about two ...

View answer
SPARK Mllib: Multiclass logistic regression, how to get the probabilities of all classes rather than the top one?
1 votes

To get all probabilities instead of all classes instead of just the labeled class, there is no explicit method till now (Spark 2.0) in Spark MLlib or ML. But you can extend the Logistic Regression ...

View answer
Riskscore creation on Numerical Data
0 votes

There are traditional ways of creating a risk scorecard using linear regression techniques. It's a whole topic in itself. A good book for beginners to read in-depth about it would be this. For sure, ...

View answer
Find recurrent dates in a small set (and get rid of non recurrent ones)
0 votes

You can use clustering algorithm to cluster closer dates together. But since you've mentioned the number of dates to be clustered won't be more than 20, seems like you can just create a simple logic ...

View answer
Will cross validation performance be an accurate indication for predicting the true performance on an independent data set?
0 votes

It is possible. Think of a simple scenario where model M1 has learnt the variance of the training dataset D better than model M2 since it's parameters are better tuned. This means M1 performs better ...

View answer
Use of Correlation score
Accepted answer
0 votes

Correlation should be as less as possible between different features, because correlated features mean that those features are giving out same kind of information/trend for the predictor to learn. ...

View answer