Theudbald
  • Member for 5 years, 8 months
  • Last seen this week
Why is the cost increasing in the linear regression method?
Accepted answer
6 votes

The reason is your learning rate alpha is too large for this optimization problem. Start with a really small value (< 0.000001) and you will observe a decrease in your cost function. Keep in mind ...

View answer
Is there any use to running Pandas on Spark?
6 votes

There is no need for pandas module to be installed because your data is generally stored in spark RDD or spark dataframes objects. The only interest I have found using Spark with pandas is when you ...

View answer
Improve Precision of a binary classifier - Decision Tree in Python
Accepted answer
5 votes

I guess differences in accuracies between class 0 and class 1 come from the class_weight parameter you have used. Class 1 will benefit from this overweighting towards class 0. You could try to play on ...

View answer
What are examples for XOR, parity and multiplexer problems in decision tree learning?
Accepted answer
5 votes

Below is an example of XOR dataset for classification. As you can see, decision trees perform pretty poorly on this dataset. Reason is decision trees splits space into rectangular regions. Therefore ...

View answer
What are limitations of decision tree approaches to data analysis?
5 votes

Simple decision trees have some limitations listed below. Fortunately, some of these can be fixed used ensemble learning techniques (think bagging, boosting...). Concerning limitations : Trees tend ...

View answer
Error 'Expected 2D array, got 1D array instead:'
Accepted answer
4 votes

This is a bit tricky. Using pandas data, sklearn only accepts input variables (features) with type pandas.Dataframe. In your code variable sex_train in pandas.Series type. Try the following code : ...

View answer
General Machine Learning Workflow Question
Accepted answer
3 votes

I agree with most of the answer. However, I think you are missing some points including the cross-validation step. I try below to provide an overview of a common machine learning project. I assume a ...

View answer
API to find out how many executors are running my Spark jobs?
3 votes

Keep in mind that the number of executors is independent of the number of partitions of your dataframe. You set the number of executors when creating SparkConf() object. https://spark.apache.org/docs/...

View answer
Weights in neural network
Accepted answer
2 votes

In parametric models such as linear regression, logistic regression and multi-layers perceptrons, weights are updated with regards to the "difference" between the output of your model and the real ...

View answer
When using numerical duplicates for categorical data, new columns should be added or values be converted?
Accepted answer
2 votes

It depends of the algorithm you are using. For a linear model (linear / logistic regression, SVM...), you need to create dummy variables meaning features "Sex_M" and "Sex_F" as you noticed. However, ...

View answer
Why Liblinear performs drastically better than libsvm linear kernel?
2 votes

Here is just a guess, but according to me, the linearSVC might perfoms better than SVM with linear kernel because of regularization. Because linearSVC is based on liblinear rather than libsvm, it has ...

View answer
Spark development on local machine with PyCharm
2 votes

In my opinion, I recommend to use the following approach to develop spark jobs for a big Data Context : First, develop your spark in local mode on your computer. Use a simple subset of data from your ...

View answer
Interpreting Machine Learning Classification Metrics
1 votes

I assume the results you show have been evaluated according to a train-validation-test split approach. With the information you have provided, it is possible to figure your confusion matrix. It has to ...

View answer
Understanding regularisation and a preference for small weights
Accepted answer
1 votes

Ridge or L2 regularization is used to prevent over-fitting when having multi-col-linearity in your features. Here is an example : suppose you want to train a linear regression model on following ...

View answer
Use a dataframe of word vectors as input feature for SVM
Accepted answer
1 votes

In my opinion, scikit-learn raises an error because updated_df is composed of 2 features (columns) with list formats. Therefore, for a given observation x_i : x_i = [arg1_i, predicate_i] = [[...

View answer
For what condition boosting work better than bagging in Ensemble Learner?
1 votes

Bagging (and features sampling) aim to reduce variance by providing low-correlated trees. Estimators can then be aggregated together to reduce variance. Reason is simple decision trees tend to quickly ...

View answer
Bootstrapping or Randomly Dividing Dataset to reduce variance?
Accepted answer
1 votes

I think the second method will yield less correlated models than the first method. It is particularly true with decision trees which tend to quickly overfitting in the bottom nodes. It will help ...

View answer
Which one of these tasks will benefit the most from SPARK?
Accepted answer
1 votes

I think second job will benefit more from spark than the first one. The reason is machine learning and predictive models often run multiple iterations on data. As you have mentioned, spark is able to ...

View answer
Simple logistic regression wrong predictions
1 votes

This is because you have an imbalanced dataset towards class 0. I have taken a look on the logistic regression coefficient you get. On the below chart 1, I have plotted the decision boundary you get ...

View answer
How do I use recency/time as a feature after I've a word2vec model
1 votes

The purpose of the Word2vec is it will itself learn a hidden structure in your text data. If "recovery" comes after rap and music is because Eminem is more frequent with these context words than with ...

View answer
Pyspark code is not performant enough when compared to pure python alternative
1 votes

You have instantiated a sparkContext object with "local" mode configuration. It means you have allocated ressources for a single multi-core Java Virtual Machine on your computer. In this configuration,...

View answer
Product Recommendation based on purchase history
Accepted answer
1 votes

Even if you don't have ratings or reviews, you can use the customer purchases to help creating your model and selecting the most appropriate one. If the customer has bought the product, you can ...

View answer
How to fix Sagemaker's "No finished training job found associated with this estimator" error?
0 votes

I think that you have to train your model first. To do that in sagemaker, you can do the following. I assume that you want to train a machine learning model without using a python script as an entry ...

View answer
Ridge regression - varying alpha and observing the residual
0 votes

I have taken a look on your code. You obtain same errors results for each alpha value because your regularization strength is too small. Replacing : alphas = np.logspace(-40, -18, n_alphas) with : ...

View answer
Set for building ROC curve and and choosing logistic regression cut-off
Accepted answer
0 votes

Usually you need to generate the ROC curve and choose the threshold within the training data. Then, with the selected threshold, you have the possibility to report accuracy, sensitivity, recall ...

View answer
Evaluating machine learning model with missing features
Accepted answer
0 votes

If less than 20-25% of behavioral data is missing, maybe you could try to impute missing data using one of the following solutions : Impute missing behavioral data using some business rule or by ...

View answer
Apache Spark Question
0 votes

Using Spark, there is no notion of "mappers" or "reducers". Each task you perform is Spark is achieved by executors (JVM with allocated ressources). Executors also have the ability to split themselves ...

View answer