32

Python is more "general purpose" while R has a clear(er) focus on statistics. However, most (if not all) things you can do in R can be done in Python as well. The difference is that you need to use additional packages in Python for some things you can do in base R. Examples: Data frames are base R while you need to use Pandas in Python. Linear ...


20

In linear regression overfitting occurs when the model is "too complex". This usually happens when there are a large number of parameters compared to the number of observations. Such a model will not generalise well to new data. That is, it will perform well on training data, but poorly on test data. A simple simulation can show this. Here I use R: ...


13

This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the split at 3. in your flow chart. A common step I think you have missed is imputation of missing values. I would put that before feature engineering. Overall I ...


13

Python being more widely used is an important consideration. This will especially become important when applying for a job. Also Python has as many if not more key statistical and ML/AI tools as R, and a larger open-source base to utilize. Python is designed for programmers, R is designed for statisticians. Originally I was a R programmer, but most of my ...


12

Assuming that these results are obtained on a valid test set with no data leakage, these results don't show overfitting because overfitting would cause great performance on the training set but significantly lower perfomance on the test set. Make sure that your instances between the training and test set are truly distinct: there might be some data leakage, ...


11

Simple explanation with images We know that an activation is required between matrix multiplications to afford a neural network the ability to model non-linear processes. A classical LSTM cell already contains quite a few non-linearities: three sigmoid functions and one hyperbolic tangent (tanh) function, here shown in a sequential chain of repeating (...


11

The topic you are interest in is called "PU learning" or "positive and unlabeled learning". You can start by having a look into survey literature.


10

The X axis is the number of instances in the training set, so this plot is a data ablation study: it shows what happens for different amount of training data. The Y axis is an error score, so lower value means better performance. In the leftmost part of the graph, the fact that the error is zero on the training set until around 6000 instances points to ...


10

An easy way to think about autoencoders is: how well a prticlar pice of infrmaton can be reconstrcted frm its reducd or otherwse comprssed reprsentaton. If you made it this far it means that you sucessfully reconstructed the previous sentence by using only 92 of its original 103 characters. More specifically, autoencoders are neural networks that are trained ...


10

what does an auto-encoder do? The simplest auto-encoder takes a high dimensional image (say, 100K pixels) down to a low-dimensional representation (say, a vector of length 10) and then uses only those 10 features to try to reconstruct the original image. You can imagine an analogy with humans: I look at someone, describe them ("tall, dark-haired, ...&...


9

To decide which strategy is appropriate, it is important to investigate the mechanism that led to the missing values to find out whether the missing data is missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR). MCAR means that there is no relationship between the missingness of the data and any of the values. MAR ...


9

You don't always need 3 separate datasets. You usually split a dataset into 3 if you are doing some parameter or hyperparameter tuning before choosing a final model. Tuning will usually add bias from the 2nd dataset into your model, decreasing it's performance. For instance: If you are manually tuning a model over several iterations and using the results ...


8

Why is Multicollinearity a Potential Problem? A key goal of regression analysis is to isolate the relationship between each independent variable and the dependent variable. The interpretation of a regression coefficient is that it represents the mean change in the dependent variable for each 1 unit change in an independent variable when you hold all of the ...


8

As the other answers already state: Warmup steps are just a few updates with low learning rate before / at the beginning of training. After this warmup, you use the regular learning rate (schedule) to train your model to convergence. The idea that this helps your network to slowly adapt to the data intuitively makes sense. However, theoretically, the main ...


8

The changes in distribution with respect to training time are sometimes referred to as concept drift. It seems to me that the amount of information available online about concept drift is not very large. You may start with its wikipedia page or some blog posts, like this and this. In terms of research, you may want to take a look at the scientific ...


8

Other options would be to... Compare similar text sequences, Compare similar string sequences, Use fuzzy matching. Fuzzy Matching: library(fuzzyjoin) # https://stackoverflow.com/questions/26405895/how-can-i-match-fuzzy-match-strings-from-two-datasets a <- data.frame(name = c('Ace Co', 'Bayes', 'asd', 'Bcy', 'Baes', 'Bays'), price = c(10,...


8

A weak learner can be either a classification or a regression algorithm: Boosting (Schapire and Freund 2012) is a greedy algorithm for fitting adaptive basis-function models of the form in Equation 16.3, where the $\phi_m$ are generated by an algorithm called a weak learner or a base learner. The algorithm works by applying the weak learner sequentially ...


8

This is a very ambitious project. First it's important to realize that ML cannot really solve this kind of problem in general, it can only help detect the posts which are likely fake news (see for example this other post about measuring credibility, i.e. the same question seen the other way around). Assuming you work on the text of the message (I'm not ...


8

What are Bias and Variance? Let's start with some basic definitions: Bias: it's the difference between average predictions and true values. Variance: it's the variability of our predictions, i.e. how spread out your model predictions are. They can be understood from this image: (source) What to do about bias and variance? If your model suffers from a bias ...


7

Let's assume that we are talking about 2D convolutions applied on images. In a grayscale image, the data is a matrix of dimensions $w \times h$, where $w$ is the width of the image and $h$ is its height. In a color image, we normally have 3 channels: red, green and blue; this way, a color image can be represented as a matrix of dimensions $w \times h \times ...


7

Not every seed is the same. Here is a definitive function that sets ALL of your seeds and you can expect complete reproducibility: def seed_everything(seed=42): """" Seed everything. """ random.seed(seed) os.environ['PYTHONHASHSEED'] = str(seed) np.random.seed(seed) torch.manual_seed(seed) torch.cuda.manual_seed(seed) ...


7

Practically everything related to statistics (including Machine Learning) has to do with studying chance, i.e. trying to determine to what extent an observation is due to chance or not. For example one might want to know whether a drug actually helps with a particular disease or not. If we observe that one patient improves after taking the drug, there's ...


7

This is an implementation detail, and I wouldn't necessarily rely on this behavior, but presently in sklearn, it will choose the "first" class. The predict method calls for the probability prediction, then takes the argmax, which in case of ties takes the first one: https://github.com/scikit-learn/scikit-learn/blob/fd237278e/sklearn/tree/_classes....


7

The shaded area likely shows the dark green line plus or minus some error/uncertainty estimate. Common error estimates may be based on the standard deviation, a confidence interval, or the interquartile range depending on the data and the analysis being done. Without more information, we cannot know what the shaded area represents.


7

The direct way to check your model for overfitting is to compare its performance on a training set with its performance on a testing set; overfitting is when your train score is significantly above your cv score. According to your comments, your r2 score is 0.97 on the training set, and 0.86 on your testing set (or similarly, 0.88 cv score, mean across 10 ...


7

In the paper of Hinton - Distilling the knowledge of Neural Networks, the following is mentioned (Section 5) when defining specialist models: When the number of classes is very large, it makes sense for the cumbersome model to be an ensemble that contains one generalist model trained on all the data and many “specialist” models, each of which is trained on ...


7

The cleanest solution would be to apply scikit's OneHotEncoder with the handle_unknown parameter set to "ignore": handle_unknown{‘error’, ‘ignore’}, default=’error’ Whether to raise an error or ignore if an unknown categorical feature is present during transform (default is to raise). When this parameter is set to ‘ignore’ and an unknown category ...


6

Check this post. In the cases where the frequency is related somewhat with the target variable, it helps the model to understand and assign the weight in direct and inverse proportion, depending on the nature of the data. Check also this thread. What's the rationale behind it? High cardinality may result in dimensionality curse and actually decrease ...


6

Feature selection is a combinatorial optimization problem. And genetic algorithms is an optimization technique. So there really isn't anything special, you just need to formulate your problem as an optimization one, and understand how do genetic algorithms optimize. There are enough tutorials on this. Whether it's better or worse you already know the ...


6

If you want to move from theory to application then I suggest to do exactly that: get your handy "dirty"! UCI Machine Learning Repository has some easier datasets to get started. Kaggle is great too but before going for any competition look for an easier dataset from their repository. If you prefer something with more guidance the book "Introduction to ...


Only top voted, non community-wiki answers of a minimum length are eligible