New answers tagged classification
0
There are a number of Machine Learner explainers and diagnostics.
Let's set up a sample problem.
Mnist is a fair dataset, so let’s first use a random forest to describe it, and then vivisect the learner to understand what, why, and how it works.
Here is code for mnist using h2o.ai:
Here is how it did. The fastai example uses
(note: this answer is mid-edit)
1
This approach makes some sense but it's not the best approach for several reasons.
First, this control variable might not always be last in importance because it's possible that some other variable also don't have any impact at all on the target variable (outcome).
More importantly, the concept of a control group/control variable is useful in cases where one ...
0
First you have to understand what are those records. What is the mean of each number from the sensor. If the order of each number is useful or useless. Then, you have to determine the number of input you've got, and how to make them more comprehensive.
And then, you've got to build a classification. But, in those kind of case, I think it would be imbalanced.
...
0
Well, sometimes the features simply do not provide enough information to get 100% accuracy (like in your case), even with a model a flexible as the Decision Tree.
The Decision Tree works by trying to split the data using condition statements (e.g. A < 1), but how does it choose which conditional statement is best? Well, we want the splits (conditional ...
1
I tried with different features: in one dataset with less features engineering, i.e., using only features from Text, I got a maximum value of F1-score equal to 68%. With more features, that I thought to be significant for improving the model, I am getting max 64%, that is weird considering the problem (email classification for spam detection).
Typically ...
1
About the question whether to scale only a subset of features, I would tell you to do it over all the features (at least the continuous numeric ones) since the goal of data-scaling is to put these data on the same "reference scale" to be fairly compared.
Nevertheless, having mixed data types (continuous numerical, categorical...) for your ...
0
So based on the definition we need to find $2^n$ arrangements that can be shattered by a linear boundary, which you have shown in the figure. Therefore, it does not matter if there are other arrangements that cannot be shattered.
For $n = 4$ we cannot find $2^n = 16$ arrangements that can be shattered by a linear boundary hence we say we cannot shatter $4$ ...
0
Well, sometimes the features simply do not provide enough information to get 100% accuracy (like in your case), even with a model a flexible as the Decision Tree.
The Decision Tree works by trying to split the data using condition statements (e.g. A < 1), but how does it choose which condition statement is best? Well, it does this by measuring the "...
0
@Erwan probably gave you a better idea, but for a simpler problem, when each token has a separate class, this can be viewed as a Part of Speech Tagging, which is essentially a per-token multiclass problem. This assumes each token can be mapped only to one correct class. In such case the LSTM's output would be size, e.g. (1, seq_length, hidden_dim), which you ...
0
If in each file the letters are spoken separately, with silence in between, and always in the same order (A,B...,Z) then one can try to automate finding each section and its label.
Use a Voice Activity Detection (VAD) module to detect each spoken character. Then assign A to the first voiced area, B to the next etc. If you have very clean and uniform audio, ...
0
There are several different ways to frame the problem.
One way is multiclass classification. The goal would be to assign a single discrete label to every phrase. In order to get phrases, you'll have to build a parse tree first. You did not list of all of the labels but let's assume they are all nouns. Then you'll need a Part-Of-Speech Tagger (POS Tagger) to ...
0
Generally, feature selection is somewhat of a fuzzy process. Since you usually don't have a ground truth in predicting biology, you will always have to consider how realistic whatever you came up with is. I would recommend to start with the most simple method and see how your model performs. After establishing this as a baseline, you can then explore other ...
1
Misclassification error will not help in splitting the Tree.
Reason-We consider the weighted dip of error from parent Node to the child node and misclassification error will always result in 0(Other than pure splits).
Let's consider an example
Data = 1, 1, 0, 1, 0, 1, 0, 1, 0, 1
Parent Classification error= 4/10 = 0.4
Parent Gini Impurity = 1-(0.4x0.4+0.6x0....
1
Technically this is sequence labeling, the most common application being Named Entity Recognition.
However it looks like in this case you're trying to solve a problem of coreference resolution, which is a quite difficult task in general. I think this usually involves a more complex model than simple sequence labeling, but I'm not an expert in this. You might ...
1
have you already tried using only very little of the -ve cases? So for example to train your model on 900 points total, 600/300? Then stratified sampling should still work fine. Then I'd evaluate your model based on it's ability to predict -ve cases and just monitor the performance it get's on the (in your case) gigantic test dataset that the model hasn't ...
0
Since I only had a gut feeling about this, I decided to implement it in code. For this, I followed the method described at https://victorzhou.com/blog/gini-impurity/. Generally, calculating the GI provides you with a metric for which you don't have to know the underlying distribution which I think is the reason why your example works.
Generally, my ...
1
There are some distance/dissimilarity functions for two binary/boolean vectors such as Jaccard distance and Hamming distance. You can check them from scipy's documentation. You can calculate similarities between two boolean columns by using these distance functions.
from scipy.spatial import distance
distance.hamming([1, 0, 0], [0, 1, 0])
0....
0
In general, the further away the green line is from the red line, the more the model is overfitting, however eventually enough data will cure all overfitting (there will be so much data the model can't possibly memorize all of it), and that's why the lines converge to being together (stops memorizing, red line goes down, starts generalising, green line goes ...
0
I over complicated the answer I was looking for. If you have the equation of the hyperplane, you can test it against each point in the dataset. If all points of class 1 are ALL (above or below) the intercept and the converse is true for all points of class -1. Then the hyperplane separates the two classes
0
The solution for my problem was implementing Batch Renormalization: BatchNormalization(renorm=True). In addition normalizing the inputs helped a lot improving the overall performance of the neural network.
1
i did it using matplotlib by showing the percentage of each country [ target class] is taking
ratio is almost 62:18:19 which shows its a balanced data set
if u want the proper code comment down
1
Answering your questions, it is important to remember that this kind of transformations, in this case the change in the minority class distribution by oversampling or in the majority class by undersampling, must be only done in the training dataset, so you do not alter the real situation when you want to apply your model (which, at training time, is your ...
1
Do the undersampling after the train/test split and only in the train split: you want to somehow "weigth" your learning algorithm in order to prevent it to be biased towards the majority class, indeed, other techniques can be applied, and always on the train split, what eventually is the data for the learning algorithm. But... your test set should ...
1
The fact that a feature has low correlation with the target variable shows that it's not a good indicator on its own, but that doesn't mean that it can't be useful for the model when combined with the other features.
The only way to know if these features are useful is to use them to train a model, then evaluate on a validation set and see if it improves ...
1
I will raise several issues that could arise if you are selecting features based on chi-2 tests
Repeated use of chi-2 test can lead to spurious results unless you correct for the number of times you run it
You can include features that are correlated with each other, i.a. A is correlated with B, and both are correlated with label. Not sure, but I think, ...
4
What this shows is that the protocol is not a very discriminative feature:
the probability of class 1 given http is 10/(109+10)=0.084
the probability of class 1 given https is 180/(180+1560)=0.103
If these conditional probabilities were very different this feature would be more helpful to predict the class, but they differ only slightly. Note that the ...
1
A common approach is what you suggest in 1. - apply time-shift as a Data Augmentation strategy. The augmentation is generally beneficial with deep learning models, and GPUs are fast so the compute time is rarely a big problem.
Another strategy, less common, would be to make sure that the event is always located at the same position inside the analysis window ...
1
Correct, all categorical features should be encoded into binary digits so ML algo have more predictive power as categorised features cannot have order or magnitude ( be careful also about
multicollinearity if you use regression)
Some ML framework such as Catboost automatically encode features for you if you specify the feature index. I also like using ...
0
Please, edit your figure to be more concise... adding axis labels will help a lot. I guess Y-axis is the target label, and X-axis is one of the nine parameters. But looking at the plot, I wondering if you are plotting the labels of a dataset with more than 10K samples...
Anyway, regarding the scatter plot of your nine parameters, which I belive they are the ...
0
Have a look at the comment in that notebook:
# Define preprocessing for categorical features (encode the Age column)
It seems that the data you have is different (maybe only in order) to what is used in the notebook as the Age column is the eigth column (index seven) in the notebook. See also the cell where data is selected from the input file:
# Separate ...
0
using lstm or rnn's for time series data is like using a hammer to swat a fly. have you tried time series modeling using classical stat techniques ARCH, ARIMA etc ? the issue of using individual number as inputs (which is what your speedometer is going to give you) means that the states in each lstm / gru cell or unit will have like a 1x1 matrix , meaning 1 ...
0
Gated recurrent units (GRUs) is an advanced version of RNN and LSTM. It has less time complexity and good prediction results than LSTM ann RNN. You can use it for dealing with time series dataset.
You can boosted it performance by combining advanced version 1D CNN like ResNet, etc.
1
The learning rate is one of those first and most important parameters of a model, and one that you need to start thinking about pretty much immediately upon starting to build a model. It controls how big the jumps your model makes, and from there, how quickly it learns.
There are learning rate technique called Cyclical Learning Rates.
Training with cyclical ...
0
Creating a new variable as you described would be redundant as it is a function of the other two variables. In other words it is not adding any information.
The below suggestion assumes the model cannot deal with missing values, but a lot of the best models (ex. xgboost is typically one of the best for classification) will deal with this in a smart way for ...
0
Every Tree gets its OOB sample.
So it might be possible that a data point is in the OOB sample of multiple Trees.
oob_decision_function_ calculates the aggregate predicted probability for each data points across Trees when that data point is in the OOB sample of that particular Tree.
The reason for putting above points is that OOB will give you the mean of ...
1
First of all, I do not think you need a DNN if task is as simple as you are describing it. There will be lot of digital image processing solutions or maybe check opencv-python (cv2). But, if you are particularly interested in DNN, I would suggest starting with Keras. According to me it should not be too tough to perform this task as CNN can do complex jobs ...
0
You can look at this article. Helmreich, James & Pruzek, Robert. (2008). PSAgraphics: An R Package to Support Propensity Score Analysis. Journal of Statistical Software. 29. 10.18637/jss.v029.i06.
In section three, they show an example of how you can estimate propensity scores and stratify the data based on them.
There is also a good example in Valliant, ...
0
import numpy as np
import matplotlib.image as img
import matplotlib.pyplot as plt
from skimage.metrics import structural_similarity as ssim
def rgb2gray(rgb):
return np.dot(rgb[...,:3], [0.299, 0.587, 0.144])
full_image = rgb2gray(img.imread("img/full_image.png"))
sub_image = rgb2gray(img.imread("img/sub_image.png"))
full_w,full_h =...
Top 50 recent answers are included
