You can process the data with dynamic time warping (DTW). DTW has shown to be effective in mapping different speeds of gait to the same time grid. Once the data is mapped to the same time grid, standard time series analysis can be applied.
I would recommend to use UMAP. It is a superior algorithm to both of them:
YOUTUBE: UMAP explained | The best dimensionality reduction?
Publication: UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
Guide: How to Program UMAP from Scratch
CODE: GitHub - NikolayOskolkov/HowUMAPWorks: Here I explain the math behind UMAP and show how ...
Imbalanced data is a hot topic and in my opinion there are a couple misconceptions around.
For the metric: You should always be aware of the meaning of the metric you are using and the ratio of classes.
For the optimization, Unless your ratio is more than 1:999 I wont consider changing the optimization of your algorithm (and in my opinion never use synthetic ...
You have 3 variables that refer to a particular station.
In your training set, you have only one station? -- If yes then the best is to drop them
In your training set, you have more? -- Then they can have distinct values so leave them.
If your training set has one station, and then in your test set you have another station -- Drop them. Your model wont be ...
You can calculate the text-similarity using Transformers. With transformers, we can get better accuracies. Try the following code:
pip install sentence-transformers==1.2.1
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('distilbert-base-uncased')
sen = [sentence1 , sentence 2]
sen_embeddings = model.encode(sen)
Disclaimer: please bear in mind that I'm no expert in this kind of application.
How could I be sure this is not caused by the training error?
You can be sure because the evaluation process is valid. Errors are expected in any ML process, what matters is to correctly estimate the expected level of error (performance evaluation).
As far as I can tell, your ...
One option is to look at the set membership information associated with "BAD" target value.
Which quality numbers are only associated with that label? Which quality numbers are never associated with that label? Which quality numbers are associated with that label and other labels?
Yes, but one must use a proper representation of the numerical range: this is an ordinal variable, i.e. it has an order but it's not continuous. Common options are:
A range can be represented as a simple integer: 1 for the first range, 2 for the second range, etc.
Or it could be represented with a value representing the range, for instance the middle of the ...
Here is a good explanation of Gini impurity: link. I don't see why it can't be generalized to multinary splits.
The binary split is the easiest thing to do (e.g. discussion: link). That's why it is implemented in mainstream frameworks and described in countless blog posts.
A non-binary split is equivalent to a sequence of binary splits (e.g. link). However, ...
There are multiple ways you can do this some of which are:-
1.) Use L2 Regularization to determine which out of the 5 features are contributing
more in predicting the target.
2.) Use Tree based and gradient boosting based model to calculate the feature importance of each features. Here is an article that uses various models to calculate the feature ...
You don't need to form Regression equations and then check for the coefficients. Instead, you should do the following:
Check if the data is normally distributed:
If it is normally distributed then you can run t-test. This will tell you if the difference between both of the groups are statistically significant or not.
If the distribution isn't normal then ...
You can model your problem with Bayes' theorem. In particular, naive Bayes classifer can be used for binary classification of text data.
Priors in a naive Bayes classifer refers to the base rate for the different classes (i.e., Are there more English or French sentences?)
Naive Bayes may or may not be the best solution. Typically, the best solution is chosen ...
In essense all ML models are numerical algorithms which correlate numbers in various ways in order to arrive at other numbers.
So, backstage, all algorithms somehow use only numerical data, even if they do not force the programmer to do so from the start (like CARTs, for example).
Some models, eg all types of Neural Networks, require the programmer to ...
One option is encoding each name as a separate input node and the activation of that node would be the standardized value (scaled between 0 and 1). This would convert the frequency to rate. If the total number of names is 8723, the Madura input node would have a value of 0.12197 (1064/8723).
You need to distinguish between uncertainty on the prediction and uncertainty on the class.
One example, lets say that we are tossing a coin. I am 100% confident that the probability of getting "tails" is 50%
On the other hand, there is a 90% probability that tomorrow will rain but the weatherman is not very certain of this to happen.
To get this ...
Regarding your first question, no your dataset is not small to give you bad results, although adding more data would definitely help. I have worked with smaller datasets than yours.
The answer to your second question depends on weather you have performed feature selection/engineering. Usually the best type of feature selection is with the help of domain ...
multimodal learning can be complex (like anything) but it can also be fairly simple.
The general idea of multimodal modeling is to take data consumed in parallel, which has different "modes" which are very different to one another (like audio, video, and a text description) to predict something (if the video is about cats, for instance). This can ...
The simplest option in order to represent the two sentences independently of each other is to represent each of the two sentences with its own TFIDF vector of features and concatenate the two vectors. In other words you obtain 2 * N features where N is the size of the vocabulary.
But at first sight it looks like the wrong approach for the problem that you're ...
1.) Faster than RF.
2.) Relatively easy to interpret than other algo's, although most of the
algorithms are interpretable more or less.
3.) Easy to visualize.
4.) Can have control over feature selection if you don't want to go over the
filter based or wrapper based feature selection.
5.) Easy to implement.
6.) Easy to tune the hyperparameters.
In principle, I do not see any problem with using the RFM score as dependent variable ($y$) since it is just an aggregate or balanced score of R, F, M.
I suggest using random forest or (tree based) boosting (like xgboost, lightgbm), since these techniques are very robust and usually deliver relatively good results (compared to other methods). You can look at ...
In linear regression models (aka OLS), you can interpret the estimated coefficient(s) as the percentage change in case $y$ and $x$ are log-transformed using the natural log (see this post "Both dependent/response variable and independent/predictor variable(s) are log-transformed" or see also this post).
For a model like:
$$ log_e(y) = \beta_0 + \...
The main reason for data preprocessing is to ensure that the datasets are formatted in such a way that the data they contain can be interpreted and parsed by the machine learning algorithms.
As you train the machine learning models with training data and predict with machine learning models with the test data.
Data preprocessing has to be applied to both ...
The accuracy is likely to go down if you change the cutoff point to 0.9, since any model tries to separate the classes so that the probability of the correct class is higher than 0.5. But the only way to know would be to actually do the experiment (I assume that the results that you show are obtained with the default cutoff).
AUC is a complex measure for a ...
Consider the input sentence I am good. In RNNs, we feed the sentence to the network word by word. That is, first the word I is passed as input, next the word am is passed, and so on. We feed the sentence word by word so that our network understands the sentence completely.
But with the transformer network, we don't follow the recurrence mechanism.
Your goals include two criteria that interact and may conflict. It is not possible to write a single reward function to solve this perfectly. You have to decide first on relative importance of the two goals. As one is effectively a constraint, you need to decide on how hard you want to apply this constraint.
As the revenue is easy to measure, and already ...
Preprocessing is needed for both train and test sets. But you should be aware of data leakage, meaning no information from the test set should be used to preprocess the training set.
For example, if you are trying to apply One-Hot encoding to your classification labels you should train the encoder (e.g. sklearn.preprocessing.OneHotEncoder) on training set ...
First of all, in problems like you brought, you typically start with preprocessing. Specifically, in your case, you need to normalize it. That means, using some processing, you have to change both Are you there and are you there??? to just are you there. By doing that, you are removing duplicate examples there. Now, unlike Erwin, I suggest that in general. ...
1.) You only get a limited number of algorithms and preprocessing steps to work with in Azure ML. This is really not a disadvantage as you can use Azure ML Python SDK to build your own model (just like you would on your laptop).
1.) You do not need to buy a high end computer if your dataset is large. You can use a budget laptop ...
In general it's not recommended to get rid of duplicates because it modifies the distribution of the data and this could bias the model. In other words, if the final application (or any test data) is expected to contain cases like these in similar proportions then it is preferable to train the model with these cases.
So the duplicates by themselves are not ...
Let's stick with reducing 3 neurons to 2 neurons for simplicity (the mechanism will be the same for any number of neurons). Take the image below (taken from a StackOverflow post) as an example.
Consider the transition from the second layer with 3 neurons to third layer with 2 neurons. All that happens is that the output of each of the 3 neurons of the 2nd ...
When you have a case of missing not at random, the best thing to do is create a new feature. You can add a new feature which has value 0 if the person doesn't own a house and 1 if he owns a house.
Also K-means is usually not used for clustering. You could go for other algorithms such as Hierarchical or DBScan.
You can either use k-means or Hierarchical clustering for your use case.
Hierarchical clustering method works via grouping data into a tree of clusters
TYPES of Hierarchical Clustering
Agglomerative : An agglomerative approach begins with each observation in a distinct (singleton) cluster, and successively merges clusters together until a stopping criterion ...
You do not need to plot data first to fit k-means clustering. You can fit k-means clustering and then decide to plot it or not.
K-means clustering requires a distance metric, typically euclidean distance. Euclidean distance requires continuous-valued features. It appears that only "Time" and "Length" are continuous-valued. The other ...
According to this article, both of the learning(Online and incremental) methods aim at learning (updating) a model when the data comes on the fly to obtain the same
model as the one learned in a batch setting (i.e. on static data). The difference is that on-line
learning learns a model when the training instances arrive sequentially one by one (1-by-1),
Code Interpretation: Lets try to interpret from innermost to out
training <- training[, 6:dim(training)]
dim() is an inbuilt R function that either sets or returns the dimension of the matrix, array, or data frame. The dim() function takes the R object as an argument and returns its dimension, or if you assign the value to the dim() function, then ...
This can be understood with a simple example
Let's create a data-frame for this:
a <- runif(20,5,100)
b <- runif(20,51,100)
c <- runif(20,57,89)
d <- runif(20,41,78)
e <- runif(20,3,133)
f <- runif(20,74,700)
g <- runif(20,45,100)
h <- runif(20,69,400)
training <- data.frame(a,b,c,d,e,f,g,h)
Now, let's see how dim ...
(Answering just the first part of the question, based on my own understanding.)
Multi-head attention is a randomly-initialized multi-dimensional indexing system where different heads focus on different variations of the indexed token instance (token = initially e.g. a word or a part of a word) with respect to its containing sequence/context. Such encoded ...
That is problem commonly called time series anomaly detection / outlier detection. Most systems start with combinations of static and dynamic thresholding. Dynamic thresholding for example can use percentile value as the threshold. One common algorithm is Isolation Forest where the features can including different length moving average windows.
One of the ...
It's the usual XGBoost boosting, but with linear models instead of decision trees as the base learner.
There are several questions about this over at stats.SE; here's a quick sampling:
What exactly is the gblinear booster in XGBoost?
How does linear base learner works in boosting? And how does it works in the xgboost library?
Difference in regression ...
Interesting question. I'd say it is correct not to divide, due to the following reasoning...
For linear regression there is no difference. The optimum of the cost function stays the same, regardless how it is scaled.
When doing Ridge or Lasso, the division affects the relative importance between the least-squares and the regularization parts of the cost ...
Most of the work done in data science is experimental in some way (from data acquisition to a model being used to do something), partially due to how most techniques work (hyperparameters, distributional assumptions, etc.), after all most models are very tied to the task they were built to solve, so they don't translate very well even to tasks that are very ...
Although both methods are correct, it's better to train the old model since we already trained one model, and starting over from scratch would cost more. The old model already had optimal weights, so with new data, it only needs to fine-tune the model.
The model performance is not affected by these approaches as long as we have the same distributed train ...
I finally solved this by using the Virusshare website. It has millions of malwares, and is free.
Note that around 1-2% of their PE files are probably benign, meaning less than 1-2 detection on VirusTotal, so just labeling every single PE file as malware might not be academically complete.