4

The real problem is that you should not try to fit all your images in memory. Instead, you should small groups of images, normally called "minibatches", which can fit in the GPU/CPU memory. For that, tensorflow offers the function tf.keras.preprocessing.image_dataset_from_directory that loads images from a directory. I suggest you take a look at ...


3

The principle in supervised ML is quite simple: the "method" which is going to be used to predict the response variable must be fully determined from the training set and only from the training set. In other words, anything which doesn't belong to the training set cannot be used. As a consequence, feature engineering, i.e. choosing how to prepare/...


3

Is this approach a correct approach, or logical with respect to machine learning principles? It will affect the performance of the model in the sense that your algorithm learned to separate the clusters based upon distance according to all the features. I have read discussions about how to calculate feature importance on unsupervised problems like yours, so ...


3

I would recommend looking into FBProphet. It's a good starting point for automating the creation of forecasts. It's very easy to use, and often produces better results than classical forecasting methods (ARIMA, Holt-Winters, etc.) right out of the box. The default settings offer an additive or multiplicative model, comprised of trend and seasonality. This ...


2

I am afraid that such situations are fundamentally inherent in predicting/forecasting contexts; quoting from the very recent paper by Taleb et al., On single point forecasts for fat-tailed variables (open access, para 3.7): 3.7. Forecasts can result in adjustments that make forecasts less accurate It is obvious that if forecasts lead to adjustments, and ...


2

How about using the difference of an exponential and a linear or a polynomial functions? For example, if $x = actual - predicted$, then $$L = w_1 (e^x-1) - w_2 x$$ The cost of error increases faster for positive values than for negative ones, and is a differentiable function of $x$. You can tweak $w_1$ and $w_2$ depending on your typical errors and your ...


2

Personally I think linear (through model's coefficients/weights) and tree-based models (gain importance) are the best for explainability But this is not restricted to those models since you can use model agnostic techniques to explaine any model, even those consider as "black-box" Like: SHAP Values Partial Dependence plot LIME You can check this ...


2

An approach that is not specific to the image domain is to use a probabilistic data structure like a Count-Min Sketch. A Count-Min Sketch data structure can accumulate information to estimate the observed frequency of an input value based on the past set of input values by using multiple hashing functions over the input.


2

Taking the difference (ie speed1-speed2) as the target variable effectively dismisses any low-frequency variablitiy and targets only high-frequency variability, even noise. One approach would be to bin the (highly-variable) target variable into fixed range bins and take the mid point (or any other fixed point) of each bin as the new target (stabilised) ...


2

One option would be to transform the y / target variable to be distributed more like a Gaussian, the most common transformations are log and quantile transformation. Gaussian transformation often increases the model fit statistics.


2

Everybody understands how to perform $k$-fold cross-validation but there is often quite a lot of confusion about where/how to use it. So thanks for this good question :) First, cross-validation is a statistical method for evaluation, not for training: Of course training is performed during cross-validation, but it is performed $k$ times and therefore there ...


2

Since you specifically mention Python, one option is the Prophet package. The model fitting would be something like: # Create the pandas DataFrame import pandas as pd data = [['2021-01-01', 11, 20, 30], ['2021-01-02', 22, 40, 60], ['2021-01-03', 33, 60, 90]] df = pd.DataFrame(data, columns = ['Day', 'X', 'Y', 'Z']) df['ds'] = pd.to_datetime(...


2

You are overwriting the model variable in the following statement: model = request.form['model'] You should not use the same variable name.


1

I think these are often used colloquially as synonyms, but let's try to find the differences. Each of them begins with "Time Series" (TS). So the difference lies in the three following terms. here with my interpretation: Analysis - wanting to describe and understand characteristics the observed data coming from the generating function$^1$. ...


1

Okay so let's start with the first question: Is that mean bias(b) is the distance between some particular point of the red line(as per picture) to the true point(say a blue or green point). You'd be correct had you used the word difference rather than distance. Bias is the difference between the estimated value and the true value. Think of it in this way, ...


1

The obvious answer is to use the last 4 data points of your training dataset. Note that there is no harm or bias in doing it. The purpose of breaking the data set into training and test datasets is to estimate and forecast on different datasets. In your model, the dependent variable is $y_5$ to $y_{183}$. On the other hand, the four explanatory variables are:...


1

I'm not familiar with the way you are obtaining the optimal threshold, but It might be a little bit easier. What you are looking for is the leftmost point in the x-axis (false positive rate) and the rightmost point in the y-axis (true positive rate) So by calculating the difference between the two you will have so. from sklearn.metrics import roc_curve yhat =...


1

If you want to make an inference on an order where a categorical variable was not seen in the training data, you could train the model on a hash bucket representation of that variable. If using tensorflow, you can leverage: https://www.tensorflow.org/api_docs/python/tf/feature_column/categorical_column_with_hash_bucket Or implement yourself.


1

Interesting question. The answer is: It depends. The best way to find out how it would affect your model is with the shap package. You can use it to uncover the importance of features and reveal interaction effects in the model. There could be a very different effect depending on how „important“ the excluded features are. Let‘s assume a very simple decision ...


1

Just to expand on @Ankita Talwar answer and give some slightly more formal intuition you can write a linear model with to regressors and their interaction as follows: $$ y = w_0 + w_1 x_1 + w_2 x_2 + w_3 x_1 x_2$$ where $x_1 x_2$ is the interaction term. Now refactoring you can see that the interaction can be absorbed into the coefficient for $x1$ making it ...


1

Another straightforward approach is to use the Pinball Loss function, which you can use to predict quantiles. Let $τ$ denote the target quantile, $y$ the target and the $z$ the quantile target, the pinball loss is defined as: $L_τ= \begin{cases} (y-z)τ,\text{if}\ y\geq z \\ (z-y)(1-τ), \text{if}\ y<z\\ \end{cases}$ This is the standard loss for quantile ...


1

Yes, by customizing the model with new information. But, you have to run atleast one training, with complete cycle and export output. the general example from tensorflow is here


1

The fastest way to train a model to predict each item's label is using Conditional Random Fields (CRT) like in this example. h/t @erwin


1

You don't say what the amount of available data is and if you use a test set. When you are up to prediction, always use a test set (some randomly chosen part of the data, say 20-30% NOT used for model training) to test your model predictions. With sklearn: import numpy as np from sklearn.model_selection import train_test_split X, y = np.arange(10).reshape((5,...


1

In case you are talking about providing certain interval to your predictions, what you might need is adding some confidence interval to your linear regression predictor, something which you can make via a resampling method like bootstrapping as a robust way to find predictions intervals. One key advantage is that it does not assume any kind of distribution, ...


1

Is it possible to create a predictive model for a dataset that consists of only positive occurrences of the dependent variable? One-class classification is a type of classification algorithm which does exactly that. In one-class classification the principle is to discover the patterns which characterize the instances of the class, assuming that everything ...


1

Given the example of earthquakes I assume that your aim is to predict a dichotomous variable but you actually only observe (or record) samples with one of the two labels. In this case is tough to make predictions. The best would be to actually get those samples with the other label. If you really can't get those samples then Anoop A Nair answer points in the ...


1

My dataset would only contain data about earthquake occurrences and no data about non-earthquake occurrences as that would basically be any other period of time which is not kept in the dataset. It would be nice if you could specify the features used for the prediction. In that case, I assume a decision tree or logistic regression would not work as we don'...


1

The reason for the rules based approach is the simpson paradox. In short, the way a test group is sampled can easily overturn the conclusion of a study. It is very important to include sufficient patients from a particular risk group in the study. This is why rules are set. This approach however is the basis of causality theory. A data driven-approach would ...


1

The main idea of Machine Learning is that your training data is similar to test data. If that hypothesis does not hold, then big chances of failure are guaranteed. Given that, you still can handle unseen categories even thought the solution is not good and more with One Hot Encoding In the category encoders library you have the following hyperparameter with ...


Only top voted, non community-wiki answers of a minimum length are eligible