New answers tagged

0

You might consider exploring Time-series analysis. As you've correctly pointed out, any model you will build will be based on the assumption that past factors are the ones that determine the future - therefore feature engineering is key. If calculating a ratio/percentage instead of a fixed value is more important to your analysis, then that would be the ...


0

Regularization tries to reduce the chance of overfitting by reducing the sensitivity to small changes in the input data. This is not as much of an issue for the intercept term (relative to the coefficients) so it is often not included.


1

Instead of using the estimator attribute you should be using the best_estimator attribute, after which you can access the underlying estimators of the MultiOutputRegressor using the estimators_ attribute. You can then access the coefficients as follows: coefficients = [estimator.coef_ for estimator in best_model.best_estimator_.estimators_] # [array([-0. ...


0

a) Any intelligent way to assign likelihood to this table? Your conversions are essentially an outcome of two functions, your biz team and the procurement team/contact of the client. The common denominator is your biz team. Consequently, in order to be able to predict likelihood you need recorded metrics of your biz team's both successful and unsuccessful ...


1

Provided you have the data already, and the data is labelled (i.e., split into the two classes $A$ and $B$), it makes sense to produce a number of visualisations to gauge what the model output would be. If you start with traditional classification algorithms like logistic regression, then the model output is going to be the probability of belonging to a ...


0

As stated in the docs, the KNeighborsClassifier from scikit-learn uses minkowski distance by default. Other metrics can be used, and you can probably get a decent idea by looking at the docs for scikit-learn's DistanceMetric class


1

You could simulate data and fit a model to it as if it were real data. there are packages and functions in R and Python to do this. You'd have to be very clear that the data is faked. You could then examine the model and produce graphs as if it were a real one. This has the downside that it involves writing all the code and writing code to sim data, which ...


1

Look at your past experience. Even though you're a novice, you were hired as a data scientist, so you'll probably have some experience with data science projects. A simple binary classification problem with a few hundred datapoints can be solved in a productive afternoon, whereas a large project that requires significant upfront engineering for the ...


0

In regression, it makes perfect sense to have multiple hidden layers to model the complex relationships between inputs and outputs. However, those layers need to have non-linear activation functions, otherwise, they would be equivalent to a single layer. When your textbook said that "linear activation function is commonly used (along with MSE loss ...


0

Yes it does, because if the relation between target and training data is a very complex function we may not be able to capture it in one layer with limited units. Increasing the number of hidden layers may help you model a highly complex function with limited units on each layer. Number of hidden layers varies from problem to problem and mostly is a hot and ...


0

You might consider a log transform or a square root transform to reduce the skew but they work on positive values only. This might also help you with the long tail of values by "drawing them in closer".


1

These string data, called categorical data can be converted to numerical data using many Categorical Encoding Techniques. Encoding categorical data is a process of converting categorical data into integer format so that the data with converted categorical values can be provided to the different models. Types of Categorical Techniques: Backward Difference ...


1

You can multiple things here : Converting them to numerical introduces some sense of ordering For example if you say slovenia is 1 and USA is 2 ans ordering is introduced instead you can use one hot encoding. Pandas getdummies function will do it for you If one of your string has a lot of values say 1000 one hot encoding does not makes sense. In those ...


0

If you have a labeled dataset $f(X) =Y$ then you have a supervised learning problem, so you may try to solve it as a "usual" binary classification problem by using metrics like $F1$ or $AUC$ and Cross-validation to evaluate your model's performance, and what I mean by usual is that you do not need to apply something special for anomaly detection ...


0

Not sure what is meant by a paper. Are you asking if there is mathematical proof that this is the best setting all of the time? There are some experiments pointed to from here, a text book quote in that answer, and in the link you posted. The answer is "it depends". You can tune this parameter for your data and problem. Perhaps n/3 is a good place ...


0

Some of your assumptions are incorrect, let me try to explain this: To predict mortality you have to feed the model with mortality data and I think if you have mortality data, then it is probably too late to use the model. In supervised learning it's crucial to distinguish two very different kinds of "input": The training data is made of many ...


1

I think the issue is mostly with your network architecture. You are using only one convolutional layers and you are using all sigmoid activiations. Adding more convolutional layers, changing the activations from sigmoid to relu, and changing the optimizer to Adam gives me a loss below 5 after 30 epochs: model = tf.keras.Sequential([ tf.keras.layers.Conv2D(...


1

I assume the "dose" $y$ is limited to $y \in [0,1]$. So in the moment you have "bunching" in your target value $y$ which you try to remove. In this case, a linear regression could lead to "overshooting" (see here for more details). So it could be beneficial to use some estimator which "restrics" $\hat{y} \in [0,1]$ as ...


9

The procedure that you can use is the following. First cluster your data with gaussian mixture models. This method should also work with multiple lines with different slopes. It should be able to deal with intersections as points near an intersection can belong to both clusters and a wrong classification will not lead to huge differences in the results of ...


1

You could try to use the following method. $$y=a\sin \left[\dfrac{\pi}{12}x-b\right]+ c$$ $$=a\left[\sin \dfrac{\pi}{12}x \cos b - \sin b \cos \dfrac{\pi}{12}x \right] +c$$ $$=a\sin \dfrac{\pi}{12}x \cos b - a \sin b \cos \dfrac{\pi}{12}x +c$$ The last equation indicates that we can switch to new parameters $w_1 = a\cos b$, $w_2=-a\sin b$, and $w_0=c$. ...


0

Option include segmented regression or decision tree regression. Both of those algorithms are able learn to predict different targets values conditional on feature values.


1

If I understand your question right the answer is yes. For example in tensorflow functional API could you have as your final layers: out = tf.keras.ayers.Dense(10*10)(previous_layer) out = tf.keras.layers.Lambda(lambda t: tf reshape(t, [..., 10, 10]))(out) out = tf.keras.Lambda(lambda t: tf.nn.softmax(t, axis=-1)) Your output would have shape [batch, 10, 10]...


1

If your goal is to set the optimum value of hyperparameters (weather it be learning rate, no of layers, activation function etc.) you should look into Keras Tuner. The reason as to why the learning curve is oscillating is not clear to me without seeing your code/data. But if you want to set an optimum value of learning rate then look into Kera Tuner provided ...


Top 50 recent answers are included