New answers tagged

0

P Value of 0 is rare but theoretically possible. However in reality, p value can very rarely be zero. Any data collected for some study are certain to be suffered from error at least due to chance (random) cause. Accordingly, for any set of data, it is certain not to obtain "0" p value. However, p value can be very small in some cases. Lets look at ...


0

Is it normal to have p equals to absolute zero? I don't know about "normal", however it is completely possible, and in your case it makes sense, your frequencies are vastly different between the classes, so one would expect this result to be extremely unusual. I'll repeat this test in R ct=rbind( c(315,37,2), c(665,2661,665), c(3,49,285) ) ...


0

This is due to the fact that first time is very important, everything is set up during the first pass like cache, memory on the GPU, graph optimization, etc. Also 1 seconds is not that long it can take few seconds just to run matrix multiplication on GPU


0

The article you have shared answers the question correctly - your algorithm could input both int and float, and its internals (weights, biases, etc) will all be float matrices. A common example that you will find is the "customer churn" datasets, in which you need to classify whether a customer will stay/leave a company to go to a competitor. The ...


0

You might consider exploring Time-series analysis. As you've correctly pointed out, any model you will build will be based on the assumption that past factors are the ones that determine the future - therefore feature engineering is key. If calculating a ratio/percentage instead of a fixed value is more important to your analysis, then that would be the ...


0

Since PyTorch version 1.10, nn.CrossEntropy() supports the so-called "soft’ (Using probabilistic) labels the only thing that you want to care about is that Input and Target has to have the same size.


0

One possibility is that DNNRegressor is a TensorFlow Estimator. TensorFlow Estimators have been deprecated because they "can behave unexpectedly". It might be better to explicitly define a model.


1

You can access the steps within a pipeline by their name using the named_steps attributes. After getting the preprocessing step you can then use the transformers_ attribute in combination with standard python indexing to get to the OrdinalEncoder. Using the categories_ attributes then gives you the attributes for the encoder and, since the index of each ...


0

This should work, you dont need to append. def meanEmbeddings(model, words): # remove out-of-vocabulary words words = [word for word in words if word in model.vocab] if len(words) >= 1: return np.mean(model[words], axis=0) else: return []


0

Changing the batch size value seemed to be work. I put batch_size=10 and now the model is predicting properly. Not super accurate but atleast now its giving separate predictions


0

If you want, you can use our tutorial on how to do it. Worked pretty well for our test reviews.


1

This is more of a programming question than a data science question and would therefore be better suited for stackoverflow stackexchange, but the following code should do what you're looking for: df[["A", "C"]] = ( df # create groups .groupby(["B", "D"]) # transform the groups by filling na values with ...


0

I ran this movie review using my code, and it correctly classifies as a positive review. Please feel free to test it on this link: https://swetakesurnlp-playground.herokuapp.com I first recommend you to have a look at my code on this link: https://github.com/sweta-kesur-nlp-playground/nlp-play1-movie-reviews/blob/master/Sweta-1-NLP-play-Movies.ipynb and then ...


0

According to your requirements, this dataset will be a good choice for you to work on. It is available on Kaggle: https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews?select=IMDB+Dataset.csv IMDB dataset has 50K movie reviews for natural language processing or Text analytics. This is a dataset for binary sentiment classification ...


0

GPS data includes positional and time data. If the n+1 position at t+1 is too far away from the n position at t (i.e. d>0.5m for instance), you should be able to detect an anomaly. Same topic about the angle: if the angle between d1 and d2 is grater than a normal value (ex: 2 degree) then it should be considered as an anomaly. You should consider the ...


0

It maybe due to the fact that some of your columns may not have complete integer values, before fitting you should convert it X = np.asarray(X).astype(np.int_) Y = np.array(Y).astype(np.int_)


0

Scikit-learn MLPRegressor has a .partial_fit method for training on batches which will overcome this memory issue.


0

I appreciate that this response is many months late. However, I have faced the same problem and I hope that others might find this response useful. Ishaan, one of the ways to create personalised model per company is to use Partitioned ML Model. If this approach is used, then it would look like this: from sklearn.linear_model import LinearRegression import ...


0

This answer was submitted by the user @Vlad_Z These values represent the weighted observations for each class, i.e. number of observations per class multiplied by the respective class weight. Since your class weights aren't integers, the resulting values are the way they are. If you want to get class counts, you can simply divide your values by class weights....


0

Machine learning models take vectors (arrays of numbers) as input. When working with text, the first thing you must do is come up with a strategy to convert strings to numbers (or to "vectorize" the text) before feeding it to the model. The performance of a Machine Learning Model not only depends on the model and the hyperparameters but also on how ...


1

These string data, called categorical data can be converted to numerical data using many Categorical Encoding Techniques. Encoding categorical data is a process of converting categorical data into integer format so that the data with converted categorical values can be provided to the different models. Types of Categorical Techniques: Backward Difference ...


1

You can multiple things here : Converting them to numerical introduces some sense of ordering For example if you say slovenia is 1 and USA is 2 ans ordering is introduced instead you can use one hot encoding. Pandas getdummies function will do it for you If one of your string has a lot of values say 1000 one hot encoding does not makes sense. In those ...


0

import pandas as pd import numpy as np old = pd.DataFrame({ "ID": ["AA", "BB", "CC"], "Rating": ["High", "Low", "Medium"], "Status": ["On track", "Monitor", "On track"] }) new = pd.DataFrame({ "ID": ["AA&...


0

Transformers can be used for classification tasks. I found a good tutorial where they used a BERT Transformer for the encoding and a Convolutional Neural Network for a sentiment analysis. You can also fine-tune a whole Transformer for the classification but this is usually pretty intense when it comes to the training and you definitely need a GPU.


0

Brute force, but it solves the problem. This might be a good start: old = pd.DataFrame({"a":[1,2,3],"b":["a","b","c"],"c":["23-11","11-90",None]}) new = pd.DataFrame({"a":[9,2,3],"b":["a","x","y"],"c":["23-11",...


0

Please merge (left Join) the current table to previous table, Now you will have all the 4 columns in one dataframe. You can apply concatenate of columns to get desired results. Please share dataframe creation code if you need help with code creat


1

It seems that the function tries to call the SPLAT! command line using srtm2sdf on all files in in_path. Trying to run a command line program using subprocess.run when the command line program doesn't exist (i.e. returns a 'command not found' error when trying to run a command on the command line) gives a FileNotFoundError instead of the actual error you get ...


0

The first error you're getting is likely because the input becomes too small for the network to perform a 5 by 5 convolution on. The second error is caused by the fact that you are placing the padding argument in the wrong place. You are currently using it for the model.add call, whereas you should use it with the Conv2D classs: model_cnn.add(Conv2D(filters =...


1

It is crucial to measure the final result reached with prepropressing as best as possible. Therefore, there is a lot of different options depending on the datasets and depending on the algorithms/models. For instance, some models needs data normalization, some models needs logarithm or other transformation to improve the final results. Sometimes, you can ...


0

Another option would be to use integrated gradients to get an attribution for each word in a review and add them up over all reviews. Then you know for each word whether it let to a positive or negative review. This is a practical use case on how to use integrated gradients.


2

Yes, you can train XGBoost in parallel using the Dask backend. Short Solution Training XGBoost in parallel with Dask requires 2 changes in your code: substitute dtrain = xgb.DMatrix(X_train, y_train) with dtrain = xgb.dask.DaskDMatrix(X_train, y_train) substitute xgb.train(params, dtrain, ...) with xgb.dask.train(client, params, dtrain, ...) Have a look ...


0

Some of your assumptions are incorrect, let me try to explain this: To predict mortality you have to feed the model with mortality data and I think if you have mortality data, then it is probably too late to use the model. In supervised learning it's crucial to distinguish two very different kinds of "input": The training data is made of many ...


1

This can be solved by simply changing the method that is called within transform to the transform method of the vectorizer. In addition you would also have to add a call to fit within the fit method to make sure that the vectorizer is actually fitted before being used to transform any data: class Vectorizer(BaseEstimator, TransformerMixin): def __init__(...


0

To answer my question, I will use three types of models - KNN Regression Regression Trees Complex models like NN or SVM KNN Regression It is a non-parametric regression model, and the confidence might be explicitly modeled using mean absolute error or mean squared error. At the test time, for a given instance, K nearest instances will be found, and ...


0

This is evaluation and it's done experimentally: with a test set of fresh instances containing the true target value, apply the model and measure the error across all the instances (e.g. with MAE, MSE, RMSE...). Assuming that the test is a sufficiently large representative sample of the data, it's possible this way to estimate the quality of the model ...


0

I would look at "Fuzzy-C Clustering". This type of clustering is "soft" in that it provides a likelihood of a given point in a given cluster based upon weights, etc. Below are some links to get into the weeds a little... Towards Data Science, Wikipedia and the Python docs.


1

I think the issue is mostly with your network architecture. You are using only one convolutional layers and you are using all sigmoid activiations. Adding more convolutional layers, changing the activations from sigmoid to relu, and changing the optimizer to Adam gives me a loss below 5 after 30 epochs: model = tf.keras.Sequential([ tf.keras.layers.Conv2D(...


0

The problem seems to be that you are dividing by 10 (k) at each iteration, I can think to try to calculate the average, this is incorrect and probably it is what is causing you to see a very low metric value. It would be simpler and correct, to only store the values for the metric in each iteration at the validation set and finally just calculate the average ...


1

Some pure scikit approaches: When pre-processing relates to data balancing & sampling strategies, consider using Imbalance-Learn components (ie: RandomUnderSample) you embed right into your pipelines. This lets you hyper tune the parameters. Rely on passthrough functionality of grid search when deciding if certain pre-processing steps are needed at all....


1

$$R^2_{out}=\dfrac{\sum \big( y_i-\hat y_i \big)^2 }{ \sum\big( y_i-\bar y_{in} \big)^2 } $$ If your out-of-sample performance (measured by squared residuals) is worse (bigger) than performance of a naïve model that always predicts the in-sample mean of $y$, then your out-of-sample $R^2_{out}<0$. This is not unique to polynomial regression.


0

The scores you are seeing indicate that a linear regression would with multiple polynomial features does not fit the data well, with performance decreasing drastically on new data when using features polynomial features of degree 5/6 and higher (likely because of overfitting and/or multicollinearity). R-squared can be negative, for what this exactly means ...


2

A single call to partial_fit is very unlikely to get you a good fit, as it only performs one iteration of stochastic gradient descient. As stated in the docs: Internally, this method uses max_iter = 1. Therefore, it is not guaranteed that a minimum of the cost function is reached after calling it once. Matters such as objective convergence and early ...


0

If you would already have those datapoints before a company actually goes into bankruptcy then you can then them in your model since when predicting to the future you could have access to that data. However, if you would only know the data once the bankruptcy event happens (e.g. date of bankruptcy) then you cannot use this variable in your model since you ...


0

From the info you provide, I think you can do: average the test scores you get from your splits instead of printing per split, or use the cross_validation_score option given by sklearn, and then average such scores: cross_val_score(clf, X, y, cv=5).mean(), or use the sklearn GridSearchCV class, with which you can access the details of your cross ...


3

There are at least two reasons why the split should be made first: In theory at least, there is a true distribution of the data for the target task. Any model should always be evaluated on the true distribution of the data, because the goal is to predict on this distribution. Since data augmentation modifies this distribution, it's as if the model is ...


0

If the column contains actual dictionaries you can simply use the .str accessor to access the data in the dictionary by simply using the keys. For example, if you want to get the location from the dictionary you would use df_json["metadata"].str["area"].str["name"] to get the value 'Southern Suburbs & Logan'.


0

Converting to lower case is a historical method to combat data sparsity. The idea is that if you don't have a lot of data, case usually does't matter, so remove the meaningless variable. But for NER case is an important clue - capital words are more likely to be proper nouns, for example. So you definitely don't want to lower case things. In general, ...


1

In my opinion there are two main problems with your approach: The clustering is extremely unlikely to correspond to sentiment, unless the features that you use for clustering are specifically engineered to represent sentiment. In general text clustering tend to group documents by common words, i.e. similar topic. This might lead to different categories of ...


1

In general, yes: decreasing the number of classes mechanically increases the probability that the classifier finds the right one. Even in the worst case scenario where the class is picked randomly, the probability of every remaining class increases when there's one less class. Another way to look at it: all other things equal, the number of errors can only ...


0

I think what you are looking for is numpy.concatenate. From the documentation this allows you to: Join a sequence of arrays along an existing axis. It has the function signature: numpy.concatenate((a1, a2, ...), axis=0, out=None, dtype=None, casting="same_kind") See the documentation for more.


Top 50 recent answers are included