New answers tagged

0

It's because 80'000 samples is probably too much training data. Usually, statistical forecasting models are not trained on the entire training set like ML models. Try reducing your training data significantly, and incorporate only a certain number of seasons as your training data. Example: If your data is hourly, one season can be 24 samples. Then, the ...


0

You are using np.nan_to_num(x_train) which would convert the null values to zeroes and also will take care of infinites. But you are not assigning back. can you try x_train = np.nan_to_num(x_train) and similar to y_train as well? I just test this with one example: a = np.array([[1,np.nan,3],[np.nan, 0, np.nan]]) a=np.insert(a, a.shape[0],[[1, np.nan, 1]], ...


0

You should imput missing data in your dataset, or delete thos rows if they are not many. Alternatively, if the values are too big to be represented by float32 datatype, try to convert them to float64 (takes more RAM).


0

I think the first question you should ask before you start going off on a deep learning model is, can you tell when the failure is going to occur just by looking at a plot of your data? If you can't, then no model will help you deduce when a failure will occur. You shouldn't overlook some basic models also such as exponential or poisson distribution models ...


1

You could always call on the pandas DataFrame's columns and work with that. values = df['price'] * df['quantity'] sum(values) if you want more information, I recommend https://stackoverflow.com/questions/14059094/i-want-to-multiply-two-columns-in-a-pandas-dataframe-and-add-the-result-into-a-n


1

If I understand the problem correctly, you want to fill all missing values in the Fare column by the median value of the Fare column where Pclass=3. This can be achieved by putting the extra row filter test['Pclass']=3 on median of the fare column, see below. test['Fare'] = test['Fare'].fillna(test.loc[test['Pclass']=3,'Fare'].median())


1

I think you have misunderstood the koalas library. You can say its Pandas on Distributed System. You can use Koalas similar to pandas. There are few drawbacks with respect to APIs which is documented in their docs and few articles already written on medium. You can do your EDA and straight away use them in all the libraries you have mentioned. Recent ...


1

Experimentally: using cross-validation on a subset of your training data, compute the performance of every option that you want to consider. Then select the best option and train the final model using this option. // different settings for hyper-parameters, // for instance different pruning criteria: hpSet = { hp1, hp2, ...} trainSet, testSet = split(...


1

If you have a one dim data, why do you need to use K-means? In such a case, to detect the outlier I would recommend creating a simple histogram and then based on its shape you can visually find the outliers. To get a proper outlier threshold you can use np.quantile() function.


0

BERT is trained on a combination of the losses for masked language modeling and next sentence prediction. For this, BERT receives as input the concatenation of the special token [CLS], the first sentence tokens, the special token [SEP], the second sentence tokens and a final [SEP]. [CLS] | First sentence tokens | [SEP] | Second sentence tokens | [SEP] ...


0

Your problem deals with forecasting of the sales from the historical data, you can use any time series forecasting like ARIMA, Facebook Prophet. They have ability to predict the sales based on the seasons as well. Refer here https://facebook.github.io/prophet/docs/quick_start.html


0

You will.find the whole explanation of ensemble learning with code https://link.medium.com/uADFoKwi33


0

Using a CNN in an autoencoder (mentioned by S van Balen), is the most established way of doing anomaly detection. You can possibly use a pre-trained network as a base for this. And it should be possible to train only the decoder, keeping the encoder frozen. There are some works that show using regularization constraints to make the decoder layers the inverse ...


2

You can try with this: import pandas as pd import nltk df = pd.DataFrame({'frases': ['Do not let the day end without having grown a little,', 'without having been happy, without having increased your dreams', 'Do not let yourself be overcomed by discouragement.','We are passion-full beings.']}) df['tokenized'] = df.apply(lambda row: nltk.word_tokenize(row[...


2

There is not a reshape problem. You need to transform your text in a set of features, say, vectorize it in the same way you created your dataset, in this case using TF-IDF. Just prepare a query vector applying the same TF-IDF and will work.


2

You cannot do that directly because your training data is not a text but set of features extracted from the text.You need to convert the text to list of features and then try to predict it


0

b = BernoulliNB() b.fit(b, X_train,y_train) This doesn't work because you're calling the fit method of the BernoulliNB class object you instantiated with BernoulliNB(). This argument says you have an extra parameter because it only accepts X, y. It's hard to diagnose your second issue without seeing your data. But your y_train is probably formatted ...


0

You are not getting the results you're expecting because you're explicitly comparing the value in the Answer column to 'Coca-Cola' (i.e x == 'Coca-Cola') but the value in that column is (I'm assuming) a list or a string. In either case comparing it to 'Coca-Cola' would result in False. What you need to do is change your condition to check in 'Coca-Cola' ...


0

I haven't tried, but it might be possible to load a single categorical feature at a time and train an encoder on that data, before deleting it from memory and loading the next feature to train another encoder. In the end, I just did this.


0

Long story short is to use cloudpickle instead of joblib or pickle to dump thing to disk, and this all works much more cleanly.


0

Yes there multiples ways to do it. Simple ones TF-IDF, SIF and quick/skip thought use encoder-decoder structure and the output of encoder is the embedding. Then the similarity between documents is simply the cos of embeddings. Doc2vec ultimately generate embeddings too.


0

It's a supervised classification problem: you're trying to predict the destination (class) based on some categorical features (input columns). I would suggest starting with some simple algorithms such as decision trees or Naive Bayes. However I'm guessing that logistical shipments can evolve over time: maybe a shipper business grows with country X but ...


0

Your question is not very clear: do you mean that your test data never contains this feature? If yes, you should remove this column from the training data. The train and test data must have the same features. If no, i.e. only some instances might not have a value for this column, then it's about having missing values in your data. In this case you could ...


0

You should be able to use the column name like: df_1 = df_1.drop('furniture')


0

That dictionary can be created with two lines: A nested generator expression to create the keys. The dictionary method fromkeys with a default value of 0. year = [2014, 2015, 2016, 2017, 2018, 2019] quarter = ['Q1', 'Q2', 'Q3', 'Q4'] keys = (str(y)+'_'+str(q) for y in year for q in quarter) change_it = dict.fromkeys(keys, 0)


0

Alternatively, you could just use a defaultdict from python, which will return the 0 value if you ever look for a key that isn't there. So we make 0 the default value (below using an anonymous/lambda function). Otherwise, is behaves like a normal dictionary! In [1]: from collections import defaultdict In [2]: d = defaultdict(lambda: 0) In [3]: d.keys() ...


0

You were close, but you needed a few changes, the biggest issue is that your dictionary and date value are both called change_it and your never updating your dictionary, try: years = [2014, 2015, 2016, 2017, 2018, 2019] quarters = ['Q1', 'Q2', 'Q3', 'Q4'] change_dict = {} for year in years: for quarter in quarters: change_it = str(year) + '_' + ...


1

You are using different random starting points each time you run k-means (random_state=None) This means you may get different clusterings, different clustering metrics each time. That's expected. What you may wish to do is average the results over several different runs to get a more reliable estimate of the loss at each k.


0

I would remove the classes with very few samples, as they create discrepancies in the model and also help with skew. I would try to create new features by combining the features which are similar/ or have similar influence on the outcome. This is because as compared to the number of samples, you have too many features. Try using logistic regression and see ...


1

Reason for the discrepancy Two aspects have to be considered regarding the split: Is the split done in a stratified manner? (it should) Is the data shuffled? (it should) The line X_train, X_test, y_train, y_test, i_train, i_test = train_test_split(feature_matrix, y, indices, test_size=0.33, random_state=random_state) splits the data in a stratified ...


0

I want to build a deep learning model to predict the next job title when a current title is given. Are there any ways that I could achieve this using some deep learning model? I think that approaching this problem in the classification way (input:- current job embedding, output:- getting next job title as a class) can somewhat be time-consuming (not ...


2

A quick search of the source code, e.g. RandomForestClassifier's, doesn't find anywhere that sample_weight gets saved as a class attribute. I suspect that was a conscious decision: the sample weights are directly tied to the dataset, which also doesn't get saved for later use; that's why sample_weight appears in the fit method rather than the class ...


0

Generally speaking, you need a string to be executed that contains placeholders to accept the values contained within the tuple. Also, what you are calling your "tuple" is actually a Python dictionary, but that isn't a problem. From the sqlite3 documentation, we can see there are two ways to correctly use the execute() method: an example command is given ...


0

Whether this particular performance is stable is mostly a matter of opinion, and it depends on the context as well (size and complexity of the data). It's often more meaningful to use a baseline system and compare the performance against the tested system: this way it can objectively be said whether the performance is more or less stable than the baseline.


1

Accuracy is a statistic that you can compute on a dataset if you know the true labels. For a single image, the accuracy is either 0% or 100% based on if you get it right or wrong. In the newer versions of Keras, the predict method returns the probabilities of the classes, what you want to print is (if I guess correctly), the probability score for the best ...


1

Try the following code: import requests import pandas as pd import io params = { "api_key": "abc", "format": "csv" } r = requests.get('https://www.parsehub.com/api/v2/runs/ttx8PT-EL6Rf/data', params=params) r = r.content rawData = pd.read_csv(io.StringIO(r.decode('utf-8')))


0

Try the simplest approach first - deterministic check looking for intersection overlap between the set of fruit names and the set of items bought. Set comparisons are scalable because the look-up time for each item is constant. If scaling is an issue with regular set membership check, bloom filter is an option.


0

One of the default algorithms to use for this use case (set of search strings to be searched simultaneously in a text) is Aho Corasick. From the Wikipedia page: "The complexity of the algorithm is linear in the length of the strings plus the length of the searched text plus the number of output matches." Implementations of this algorithm exist in all common ...


0

The scaling for both plots is different since the feature importance in Python is normalized and sums to 1, this normalization is not done in R. You can try setting the random state for the random forest in both languages since this would control for any randomness.


1

Looking at the documentation example here, a xgboost custom loss function needs to return the gradient and second-order gradient. Your function does not return those values for the stated goal.


0

I just faced the same situation. If you need to explicitly build the inverse, check this paper: https://pdfs.semanticscholar.org/f278/b548b5121fd0d09c2e589439b97fad16ece3.pdf In particular, given a Matrix M that you need to invert, you can just do: A = tf.math.real(M) C = tf.math.imag(M) r0 = tf.linalg.pinv(A) @ C y11 = tf.linalg.pinv(C @...


2

I had the same problem, solved by moving y axis: ax.set_ylim([0,2])


0

I have the same problem, too. It is the dataset error. I find that the training dataset loaded is the Fashion-MNIST dataset, while the test dataset is the the MNIST dataset. So, I download the original Fashion-MNIST dataset from the official site https://github.com/zalandoresearch/fashion-mnist


0

I will share my experience in image optimization ... At first, I had to manually compress all the pictures through Photoshop. The most free option by the way (except for the cost of a license for Photoshop). But this process takes a lot of time if there are more than 10-20 pictures on the site. After all, each picture must be manually processed, and then ...


0

May be you should consider using the .loc method of a dataframe. It allows to select rows using a logical expression. Have a look at this link. In your case something like df.loc[df.A>=3] should do the work. Selecting columns is also an option.


2

There are several ways df[df['A']>=3] Maybe using query df.query('A >=3') Or using the .loc df.loc[df['A'] >= 3] This question is posted on several places.


1

I would suggest you to scale your data using standard scaler and do it before you split it into X and Y here in your case. Why ? Please check this answer on stats sc. Also, keep the target variable (Front in your case) as it is. So, according to me, the right choice looks like this, however you can try and experiment with min max scaler too: from sklearn....


0

You could also use {:.3%} instead of {:.3f}%. It will transform the value into percentages automatically. That means "{:.3%}".format(0.3) will print "30%" while you have to write "{:.3f}%".format(0.3 * 100) to get "30%" as well.


0

Try predict_generator imgen = ImageDataGenerator(rescale=1/255.) testGene = imgen.flow_from_directory(<path to testing images>, target_size=(150, 150,), shuffle=False, class_mode="input", ...


1

I can give you some hint of doing so with deep learning approaches. It's easy to use gensim and sklearn python libraries. First, you need to extract the word embeddings which are vector of numbers to represent a word, and then take the average of the words within a sentence is a way of fining that vector representation for your sentence. So extract ...


Top 50 recent answers are included