New answers tagged

0

Both bidirectional LSTM (like ELMo) and BERT seem appropriate for this kind of task. Whether one or the other performs best can only be known by testing. If you use BERT, ensure to apply the typical measures to avoid overfitting. If you apply LSTMs, you will probably need regularization measures.


0

The column df['text'] was of type 'object', which was a byte type, so the pandas Series contained b"foo", etc. The only change to make was to decode the object using: .str.decode('utf-8') df = tfds.as_dataframe(ds.take(4)) reviews = df['text'].str.decode("utf-8") corpus = reviews.tolist() print(corpus) tokenizer=Tokenizer(num_words=100) ...


0

Your solution is close Maybe just needed to add an apply. Try: df = pd.DataFrame({"Name":["Name1", "Name2", "Name#DATE#"], "Date":[20200126, 20200127, 20200210]}) df["NewColumn"] = df.apply(lambda row: row["Name"].replace("#DATE#", str(row["Date"])), axis = 1) Outputs:...


1

Keras' one_hot function has many limitations. The biggest issue is that the function does not actually do one hot encoding, it does the hashing trick. One possible fix is to use keras' hashing_trick function. It allows the hashing function to specified. If you pick a stable hashing function like md5, then the values will be consistent across runs. Here is an ...


1

It's normal: LDA tries to maximize the likelihood of the data according to the parameters by finding the right probabilities for the parameters. Usually at the beginning increasing the number of topics allows the model to separate topics more precisely and therefore obtain a higher likelihood. But at some point (depending on the data), increasing the number ...


1

As far as I know, scikit-learn has no library for ensemble clustering. On the other hand, you can apply the method on your dataset as follows: import numpy as np import ClusterEnsembles as CE kmeans1 = np.array([1, 1, 1, 2, 2, 3, 3]) kmeans2 = np.array([2, 2, 2, 3, 3, 1, 1]) kmeans3 = np.array([4, 4, 2, 2, 3, 3, 3]) kmeans4 = np.array([1, 2, np.nan, 1, 2, ...


0

You should be able to use keras.models.load_model to load the hdf5 model file. See also the tensorflow documentation. # saving the model model = ... # Get model (Sequential, Functional Model, or Model subclass) model.save('weights.hdf5') # loading the model from tensorflow import keras model = keras.models.load_model('weights.hdf5')


2

You can use groupby together with shift and cumsum as follows: df['header_contract'] = df['contract'] + '_' + df.sort_values(['contract', 'date']).\ groupby('contract')["date"].\ apply(lambda x: (x.shift() != x).cumsum()).astype(str) In the apply, x.shift() != x is used to create a new series of booleans corresponding to if the date has ...


2

Welcome to the biomedical domain, one of the few domains in NLP where there are too many resources to choose from :) Data resources: Medline is a database corpus of 30 millions abstracts. Each Medline abstract is annotated with Mesh descriptors, Mesh being a structured hierarchy of medical concepts. PubMed Central (PMC) is a database of around 6 millions ...


0

According to your questions: Labels should be long and advised. [num_samples, ] It should have two outputs. If your batch_size=200 then target somehow similar to this: [0, 1, 0, 1, 1, 0, ....1] 3rd ^


1

That is indeed a drawback with grid search strategy, since you must know in advance each one of the possible combinations to try out, and that might be not optimal neither to get the best evaluation metric value nor in computation performance. You have other interesting strategies, not exhaustive hyperparameter search, for instance random search or based on ...


2

Assuming you cannot add more memory to your computer (or free up some of the memory), you could try 2 general approaches: Read only some of the data into memory e.g. a subset of the rows or columns. reduce the precision of the data from float64 to float32. From your error, it looks like you are loading data into a numpy array, so somewhere in your code, ...


2

The only thing that you need to do is to start your agent and the goal/end at a random (non-overlapping) location. You can try your setup initially with an empty grid (no walls). If DQN learns, your set up is good and you can start introducing obstacles into the grid. Gradually, the agent will start associating the end location inputs as something rewarding ...


0

I had also run into this problem several times, so I've created an open source modelstore Python library which seeks to tackle the problem of simplifying the best practices around versioning, storing, and downloading models from different cloud storage providers. The modelstore library unifies the versioning and saving of an ML model into a single upload() ...


4

The way you have set your DQN up, it is designed to solve just one maze at a time. It has not (and cannot) learn to solve mazes in general, because it has no access to data about the layout of the maze, and a basic DQN agent has no capability to memorise layout seen so far. You could view the training process as general algorithm for "solving the maze&...


0

I don't think it needs to be ML based, it could just be threshold based. For example, your pseudo code could look like this: In a moving window of n=100, if 80% of the data is above/below threshold, then flag a warning.


0

fit, transform, and fit_transform. keeping the explanation so simple. When we have two Arrays with different elements we use 'fit' and transform separately, we fit 'array 1' base on its internal function such as in MinMaxScaler (internal function is to find mean and standard deviation). For example, if we fit 'array 1' based on its mean and transform array 2,...


2

For deep learning, there are a few model hubs where folks share models that are suitable for further fine-tuning or usage in given areas. None of these will be in the pickle format (from your question), but they are great resources nonetheless: PyTorch Model Hub Tensorflow Hub Hugging Face Models


0

I had also run into this problem several times, so I've created an open source modelstore Python library which seeks to tackle the problem of simplifying the best practices around versioning, storing, and downloading models. As others have pointed out, the best practices around this area are still forming. The whole area is fairly straightforward if you are ...


0

To visualize a decision boundary of a classifier, specifically a binary classifier as in your case, you can instantiate a grid of points that spans the domain of interest that you want to classify. Then use your trained classifier to predict the class of each and every point in the grid. If the grid resolution is fine enough, when you plot the contours of ...


0

You are right in saying that replacing with a simple mean, mode... is a common but unreliable imputation strategy in many cases. You have in scikit learn some utilities for imputation of missing values (have a look at https://scikit-learn.org/stable/modules/classes.html#module-sklearn.impute) using for instance the knn imputer as an additional strategy. Take ...


0

To convert numpy array to tensor, import tensor as tf #Considering y variable holds numpy array y_tensor = tf.convert_to_tensor(y, dtype=tf.int64) #You can use any of the available datatypes that suits best - https://www.tensorflow.org/api_docs/python/tf/dtypes/DType


1

Let's say you pass in output_shape as a tuple (50, 50, 10) where we can call the values (height, width, channels)` to the lambda layer: your_layer = tf.keras.layers.Lambda(lambda x: x, output_shape=(50, 50, 3)) The part of the documentation: If a tuple, it only specifies the first dimension onward; means that the batch dimensions itself is simple carried ...


1

To address your points: There are no significant autocorrelations The correlation is low (~0.25), but there are significant autocorrelations. The data is random & most of the correlations (except for 2 lags) fall within 95% confidence limits The confidence intervals are used to show which autocorrelations are significant. As you rightly observed, a ...


0

As far as estimating similarity between the time series series there is a variety of methods you may want to investigate. Some of those are: cross correlation: this will be affected by the amplitude and will not be able to estimate lagged correlations, prone to noise. coherence: normalised frequency based correlation (cross-spectrum), not prone to amplitude ...


0

SVM is effectively a 2-layer NN. It is better to use a neural network for creating an embedding. Since, you don't have negative examples, you can't use something like a Siamese Network. One good way for you to create those embeddings would be use the bottleneck layer of an autoencoder. Or if you have image data, use ResNet to get the embeddings (the ...


0

Doc2vec and similar algorithms are useful methods to create document embeddings. You should try to include as much data and metadata as possible. The size of the documents does not matter much because they will be project into a fixed dimensional embedding space. As far as noise, the effect will be task specific. If you are doing similarity analysis and ...


0

For TypeErrors it is always good to check the exact type of the variable causing issues, especially when troubleshooting TensorFlow and Keras: print(type(df['text'])) fit_on_texts expects a list of string or similar, but you are providing a dictionary, so you'll want to convert accordingly. Example: from typing import List, Dict foo_dict: Dict[str, str] = {...


0

Yes, in scikit-learn, you can find the correlation between the elements using LedoitWolf Estimator. For dimensionality reduction, I assume you will use PCA but then, you want to backtrack the reduced to the original data, for that I don't have a solution, since, PCA transforms your data and computes the largest variance direction. But, yes both of these won'...


1

Following on Thomas on the relation between the Bray Curtis distance and the F1 score and the calculation of the first and second-order derivatives. If one defines the Bray Curtis distance between vector X and Vector Y as: $\sum |X_i-Y_i| \over {\sum (X_i+Y_i)}$, than the first derivative to $x$ is $d \over (dx)$ $|x - y| \over {(x + y)}$ = $2y(x - y) \over{\...


2

For this type of issue, I typically add the reciprocal of the log base. For data that's being log10-scaled, this results in adding 0.1 to all values. For data that's being log2-scaled, this results in adding 0.5 to all values. This has the nice property of mapping all of your 0 values to -1 in the log scale, regardless of what log base you use. If your data ...


0

There might already be a built-in function to compare these outputs you've shown, but one solution would be to just threshold the lists into Boolean lists, and then use logical_and to compare them: import numpy as np def threshold_clusters(teacher_list, threshold = 0.85): return [i>threshold for i in teacher_list] def compare_clusters(first,second): ...


2

Your suggestion is a valid one, encoding variables with a known outcome once the scaling is applied. Log(1) will become zero, so just keep that in mind for your next stage. You can use clip or replace for this: df.clip(1, df.max()) or try replacing with a NaN df.replace(0, np.nan) Alternatively you could do one of the following: Drop the zero value rows e....


0

Some suggested alternative plotting methods to visualise this data: Histogram of the y-axis. Check the distribution of time intervals df.plot.hist(by='interval', bins=10) #test varying the bin size Plot smaller subsets of the data if the order is important e.g. df[:100].plot() furthermore, if there is periodicity in the data, e.g. daily, hourly etc. you ...


0

Try this: baseline_df.pivot_table("product_count",columns="Product",aggfunc=sum)


0

You should be able to use pandas.DataFrame.pivot for this after resetting the index as follows: import pandas as pd baseline_df.reset_index().pivot(index="RtrId", columns="Product", values="product_count")


0

As per the above answer, the below code just gives 1 batch of data. X_train, y_train = next(train_generator) X_test, y_test = next(validation_generator) To extract full data from the train_generator use below code - # Store the data in X_train, y_train variables by iterating over the batches train_generator.reset() X_train, y_train = next(train_generator) ...


1

The normalisation you do does not re-scale to $[0,1]$ range! It normalises to have mean $0$ and std $1$ instead. To scale the tensor to be in $[0,1]$ range you should subtract min value and divide by absolute max-min value.


0

One approach is to have a dummy class that represents no treatment and use the accuracy score via a threshold (lower than threshold) correspond to no treatment at all. Threshold as used above becomes a new hyper-parameter and you have new input-output pairs that are now exact (depending on threshold).


0

I used this code and get output from this(with the help of Oxbowerce). df2=pd.DataFrame(testX,columns=['p','M','Date']) df3=pd.DataFrame(pred,columns=['pred']) df4=pd.concat([df2,df3],axis=1) df4.to_excel(r'/content/Book1.xlsx', index = False) I saved the output in an excel file. You can see in the below picture.


0

It seems that there is a way of using sample weights which is required a little more work than just using a single argument, see this stackoverflow answer.


0

X needs to be the features for your Model and Y needs to be a target variable. As you mentioned, you are using a IMDB dateset so, all the features which you want your model to use will be stored in X variable whereas the 'LABEL' columns will be stored in the Y variable. Instead try this code: X = vectorizer.fit_transform(df_train['text']) Y = vectorizer....


0

Depending on your dataset, I would say you can set the threshold yourself. For example, a 50% split is a perfect balance so by definition any deviation away would be an imbalanced problem. That being said, this is quite typical, so it would make more sense to set your threshold to be somewhere around 80% imbalance or higher. Count your classes and set a ...


0

First of all, if you are in a classification problem as you said, R2 score is not good, it should be used for regression problem. For classification problem you have to use something like accuracy, precision, recall and F1 score. First question Anyway, I will answer your first question about R2score: R2 mean: you are calculating the R2 for each split you ...


0

The outputs of your model are in the same order as your inputs, so the first row in your output array corresponds to the first row in the testX array. If you want to have both the inputs and the model prediction in one table you can just concatenate them along the column axis.


2

Well, there are a few ways to do the job. Here are some I thought of: Scatterplots with noise: Normally, if you try to use a scatter plot to plot two categorical features, you would just get a few points, each one containing a lot of instances from the data. So, to get a sense of how many there really are in each point, we can add some random noise to each ...


0

Keras included in their library to predict the class label. You can get the class label directly by using model.predict_classes(img). Ref: https://datascience.stackexchange.com/a/40415/109134


1

The error comes from attempting to fit a classifier (logistic regression) on a regression problem. If you are trying to predict prices (continuous outcome), you should use linear regression.


1

When imputing data, one is looking not to modify the true distribution of your data. So a way to test how good your imputation was is to make a test to contrast the true distribution of every feature that has been imputed vs the true (via KS test for example) distribution of the feature (prior imputing) if you can sate with a level. of confidence that your ...


0

Model prediction output is a bunch of probabilities. In order to get category name you need use following snippet. It calculates the argmax of predicions and give it to CLASSES list: print(CLASSES[np.argmax(predictions)])


Top 50 recent answers are included