New answers tagged

0

With reference to this answer, here's a running example to solve your problem, nan_obj = float( 'nan' ) # dict as mentioned in the question dictionary = { '$175000-199999': nan_obj, '$698506': nan_obj } # Loop through key-value pairs # For different ways to check if a number is NaN, # see https://stackoverflow.com/questions/944700/how-can-i-check-...


1

3 points: If the feature is certainly or very most of the time is not available during prediction the no you cant use it If it is sometimes available and sometimes not, you must include no-comment bugs into your training as well and choose a default value which means no-comment (e.g. 'no-comment' string! or None) In case they are available only for training,...


0

There are multiple errors in the messages you showed. Maybe you tried running your code multiple times, commenting out or changing the lines causing errors to see if it worked? Anyway, here are my guesses: TypeError: flatten(): argument 'input' (position 1) must be Tensor, not Linear: this is because at line 22 of file DQN_NEW_Original.py you typed self.fc1 ...


1

Just multiply every value by $0.95$. Your original $0$ will stay at $0$; your original $100$ will be reduced to $95$; and the original values in between will be reduced a bit, too.


0

Assuming that there are almost always clear markers at the beginning of the enumeration, i.e. either "required skills" or "nice to have" (or any variant of these two), I would suggest trying to add custom features for example: last marker seen from the current position, a categorical value for either "Nicetohave" or "...


0

df3 = df1[df['series_name'].isin(df2['series_name'])] EDIT: Adding details per request. This simple approach can compare across 2 different dataframes, placing the output in a third.


0

All Deep Learning libraries have data loading APIS that can lazily way to load data. You mention TFRecords so I assume you are using TensorFlow. You can use TensorFlow's data API.


0

To install a Python package select: options > add-ons > add more > insert package name So to install the imblearn package : options > add-ons > add more > imblearn


1

Backward Elimination Step 1: Select a significance level to stay in the model (Eg: SL=0.05). Step 2: Fit the full model with the possible predictors. Step 3: Consider the predictor with the highest p-value. If P>SL, go to step 4 otherwise your model is completed. Step 4: Remove the predictor Step 5: Fit models without this variable and move to step 3. ...


5

This is a very strange design: The goal is to train an ensemble classification model. In general there is no strong reason to use only subsets of the data to train the individual learners, let alone to use a strict partition of the data. It might make sense to train different learners with different subsets in order to make the final model more stable, but ...


0

If you want to convert the categories to embeddings using tf.feature_column the best option is tf.feature_column.embedding_column.


0

For all the three samples the training data accuracy saturates around 85℅. However, with more training data, the gap of training and test data widens. This is a clear sign of overfit. The hyper-parameter tuning has not alleviated overfiting yet. Try hyper-parameter tuning with k-fold cross validation. Here is an article on the same: https://...


0

In order to predict the first out-of-sample datapoint you should take a sequence of the data and pass it to the LSTM model (example in pseudo-code): pred = model.predict(X[-10:]) For the next predictions you'll have to include the current prediction into the data passed to the model. X = X + [pred] next_pred = model.predict(X)


0

Error may increase with number of samples because of non correlated features. Have you done a correlation heatmap in order to know if there are non correlated features? Note: non correlated are close to 0, but you should keep the anti-correlated (<0) because they have a kind of correlation. https://medium.com/@szabo.bibor/how-to-create-a-seaborn-...


3

This kind of problem is called record linkage (or sometimes entity matching or other variants). The task consists in finding among a list of strings representing entities (persons or organizations) those which represent the same actual entity. There are two main approaches (which can be combined): String similarity matching methods. See for example this ...


0

Why not using classing distance measurement such as K-Means? Otherwise, this page has code about fuzzy c-means including a distance calculation: https://pythonhosted.org/scikit-fuzzy/_modules/skfuzzy/cluster/_cmeans.html There is also a publication but no code: https://www.researchgate.net/publication/...


0

It all depends if your data was initially randomized or not. If the data was well organized in a specific order, you must shuffle it first, and then split to train/test sub datasets. Otherwise, your prediction will be wrong because a learning model need to study various potential configurations, and the best way to do it, is to use random train data and ...


0

You may encounter that if an attribute has previously be assigned as a key it will overwrite the dictionary value if the attribute names are not unique. Storing a list can mitigate this issue. import pandas as pd # convert to dict with list of values def convert_to_dict(df): df_dict = {} # empty dict for row in df.itertuples(): # ...


0

Evaluation is always based on the task, not on the method. Since the dictionary-based method gives an output similar to the ML-based approach, you can evaluate it in the same way, using a test set with gold-standard labels (preferably the same test set as the other method, or at least similar in size). Maybe what confuses you is that the dictionary-based ...


1

There is one obvious problem with this task: the result is not a real review, it's a generated text which looks like a review. Given that the point of a product review is usually to provide the reader with some information about the product, it's not clear to me how this task would be useful: if the review can be made without even testing the product, its ...


1

Looking at the source code, it seems that the first value is the median value for the epoch and the second value in parentheses is the global average/mean. The difference between the value for loss and loss_classifier is that the value for loss is the sum of the losses of the individual parts (including that of the classifier, 0.1807 + 0.0592 + 0.6662 + 0....


0

There are two common options: Use Bayesian methods to set a confidence threshold for predicting cat or dog. Retrain your model to be multinomial. It will learn to predict one of three outcomes - cat, dog, no cat or dog.


0

I ended up creating a new Pandas DataFrame using the code below. I wash hoping for something simpler or more elegant. # Create a new dataframe df_2d = pd.DataFrame() for _, sp in df.iterrows(): count = 0 if np.isnan(sp['Count']) else int(np.ceil(sp['Count'])) df_2d = df_2d.append([{"species": sp["species"], "Altitude": ...


0

Look at every sample as a string and calculate any string similarity (one example is Hamming distance). After calculating all similarities, you will have the similarity matrix a.k.a Affinity Matrix. Then You are all set for Spectral Clustering. Comment here if you had any further questions.


0

Definitely not. Because the classifier won't bother with how you name your classes, but how are they encoded. Basically, if you can have only one class per input value, you can use one-hot encoding, that is for each input value you will have a target converted from the string "Cat", for example, to [0, 1] given that you have two classes, where the ...


3

It's similar to a violin plot, which shows the shape of the distribution of a variable. However here the X axis shows only categorical values, it's not clear if the shape is based on some underlying numerical variable (this is a requirement for a violin plot). Violin plots can be made in Python or in R.


0

In order to just observe the shape of the DataFrame (named "iris" in your case), you can simply use the iris.shape or iris.values.shape and both should be fine. The difference in both being that iris.shape would simply print the shape of the DataFrame, whereas iris.values.shape would print the shape of the NumPy vector behind the DataFrame. ...


0

I'm not sure to understand well your problem. But many you could try using filter functions from Data Frame in Pandas. In addition to that, it seems more a code issue rather than a data science one: you might have better result in Stack Overflow for this kind of question.


0

Explanation is mentioned in comments # Created some data like yours data = { 'attribute_one':['male','34-45','graduate'], 'value_one':[10,17,32], 'attribute_two':['female','55-64','high school'], 'value_two':[15,8,5] } # Pandas for handling dataframes import pandas as pd # Created a dataframe from the given data df = pd.DataFrame(data) ...


1

This Universal Sentence Encoder that you link is trained specifically on English data, so it's going to work very poorly on any other language (to be clear, it's likely to produce garbage). Unfortunately it's quite unlikely that you'll find a similar pre-trained model for Macedonian. You would have to train your own model from Macedonian data, and you need a ...


2

Ideally you should have the historical data for your model to gain trust. However, since you don't have data you should atleast add the tenant/region feature to your model and train again. Also keep on retraining the model, as and when failures are detected in prediction. If different tenant users show very different behaviour then you should have different ...


2

I have found a way to create the corpus of potentially medical institutions by requesting the NCBI RESTful server, following the description in this link. First, you send an ESearch request containing some searching criteria (e.g. 'radiology', 'dicom', 'segmentation' - or whatever). As a response you obtain an XML document with a list of PubMed Ids. Then you ...


2

I can think of a couple options to collect a sample of medical institutions: Wikipedia has a list of hospitals by country (isn't Wikipedia amazing?) Many countries have some kind of national directory of medical institutions, but that would probably be difficult to scrap and specific to each country. UMLS has a category ("semantic group") for &...


1

This repo has list of tools for visualizing which might be useful. Though i would recommend to use monitoring tools like wannb with which you can visualize how each layer is behaving.


0

It is easier to understand if you use keyword arguments: model = xgb.train(params=param_list, dtrain=xgb_train, num_boost_round=num_rounds, evals=watchlist, obj=None, feval=customised_rmse, early_stopping_rounds=30) obj can be the objective function. feval can be the customized evaluation function. rmse is being used to minimize error on the trainning data. ...


0

In relation to the issue being related to the trainable attribute https://datascience.stackexchange.com/a/84067/51317 and if it's difficult to figure out which weights were set to trainable, one option is to try loading the weights by name with something like this (this doesn't cover all scenarios): def load_weights_by_name(model, path, verbose=False): ...


0

I would try to train a neural network with some sort of self-supervised approach, where you take all of your images and you change them in some ways (mess with colors a bit, rotate, rescale, etc.) and the task of the network is to create embeddings for these two to be close together and far from all the other images. The network will probably have harder ...


0

This would be another approach a little bit shorter: df.assign(resultado = lambda x: x.rolling(2).max()) EDIT: For your comment try: def idx(x): return x.index.values[np.argmax(x.values)] df.rolling(2).agg(['max', idx]) will return both, the pairwise maximum and the index that corresponds to that value.


0

You can create a new column like the one you have but "shifted" one position down, and the compute the maximum of these two columns: import pandas as pd import numpy as np data = np.random.randint(0, 50, size=20) df = pd.DataFrame(data, columns=['values']) df['prev'] = df['values'].shift(1) df['max'] = df[['values', 'prev']].max(axis=1) The ...


1

Why is the training accuracy so low? This is because your model is underfit. Few of the reasons for this could be, you might be using small learning rate. your model architecture is simple (small) and not big enough to recognize patterns from the data. Try increasing layers. try removing regularization if any. As per the best of my knowledge and ...


0

OOP in Python, pointing to advanced part of designing of class If you apply type on the name of a class itself, you get the class "type" returned (metaclass of MSELoss) and applying "type" to an object returns the class of which the object is an instance of . going back to your question: CLASS torch.nn.MSELoss(size_average=None, reduce=...


0

If the two data columns in your dataset contain similar type of information, you can find the corresponding columns by using the re.search() function. This function takes a regular pattern and a string and searches for that pattern within the string. This will return a match if the search is successful or else None. You can try using the following function ...


1

It clearly states that you are dealing with simple classification problem. So you don't need to go with CNN you can use Machine learning classification Algorithms like Logistic Regression. SVM KNN (k-Nearest Neighbor)


1

You could start off with something as simple as logistic regression to model your problem. You could then experiment with Random Forest Classifiers and then graduate to CNNs and RNNs.


0

The problem is that ImageDataGenerator.flow_from_directory shuffle argument is True by default, so you must set it False and get the data in alphanumeric order. This causes consistent label assignment to images and also because of data is totally unseen in training phase it won't causes any issue on generalization of model. test_batches = ImageDataGenerator(...


0

nocibambi is correct...to piggy back on his a simple implementation would look something like the following. I haven't run this as I'm at work, but hopefully it points you in the right direction. # example classifier svclassifier = SVC(kernel='linear') # fit svclassifier.fit(X_train, y_train) # predict y_pred = svclassifier.predict(X_test) # score print(...


2

I do not see your whole example but this usually happens when you have not initialized your classifier. Even more, to test, you first have to train your classifier (e.g. clf().fit(X_train, y_train)).


0

I found one solution thanks to this SO thread. What I ended up doing was the following: First, I made the indicator variable described in the question: df['High'] = np.where((df['Systolic'] >= 140) & (df['Diastolic'] >= 90) , 1, 0) Then I made a cumulative sum to count how many times this person has had a blood pressure spike by a specific week. ...


0

You can for instance tackle that as an Object Recognition task. Then, one possible solution is to use YOLO for object detection; Such networks split input images into many cells, and for each cell they predict a set of bounding boxes (and the corresponding object label, if any). You can for instance configure your network to recognize as much classes as ...


2

Performing such benchmark is not that easy. Meaning one can not just pick a few data set and run these models as there is a data dependency. In such cases, one need to simulate data through various process - the simulation helps to design various data in various condition. for example perhaps model one is doing a better job at binning so, the data with ...


Top 50 recent answers are included