New answers tagged

0 votes

Could not identify NUMA node of platform GPU ID 0 on M1 MacBook

Apple silicon is UMA, not NUMA. It's a unified memory architecture. So ignore this message. TF will work despite this annoying informational message.
user avatar
0 votes
Accepted

How can I implement lambda-mart with lightgbm?

Looks like the implementation of lambdaMART in the notebook referenced in the question is correct. From the paper titled, From RankNet to LambdaRank to LambdaMART: An Overview, it is clearly mentioned ...
user avatar
1 vote

Manipulating noise to get some data in right format and apply it to task using PPO

From my understanding of your question, you are looking to implement a learning-to-sort algorithm. There are current learning-to-sort machine learning solutions that do not require reinforcement ...
user avatar
2 votes
Accepted

Creating a grid type 3D data array from data points

You can use itertools.product to get a possible combinations of x, y, and z and then convert to resulting list to a numpy array: ...
user avatar
  • 5,844
0 votes

How to find the size of a tensor in bytes?

To do this in a line of code, use: size_in_bytes = encoding.nelement() * encoding.element_size() This multiplies the number of elements in your tensor by the size ...
user avatar
2 votes

How do I train a CNN using my own dataset?

That largely depends on the framework you're using. For Torch, there's torchvision.datasets.ImageFolder with ToTensor() ...
user avatar
  • 156
1 vote

Features and LSTM

Did you normalize your data with a min-max scaler? LSTM is a complex neuron, and its size should be adapted enough to your data: very simple models could under-perform because LSTMs are not suited for ...
user avatar
2 votes
Accepted

Plotly graph does not show x-axis values correctly

As per the plotly documentation on time series, you can use the update_xaxes method to change the ocurrence and format of the x-...
user avatar
  • 5,844
1 vote
Accepted

Manipulating noise to get some data in right format and apply it to task using PPO

In terms of process optimization, RL is an excellent option but the environment definition and its policy could be difficult to implement. That's why a genetic algorithm is a good alternative as it ...
user avatar
0 votes

python compare 2 dataframes

...with real datas ....don't work : Here is my real dataframes df1 and df2 (with columns) ...
user avatar
0 votes

Odd error when training neural network with Keras - Error occurred when finalizing GeneratorDataset iterator

If it stops always after 36 epochs, it could be due to the data in input. You should check if the generated data has always the same dimension and same format, without any outlier (like an empty ...
user avatar
0 votes

python compare 2 dataframes

Your files are delimited by semicolons, not commas. You need to specify this in the read_csv function. Some other problems: There is no need to convert a data_1 dataframe to a df1 dataframe. You do ...
user avatar
0 votes
Accepted

I want to make a model that minimizes the heat supply, what should I do?

Modeling any industrial process is quite complex because there are a lot of physical and non-linear events. That's why I usually recommend simulating the most important processes first with a ...
user avatar
0 votes

Small difference in metrics in KERAS for the same model

I found explanation here: https://github.com/tensorflow/tensorflow/issues/29964 https://stackoverflow.com/questions/59118430/keras-model-evaluate-on-training-and-val-set-differ-from-the-acc-and-val-...
user avatar
1 vote

Why use fit when already have fit_transform?

A relatively late answer, but it is also very convenient to first fit all the data then during the deep learning training loop ...
user avatar
  • 11
1 vote

Clustering 3D with survey data

Convert your categorical choices into a range of 0 to 1. Convert your 1-10 scale to 0-1. Throw sklearn k-means at it. Use the elbow method for deciding how many clusters there are. For plotting the ...
user avatar
0 votes
Accepted

Sampling from earths landmass

You can use cartopy to achieve this easily. pip install cartopy For instance: ...
user avatar
1 vote

How to further improve on overfitting?

This was an issue I was struggling with for over a week but the eventual problem seemed to be perhaps something in the way the function was done; Initially I used ...
user avatar
  • 111
0 votes

PicklingError in pyspark (PicklingError: Can't pickle <class '__main__.Person'>: attribute lookup Person on __main__ failed)

My solution: Put class Person into a separate e.g. "utility.py" import utility import pickle new_person = pickle.loads(pickle.dumps(utility.Person())) ...
user avatar
0 votes

ValueError: cannot reshape array of size 36276416 into shape (96,227,227,1)

It is normal that it can't be reshape, because: 36276416 / (96227227*1) = 36276416 / 4946784 = 7.33333333 which is not an integer result. Maybe there is a problem with some images' size or color ...
user avatar
2 votes
Accepted

How do I not display rows that have an empty value when trying to output a dataframe with pandas

You can simply filter out those rows using pandas indexing: df[df["Value2"].notna()]
user avatar
  • 5,844
1 vote
Accepted

Optimize daily ice cream profit beased on simulation of all combinations input variables

I would propose a solution like this: Train a regression model which predicts the sales (target variable) based on all the features (both types: those you have control on and those you don't). ...
user avatar
  • 21.8k
1 vote

Prohibitive size of random forest when saved to disk

I ran into a similar issue and was surprised to find out that indeed decision trees can easily take a lot of memory (range of MBs) and random forests will easily multiply that in the GB range. Details ...
user avatar
0 votes

Fasttext error while loading wiki pre-trained data

The first answer is good but might make you confused. Those two steps should be implemented together. See this link: https://github.com/RaRe-Technologies/gensim/issues/2378#issuecomment-791999124 ...
user avatar
1 vote

How to measure similarities between two datasets with same features?

You can use statistical approach and try computing KL-divergence between the 2 datasets (Distributions). However, the KL-Divergence output is between 0 and ∞ (0 meaning two distributions perfectly ...
user avatar
0 votes

Threshold tuning with one-vs-rest for multi classification python

In general it's not possible to tune any threshold in multiclass classification: In binary classification, modifying the threshold means predicting more or less instances as positive, because the two ...
user avatar
  • 21.8k
1 vote

All classification models except neural network giving 100% accuracy

It could be due to a lack of initialization of your neurons. Did you initialize them randomly? For instance: ...
user avatar
0 votes

Why is the type of pytorch MSELoss class 'type'?

it is because you haven't instantiated the object, add () after the name of the class ...
user avatar
1 vote

Isolation Forest Feature Importance

Interpretable Anomaly Detection with DIFFI: Depth-based Isolation Forest Feature Importance The Isolation Forest is one of the most commonly adopted algorithms in the field of Anomaly Detection, due ...
user avatar
  • 2,336
0 votes

How is cyclic time series data stationary

According to Forecasting: Principles and Practice by Rob J Hyndman and George Athanasopoulos, time series with trends, or seasonality, are not stationary. On the other hand, a time series with no ...
user avatar
-1 votes

Techniques for classification AI with sparse labels

I need an AI/training method which accounts for these inaccuracies by giving more weight to present labels and less weight to labels not present. This is a multi-label classification problem. One ...
user avatar
  • 2,309
1 vote

Binary classifier high overall ROC AUC but low in different bins

One interpretation of the AUROC is "the probability that a randomly selected positive instance is given a higher probability by the model than a randomly selected negative instance." With ...
user avatar
  • 9,591
3 votes

XGboost Classifier predicits different results for same samples depending on test dataset size

So the mistake was to use the DMatrix datatype while using categorical data in the dataset. I split the data and then stored it in a DMatrix with ...
user avatar
  • 41
0 votes

Sub labelling of an object

Sounds like face segmentation is what you are looking for. This one looks pretty promising at first glance: https://github.com/zllrunning/face-parsing.PyTorch Does it improve inference efficiency? ...
user avatar
  • 2,309
1 vote

What is the right way of training Regression model having various categories involved?

The question of the role of this 'category' column matters: If the categories are independent, i.e. instances in category X have no relation with instances in category Y, then it makes more sense to ...
user avatar
  • 21.8k
0 votes

How to get different results running sklearn's MeanShift in a single program? (Python3)

A couple of work arounds: 1- Set bin_seeding to True, and/or using random seeds at each iteration. 2- Shuffling the data at each iteration.
user avatar
2 votes

Plot distribution of multi classification with features - Python

Before you think about visualization, you need to come up with the questions you would answer using visualization techniques. 1- For example, you could ask how is X feature different in different ...
user avatar
1 vote

Which Model for predicting flight delays is appropriate except Random Forest and Decision Tree? (Monte Carlo?)

Weather is responsible for 90% of the flight delays. How is it possible to make reliable predictions with just 10% of the remaining causes? (if their data is available) You have an existing map called ...
user avatar
0 votes

Generating synthetic data based off existing real data (in Python)

One option is the Python package imblearn which contains the SMOTE algorithm. SMOTE generates synthetic samples from a real dataset by interpolating plausible new datapoints based on observed data.
user avatar
0 votes

How to perform entity level train-val-test split for NER task?

I know it's a bit late, but I had the same question and developed a method which is available here: ...
user avatar
1 vote

Interpreting a curve val_loss and loss in keras after training a model ; help

The training loss decreases while the validation loss increases, which is a clear sign of overfitting. This means that your model is too specific to the training dataset and do not generalize to the ...
user avatar
  • 489
2 votes
Accepted

Seaborn Heatmap with month & hour of database entry

...
user avatar
0 votes

Merging two datasets with different features for machine learning prediction

...
user avatar
0 votes

Classification - get some label value to check how close to another class (Python)

With Naive Bayes you can predict the probability of a text to belong to each class using the predict_proba method. Using this method you'll get a vector of ...
user avatar
  • 489
0 votes

How to examine effect of variable not used in training a neural network

If you have df the dataset with all the columns df_train and df_test the datasets for train ...
user avatar
  • 489
3 votes

Binary classifier high overall ROC AUC but low in different bins

My two cents: If the goal is to predict the different bins corresponding to the probability of predicting the positive class, then this seems a strange design: why use binary classification if the ...
user avatar
  • 21.8k
1 vote

precision and recall at k for movielens dataset

I used a regression model and then considered every movie with a true rating higher than 3 as relevant. To me, it seems you are introducing bias here (above quote)...
user avatar

Top 50 recent answers are included