Questions tagged [weighted-data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Negative values when calculating weighted Jaccard similarity

I have a bilateral dataset that includes countries and their the weight of their relation. I'd like to calculate the similarity of countries in 1) who their trade parterns are and 2) the weight of the ...
Johanna's user avatar
1 vote
1 answer
28 views

Neural Network Weights - How do they know their position?

I am a copyright scholar so please forgive my ignorance. When weights are stored external to a model what is the mechanism by which the weight knows which neuron or node in a decision tree it is ...
Benjamin White's user avatar
0 votes
0 answers
17 views

How to calculate the sum of all the possible answers in a single-choice questionnaire?

I am not sure if this is the right place to ask this question but imagine there are a number of single-choice questions with each answer having a numeric value (weight) assigned to it. For example: <...
xbmono's user avatar
  • 101
1 vote
0 answers
30 views

With infinite observations, would the weights resulting from ridge regression be the same as simple linear regression?

As the number of observations approaches infinity, do the weights of a linear regression approach the weights of a linear regression with L2 penalty?
BigMistake's user avatar
0 votes
0 answers
38 views

How to properly linearize data (if possible)

I was assigned the task of linearizing some of my data, which exhibits a non-linear appearance. When using the distfit library, it indicated that my data's distribution is closest to a gamma function. ...
Guilherme Raibolt's user avatar
0 votes
0 answers
10 views

How to account for exponential oversampling?

I have a dataset of frequencies from 20Hz to 20kHz. The measurement rig that created the dataset doesn't sample evenly across this frequency range. There is a much higher density of sampling happening ...
db2791's user avatar
  • 101
0 votes
1 answer
47 views

Train CNN weights by using FFT - Reinforcement Learning?

Assume that you are doing convolution inside a CNN network, by using FFT because FFT is much way faster than using 4-5 for-loops etc. But how should I train the weights if I know the output of my CNN ...
euraad's user avatar
  • 115
0 votes
1 answer
114 views

Learning with duplicate count as sample weights

I have a dataset D: X = D.drop(columns=['target']) y = D['target'] D is large, but contains huge number of duplicates - and I want to speedup the learning process. ...
Ars ML's user avatar
  • 61
0 votes
0 answers
44 views

How to calculate the coefficients of a weighted nonlinear regresion model

I'm trying to build an Excel-sheet that's able to find the a,b and c coefficient of a simple y=ax^2+bx+c model. This is clear enough by the following math. But the data I'm using is heteroscedastic, ...
DrDirk's user avatar
  • 1
0 votes
1 answer
35 views

How to assign weights in data analysis

I have a dataset that has a field 'Course' and for each course, there is a varying number of lessons. I am trying to compare the number of quizzes accessed for each of the lessons and group by course ...
hyeri's user avatar
  • 141
0 votes
0 answers
436 views

How to calculate the weighted sum for Neural Network

I like to clear the concept for neural network. I have a dataset. It is a toy dataset ...
Encipher's user avatar
  • 359
0 votes
0 answers
43 views

Custom layers in Keras -- custom weights

I am trying to understand how to build custom layers in Keras and I went through a couple examples: here and here. The syntax is, of course, similar, but in non of the cases it is addressed why ...
MsTais's user avatar
  • 101
1 vote
0 answers
215 views

XGBoost: How to obtain scale_pos_weight for multi classes?

I know there is a similar Qn at Unbalanced multiclass data with XGBoost. But I don't understand the reply provided by @Esmailian. What is the actual formula to obtain 1, 0.333 and 0.167? ...
Peter's user avatar
  • 179
0 votes
0 answers
94 views

Is there R functions that allow to test for overdispersion when fitting a model with survey design?

I realized I need to use the package survey to be able to include sample weights in my regression analysis. Initially, I wanted to use a negative binomial regression on each one of my outcomes as ...
airpoll_epi's user avatar
0 votes
0 answers
42 views

Weighting the loss function based on previous seen true positive rates

Similiar to class imbalance there is always something I would call "learnability imbalance" in multi-class classification. What I mean by that: Even when the classes are evenly distributed ...
T Piper's user avatar
  • 123
1 vote
1 answer
63 views

Reducing (Variance) | the gap between my weights

I have ML ready samples. And each sample has a weight. The weights distribute between [0-1] My problem arise because there are a lot of samples which are ...
Eran Moshe's user avatar
3 votes
1 answer
3k views

Specifying class or sample weights in Keras for one-hot encoded labels in a TF Dataset

I am trying to train an image classifier on an unbalanced training set. In order to cope with the class imbalance, I want either to weight the classes or the individual samples. Weighting the classes ...
user1158795's user avatar
0 votes
2 answers
66 views

Do we have any method which handles the imbalanced classification with sample weighting instead of class weighting?

I am looking for methods that use sample weighting instead of class weighting for the imbalanced classification. I think sample weighting is more precise than weighting all the samples from one class ...
user137927's user avatar
0 votes
1 answer
1k views

Weighted accuracy, sensitivity and specificity

I have a confusion matrix TN= 27 FP=20 FN =11 TP=6 I want to calculate the weighted average for accuracy, sensitivity and specificity. I know the equation but unsure how to do the weighted averages.
Francesca Gavins's user avatar
1 vote
1 answer
35 views

Integer encoding and weighing when one feature consists of more names [closed]

Hello I am trying to make a content based movie recommendation system and one feature is genre of the movie. I will give an integer number to each genre randomly. However, some movies are of more than ...
OnurTR's user avatar
  • 25
4 votes
3 answers
131 views

Understanding Weighted learning in Ensemble Classifiers

I'm currently studying Boosting techniques in Machine Learning and I happened to understand that in Algorithms like Adaboost, each of the training samples is given a weight depending on whether it was ...
AnonymousMe's user avatar
1 vote
0 answers
17 views

Class weights in RSNSS package?

New to ML in general and have been Googling on this. I am working on a dataset that will rate each customer features to a credit worthiness class attribute. Is it possible to add class weights in the '...
datanewbie96's user avatar
0 votes
1 answer
60 views

Methods of disaggregating data to smaller units?

I have a relatively straightforward question that I know poses some difficult challenges. Let's say I have a state-level rate of X. I would like to disaggregate the state-level rate to the county-...
bashity's user avatar
  • 103
3 votes
1 answer
3k views

How to interpret the sample_weight parameter in MiniBatchKMeans?

I am using scikit-learn MiniBatchKMeans to do text clustering. In the fit() function there is a parameter sample_weight described as follows: The weights for each ...
robertspierre's user avatar
0 votes
0 answers
99 views

How to balance dataset using fit_generator() in Keras?

I am trying to use keras to fit a CNN model to classify 2 classes of data . I have imbalanced dataset I want to balance my data equally. How I can do that ?? Any help would be appreciated The main ...
Edayildiz's user avatar
1 vote
1 answer
539 views

How and where to set weights in case of imbalanced cost sensitive learning in machine learning?

I confront with a binary classification machine learning task which is both slightly imbalanced and cost sensitive. I wonder what (and where in the modeling pipeline, say, in sklearn) is the best way ...
Fredrik's user avatar
  • 1,007
1 vote
2 answers
725 views

Assign more importance to recent records during training

My goal is to build a classification model in order to predict if a customer will buy a product or not (binary classification). Since in the last months (let's say 3-4) I know that the advertising of ...
A1010's user avatar
  • 193
4 votes
1 answer
4k views

Weighted loss functions vs weighted sampling?

For image classification tasks, is there a practical difference between using weighted loss functions vs. using weighted sampling? (I would appreciate theoretical arguments, experience or published ...
Jean-Pierre Coffe's user avatar
1 vote
0 answers
43 views

Non-monotone missing data, and inverse probability weighting

I'm having difficulty identifying whether or not my missing data pattern is 'monotone'. I have two variables with missing data, and the missing data patterns in each variable do not completely ...
Alice's user avatar
  • 11
0 votes
1 answer
286 views

When exactly should I use weighted loss?

Do I have to use it in any case when the class distribution is imbalance (Train: class A:10%, B:90% and Test: class A:10%, B:90%) or when it is different (Train: class A:10%, B:90% and Test: class A:...
user97950's user avatar
1 vote
1 answer
249 views

Heterogeneous clustering with text data

I have a dataset which consists of multiple user ratings. Each rating looks similarly to: ...
qbit-'s user avatar
  • 13
0 votes
1 answer
53 views

Closed form of Weighted Ordinary Least Squares calculation of the trend line

I would like to know if there is a closed form version of this equation: $\beta = \frac{n\sum{xy}-\sum{x}\sum{y}}{n\sum{x^2}-(\sum{x})^2}$ But for weighted data, where the weight $w_i$ is the value ...
Juan Esteban de la Calle's user avatar
1 vote
0 answers
251 views

Grouping by 2 Columns in R to Sum, Count, Percentage, Weighted mean, mode [closed]

I am trying to use dplyr or data.table to calculate sums, percentages, counts, and weighted means for various sub groups. Each Name can have subgroups High Medium and low, and potentially others, and ...
DataGuy23's user avatar
1 vote
1 answer
940 views

Counting Number of Parameters in Neural Networks [closed]

Note: This is an academics based problem. So in a recent in-class quiz, we were asked that if we have an input layer consisting of 20 nodes along with 2 hidden layers (one of size 10 and the other of ...
x89's user avatar
  • 191
1 vote
1 answer
49 views

On assiging weights for unbalanced classes

Consider a dataset that will be split into train and test. The model will be learned using the train set and evaluated using the unseen test set. Now the dataset is unbalanced -- it contains more ...
Sm1's user avatar
  • 531
2 votes
0 answers
1k views

XGBoost predicting everything as null when sample weights are passed

I am trying to build an Uplift model using observational data. The data is consists of collections calls to customers and my objective is to predict the incremental probability due to the treatment (...
Arindam Bose's user avatar
0 votes
1 answer
765 views

How to generate 12 independent random weights which all add up to one

I'm using Palisade's @Risk software with a triangular distribution to generate 12 random weights which must add up to one, but I get a lot of negative numbers. Is there a straightforward way to set ...
Angus's user avatar
  • 103
0 votes
0 answers
505 views

How to visualize the weights of the output(classification) layer?

I'm doing a Convolutional Neural Network of MNIST data set, and how can I visualize the weights of the output(classification) layer? I only found several websites visualizing the filters in CNN model ...
Shania Bu's user avatar
0 votes
1 answer
259 views

Clustering with custom criterion (minimum cluster weight)

Edit: following comment from @anony-mousse, I'm changing the question to search for a general clustering approach that matches this criterion (minimum weight per cluster). I am to use a clustering ...
couturierc's user avatar
2 votes
1 answer
103 views

Training a model where each response in the observation data has a different known varience

I have a dataset where each response variable is the number of successes of N Bernoulli trials with N and p (the probability of success) being different for each observation. The goal is to train a ...
dln's user avatar
  • 21
11 votes
1 answer
8k views

CNN - imbalanced classes, class weights vs data augmentation

I have a dataset with a few strongly imbalanced classes, eg. the smallest class is about 54 times smaller than the largest. Therefore, data augmentation in order to equalize the size of classes seems ...
I.D.M's user avatar
  • 175
1 vote
1 answer
1k views

How are weights calculated in a feed-forward neural network before they are summed up with bias?

I have read a lot of papers and watched different videos, it seems like they explain how they are summed up with bias before entering the activation function. What I am trying to understand is the ...
Mohamed Mahyoub's user avatar
13 votes
1 answer
7k views

Why doesn't class weight resolve the imbalanced classification problem?

I know that in imbalanced classification, the classifier tends to predict all the test labels as larger class label, but if we use class weight in loss function, it would be reasonable to expect the ...
user137927's user avatar
2 votes
1 answer
1k views

Calculate weighted mean for two columns and hundreds of rows?

I'm trying to teach myself basics of R and I couldn't find the answer: Say, I have a csv file and I want to calculate weighted mean for each subject such that I have a mean mu = 0.015*1030+0.16*26930+...
InfiniteLoop's user avatar
12 votes
7 answers
22k views

How to apply class weight to a multi-output model?

I have a model with 2 categorical outputs. The first output layer can predict 2 classes: [0, 1] and the second output layer can predict 3 classes: [0, 1, 2]. How can I apply different class weight ...
Gal Avineri's user avatar
6 votes
3 answers
5k views

using sklearn class weight to increase number of positive guesses in extremely unbalanced data set?

Hi I have a poorly correlated and unbalanced data set I have to work with. The set is 2 classes, 0 has 96,000 values and 1 has about 200. When I run random forest or other methods I get an output like:...
user3107977's user avatar
2 votes
2 answers
1k views

How to weight loss in regression

I've got a regression problem where a model is required to predict a value in the range [0, 1]. I've tried to look at the distribution of the data and and it seems that there are more examples with ...
Gal Avineri's user avatar
0 votes
1 answer
2k views

Class weights for imbalanced data in multilabel problems

I am trying to train a CNN for a multiclass - multilabel classification task (20 classes, each sample can belong to 1+ labels) and the dataset is highly imbalanced. In single-label cases I would use ...
Costas Papastamos's user avatar
5 votes
2 answers
2k views

How to use class_weight parameter for validation set?

I am using Keras' class_weight parameter to deal with an imbalanced class problem. I am doing this to define the weights : ...
MysteryGuy's user avatar
0 votes
1 answer
99 views

Differentiating roadmap of a loss function

Let's say I'm performing Stochastic Gradient Descent (SGD) on binary cross entropy error while optimizing weight $w_{2}$. Binary cross entropy error: $$L(y|p(x_{i}))=-y_{i}*ln(p(x_{i}))-(1-y_{i})*...
ShellRox's user avatar
  • 409