Questions tagged [noise]
The noise tag has no usage guidance.
38
questions
1
vote
1
answer
19
views
Which loss function to use for a convolution NN for noise removal of high resolution images
My task is to remove small random spots from my 4 mega pixel images.
My strategy was to feed a convolution network these images as I have the true images without the spots in them.
The current loss ...
- 11
0
votes
0
answers
16
views
Fixing/up-rezzing images is easily possible these days in practice. Is there any analogous tool available for low-quality audio?
Tools like waifu2x and Anime4k (for the MPV video player) can already easily de-noise and up-rez low quality images to 4k quality pretty darn well. Similar tools like DLSS and FSR are used for video ...
- 111
0
votes
0
answers
19
views
Practical application of denoising autoencoders
I have been reading into autoencoders for the purpose of denoising data. In the examples i found (eg. [1, 2, 3], which are the first few google results) they have the following input/output:
Input ...
0
votes
0
answers
19
views
What is the minimal number of examples for BERT-like language model for the model to train a word
I have heard rumors of a particular count of positive examples that allowed the model to train a given word (or context of it - when talking about MLM) to be ~40. I am wondering though about the ...
- 207
0
votes
0
answers
10
views
Metrics for changes in datapoints
I am looking for a metric which quantifies the noise in data with respect to neighbouring data points. I have time series data and I want to check how much values change from a previous to a following ...
0
votes
0
answers
39
views
Time Series Data Noise Handling Questions
There are manufacturing time series data as shown in the picture.
The average of the two variables is about 100.
However, the noise value is 6500 and 65000, which is too different from other values.
I ...
- 11
0
votes
0
answers
9
views
Finding the delayed effect of a change in input
I am trying to figure out the delay between changing the speed of a pump that pumps a modifier into a process and the change in Amps drawn by an extruder at the end of the process. The amps drawn are ...
- 1
0
votes
0
answers
17
views
KDE Sampling with negative density and/or class-specific weighting
I have a dataset which contains two overlapping distributions/classes of points. I have been trying to sample from just one of these distributions/classes using the scikit learn Kernel Density class, ...
- 45
0
votes
0
answers
16
views
Separating image signal from constant noise sources
I'm working on image signal from a sensor where the incoming signal consist of high degree of constant noise. The noise patterns are multiple, both with very low frequency and very high frequency but ...
- 1
0
votes
0
answers
48
views
Multi-task learning for improving segmentation
I am building a multi-task model, where my main task is segmentation and my auxiliary task is either denoising or image inpainting. The goal is to try to improve the quality of the segmentation with ...
- 11
0
votes
0
answers
13
views
Deep learning model for very sparse object detection in noisy images
I am trying to build a model that takes in very noisy 200x200 greyscale images of spatially sparse objects and attempts to localise them with bounding boxes. The objects are very thin streaks (data of ...
- 21
0
votes
1
answer
213
views
Using numpy to enter noise into data
I am new to data science and have to generate 200 numbers from a uniform distribution
set this as x and generate y data using x and injecting noise from the gaussian distribution y = 12x-4 + noise
...
- 107
0
votes
1
answer
993
views
TypeError: __init__() missing 1 required positional argument: 'num_features'
I was trying to denoise image using Deep Image prior. when I use ResNet as an architecture i am getting error.
...
- 103
1
vote
0
answers
74
views
Model Tree M5 - Robustness to Data Quality Issues
I am currently investigating the M5 tree algorithm by Quinlan(1992) link here: https://sci2s.ugr.es/keel/pdf/algorithm/congreso/1992-Quinlan-AI.pdf
An example of a linear regression model of the ...
- 53
0
votes
2
answers
308
views
How to group every data point with HDBSCAN to some group to have no noise?
TASK
I am clustering products with about 70 dimensions ex.: price, rating 5/5, product tag(cleaning, toy, food, fruits)
I use HDBSCAN to do it
GOAL
The goal is when users come on our site and I can ...
- 145
1
vote
0
answers
21
views
Denoising Prior to Image Classification
From what I have read, Denoising during preprocessing for image classification tasks seems to be a bit controversial.
While on one hand it might improve classification accuracy, the computational ...
- 123
2
votes
1
answer
2k
views
Choosing attributes for k-means clustering
The k-means clustering tries to minimize the within-cluster scatter and maximizing the distances between clusters. It does so on all attributes.
I am learning about this method on several datasets. To ...
- 189
2
votes
2
answers
38
views
Which algorithm to use to identify clusters with a similar value?
Here, an example of my problem:
10000 observations of people with several features [age, gender, region, number of sons, ...] and a value to predict "income".
There is not a general relationship ...
- 61
2
votes
1
answer
106
views
Methods for learning with noisy labels
I am looking for a specific deep learning method that can train a neural network model with both clean and noisy labels.
More precisely, I would like this method to be able to leverage noisy data as ...
- 156
2
votes
1
answer
210
views
Remove noise by clustering on which step of pre-processing is better?
I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes.
Part of my model includes the following preprocessing steps:
remove ...
- 221
1
vote
0
answers
33
views
Classification Algorithms For Noisy Problems
Dear fellow data scientists
I work in finance and have decent experience with neuronal networks. Right now I am facing a challenge where I need to classify a set of signals.
The data set looks ...
- 11
2
votes
2
answers
1k
views
If my model is overfitting the training dataset, does adding noise to training dataset help regularizing the machine learning model
I would like to know if this is a best practice or not. Can we add noise to the training data to help the model "fit less the training data"; as a result, hoping to generalize better on new unseen ...
- 239
3
votes
0
answers
112
views
How to remove noise using morphological filtering
I have two groups of dots that both contain noise between them:
The line that separates the two groups in the picture is diagonal in shape.
I tried to use morphological filtering on this image to ...
- 43
1
vote
1
answer
207
views
Noise Elimination with majority vote filtering
I have a dataset with label noise which I wan't to clean with majority/consensus vote filtering. This will mean I will divide the data in K-Folds and train an ensemble model. Than using the ...
- 11
1
vote
0
answers
20
views
Isolated/noisy instances that have outsized effect on SVM hyperplane selection
Consider two linearly separable classes and the optimal separating hyperplane (image credit Prof Jiawei Han of UIUC):
Now consider if there were a single "rogue" point for the Yellow class - which ...
0
votes
1
answer
2k
views
what is correct way to perform normalization on data in Auto encoder?
working on anomaly detection problem. i'm using auto-encoder to denoise given input. I trained network with normal data(anomaly free). so model predict normal state of given input. Normalization of ...
3
votes
0
answers
209
views
Training deep CNN with noisy dataset
I am training a Mask RCNN model with a train dataset that has been generated from some simple computer vision operations (color thresholding) and some morphological filtering.
The train set captures ...
- 161
3
votes
1
answer
2k
views
Effect of adding gaussian noise to the input layer in a CNN
I often come across Keras code that adds GaussianNoise to the input, however its not clear to me what advantages does it offer to the learning.
...
- 183
1
vote
1
answer
2k
views
How to create a complex Gaussian random noise with a specific covariance matrix
I am trying to generate a complex Gaussian white noise, with zero mean and the covariance matrix of them is going to be a specific matrix which is assumed to be given.
Assume i to be a point on the ...
- 111
0
votes
1
answer
31
views
How can I avoid requiring global information for performing regression on meter variables?
Note: With a meter variable a timestamped value is the sum of all previous differences plus a difference to the most recent value. Think of a electricity meter counting the use of energy.
The goal ...
- 103
2
votes
1
answer
109
views
Paramaeter estimation in noisy conditions with Machine Learning, possible?
Let's take two constants, $\alpha$ and $\beta$, both are given by two functions $f_1(\vec{\theta})$ and $f_2(\vec\theta$) (the model). These functions are known: we have an analytical closed ...
- 1,628
1
vote
0
answers
284
views
Rain radar image noise reduction and cleanup
An application that I am building is plotting rain radar images on map. The images are transparent PNGs that are sourced straight from the local meteorological service.
As can be seen in the example, ...
- 133
5
votes
1
answer
1k
views
Training on accurate data versus noisy data
I have data currently available that is very accurate and I would like to train my classification methods on this set of clean data to learn the important markers for distinguishing between classes. ...
- 185
0
votes
1
answer
70
views
Reducing noisy data from non normal distribution of data with std deviation?
I have used MATLAB code and get the two different row vectors A=1×18 and B=1×350. From both row vectors separately I need to remove the noisy data by using standard deviation. But the problem is that ...
3
votes
2
answers
120
views
What is the loss function defined by Mnih and Hinton in their paper “Learning to Label Aerial Images from Noisy Data”?
In section 3.3 of the paper, they state that they use the cross entropy.
Then they define the probability for a label to be a false positive as $\theta_0$ and a false negative as $\theta_1$.
They use ...
- 141
1
vote
1
answer
37
views
When we should use binning to redure noise? or How we find out we have noise?
I read several times which binning is helpful for reducing the noise of data. But how can we find out our data has noise? What if our data is clean and we reduce the accuracy of data?
Is there any ...
- 751
2
votes
0
answers
94
views
Rprop is too noisy
Is there a way to reduce the noisiness and stochasticity of Rprop (and for that matter the iRprop+)?
Specifically, in deep networks (with 8+ layers) this effect starts to become apparent, as the ...
- 2,586
6
votes
2
answers
6k
views
Can Neural Networks be trained to smooth values / output the average?
Let's say we have a neural network with one input neuron and one output neuron. The training data $(x, f(x))$ is generated by a process
$$f(x) = ax + \mathcal{N}(b, c)$$
with $a, b, c \in \mathbb{R}^...
- 17.9k