Questions tagged [noise]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

How do design a 2 class+noise classifier system

I need to train a 2 class classifier for a 30 x 6 frame. In the dataset, there exists data for class A and Class B, but there is a lot of junk data as well which particularly does not classify into ...
Fr_nkenstien's user avatar
0 votes
1 answer
18 views

What are some Models/Methods to reduce noise using environmental data?

I have a set of pressure datasets from a mechanical device that frequently moves around the country. I also have several sets of environmental data (Altitude, ambient temperature etc.) from those ...
PressureQuery's user avatar
0 votes
0 answers
11 views

Dealing with noise in softmax output

I have a device with an accelerometer and gyroscope (6-axis). The device sends live raw telemetry data to the model 40 samples for each input, 6 values per sample (accelerometer xyz, gyroscope xyz). ...
Sterling Duchess's user avatar
0 votes
0 answers
22 views

Should I remove noise by fitting a curve in my case?

I have an object that is moving towards target point and I want to determine when it will reach or pass through it. I receive points in real time, which represent the current location of the object. ...
theateist's user avatar
  • 141
0 votes
0 answers
14 views

Is there an unified pipeline to convert a 4D nifti noisy pet scan image to an averaged, denoised, 3D Nifti image?

What I'm looking is to do at least the step 2 of ADNI pre processing (coregistered, averaged): https://adni.loni.usc.edu/methods/pet-analysis-method/pet-analysis/ OASIS-3 dataset provides FDG PET ...
hhaamm's user avatar
  • 143
0 votes
1 answer
22 views

Investigating the Impact of Additive Gaussian Noise on EEG Signal Classification: Analyzing the Relationship between Augmented and Original Data

Definition: I have conducted research on EEG signal classification, specifically focusing on distinguishing between two different classes using raw EEG signals. Data availability poses a significant ...
Armin Amini's user avatar
0 votes
0 answers
15 views

Is there a name for this metric I came up with?

I am working with a data set that tracks a daily distinct count of user_ids that meet a specific criteria. There is some noise in the data and I want to differentiate groups that have high and low ...
Matt Johnson's user avatar
0 votes
1 answer
22 views

How to denoise overrepresented lines in sample of 2D (geospatial) data?

I have some geospatial data in lat/lon form accurate to 6th decimal place. As shown in the picture below, there are some over-represented lines of points at specific latitudes which appear in the ...
Brendan Hill's user avatar
0 votes
1 answer
193 views

How to smooth noisy data set?

I have a univariate dataset that is locally jagged (lots of ups and downs) that I need to smooth. I've also tried polynomial features with linear regression, but because the curve is extremely non-...
ron burgundy's user avatar
1 vote
1 answer
29 views

Which loss function to use for a convolution NN for noise removal of high resolution images

My task is to remove small random spots from my 4 mega pixel images. My strategy was to feed a convolution network these images as I have the true images without the spots in them. The current loss ...
ando's user avatar
  • 11
0 votes
0 answers
54 views

Practical application of denoising autoencoders

I have been reading into autoencoders for the purpose of denoising data. In the examples i found (eg. [1, 2, 3], which are the first few google results) they have the following input/output: Input ...
Leander Moesinger's user avatar
0 votes
0 answers
40 views

Time Series Data Noise Handling Questions

There are manufacturing time series data as shown in the picture. The average of the two variables is about 100. However, the noise value is 6500 and 65000, which is too different from other values. I ...
hahaha's user avatar
  • 11
0 votes
0 answers
79 views

Multi-task learning for improving segmentation

I am building a multi-task model, where my main task is segmentation and my auxiliary task is either denoising or image inpainting. The goal is to try to improve the quality of the segmentation with ...
A. Cimet's user avatar
0 votes
1 answer
562 views

Using numpy to enter noise into data

I am new to data science and have to generate 200 numbers from a uniform distribution set this as x and generate y data using x and injecting noise from the gaussian distribution y = 12x-4 + noise ...
joseph's user avatar
  • 107
0 votes
1 answer
2k views

TypeError: __init__() missing 1 required positional argument: 'num_features'

I was trying to denoise image using Deep Image prior. when I use ResNet as an architecture i am getting error. ...
harsha's user avatar
  • 103
1 vote
0 answers
146 views

Model Tree M5 - Robustness to Data Quality Issues

I am currently investigating the M5 tree algorithm by Quinlan(1992) link here: https://sci2s.ugr.es/keel/pdf/algorithm/congreso/1992-Quinlan-AI.pdf An example of a linear regression model of the ...
chrisper's user avatar
0 votes
2 answers
588 views

How to group every data point with HDBSCAN to some group to have no noise?

TASK I am clustering products with about 70 dimensions ex.: price, rating 5/5, product tag(cleaning, toy, food, fruits) I use HDBSCAN to do it GOAL The goal is when users come on our site and I can ...
sogu's user avatar
  • 145
1 vote
0 answers
32 views

Denoising Prior to Image Classification

From what I have read, Denoising during preprocessing for image classification tasks seems to be a bit controversial. While on one hand it might improve classification accuracy, the computational ...
madprogramer's user avatar
2 votes
1 answer
2k views

Choosing attributes for k-means clustering

The k-means clustering tries to minimize the within-cluster scatter and maximizing the distances between clusters. It does so on all attributes. I am learning about this method on several datasets. To ...
Borut Flis's user avatar
2 votes
2 answers
46 views

Which algorithm to use to identify clusters with a similar value?

Here, an example of my problem: 10000 observations of people with several features [age, gender, region, number of sons, ...] and a value to predict "income". There is not a general relationship ...
A M's user avatar
  • 61
2 votes
2 answers
176 views

Methods for learning with noisy labels

I am looking for a specific deep learning method that can train a neural network model with both clean and noisy labels. More precisely, I would like this method to be able to leverage noisy data as ...
Mathias Müller's user avatar
2 votes
1 answer
299 views

Remove noise by clustering on which step of pre-processing is better?

I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes. Part of my model includes the following preprocessing steps: remove ...
motevalizadeh's user avatar
1 vote
0 answers
37 views

Classification Algorithms For Noisy Problems

Dear fellow data scientists I work in finance and have decent experience with neuronal networks. Right now I am facing a challenge where I need to classify a set of signals. The data set looks ...
Neo's user avatar
  • 11
3 votes
2 answers
2k views

If my model is overfitting the training dataset, does adding noise to training dataset help regularizing the machine learning model

I would like to know if this is a best practice or not. Can we add noise to the training data to help the model "fit less the training data"; as a result, hoping to generalize better on new unseen ...
I. A's user avatar
  • 249
3 votes
0 answers
176 views

How to remove noise using morphological filtering

I have two groups of dots that both contain noise between them: The line that separates the two groups in the picture is diagonal in shape. I tried to use morphological filtering on this image to ...
tamarlev's user avatar
1 vote
1 answer
272 views

Noise Elimination with majority vote filtering

I have a dataset with label noise which I wan't to clean with majority/consensus vote filtering. This will mean I will divide the data in K-Folds and train an ensemble model. Than using the ...
Jeuszt's user avatar
  • 11
1 vote
0 answers
24 views

Isolated/noisy instances that have outsized effect on SVM hyperplane selection

Consider two linearly separable classes and the optimal separating hyperplane (image credit Prof Jiawei Han of UIUC): Now consider if there were a single "rogue" point for the Yellow class - which ...
WestCoastProjects's user avatar
0 votes
1 answer
3k views

what is correct way to perform normalization on data in Auto encoder?

working on anomaly detection problem. i'm using auto-encoder to denoise given input. I trained network with normal data(anomaly free). so model predict normal state of given input. Normalization of ...
Milan_Harkhani's user avatar
3 votes
0 answers
230 views

Training deep CNN with noisy dataset

I am training a Mask RCNN model with a train dataset that has been generated from some simple computer vision operations (color thresholding) and some morphological filtering. The train set captures ...
Gouda's user avatar
  • 161
3 votes
1 answer
2k views

Effect of adding gaussian noise to the input layer in a CNN

I often come across Keras code that adds GaussianNoise to the input, however its not clear to me what advantages does it offer to the learning. ...
KarateKid's user avatar
  • 183
2 votes
1 answer
2k views

How to create a complex Gaussian random noise with a specific covariance matrix

I am trying to generate a complex Gaussian white noise, with zero mean and the covariance matrix of them is going to be a specific matrix which is assumed to be given. Assume i to be a point on the ...
gurluk's user avatar
  • 121
0 votes
1 answer
35 views

How can I avoid requiring global information for performing regression on meter variables?

Note: With a meter variable a timestamped value is the sum of all previous differences plus a difference to the most recent value. Think of a electricity meter counting the use of energy. The goal ...
Etienne Ott's user avatar
2 votes
1 answer
114 views

Paramaeter estimation in noisy conditions with Machine Learning, possible?

Let's take two constants, $\alpha$ and $\beta$, both are given by two functions $f_1(\vec{\theta})$ and $f_2(\vec\theta$) (the model). These functions are known: we have an analytical closed ...
ignatius's user avatar
  • 1,658
1 vote
0 answers
382 views

Rain radar image noise reduction and cleanup

An application that I am building is plotting rain radar images on map. The images are transparent PNGs that are sourced straight from the local meteorological service. As can be seen in the example, ...
Yuval Adam's user avatar
5 votes
1 answer
1k views

Training on accurate data versus noisy data

I have data currently available that is very accurate and I would like to train my classification methods on this set of clean data to learn the important markers for distinguishing between classes. ...
Mathews24's user avatar
  • 185
0 votes
1 answer
85 views

Reducing noisy data from non normal distribution of data with std deviation?

I have used MATLAB code and get the two different row vectors A=1×18 and B=1×350. From both row vectors separately I need to remove the noisy data by using standard deviation. But the problem is that ...
user57546's user avatar
3 votes
2 answers
132 views

What is the loss function defined by Mnih and Hinton in their paper “Learning to Label Aerial Images from Noisy Data”?

In section 3.3 of the paper, they state that they use the cross entropy. Then they define the probability for a label to be a false positive as $\theta_0$ and a false negative as $\theta_1$. They use ...
Borbag's user avatar
  • 141
1 vote
1 answer
101 views

When we should use binning to redure noise? or How we find out we have noise?

I read several times which binning is helpful for reducing the noise of data. But how can we find out our data has noise? What if our data is clean and we reduce the accuracy of data? Is there any ...
parvij's user avatar
  • 771
2 votes
0 answers
108 views

Rprop is too noisy

Is there a way to reduce the noisiness and stochasticity of Rprop (and for that matter the iRprop+)? Specifically, in deep networks (with 8+ layers) this effect starts to become apparent, as the ...
Kari's user avatar
  • 2,706
6 votes
2 answers
6k views

Can Neural Networks be trained to smooth values / output the average?

Let's say we have a neural network with one input neuron and one output neuron. The training data $(x, f(x))$ is generated by a process $$f(x) = ax + \mathcal{N}(b, c)$$ with $a, b, c \in \mathbb{R}^...
Martin Thoma's user avatar
  • 18.8k