Questions tagged [data-cleaning]
Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.
217
questions with no upvoted or accepted answers
3
votes
0
answers
22
views
Why RANDOM noise images always predicted as BIRD?
Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%.
However when feeding into 5000 random noise images (Gaussian noise with the ...
- 31
3
votes
0
answers
190
views
How to use zero-inflated negative binomial regression for binary classification task?
I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly ...
- 321
3
votes
1
answer
61
views
Data Exploration Led Conversion to Ordinal Variable
[I encountered this question at an interview few weeks ago and I am still not clear.]
If all the values in a categorical column fuel_mileage come from the set <...
- 41
2
votes
0
answers
45
views
change target variable value to reflect better affordability
Context
I am working on a regression problem trying to predict affordability.
My dataset contains daily installments repaying a purchase in a form of contract. Essentially, a minimum daily rate the ...
- 21
2
votes
0
answers
55
views
Shuffling data yields significantly worse performance
Edit: I've experimented a few times, shuffling the data at various steps. It seems that as long as I restart the python kernel and reset the dataframe indices, the performance is good. I'm still not ...
- 63
2
votes
1
answer
32
views
Manual Data Cleanup Tools
I am writing an ETL pipeline for geospatial data of the form
place_name,address,longitude,latitude,id_linking_to_other_dataset
As the last step in the pipeline, I ...
- 21
2
votes
0
answers
203
views
How to process categorical variable having lots of unique values in linear regression?
I have House Price dataset and I am using linear regression to predict the house price.
while data preprocessing I found a variable called "Location" and it have around 342 unique value. For ...
- 21
2
votes
1
answer
33
views
Architectures that take inputs of mixed sampling rates
Let's say a model is trained on multiple datasets of 1D time series. These datasets have been gathered with different sampling rates. I plan to use a convolution neural network to process these time ...
- 21
2
votes
2
answers
213
views
Missing value Imputation in dataset
I have two separate files for Testing and Training.
In the training data, I am dropping rows that contain too many missing values .
But , In the test data , I cannot afford to drop the rows so I have ...
- 75
2
votes
0
answers
21
views
Correlation matrix on multi-time scales
I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams.
For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (...
- 217
2
votes
0
answers
241
views
Tools for reading data from large, irregular csv files (aka excel file hell)
I have a csv file with 1000 columns and 50 rows that was collected over a few months. Embedded randomly in the file are ~1000 small datasets with semi-standard formats (some columns differ, so does ...
- 141
2
votes
0
answers
73
views
Keras Binary Classification val_acc won't go past ~67; Full data and code included
I'm working on a binary classification in Keras with a Tensorflow backend. No matter how much I tweak, I can't seem to get my model past a val_acc of 67%. Is there something I'm missing, or is this ...
- 33
2
votes
0
answers
1k
views
Issues with pandas chunk merge
I'm trying to solve a kaggle competition
Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back to disk. This is the code I'm using:
...
- 21
2
votes
0
answers
118
views
Prediction questions related to the dataset
I have been self-learning data science from different sources. I have a dataset which was sent to me by my friend from one of her college courses. The work is to be done in R preferably.
Description: ...
- 21
2
votes
0
answers
396
views
Online noise removal techniques
There are lots of noise removal algorithms for offline dataset. I would like to ask if there are some suitable online implementations to remove noise of the data in real-time (similar to Kalman ...
- 161
2
votes
0
answers
119
views
How to read data in DSX from Data hub object storage container?
I am creating a new notebook with DSX. I have a data set that I want to use. It is not CSV. I found that I could go to Data hub by clicking top-right tile near my logged in name, then object storage ...
2
votes
0
answers
76
views
What tools are available for semi-automated matching of dirty columnar data
Are there any automated or semi-automated tools for finding matching "similar" or data in two columnar data sets? The data I'm working with was collected (and handled) by different organizations. ...
- 121
2
votes
0
answers
166
views
Combining parameters for Douglas-Peucker Simplification
NOTE : I'm not sure if this is the right forum for this question. if not, please advice.
Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect ...
- 121
2
votes
0
answers
198
views
Natural Language text categorization using RapidMiner
I'm new to data mining, so this might sound like a very simple task to some.
I work in reliability engineering in aviation and have a set of data that is generated on a daily basis regarding system ...
2
votes
0
answers
89
views
Cross-sell models and additional holders
I would like to pose a question about how to treat additional holders in the propensity-to-buy models of banking products.
Up to now I was only taking into considerations the clients as first holders....
- 81
2
votes
1
answer
32
views
How to make smaller categories with factor character variables
I have this data set with consist of ISO3166 Alpha-2 codes for countries.
Example: DE, AD, AE etc
They are coded as factor variables in R and there are about 173 observations.
Now because there are ...
- 29
1
vote
1
answer
26
views
Should I drop duplicates over features but no target
I'm in a debate with someone about a problem where there are duplicates over features (i.e. $ X_1 = X_2 $ but $ Y_1 != Y_2 $).
My point of view is that we should keep those datas, as they can be ...
- 323
1
vote
0
answers
48
views
How can I detect errors/irregularity in a dataset
I've been working on a pathfinding project using topographic data for my software development course, even though I've never really had calculus lessons.
The algorithm works fine by itself, however, ...
- 11
1
vote
1
answer
33
views
Handling missing value in column having textual data
I have been working on a supervised ML use case where dataset has Numerical (Price), Categorical(Category) and Textual data(Description) as features. Description feature has about 30% missing values. ...
1
vote
0
answers
51
views
Data Leakage (specific case with standardization)
I read a few blog posts and found some books that caution against data leakage when adjusting the training data using any information from the held out datasets. For example, if I were to standardize ...
- 11
1
vote
0
answers
14
views
Data filtering framework
I have procurement data that needs to be labeled with product categories. It's tabular data, containing 700k rows and a mix of data types (dates, free text, floats, etc.) The product set we currently ...
- 245
1
vote
0
answers
43
views
supply chain analytics dataset?
I need to develop a model which enhances the supply chain and reduces the cost and wastage for perishable products like meat, seafood, dairy products and fruits and vegetables etc. Could someone ...
1
vote
1
answer
49
views
1
vote
1
answer
16
views
How to perform linear regression on a parameter that represents state/configuration of a machinery in a production process?
I am trying to perform linear regression on a manufacturing process in order to determine the influencing parameters on a particular product. The thing is there are several production parameters, and ...
- 11
1
vote
0
answers
197
views
Is it possible to use structured(tabular) data as a reinforcement learning environment?
I want to do an RL project in which the agent will learn to drop duplicates in a tabular data. But I couldn't find any examples of RL being used that way - checked the RL based recommendation systems ...
- 11
1
vote
0
answers
25
views
How can one ensure the quality of collectively created ground truth labels?
Here is the situation: I have a small tool to assign (one or multiple) labels to image data. In the tool, you can not only assign a label(s) to the images but also assign a score for the certainty of ...
- 11
1
vote
0
answers
83
views
How to represent/aggregate data belonging to same group, and feed to model for training?
I have a problem in which there are couple of records that belong to the same group, like multiple observations. For example this data set
has three groups, For each group, I have three observations....
- 113
1
vote
0
answers
304
views
How to combine data from multiple Google Trends queries effectively?
As you might know, Google Trends works by normalising a random sample of the search term data, with the sample changing at least once per day, from my experience. This is not an issue for western ...
- 23
1
vote
0
answers
129
views
1
vote
0
answers
21
views
Tokenizer returning incorrect values and losing a lot of data
(cross posted from main stackoverflow) This is a weird situation so I hope I can explain it correctly. My partner and I are working on a ML project where we create a model that predicts whether a ...
1
vote
0
answers
24
views
Missing data for time-series forecasting/classification
I want to train a model to predict some customer behavior. I have two types of data that I have to deal with. My measurements are daily.
The first type is regular measurements without any missing data,...
1
vote
0
answers
46
views
How to handle unclassifiable data in the dataset
Premise:
Classification problem
Input is three text fields
Output classes are A, B, A&B (Note: A and B are not always exclusive though usually are, hence the 'A&B' class)
Sci-Kit Learn is the ...
1
vote
0
answers
25
views
How to handle insufficient information about the collected data in the dataset?
I am trying to create a ML model to predict foodborne diseases. I found online a few datasets with the informations I need. The problem is that none of them explains how they got the medical measures.
...
- 11
1
vote
0
answers
18
views
1
vote
1
answer
17
views
Full gaps in the curve
This is the data from Irelands vaccine program. There are these missing bits of data.
Is there away to fill these gaps with proportional data to make a smooth curve?
- 21
1
vote
0
answers
51
views
Group points to reduce data set such that the linear regression stays the same
I have a very long dataset and I'm trying to reduce it by grouping the data in periods of 24 hours. In this way, there will be a single data point that represents that day, but they must yield the ...
- 11
1
vote
0
answers
9
views
Identifying Mixed Label Duplicates In Large Data Set
I have a data set containing 250k documents and 400k labeled topics. Each document may have 200+ topics. These topics may significantly overlap in subject and create noise. I want to reduce the ...
- 111
1
vote
0
answers
30
views
How does a misrepresented disproportionate data affects modelling?
Let's say I have a dataset of the occurrence of pregnancies each time is tried, the ground truth of success to failure rate is 30:70. But the dataset with me now is a 70:30 dataset. How would that be ...
- 11
1
vote
0
answers
57
views
Dataset format for Transformer text-generation
I'm trying to find some tutorials on training Transformer for generating comments on articles. So far, I found an article showing how to train GPT2 as a chat-bot. Input files in that example are given ...
- 111
1
vote
0
answers
16
views
Does it make more sense to use central tendency methods on training and test sets separately?
I'll explain further, so I'm taking a data science course on cleaning and preparing data and I'm on the how to handle missing data section. So the question is essentially what you see above except it ...
- 11
1
vote
1
answer
22
views
Should I apply a transformation to columns with INTEGERS, in case I want to reduce the skewness of that column?
I am performing EDA on a dataset of Hotel Reservations. Target is Categorical stating if a given customer will cancel the reservation or not. Dataset has 25 features, 30244 entries.
I have two ...
- 73
1
vote
1
answer
19
views
Handling Missing Inconsistent Educational Data
I'm an educational researcher learning about machine learning so I can further explore my data beyond the usual statistics.
I currently have some assessment data but I am not sure how to appropriately ...
- 11
1
vote
2
answers
5k
views
How to remove spikes from data with Python using signal.find_peaks
I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used ...
1
vote
0
answers
530
views
Dealing with null values in categorical string data
Which of the following is a better approach in dealing with null values and why?
Fill the null values with the mode (most occurring value).
Consider the null itself as a category by converting it to ...
- 111
1
vote
1
answer
22
views
Approach to labelling new data for training
Overview:
Imagine an application that identifies cats and dogs from their phone camera. User's take a photo of their pet and it tells them if it is a dog or a cat. The data is then sent to the server.
...
- 111