Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

177 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
2answers
62 views

Missing value Imputation in dataset

I have two separate files for Testing and Training. In the training data, I am dropping rows that contain too many missing values . But , In the test data , I cannot afford to drop the rows so I have ...
3
votes
0answers
20 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
3
votes
2answers
551 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
3
votes
1answer
155 views

Extract features from a survey

I need to use the answers from a questionnaire for training a classifier. I discovered that some questions can have nested sub-questions.. Let's say (just an example) that I want to predict whether a ...
3
votes
1answer
57 views

Convert from many sub-tables to a single tidy dataframe

I have data in a bad format that I want to make tidy using an R script. I lack the skills to convert it into the format I want, so I look for an answer that (a) outlines a method, or (b) hands me the ...
3
votes
1answer
59 views

Data Exploration Led Conversion to Ordinal Variable

[I encountered this question at an interview few weeks ago and I am still not clear.] If all the values in a categorical column fuel_mileage come from the set <...
2
votes
0answers
25 views

change target variable value to reflect better affordability

Context I am working on a regression problem trying to predict affordability. My dataset contains daily installments repaying a purchase in a form of contract. Essentially, a minimum daily rate the ...
2
votes
0answers
35 views

Shuffling data yields significantly worse performance

Edit: I've experimented a few times, shuffling the data at various steps. It seems that as long as I restart the python kernel and reset the dataframe indices, the performance is good. I'm still not ...
2
votes
1answer
23 views

Manual Data Cleanup Tools

I am writing an ETL pipeline for geospatial data of the form place_name,address,longitude,latitude,id_linking_to_other_dataset As the last step in the pipeline, I ...
2
votes
0answers
48 views

How to process categorical variable having lots of unique values in linear regression?

I have House Price dataset and I am using linear regression to predict the house price. while data preprocessing I found a variable called "Location" and it have around 342 unique value. For ...
2
votes
1answer
15 views

Architectures that take inputs of mixed sampling rates

Let's say a model is trained on multiple datasets of 1D time series. These datasets have been gathered with different sampling rates. I plan to use a convolution neural network to process these time ...
2
votes
2answers
186 views

Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?

I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling. I am pretty comfortable with the sklearn Pipeline ...
2
votes
1answer
48 views

Splitting Temporal Data

I have a project where I'm required to compute (using some regression) how long a task will take. From the definition of the business problem it is clear that there is some temporal relationship in ...
2
votes
0answers
17 views

Correlation matrix on multi-time scales

I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams. For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (...
2
votes
0answers
175 views

Tools for reading data from large, irregular csv files (aka excel file hell)

I have a csv file with 1000 columns and 50 rows that was collected over a few months. Embedded randomly in the file are ~1000 small datasets with semi-standard formats (some columns differ, so does ...
2
votes
2answers
87 views

How to improve identification of outliers for removal

I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running). Unfortunately, I don't have a measurement that clearly ...
2
votes
0answers
130 views

How to use zero-inflated negative binomial regression for binary classification task?

I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly ...
2
votes
1answer
61 views

Python is reading my data with NANS and Infs, but they don't have any

I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
2
votes
0answers
64 views

Keras Binary Classification val_acc won't go past ~67; Full data and code included

I'm working on a binary classification in Keras with a Tensorflow backend. No matter how much I tweak, I can't seem to get my model past a val_acc of 67%. Is there something I'm missing, or is this ...
2
votes
2answers
33 views

What are the ways to merge data from different sources? Is there any software framework?

Is there any software library or framework that allows merging data from a different source on regular basis? As I understand it should store and support a different kind of rules for cleaning and ...
2
votes
1answer
82 views

PCA huge parts of missing data filling

I’m performing PCA on different time series’ and then using K Means clustering to try and group together common factors. The issue I’m facing is that some of the factors come in and out of the time ...
2
votes
0answers
99 views

Prediction questions related to the dataset

I have been self-learning data science from different sources. I have a dataset which was sent to me by my friend from one of her college courses. The work is to be done in R preferably. Description: ...
2
votes
0answers
370 views

Online noise removal techniques

There are lots of noise removal algorithms for offline dataset. I would like to ask if there are some suitable online implementations to remove noise of the data in real-time (similar to Kalman ...
2
votes
0answers
111 views

How to read data in DSX from Data hub object storage container?

I am creating a new notebook with DSX. I have a data set that I want to use. It is not CSV. I found that I could go to Data hub by clicking top-right tile near my logged in name, then object storage ...
2
votes
0answers
157 views

Combining parameters for Douglas-Peucker Simplification

NOTE : I'm not sure if this is the right forum for this question. if not, please advice. Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect ...
2
votes
0answers
193 views

Natural Language text categorization using RapidMiner

I'm new to data mining, so this might sound like a very simple task to some. I work in reliability engineering in aviation and have a set of data that is generated on a daily basis regarding system ...
2
votes
0answers
86 views

Cross-sell models and additional holders

I would like to pose a question about how to treat additional holders in the propensity-to-buy models of banking products. Up to now I was only taking into considerations the clients as first holders....
2
votes
2answers
851 views

PySpark: How do I specify dropna axis in PySpark transformation?

I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument ...
2
votes
1answer
29 views

How to make smaller categories with factor character variables

I have this data set with consist of ISO3166 Alpha-2 codes for countries. Example: DE, AD, AE etc They are coded as factor variables in R and there are about 173 observations. Now because there are ...
1
vote
0answers
7 views

How to combine data from multiple Google Trends queries effectively?

As you might know, Google Trends works by normalising a random sample of the search term data, with the sample changing at least once per day, from my experience. This is not an issue for western ...
1
vote
0answers
16 views

Tokenizer returning incorrect values and losing a lot of data

(cross posted from main stackoverflow) This is a weird situation so I hope I can explain it correctly. My partner and I are working on a ML project where we create a model that predicts whether a ...
1
vote
0answers
19 views

Missing data for time-series forecasting/classification

I want to train a model to predict some customer behavior. I have two types of data that I have to deal with. My measurements are daily. The first type is regular measurements without any missing data,...
1
vote
0answers
31 views

How to handle unclassifiable data in the dataset

Premise: Classification problem Input is three text fields Output classes are A, B, A&B (Note: A and B are not always exclusive though usually are, hence the 'A&B' class) Sci-Kit Learn is the ...
1
vote
0answers
22 views

How to handle insufficient information about the collected data in the dataset?

I am trying to create a ML model to predict foodborne diseases. I found online a few datasets with the informations I need. The problem is that none of them explains how they got the medical measures. ...
1
vote
1answer
21 views

Cleaning rows of special characters and creating dataframe columns

Below is my Dataframe format consisting of 2 columns (one is index and other is the data to be cleaned) ...
1
vote
0answers
16 views

Problem with FFT in Python for smaller df

...
1
vote
1answer
15 views

Full gaps in the curve

This is the data from Irelands vaccine program. There are these missing bits of data. Is there away to fill these gaps with proportional data to make a smooth curve?
1
vote
0answers
18 views

Group points to reduce data set such that the linear regression stays the same

I have a very long dataset and I'm trying to reduce it by grouping the data in periods of 24 hours. In this way, there will be a single data point that represents that day, but they must yield the ...
1
vote
0answers
8 views

Identifying Mixed Label Duplicates In Large Data Set

I have a data set containing 250k documents and 400k labeled topics. Each document may have 200+ topics. These topics may significantly overlap in subject and create noise. I want to reduce the ...
1
vote
0answers
23 views

How does a misrepresented disproportionate data affects modelling?

Let's say I have a dataset of the occurrence of pregnancies each time is tried, the ground truth of success to failure rate is 30:70. But the dataset with me now is a 70:30 dataset. How would that be ...
1
vote
2answers
22 views

What are some examples of ordered data?

Tan,Steinbech,Kumar's book says For some types of data, attributes have relationships that involve order in time or space. I want some real life examples of such data, can you provide me. I couldn't ...
1
vote
0answers
15 views

Is there any standard guidelines, best practices on handle missing data?

Is there any standard guidelines, best practices on handle missing data? E.g. with certain threshold, one can still proceed with "data cleaning". If not certain approaches would be best ...
1
vote
1answer
13 views

Want to convert a data file into a list of lists where the semi-colon is a separator

I'm doing my very first data science project ever and am likely to seem a bit naive, and I've got three .DAT files each of which I want to convert into a list of lists, and the code I've been using so ...
1
vote
0answers
29 views

Dataset format for Transformer text-generation

I'm trying to find some tutorials on training Transformer for generating comments on articles. So far, I found an article showing how to train GPT2 as a chat-bot. Input files in that example are given ...
1
vote
0answers
10 views

Does it make more sense to use central tendency methods on training and test sets separately?

I'll explain further, so I'm taking a data science course on cleaning and preparing data and I'm on the how to handle missing data section. So the question is essentially what you see above except it ...
1
vote
1answer
17 views

Should I apply a transformation to columns with INTEGERS, in case I want to reduce the skewness of that column?

I am performing EDA on a dataset of Hotel Reservations. Target is Categorical stating if a given customer will cancel the reservation or not. Dataset has 25 features, 30244 entries. I have two ...
1
vote
1answer
14 views

Handling Missing Inconsistent Educational Data

I'm an educational researcher learning about machine learning so I can further explore my data beyond the usual statistics. I currently have some assessment data but I am not sure how to appropriately ...
1
vote
0answers
67 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
1
vote
1answer
13 views

Approach to labelling new data for training

Overview: Imagine an application that identifies cats and dogs from their phone camera. User's take a photo of their pet and it tells them if it is a dog or a cat. The data is then sent to the server. ...