Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

157 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
1answer
367 views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this dataframe (df) ...
3
votes
2answers
56 views

Missing value Imputation in dataset

I have two separate files for Testing and Training. In the training data, I am dropping rows that contain too many missing values . But , In the test data , I cannot afford to drop the rows so I have ...
3
votes
0answers
20 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
3
votes
1answer
155 views

Extract features from a survey

I need to use the answers from a questionnaire for training a classifier. I discovered that some questions can have nested sub-questions.. Let's say (just an example) that I want to predict whether a ...
3
votes
1answer
57 views

Convert from many sub-tables to a single tidy dataframe

I have data in a bad format that I want to make tidy using an R script. I lack the skills to convert it into the format I want, so I look for an answer that (a) outlines a method, or (b) hands me the ...
3
votes
1answer
59 views

Data Exploration Led Conversion to Ordinal Variable

[I encountered this question at an interview few weeks ago and I am still not clear.] If all the values in a categorical column fuel_mileage come from the set <...
2
votes
0answers
24 views

Shuffling data yields significantly worse performance

Edit: I've experimented a few times, shuffling the data at various steps. It seems that as long as I restart the python kernel and reset the dataframe indices, the performance is good. I'm still not ...
2
votes
1answer
23 views

Manual Data Cleanup Tools

I am writing an ETL pipeline for geospatial data of the form place_name,address,longitude,latitude,id_linking_to_other_dataset As the last step in the pipeline, I ...
2
votes
0answers
31 views

How to process categorical variable having lots of unique values in linear regression?

I have House Price dataset and I am using linear regression to predict the house price. while data preprocessing I found a variable called "Location" and it have around 342 unique value. For ...
2
votes
1answer
15 views

Architectures that take inputs of mixed sampling rates

Let's say a model is trained on multiple datasets of 1D time series. These datasets have been gathered with different sampling rates. I plan to use a convolution neural network to process these time ...
2
votes
2answers
143 views

Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?

I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling. I am pretty comfortable with the sklearn Pipeline ...
2
votes
0answers
17 views

Correlation matrix on multi-time scales

I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams. For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (...
2
votes
0answers
167 views

Tools for reading data from large, irregular csv files (aka excel file hell)

I have a csv file with 1000 columns and 50 rows that was collected over a few months. Embedded randomly in the file are ~1000 small datasets with semi-standard formats (some columns differ, so does ...
2
votes
0answers
119 views

How to use zero-inflated negative binomial regression for binary classification task?

I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly ...
2
votes
1answer
55 views

Python is reading my data with NANS and Infs, but they don't have any

I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
2
votes
0answers
62 views

Keras Binary Classification val_acc won't go past ~67; Full data and code included

I'm working on a binary classification in Keras with a Tensorflow backend. No matter how much I tweak, I can't seem to get my model past a val_acc of 67%. Is there something I'm missing, or is this ...
2
votes
2answers
526 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
2
votes
1answer
82 views

PCA huge parts of missing data filling

I’m performing PCA on different time series’ and then using K Means clustering to try and group together common factors. The issue I’m facing is that some of the factors come in and out of the time ...
2
votes
0answers
88 views

Prediction questions related to the dataset

I have been self-learning data science from different sources. I have a dataset which was sent to me by my friend from one of her college courses. The work is to be done in R preferably. Description: ...
2
votes
0answers
369 views

Online noise removal techniques

There are lots of noise removal algorithms for offline dataset. I would like to ask if there are some suitable online implementations to remove noise of the data in real-time (similar to Kalman ...
2
votes
0answers
109 views

How to read data in DSX from Data hub object storage container?

I am creating a new notebook with DSX. I have a data set that I want to use. It is not CSV. I found that I could go to Data hub by clicking top-right tile near my logged in name, then object storage ...
2
votes
0answers
150 views

Combining parameters for Douglas-Peucker Simplification

NOTE : I'm not sure if this is the right forum for this question. if not, please advice. Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect ...
2
votes
0answers
193 views

Natural Language text categorization using RapidMiner

I'm new to data mining, so this might sound like a very simple task to some. I work in reliability engineering in aviation and have a set of data that is generated on a daily basis regarding system ...
2
votes
0answers
86 views

Cross-sell models and additional holders

I would like to pose a question about how to treat additional holders in the propensity-to-buy models of banking products. Up to now I was only taking into considerations the clients as first holders....
2
votes
2answers
779 views

PySpark: How do I specify dropna axis in PySpark transformation?

I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument ...
2
votes
1answer
29 views

How to make smaller categories with factor character variables

I have this data set with consist of ISO3166 Alpha-2 codes for countries. Example: DE, AD, AE etc They are coded as factor variables in R and there are about 173 observations. Now because there are ...
1
vote
1answer
24 views

Selecting Training Set from Model CV Error

What I would like to do is recursively: Train the model on all data Remove the sample(s) with highest error Repeat until the remaining samples have an acceptable error The hypothesis is: "To ...
1
vote
0answers
8 views

Identifying Mixed Label Duplicates In Large Data Set

I have a data set containing 250k documents and 400k labeled topics. Each document may have 200+ topics. These topics may significantly overlap in subject and create noise. I want to reduce the ...
1
vote
0answers
20 views

How does a misrepresented disproportionate data affects modelling?

Let's say I have a dataset of the occurrence of pregnancies each time is tried, the ground truth of success to failure rate is 30:70. But the dataset with me now is a 70:30 dataset. How would that be ...
1
vote
2answers
21 views

What are some examples of ordered data?

Tan,Steinbech,Kumar's book says For some types of data, attributes have relationships that involve order in time or space. I want some real life examples of such data, can you provide me. I couldn't ...
1
vote
0answers
15 views

Is there any standard guidelines, best practices on handle missing data?

Is there any standard guidelines, best practices on handle missing data? E.g. with certain threshold, one can still proceed with "data cleaning". If not certain approaches would be best ...
1
vote
1answer
12 views

Want to convert a data file into a list of lists where the semi-colon is a separator

I'm doing my very first data science project ever and am likely to seem a bit naive, and I've got three .DAT files each of which I want to convert into a list of lists, and the code I've been using so ...
1
vote
0answers
26 views

Dataset format for Transformer text-generation

I'm trying to find some tutorials on training Transformer for generating comments on articles. So far, I found an article showing how to train GPT2 as a chat-bot. Input files in that example are given ...
1
vote
0answers
8 views

Does it make more sense to use central tendency methods on training and test sets separately?

I'll explain further, so I'm taking a data science course on cleaning and preparing data and I'm on the how to handle missing data section. So the question is essentially what you see above except it ...
1
vote
1answer
17 views

Should I apply a transformation to columns with INTEGERS, in case I want to reduce the skewness of that column?

I am performing EDA on a dataset of Hotel Reservations. Target is Categorical stating if a given customer will cancel the reservation or not. Dataset has 25 features, 30244 entries. I have two ...
1
vote
1answer
13 views

Handling Missing Inconsistent Educational Data

I'm an educational researcher learning about machine learning so I can further explore my data beyond the usual statistics. I currently have some assessment data but I am not sure how to appropriately ...
1
vote
0answers
38 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
1
vote
1answer
13 views

Approach to labelling new data for training

Overview: Imagine an application that identifies cats and dogs from their phone camera. User's take a photo of their pet and it tells them if it is a dog or a cat. The data is then sent to the server. ...
1
vote
0answers
15 views

How to take the keywords from the given dataset to train GPT-2 based chatbot?

I am working with a dataset that contains Questions on various Events conducted by a college and the corresponding answers for the queries. I am using this dataset to train a GPT-2 355M model to ...
1
vote
0answers
36 views

Should I drop Nan before before creating a scatter plot and computing the correlation matrix?

I have some NaN values in my data and I cannot replace it with median or something else. Now I have to check out the correlation, using scatter plot and Pearson correlation. Should I drop these NaNs ...
1
vote
0answers
24 views

Modeling time spent on different websites

Apologies in advance for the poor title. I didn't know exactly how to phrase what I want in a succinct manner so hopefully I can elaborate a bit more. I have a dataset where I have customer_id, ...
1
vote
0answers
20 views

Handling missing data - secondary driver characteristics in insurance data

I have an insurance dataset which includes an indicator that indicates whether the policy insures a secondary driver, and the secondary driver's age/sex. Problem is most policies do not have secondary ...
1
vote
1answer
44 views

Hierarchical dirichlet process results

I am thinking about using hierarchical dirichlet process to model a patent dataset. I've seen that HDP uses a base distribution and assumes that every topic comes from that base distribution. The ...
1
vote
0answers
21 views

NLP Classification labels have many similarirites, converge to and replace to only have one

I been trying to use the fuzzywuzzy library in Python to find the percentage similarity between strings in the labels. The problem I am having is that there is still many strings that are really ...
1
vote
1answer
15 views

Grouping profiles strings having the same words, but occurring out of order

I have a dataframe containing a column of profile types, which looks like this: ...
1
vote
0answers
11 views
1
vote
1answer
43 views

Splitting Temporal Data

I have a project where I'm required to compute (using some regression) how long a task will take. From the definition of the business problem it is clear that there is some temporal relationship in ...
1
vote
0answers
21 views

how to deal with columns that has different value in only 1 or 2 rows?

I have very high dimensional data. Almost 20% of the columns has different value in less than 1% of rows. All of these are binary columns and many columns has 0s filled in more than almost 98% of rows....
1
vote
1answer
147 views

How to prepare Audio-text data for speech recognition

I have gathered some raw audio from all the conferences, meetings, lectures & casual conversation that I was part of. The machine transcription did not offer good results (from Azure, AWS etc.) I ...
1
vote
0answers
20 views

Is there a pandoc for data manipulation?

You know how pandocs converts markdown to HTML and pdf etc. Is there a pandocs for data manipulation? Like from SQL to pandas and SQL to dplyr etc?