Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0
votes
2answers
117 views

Building predictive model with low correlated data [closed]

I have been working on a project with low features and only few entry fields ( 4 to be exact ). All the data in the dataset is barely correlated to each other. Is there some organized way or ...
0
votes
0answers
10 views

Group points to reduce data set such that the linear regression stays the same

I have a very long dataset and I'm trying to reduce it by grouping the data in periods of 24 hours. In this way, there will be a single data point that represents that day, but they must yield the ...
0
votes
1answer
19 views

How to improve regression neural network?

I am new to deep learning and data science and trying to increase my knowledge by working on some hackathons. Currently, the hackathon project I am working on has the task to predict the closing price ...
0
votes
0answers
9 views

Find if a string appear before another string [closed]

I have a string variable containing patients' addresses. My goal is to flag patients who live in "401 30th street". I would like to flags strings that contain the number "401" ...
2
votes
0answers
24 views

Shuffling data yields significantly worse performance

Edit: I've experimented a few times, shuffling the data at various steps. It seems that as long as I restart the python kernel and reset the dataframe indices, the performance is good. I'm still not ...
0
votes
0answers
8 views

Converting string values to DateTime in Azure ML

I have dataset which contains a column showing dates as a "String" value. I'm seeking to convert these values to a proper "DateTime" value in Azure ML, but upon using the Edit ...
0
votes
0answers
21 views

How to put feature names from a 4D NumPy array into a pandas dataframe with a one to one relationship [closed]

I have a 4D NumPy array containing the features: importer, product, demand sector, and ...
1
vote
1answer
27 views

How to fix my CSV files? (ValueError: Found array with 0 sample(s) (shape=(0, 1)) while a minimum of 1 is required)

I have tried to import two csv files into df1 and df2. Concatenated them to make df3. I ...
0
votes
0answers
7 views

How to deal with errors or inconsistencies in the training data?

There are inconsistant wrong labels and consistant errors in training data. For the former I tried MC-dropout and data Shapley. For the later I wonder if manual data curation is a requisite?
1
vote
1answer
24 views

Selecting Training Set from Model CV Error

What I would like to do is recursively: Train the model on all data Remove the sample(s) with highest error Repeat until the remaining samples have an acceptable error The hypothesis is: "To ...
2
votes
1answer
82 views

PCA huge parts of missing data filling

I’m performing PCA on different time series’ and then using K Means clustering to try and group together common factors. The issue I’m facing is that some of the factors come in and out of the time ...
0
votes
0answers
21 views

How to mathematically quantify the quality of a corpus?

I am working on a text classification project. I have around 60,000 text samples of 40intents. By calculating the frequency of ...
0
votes
1answer
2k views

Normalization and Outlier on Target variable which is continuous

I have doubt that should I perform outlier analysis and normalization even on target variable which is continuous ?
0
votes
1answer
17 views

ML for data processing. What are the options?

Currently I am working on improving a stage on a data processing pipeline. The source data has a large number of fields and is getting normalized into a simpler entity. This entails that in many cases ...
0
votes
1answer
290 views

Tool vs Python Script for Transforming Data in Mongo

We have a bunch of Mongo collections (data collected from APIs, web scraping, etc) that we need to transform to a cleaner data structure (standardized schema) on a monthly basis. Are there any good ...
1
vote
1answer
147 views

How to prepare Audio-text data for speech recognition

I have gathered some raw audio from all the conferences, meetings, lectures & casual conversation that I was part of. The machine transcription did not offer good results (from Azure, AWS etc.) I ...
1
vote
1answer
17 views

Should I apply a transformation to columns with INTEGERS, in case I want to reduce the skewness of that column?

I am performing EDA on a dataset of Hotel Reservations. Target is Categorical stating if a given customer will cancel the reservation or not. Dataset has 25 features, 30244 entries. I have two ...
1
vote
1answer
24 views

Encoding "histogram bins"

I am currently working on a regression problem where I have one variable (x) of the data in the form of "histogram bins". I.e. I could have value ranges ...
1
vote
1answer
425 views

Downsampling audio files for use in Machine Learning

I'm trying to use the work (Neural Networks) done in this repo: https://github.com/jtkim-kaist/VAD It says this: Note: To apply this toolkit to other speech data, the speech data should be ...
2
votes
2answers
779 views

PySpark: How do I specify dropna axis in PySpark transformation?

I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument ...
3
votes
2answers
56 views

Missing value Imputation in dataset

I have two separate files for Testing and Training. In the training data, I am dropping rows that contain too many missing values . But , In the test data , I cannot afford to drop the rows so I have ...
1
vote
2answers
2k views

Orange3 summarizing data, grouping data values

Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values? For example, instead of plotting a scatter with lots of data points, I would like to ...
2
votes
0answers
88 views

Prediction questions related to the dataset

I have been self-learning data science from different sources. I have a dataset which was sent to me by my friend from one of her college courses. The work is to be done in R preferably. Description: ...
1
vote
2answers
58 views

Dealing with high number of NAs in a classification problem

I am working on a classification problem. The dataset dimension is as 187,643 x 203. The first column contains class labels with no NA. The rest of dataset are frequency data and could be anything ...
3
votes
1answer
57 views

Convert from many sub-tables to a single tidy dataframe

I have data in a bad format that I want to make tidy using an R script. I lack the skills to convert it into the format I want, so I look for an answer that (a) outlines a method, or (b) hands me the ...
1
vote
3answers
452 views

How to deal with highly skewed (on counts) dependent variables?

I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in ...
1
vote
0answers
8 views

Identifying Mixed Label Duplicates In Large Data Set

I have a data set containing 250k documents and 400k labeled topics. Each document may have 200+ topics. These topics may significantly overlap in subject and create noise. I want to reduce the ...
1
vote
1answer
549 views

Am I supposed to scale binary features alongside other numerical features?

When I am cleaning my data, I have some features which contain large numbers and some features that are binary. Should I scale the large features and then add the binary columns or just scale them all ...
1
vote
1answer
562 views

Is there are way to impute missing values by clustering, regression and stochastic regression

I'd like to know if there are any libraries that allow imputation by clustering, regression and stochastic regression. So far, I've done imputation by mean, median and KNN. I'm trying to evaluate the ...
1
vote
2answers
35 views

How to group a unique column and find the mean value in a separate column in R

My table of data looks like this image (but with many more lines): I want to be able to find the mean age of each region, so the mean age for region SSC20790 and so on. I believe I can use the <...
0
votes
0answers
10 views

EDA on multivariate time series data

I'm working with a medical dataset in a "long" format, meaning each row represents a timestamp and I have a column named "test" representing the test, an additional column ...
0
votes
1answer
95 views

How do we choose which feature(column) should we normalize or scale

I am a newcomer to machine learning, I was going through scaling and normalization. When I tried to explore most of the documentation on web, I found people explaining that the dataset needs to be ...
1
vote
0answers
20 views

How does a misrepresented disproportionate data affects modelling?

Let's say I have a dataset of the occurrence of pregnancies each time is tried, the ground truth of success to failure rate is 30:70. But the dataset with me now is a 70:30 dataset. How would that be ...
1
vote
3answers
69 views

Do I need to manually trim 300 videos?

I wish to train a model that detects the breed of a dog based on video input. I have a dataset containing 10 classes with 30 videos in each class. The problem is that for each of these videos, the dog ...
1
vote
1answer
44 views

Hierarchical dirichlet process results

I am thinking about using hierarchical dirichlet process to model a patent dataset. I've seen that HDP uses a base distribution and assumes that every topic comes from that base distribution. The ...
3
votes
1answer
155 views

Extract features from a survey

I need to use the answers from a questionnaire for training a classifier. I discovered that some questions can have nested sub-questions.. Let's say (just an example) that I want to predict whether a ...
0
votes
1answer
55 views

How can I create a new column of binary values from my TfidfVectorizer sparse matrix?

I currently have a sparse matrix object of TfidfVectorizer which is of 1000 length. Right now it is displayed like this: ...
0
votes
0answers
15 views

formatting data from pandas into CNN-compatible array

I have a pandas dataframe of this form: time x y 0 a b 1 c d For my CNN model training, I want to slice the first 5 rows (from t=0 to t=4) and apply my custom function to label this sliced set, ...
2
votes
1answer
55 views

Python is reading my data with NANS and Infs, but they don't have any

I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
0
votes
0answers
14 views

Time series classification - subject level analysis

I have a dataset with smartphone sensors data collection. There are 300 subjects, and the frequency of the sensors collected is different for different subjects. The shape of the data is (20k, 30) -&...
0
votes
1answer
30 views

How to remove irrelevant text data from a large dataset

I am working on a ML project where data were coming from a social media, and the topic about the data should be depression under Covid-19. However, when I read some of the data retrieved, I noticed ...
3
votes
2answers
297 views

Do i need to handle missing values before EDA?

I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to ...
1
vote
1answer
155 views

Noise Elimination with majority vote filtering

I have a dataset with label noise which I wan't to clean with majority/consensus vote filtering. This will mean I will divide the data in K-Folds and train an ensemble model. Than using the ...
1
vote
1answer
65 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
0
votes
1answer
187 views

Comparing data sets with different measurements

I'm currently writing a thesis based on Cyber Crime, however I'm unsure of the proper to compare/analyse my data sets to talk about them in my thesis. One piece of data (https://www.pandasecurity....
0
votes
1answer
56 views

What is the best solution to replace NaN values?

I'm thinking about using the normal distribution of a specific column that has missing values and replace them by random values generated using the normal distribution function of numpy on that ...
1
vote
1answer
20 views

If we replace all missing values with "unknown" or "-∞", what problem will we encounter?

I am reading Han,Kamber,Pei's data mining book and I stumbled upon a section called "data cleaning". It tells we can use a global constant like "unknown" or "-∞" to ...
2
votes
1answer
179 views

How to impute missing text data?

Lets say I have a dataframe consisting of two text columns. By text, I mean the values in those columns are either sentences/paragraphs. In such a case, how do I handle missing 'NaN' values? If it ...
0
votes
0answers
20 views

A feature is still right-skewed after log scaling. How should it be normalized for machine learning?

I've attached two images below of a heavily right-skewed feature - call it x. I log scaled x, but it is still right-skewed and ...

1
2 3 4 5
13