Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Group points to reduce data set such that the linear regression stays the same

I have a very long dataset and I'm trying to reduce it by grouping the data in periods of 24 hours. In this way, there will be a single data point that represents that day, but they must yield the ...
0
votes
1answer
19 views

How to improve regression neural network?

I am new to deep learning and data science and trying to increase my knowledge by working on some hackathons. Currently, the hackathon project I am working on has the task to predict the closing price ...
0
votes
0answers
9 views

Find if a string appear before another string [closed]

I have a string variable containing patients' addresses. My goal is to flag patients who live in "401 30th street". I would like to flags strings that contain the number "401" ...
0
votes
0answers
8 views

Converting string values to DateTime in Azure ML

I have dataset which contains a column showing dates as a "String" value. I'm seeking to convert these values to a proper "DateTime" value in Azure ML, but upon using the Edit ...
2
votes
0answers
24 views

Shuffling data yields significantly worse performance

Edit: I've experimented a few times, shuffling the data at various steps. It seems that as long as I restart the python kernel and reset the dataframe indices, the performance is good. I'm still not ...
1
vote
1answer
28 views

How to fix my CSV files? (ValueError: Found array with 0 sample(s) (shape=(0, 1)) while a minimum of 1 is required)

I have tried to import two csv files into df1 and df2. Concatenated them to make df3. I ...
0
votes
0answers
7 views

How to deal with errors or inconsistencies in the training data?

There are inconsistant wrong labels and consistant errors in training data. For the former I tried MC-dropout and data Shapley. For the later I wonder if manual data curation is a requisite?
0
votes
0answers
21 views

How to put feature names from a 4D NumPy array into a pandas dataframe with a one to one relationship [closed]

I have a 4D NumPy array containing the features: importer, product, demand sector, and ...
1
vote
1answer
24 views

Selecting Training Set from Model CV Error

What I would like to do is recursively: Train the model on all data Remove the sample(s) with highest error Repeat until the remaining samples have an acceptable error The hypothesis is: "To ...
0
votes
0answers
21 views

How to mathematically quantify the quality of a corpus?

I am working on a text classification project. I have around 60,000 text samples of 40intents. By calculating the frequency of ...
0
votes
1answer
17 views

ML for data processing. What are the options?

Currently I am working on improving a stage on a data processing pipeline. The source data has a large number of fields and is getting normalized into a simpler entity. This entails that in many cases ...
1
vote
1answer
24 views

Encoding "histogram bins"

I am currently working on a regression problem where I have one variable (x) of the data in the form of "histogram bins". I.e. I could have value ranges ...
1
vote
2answers
58 views

Dealing with high number of NAs in a classification problem

I am working on a classification problem. The dataset dimension is as 187,643 x 203. The first column contains class labels with no NA. The rest of dataset are frequency data and could be anything ...
1
vote
0answers
8 views

Identifying Mixed Label Duplicates In Large Data Set

I have a data set containing 250k documents and 400k labeled topics. Each document may have 200+ topics. These topics may significantly overlap in subject and create noise. I want to reduce the ...
-1
votes
1answer
25 views
0
votes
0answers
10 views

EDA on multivariate time series data

I'm working with a medical dataset in a "long" format, meaning each row represents a timestamp and I have a column named "test" representing the test, an additional column ...
1
vote
0answers
20 views

How does a misrepresented disproportionate data affects modelling?

Let's say I have a dataset of the occurrence of pregnancies each time is tried, the ground truth of success to failure rate is 30:70. But the dataset with me now is a 70:30 dataset. How would that be ...
0
votes
0answers
15 views

formatting data from pandas into CNN-compatible array

I have a pandas dataframe of this form: time x y 0 a b 1 c d For my CNN model training, I want to slice the first 5 rows (from t=0 to t=4) and apply my custom function to label this sliced set, ...
0
votes
0answers
14 views

Time series classification - subject level analysis

I have a dataset with smartphone sensors data collection. There are 300 subjects, and the frequency of the sensors collected is different for different subjects. The shape of the data is (20k, 30) -&...
1
vote
3answers
69 views

Do I need to manually trim 300 videos?

I wish to train a model that detects the breed of a dog based on video input. I have a dataset containing 10 classes with 30 videos in each class. The problem is that for each of these videos, the dog ...
2
votes
1answer
179 views

How to impute missing text data?

Lets say I have a dataframe consisting of two text columns. By text, I mean the values in those columns are either sentences/paragraphs. In such a case, how do I handle missing 'NaN' values? If it ...
1
vote
1answer
20 views

If we replace all missing values with "unknown" or "-∞", what problem will we encounter?

I am reading Han,Kamber,Pei's data mining book and I stumbled upon a section called "data cleaning". It tells we can use a global constant like "unknown" or "-∞" to ...
0
votes
0answers
20 views

A feature is still right-skewed after log scaling. How should it be normalized for machine learning?

I've attached two images below of a heavily right-skewed feature - call it x. I log scaled x, but it is still right-skewed and ...
-1
votes
1answer
45 views

Anomaly Detection and Removal/Interpolate [closed]

I am performing a machine learning regression task on time series data. I have a data frame filled with the close prices of various assets and economic data. I am looking to perform outlier detection ...
0
votes
0answers
11 views

Removing categories with low sample size

I have a categorical column with 4 unique labels: Left ventricular hypertrophy Normal rest ecg Wave abnormality Out of 831 rows, only 4 of them include wave abnormality. It is a really low sample ...
1
vote
2answers
21 views

What are some examples of ordered data?

Tan,Steinbech,Kumar's book says For some types of data, attributes have relationships that involve order in time or space. I want some real life examples of such data, can you provide me. I couldn't ...
0
votes
0answers
7 views

How to clean/analyze relative time series data

I'm relatively new to data science, and I decided to stretch a bit and try a project with time series data. I downloaded the Human Activity Recognition from Continuous Ambient Sensor Data Data Set, ...
0
votes
0answers
49 views

Null and 0 Values Handling - Basic Statistics [duplicate]

I am trying to do some descriptive statistics and basic graphs on relatively small data sets derived from environmental sample collection and/or chemical research. It turns out that sample points end ...
0
votes
1answer
16 views

Concatenating Data in two years

I have to use a Machine Learning Model to predict the Electricity consumption and carbon emission based on some buildings' features. (Area, year of construction ...) Here is the link to the data. The ...
0
votes
0answers
43 views

Com posso resolver esse erro: TypeError: a bytes-like object is required, not '_io.BufferedReader'

msg: erro: TypeError: a bytes-like object is required, not '_io.BufferedReader' My code: import pickle with open(b'ModelosParaTrader/ModeloEurUsd.pkcls', 'rb') as modelo: lr = pickle.loads(modelo) lr
0
votes
1answer
53 views

OneHotEncoder Showing error while encoding two columns

I am encoding two columns with index number 1 and 2 that is column number 2 and 3 , using the following code, however I am facing error of invalid syntax : If I am using only 1 in the index value it ...
1
vote
1answer
39 views

What to do when the target variable does not correlate with any of the independent variable in a dataset?

I am quite new to data science. I am trying to use Logistic Regression to predict my target (either 1 or 0). But the problem is when I use a heatmap to find the correlation between the columns and the ...
0
votes
1answer
13 views

Advices to create a data set from dictionnary's screenshot

I have six thousand screenshots like the one below. I would like to create a date set to do some deep learning. My goal in the end is to create the next word prediction using only my own word data set....
0
votes
2answers
28 views

Assign value of existing column to new columns in pandas based on conditions [closed]

I am trying to create a new column in pandas data frame based on existing columns. column1 column2 column3 y1 y2 y3 100 200 300 2020 2021 2022 100 200 300 2021 2022 2023 100 200 300 2019 2020 2021 ...
0
votes
1answer
46 views

Convert any format of phone number to (111) 111-1111 using python [closed]

I have a phone number column in my dataset which has numbers in different formats. Here are some examples: ...
2
votes
1answer
24 views

Removing Outliers within R Works on One Attribute But Not on Second

(Note: this is part of a homework assignment. I am not asking for a solution to the assigned problem, I am just confused by some behavior in R.) I have a dataset loaded into an R script as a dataframe....
1
vote
0answers
15 views

Is there any standard guidelines, best practices on handle missing data?

Is there any standard guidelines, best practices on handle missing data? E.g. with certain threshold, one can still proceed with "data cleaning". If not certain approaches would be best ...
1
vote
1answer
12 views

Want to convert a data file into a list of lists where the semi-colon is a separator

I'm doing my very first data science project ever and am likely to seem a bit naive, and I've got three .DAT files each of which I want to convert into a list of lists, and the code I've been using so ...
1
vote
0answers
26 views

Dataset format for Transformer text-generation

I'm trying to find some tutorials on training Transformer for generating comments on articles. So far, I found an article showing how to train GPT2 as a chat-bot. Input files in that example are given ...
0
votes
0answers
21 views

How can I calculate total days past due between billing events?

I am dealing with a dataframe with subscription events partitioned by username, subscription status, and relative timestamps. For each of the dates, there are changes in time when the subscription ...
1
vote
0answers
8 views

Does it make more sense to use central tendency methods on training and test sets separately?

I'll explain further, so I'm taking a data science course on cleaning and preparing data and I'm on the how to handle missing data section. So the question is essentially what you see above except it ...
2
votes
1answer
24 views

Training & Test feature shape is different from number of columns in dataset

I am making a Sequential Neural Network for regression with 3 dense layers which will be trained on a simple dataset. But before I even get to that part of the code to execute the model I am getting a ...
1
vote
1answer
17 views

Should I apply a transformation to columns with INTEGERS, in case I want to reduce the skewness of that column?

I am performing EDA on a dataset of Hotel Reservations. Target is Categorical stating if a given customer will cancel the reservation or not. Dataset has 25 features, 30244 entries. I have two ...
0
votes
0answers
8 views

How to introduce a parameter for measuring change in data over time

In my project, I need to introduce a measure for 'movement' using a 3axis accelerometer (ADXL345). As sketched below: I thought about introducing some micro-changes, i.e. absolute change in ...
0
votes
2answers
45 views

How to reshape or clean data to be able to visualize it with violin plots?

My end goal is to visualize some data using a violin plot or something similar using Python. I have the following data in a file (test.csv). The first column is a ...
1
vote
1answer
25 views

Data cleaning in Pandas, where the csv file has all data of each row in 1 field [closed]

I have really messy data that looks like this: As you can see all the data in each row is contained in 1 column separated by a semi colon. How do I arrange this data so that they are spread out over ...
1
vote
1answer
13 views

Handling Missing Inconsistent Educational Data

I'm an educational researcher learning about machine learning so I can further explore my data beyond the usual statistics. I currently have some assessment data but I am not sure how to appropriately ...
1
vote
0answers
20 views

Working with others: Tidy data vs Pretty data [closed]

When working with someone whose background and skill level in data work may not be strong, how do you best make the argument for tidy data over "pretty" data? There are notes of what I want ...
0
votes
0answers
11 views

Finding the wiggles pattern in the original dataset. (Wiggles appear after performing division by another dataset)

I have multiple measurements regarding scientific observations. The problem is that there is a subtle noise pattern caused by the instrument - the wiggles. These wiggles are invisible when looking at ...
1
vote
1answer
30 views

GPA prediction of college student

I have a dataset consist of 8 columns and 15600 rows with the following columns:- 1.Entry_academic_year which have 5 discrete value (2558,2559,2560,2561,2562) 2.Faculty (It is the faculty that ...

1
2 3 4 5
13