Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
1
vote
0answers
8 views

How to combine data from multiple Google Trends queries effectively?

As you might know, Google Trends works by normalising a random sample of the search term data, with the sample changing at least once per day, from my experience. This is not an issue for western ...
0
votes
0answers
11 views

Why feature engineering and filling NaN's reduce score?

I used CatBoost for InClass Kaggle competition. I have tried various strategies to filling NaN values. Convert float binary variables to categorical. Add new categorical features (from age, for ...
0
votes
0answers
9 views

How to read multiple JSON files to a dataframe in R? can I get the code with the comments on it to understand it better?

I have multiple json files in a folder called "app users". These json files has NAN values as well in it.
2
votes
0answers
25 views

change target variable value to reflect better affordability

Context I am working on a regression problem trying to predict affordability. My dataset contains daily installments repaying a purchase in a form of contract. Essentially, a minimum daily rate the ...
0
votes
0answers
18 views

Handling repeating data from different individuals

I have a dataset that has some unique values but also includes information from multiple individuals that are repeating, meaning they are describing the same attributes and can have the same or ...
1
vote
0answers
16 views

Tokenizer returning incorrect values and losing a lot of data

(cross posted from main stackoverflow) This is a weird situation so I hope I can explain it correctly. My partner and I are working on a ML project where we create a model that predicts whether a ...
1
vote
1answer
27 views

Drop or impute the missing values?

I am working with a dataset having 45k rows and I was a bit confused on whether or not to drop the missing values OR impute the missing values. Column wise missing value distribution : As per this ...
0
votes
0answers
27 views

Should I remove rows without target value?

I am new to data science so any help is appreciated. Thanks in advance. I have a data set with a few explanatory variables and a target column that contains the label of each row. However, some labels ...
0
votes
2answers
15 views

How do I remove float object in dataframe? Ex: "winter 2021" to "winter"

I have the premiered column where I want to remove the year and only keep the season. Example: "winter 2021" to become "winter" .
1
vote
1answer
18 views

Defining a function in Pandas [closed]

Hello Data Scientists, Disclaimer: I am a Beginner learning Python and at the same time just trying out a lot of things. I am doing some data extraction on a file with with weekly deaths in ...
0
votes
0answers
19 views

Scaling columns pandas DataFrame

For a dataset having different numeric columns, they usually have different range and distributions. As an example, I have used the Iris dataset. The distributions of it's 4 columns are shown: My ...
0
votes
1answer
28 views

How to convert duration column from 1 hr. 17 min to 77 min in pandas? [closed]

I am trying to convert all the hr., min, and sec into just mins. For example, 1 hr. 17 min to 77 min, and 34 sec to 0.56 min, not 0.34 min. So I have used this code: ...
0
votes
0answers
29 views

Is it possible to derive anything useful from this piece of data?

Let's say you have online Profile A. Profile A is present on 3 websites: X, Y, Z. ...
0
votes
0answers
6 views

I am trying to sum up the "travel time in minute" for lane "GP" and grouping it under a single "Toll ID"

I am trying to sum up the "travel_time_minutes" for lane_type "GP" and grouping it under a single "Toll ID". I mean 0.11 + 0.50 + 1.21 for GP and retaining the Toll ID of ...
-1
votes
0answers
33 views

how to fill nan values based on conditions in pandas?

I am working on the Titanic dataset and there are null values in the Age column. I would like to fill them in based on there Pclass. For example, if class is 1, I ...
0
votes
0answers
13 views

Aggregating transactional data for customer segmentation

I have item-level transactional data where each row in the data represents a different item bought by a customer in a transaction (so if two different items were bought in the same transaction by the ...
0
votes
0answers
27 views

Identify the missing value in a table

How to compute the missing values x in the below table. which formula can be used to obtain the values of x. The values in the last row and last column are the sums of the rows and columns. What is ...
2
votes
1answer
45 views

Converting YYYYQQ to YYYY-MM-DD [closed]

I have been trying to convert stuff like 2002Q3 to 2002-09-01, but if I use the following, it will convert 2002Q3 to 2002-07-01: ...
1
vote
0answers
19 views

Missing data for time-series forecasting/classification

I want to train a model to predict some customer behavior. I have two types of data that I have to deal with. My measurements are daily. The first type is regular measurements without any missing data,...
0
votes
1answer
46 views

Should outliers be removed only from the target variable or from any variable where they are found?

What I often do is that I check boxplots and histograms for target/dependent variable and after much caution, treat/remove the outliers. But this is what I do only for the target variable. I.e., if ...
0
votes
0answers
42 views

Global models for time series prediction with series of different lengths

As I haven't found much on the topic, I wanted to start the discussion about local- (iteratively for every column) vs. global models (only one iteration per forecast horizon, ex ante). As this is a ...
1
vote
1answer
39 views

test data is not a good representation of train data

I have predefined train and test sets. On generating some statistics like value_counts and checking the unique values, I feel that there is a 'lot' of difference between the distributions of the ...
0
votes
0answers
33 views

How can I balance sentence data for NLP tasks

I have been given a task to train the SVM model on conll2003 dataset for Named Entity "Identification" (That is I have to tag all tokens in "Statue of Liberty" as named entities ...
1
vote
0answers
31 views

How to handle unclassifiable data in the dataset

Premise: Classification problem Input is three text fields Output classes are A, B, A&B (Note: A and B are not always exclusive though usually are, hence the 'A&B' class) Sci-Kit Learn is the ...
0
votes
0answers
15 views

fast detection of mislabelled data samples

I have about 20% mislabelled samples within my dataset and I am looking for "fast" ways to detect them, one way that occurred to me: step 1: build a classifier (with k fold) on the dataset ...
0
votes
1answer
31 views

Deleting a Column in a csv or Excel file using Pandas

I am trying to delete a column from my csv file (column 'A' called "Film Number") but have tried numerous variations of code and while it deletes the column in the dataFrame it doesn't do so ...
1
vote
0answers
22 views

How to handle insufficient information about the collected data in the dataset?

I am trying to create a ML model to predict foodborne diseases. I found online a few datasets with the informations I need. The problem is that none of them explains how they got the medical measures. ...
0
votes
0answers
11 views

When to do tokenization and does my output need tokenization after stemming?

I am working on sentiment analysis project , where there are various customer reviews. So I am trying to clean those reviews. So first thing i did is removing special characters, white spaces, ...
1
vote
1answer
51 views

Improving text classification & labeling in imbalanced dataset [closed]

I am trying to classify text titles (NLP) in categories. Let us say I have 6K titles that should fall into four categories. My questions: I do not understand why in some ML techniques categories are ...
0
votes
1answer
22 views

How to deal with data having 0 values in many columns?

I am trying to implement logistic regression but the dataset that I have have many columns with skewed data and most of them have 0 as values. I also the skewness of data for many columns its going ...
0
votes
2answers
39 views

If I have two variables with strong correlation, should I delete one and leave the other in my data

I have a large dataset, where I should make a binary prediction. The fact is that, after analyzing the data, I found that some variables are positively correlated to each other. So, I was wondering ...
1
vote
1answer
21 views

Cleaning rows of special characters and creating dataframe columns

Below is my Dataframe format consisting of 2 columns (one is index and other is the data to be cleaned) ...
0
votes
0answers
8 views

How to trim the train, test and validate datasets for LSTM to avoid shape issues?

I am trying to get an LSTM to optimize for regression of stock returns, but I keep encountering shape errors during the fit() phase. Specified a list with shape [30,2] from a tensor with shape [1,2] ...
0
votes
1answer
29 views

How to visualize data after performing OneHotEncoding and normalization?

I have a dataset and on that, I have performed OneHotEncoding and Standardization using standard scalar, Now that I have preprocessed data I have to visualize it, but on converting it to pandas ...
1
vote
0answers
16 views

Problem with FFT in Python for smaller df

...
0
votes
1answer
25 views

how to filter out and discard irrelevant tweets in simplest way possible

I have lot of tweets and from which i need to filter out and discard irrelevant tweets. the criteria for a tweet to be irrelevant is very simple. if all that a tweet has is emojis or a single hastag ...
1
vote
1answer
15 views

Full gaps in the curve

This is the data from Irelands vaccine program. There are these missing bits of data. Is there away to fill these gaps with proportional data to make a smooth curve?
1
vote
0answers
18 views

Group points to reduce data set such that the linear regression stays the same

I have a very long dataset and I'm trying to reduce it by grouping the data in periods of 24 hours. In this way, there will be a single data point that represents that day, but they must yield the ...
0
votes
1answer
26 views

How to improve regression neural network?

I am new to deep learning and data science and trying to increase my knowledge by working on some hackathons. Currently, the hackathon project I am working on has the task to predict the closing price ...
2
votes
0answers
35 views

Shuffling data yields significantly worse performance

Edit: I've experimented a few times, shuffling the data at various steps. It seems that as long as I restart the python kernel and reset the dataframe indices, the performance is good. I'm still not ...
1
vote
1answer
37 views

How to fix my CSV files? (ValueError: Found array with 0 sample(s) (shape=(0, 1)) while a minimum of 1 is required) [closed]

I have tried to import two csv files into df1 and df2. Concatenated them to make df3. I ...
0
votes
0answers
9 views

How to deal with errors or inconsistencies in the training data?

There are inconsistant wrong labels and consistant errors in training data. For the former I tried MC-dropout and data Shapley. For the later I wonder if manual data curation is a requisite?
1
vote
2answers
49 views

Selecting Training Set from Model CV Error

What I would like to do is recursively: Train the model on all data Remove the sample(s) with highest error Repeat until the remaining samples have an acceptable error The hypothesis is: "To ...
0
votes
0answers
23 views

How to mathematically quantify the quality of a corpus?

I am working on a text classification project. I have around 60,000 text samples of 40intents. By calculating the frequency of ...
0
votes
1answer
19 views

ML for data processing. What are the options?

Currently I am working on improving a stage on a data processing pipeline. The source data has a large number of fields and is getting normalized into a simpler entity. This entails that in many cases ...
1
vote
1answer
29 views

Encoding "histogram bins"

I am currently working on a regression problem where I have one variable (x) of the data in the form of "histogram bins". I.e. I could have value ranges ...
1
vote
2answers
64 views

Dealing with high number of NAs in a classification problem

I am working on a classification problem. The dataset dimension is as 187,643 x 203. The first column contains class labels with no NA. The rest of dataset are frequency data and could be anything ...
1
vote
0answers
8 views

Identifying Mixed Label Duplicates In Large Data Set

I have a data set containing 250k documents and 400k labeled topics. Each document may have 200+ topics. These topics may significantly overlap in subject and create noise. I want to reduce the ...
-1
votes
1answer
45 views

1
2 3 4 5
13