Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
1
vote
1answer
26 views

Tidy vs. untidy data

In R for Data Science, the authors explain their idea of tidy data. They give an example for data that is not tidy: ...
0
votes
0answers
14 views

Does every column need to be np.log?

I have questions regarding data cleaning for machine learning. Let's say my dataset has three columns with different skewness For example: label column skewness = 1.500, feature column 1 ...
1
vote
1answer
25 views

Knit with R markdown [closed]

I have tried running some code on RStudio Desktop, within the Chunk the codes runs smoothly, but when knitting to view as html, I get the message that something is wrong with a line of code. What can ...
0
votes
1answer
17 views

Logic Behind IQR Outlier Detection

Why we using IQR to detect outlier in Dataset , what is the logic behind the method and is there any other method to detect outliers?
0
votes
1answer
19 views

Data snooping and information leakage?

I need help in deciding whether my below implementation imposes data snooping bias and information leakage from the test/evaluation set to the train set. I have a text corpus of 10k+ short online ...
0
votes
0answers
5 views

Methods to extract / merge signal from three noisy time series (of same event)

I have three time series of same length, all containing magnitude measurements of the same event "A". But each time series is using a different method of measurement. My goal is to merge the ...
1
vote
0answers
20 views

How can one ensure the quality of collectively created ground truth labels?

Here is the situation: I have a small tool to assign (one or multiple) labels to image data. In the tool, you can not only assign a label(s) to the images but also assign a score for the certainty of ...
0
votes
1answer
23 views

Extracting the metadata form Json file making it columns

I have the following json data file which I have converted to pandas dataframe. The columns are as follows ...
1
vote
0answers
38 views

How to represent/aggregate data belonging to same group, and feed to model for training?

I have a problem in which there are couple of records that belong to the same group, like multiple observations. For example this data set has three groups, For each group, I have three observations....
0
votes
1answer
31 views

Data Preparation for dates

I have two columns, a birth year and the first year of being a customer. There is a large number of samples that have a birth year less than the first year of being a customer which does not make ...
0
votes
0answers
7 views

How to use K-NN imputer without replacing with decimal values example ( 0.75,0.6) instead of binary outcome (yes or no, 1 or 0)?

I am trying to impute some missing categorical values using K-NN imputer, after imputation the missing values are replaced with some decimal numbers. I want to use K-NN as classifier and the output (...
0
votes
2answers
42 views

How to remove rows from a data frame that have special character (any character except alphabet and numbers)

How to remove rows from a data frame that have special character (any character except alphabet and numbers) I have some unwanted labels which seems to be useless but I want to remove them, this are ...
0
votes
1answer
26 views

How to find median/average values between data frames with slightly different columns?

I am trying to combat run-to-run variance of the data I collect by combining the data from different runs and finding the mean/average. The problem is that in each run there is a chance that some of ...
0
votes
2answers
42 views

How can I know that this works and is correctly done?

Problem: This seems to be very wrong, cannot pinpoint what I am missing or doing wrong here. Should all rows for each column be Normalized? How? I am working on some small thing, and want to get a ...
0
votes
0answers
23 views

Create rows filling the sequence of numbers in a 'period column' and the 'value column' with 0 in python/pandas

I need to fill a dataframe with rows completing the values that the system doesn't show on the report (e. g. a week that the product had no sales): The report: Product Week Total_Sales 0 A 1 82 1 A ...
1
vote
0answers
32 views

How to combine data from multiple Google Trends queries effectively?

As you might know, Google Trends works by normalising a random sample of the search term data, with the sample changing at least once per day, from my experience. This is not an issue for western ...
0
votes
0answers
13 views

Why feature engineering and filling NaN's reduce score?

I used CatBoost for InClass Kaggle competition. I have tried various strategies to filling NaN values. Convert float binary variables to categorical. Add new categorical features (from age, for ...
0
votes
0answers
16 views

How to read multiple JSON files to a dataframe in R? can I get the code with the comments on it to understand it better?

I have multiple json files in a folder called "app users". These json files has NAN values as well in it.
2
votes
0answers
28 views

change target variable value to reflect better affordability

Context I am working on a regression problem trying to predict affordability. My dataset contains daily installments repaying a purchase in a form of contract. Essentially, a minimum daily rate the ...
0
votes
0answers
20 views

Handling repeating data from different individuals

I have a dataset that has some unique values but also includes information from multiple individuals that are repeating, meaning they are describing the same attributes and can have the same or ...
1
vote
0answers
16 views

Tokenizer returning incorrect values and losing a lot of data

(cross posted from main stackoverflow) This is a weird situation so I hope I can explain it correctly. My partner and I are working on a ML project where we create a model that predicts whether a ...
1
vote
1answer
34 views

Drop or impute the missing values?

I am working with a dataset having 45k rows and I was a bit confused on whether or not to drop the missing values OR impute the missing values. Column wise missing value distribution : As per this ...
0
votes
0answers
28 views

Should I remove rows without target value?

I am new to data science so any help is appreciated. Thanks in advance. I have a data set with a few explanatory variables and a target column that contains the label of each row. However, some labels ...
0
votes
2answers
19 views

How do I remove float object in dataframe? Ex: "winter 2021" to "winter"

I have the premiered column where I want to remove the year and only keep the season. Example: "winter 2021" to become "winter" .
1
vote
1answer
18 views

Defining a function in Pandas [closed]

Hello Data Scientists, Disclaimer: I am a Beginner learning Python and at the same time just trying out a lot of things. I am doing some data extraction on a file with with weekly deaths in ...
0
votes
0answers
25 views

Scaling columns pandas DataFrame

For a dataset having different numeric columns, they usually have different range and distributions. As an example, I have used the Iris dataset. The distributions of it's 4 columns are shown: My ...
0
votes
1answer
30 views

How to convert duration column from 1 hr. 17 min to 77 min in pandas? [closed]

I am trying to convert all the hr., min, and sec into just mins. For example, 1 hr. 17 min to 77 min, and 34 sec to 0.56 min, not 0.34 min. So I have used this code: ...
0
votes
0answers
30 views

Is it possible to derive anything useful from this piece of data?

Let's say you have online Profile A. Profile A is present on 3 websites: X, Y, Z. ...
0
votes
0answers
6 views

I am trying to sum up the "travel time in minute" for lane "GP" and grouping it under a single "Toll ID"

I am trying to sum up the "travel_time_minutes" for lane_type "GP" and grouping it under a single "Toll ID". I mean 0.11 + 0.50 + 1.21 for GP and retaining the Toll ID of ...
-1
votes
1answer
48 views

how to fill nan values based on conditions in pandas?

I am working on the Titanic dataset and there are null values in the Age column. I would like to fill them in based on there Pclass. For example, if class is 1, I ...
0
votes
0answers
13 views

Aggregating transactional data for customer segmentation

I have item-level transactional data where each row in the data represents a different item bought by a customer in a transaction (so if two different items were bought in the same transaction by the ...
0
votes
0answers
28 views

Identify the missing value in a table

How to compute the missing values x in the below table. which formula can be used to obtain the values of x. The values in the last row and last column are the sums of the rows and columns. What is ...
2
votes
1answer
52 views

Converting YYYYQQ to YYYY-MM-DD [closed]

I have been trying to convert stuff like 2002Q3 to 2002-09-01, but if I use the following, it will convert 2002Q3 to 2002-07-01: ...
1
vote
0answers
20 views

Missing data for time-series forecasting/classification

I want to train a model to predict some customer behavior. I have two types of data that I have to deal with. My measurements are daily. The first type is regular measurements without any missing data,...
0
votes
1answer
59 views

Should outliers be removed only from the target variable or from any variable where they are found?

What I often do is that I check boxplots and histograms for target/dependent variable and after much caution, treat/remove the outliers. But this is what I do only for the target variable. I.e., if ...
0
votes
0answers
43 views

Global models for time series prediction with series of different lengths

As I haven't found much on the topic, I wanted to start the discussion about local- (iteratively for every column) vs. global models (only one iteration per forecast horizon, ex ante). As this is a ...
1
vote
1answer
41 views

test data is not a good representation of train data

I have predefined train and test sets. On generating some statistics like value_counts and checking the unique values, I feel that there is a 'lot' of difference between the distributions of the ...
0
votes
0answers
34 views

How can I balance sentence data for NLP tasks

I have been given a task to train the SVM model on conll2003 dataset for Named Entity "Identification" (That is I have to tag all tokens in "Statue of Liberty" as named entities ...
1
vote
0answers
32 views

How to handle unclassifiable data in the dataset

Premise: Classification problem Input is three text fields Output classes are A, B, A&B (Note: A and B are not always exclusive though usually are, hence the 'A&B' class) Sci-Kit Learn is the ...
0
votes
1answer
36 views

Deleting a Column in a csv or Excel file using Pandas

I am trying to delete a column from my csv file (column 'A' called "Film Number") but have tried numerous variations of code and while it deletes the column in the dataFrame it doesn't do so ...
1
vote
0answers
22 views

How to handle insufficient information about the collected data in the dataset?

I am trying to create a ML model to predict foodborne diseases. I found online a few datasets with the informations I need. The problem is that none of them explains how they got the medical measures. ...
0
votes
0answers
14 views

When to do tokenization and does my output need tokenization after stemming?

I am working on sentiment analysis project , where there are various customer reviews. So I am trying to clean those reviews. So first thing i did is removing special characters, white spaces, ...
1
vote
1answer
55 views

Improving text classification & labeling in imbalanced dataset [closed]

I am trying to classify text titles (NLP) in categories. Let us say I have 6K titles that should fall into four categories. My questions: I do not understand why in some ML techniques categories are ...
0
votes
1answer
22 views

How to deal with data having 0 values in many columns?

I am trying to implement logistic regression but the dataset that I have have many columns with skewed data and most of them have 0 as values. I also the skewness of data for many columns its going ...
0
votes
2answers
39 views

If I have two variables with strong correlation, should I delete one and leave the other in my data

I have a large dataset, where I should make a binary prediction. The fact is that, after analyzing the data, I found that some variables are positively correlated to each other. So, I was wondering ...
1
vote
1answer
25 views

Cleaning rows of special characters and creating dataframe columns

Below is my Dataframe format consisting of 2 columns (one is index and other is the data to be cleaned) ...
0
votes
0answers
9 views

How to trim the train, test and validate datasets for LSTM to avoid shape issues?

I am trying to get an LSTM to optimize for regression of stock returns, but I keep encountering shape errors during the fit() phase. Specified a list with shape [30,2] from a tensor with shape [1,2] ...
0
votes
1answer
30 views

How to visualize data after performing OneHotEncoding and normalization?

I have a dataset and on that, I have performed OneHotEncoding and Standardization using standard scalar, Now that I have preprocessed data I have to visualize it, but on converting it to pandas ...

1
2 3 4 5
14