Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0
votes
1answer
16 views

How to get mean of column using groupby() and another condition [on hold]

For the following df = pd.DataFrame() df['1'] = 1,2,1,2,1,2 df['2'] = 3,6,5,4,7,8 df['3'] = 1,1,1,2,2,2 I want to do ...
0
votes
0answers
15 views

How do I export my data into .h5 format?

I am trying to make my data useful for to use in this model, i.e for Hierarchical Novelty Detection for Visual Object Recognition. I need to prepare my own dataset in a format like this: ...
0
votes
0answers
11 views

How to aggregate large data

I have a quite large data (frame) with 65 physical quantities and each with different time stamps. Some are gathered in intervals of several hours and some in milliseconds. Hence, the data frame ...
-2
votes
0answers
14 views

Information extraction from scanned invoice [closed]

I had a requirement of extraction of invoice number from the scanned copies. Is there any possibility of extraction. Please specify any ways or methods to extract the information.
1
vote
1answer
18 views

Noise Elimination with majority vote filtering

I have a dataset with label noise which I wan't to clean with majority/consensus vote filtering. This will mean I will divide the data in K-Folds and train an ensemble model. Than using the ...
0
votes
0answers
13 views

Keras error in dimensions when predicting

I have trained a Keras LSTM model and was now trying to use it for predictions but for some reason in is giving a dimensions error I cannot find. I processed the data in the same way as the training ...
0
votes
1answer
16 views
2
votes
0answers
22 views

Can we use log transformation for outliers?

I am doing analysis on telecom churn dataset. I have 4617 observations and 17 variables. I am using Python. I have the following questions, 1) When I skewness and kurtosis for the normality test, two ...
2
votes
1answer
42 views

Data Extraction from images using NLP and ML [closed]

Hi i'm trying to extract data like name, planType, phone#, .. from images like insurance cards or licence cards with GoogleVision / Textract using some conditions but it does not extract the correct ...
0
votes
1answer
29 views

Creating new column in dataframe based on conditions in 2 other columns [closed]

I would like to create a new column in my dataframe based on values from both the gender and experimental_grouping columns. As I have it written below, the column ...
0
votes
0answers
22 views

Creating artificial blurred image to train neural network for super resolution

I would like to create blurred images from images obtained from large pdf with the intention of creating a training data set. I would like to create the blur as shown in the uploaded picture. Which ...
0
votes
0answers
24 views

While reading the pages of a PDF file using Python, I get the following error. There are 300 pages in pdf file

CalledProcessError: Command '['java', '-Dfile.encoding=UTF8', '-jar', 'C:\Users\105051884\AppData\Local\Continuum\anaconda3\lib\site- packages\tabula\tabula-1.0....
2
votes
2answers
21 views

How do I find the count of a particular column, based on another column(date) using pandas?

I have a dataframe with 3 columns, such as SoldDate,Model and TotalSoldCount. How do I create a new column, 'CountSoldbyMonth' which will give the count of each of the many models sold monthly? <...
0
votes
1answer
14 views

CASE sentence for SQL query

I have a question abut MySQL I would like to handle a column with characters without changing the original DB. Below is the levels of the column named 'Maker'. 기아, 현대, BMW, VolksWagen, Porsche So ...
0
votes
1answer
23 views

How to transform many columns of TRUE/FALSE/NA to just TRUE/FALSE?

I have a dataframe that consists of a few columns of text, and then a bunch of columns that are TRUE/FALSE or NA (they were TRUE/FALSE but I left-joined them with ...
0
votes
2answers
35 views

Url string processing: what is the best way?

I have ~1000 different news websites and I scraped and saved all the internal url links for each website. For instance, the website dcgazette.com has a 2MB text file with associated urls: 1) https://...
-2
votes
0answers
21 views

Issue with OnehOtEncoder-

Here is my code: ...
1
vote
1answer
40 views

Feature engineering for categorical variables

I have some categorical variables in my dataset for a regression problem. 1) One of the variable can take 3 values (Girls, Boys, Girls&Boys). Converting it into one-hot encoding or binary ...
0
votes
0answers
21 views

Reading pdf file with all pages in Python error: CalledProcessError:

While reading the pages of a PDF file using Python, I get the following error. There are 300 pages in pdf file. ...
0
votes
0answers
10 views

Do I discard all my dependent variables as proved by chi-squared test of independence?

I have 134 categorical columns in my data. 7 of which are categorical variables [ one variable is highly unbalanced and has 34 classes while all other variables just has 3-5 classes in each variable ...
2
votes
1answer
33 views

How valuable is a categorical feature that has a predominant category over all other ones?

Is a categorical feature that has almost equally distributed in it's category more important or the one which one of it's category is predominant over all other ones? In data prepossessing step for "...
0
votes
1answer
13 views

how to export the data edited to a new excel file

The provided code imports the excel file and removes the not NaN values and store it in df2. Now, How do i save the changes to an another file. ...
0
votes
1answer
16 views

How to delete a row if a values in a column is not NaN

In the given code it displays only the rows without NaN values but i want only the rows with NaN values in a provided column everything has to be removed ...
2
votes
1answer
32 views

How to scale or standardize data that is mostly 0 (ranges from 0-1)?

I am relatively new to data science and big data munging in general. I currently have various columns of data that range from $0-1$, but most of the values in each ...
0
votes
0answers
23 views

Detect Typical Customer Mistakes in the Shopping Cart

I would like to ask your advice on solving this problem. Problem: There is an online store which sells furniture. There are millions of products on the store like furniture parts and furniture ...
2
votes
1answer
24 views

how to balance the data set with different number of observations

I am pretty new to ds field and recently I was working on a self project of the clustering model. My goal is to create clusters and see in each cluster what the common features are between customers. ...
3
votes
0answers
36 views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this dataframe (df) ...
0
votes
0answers
21 views

How to reduce / avoid false predictions with sklearn and MultinomialNB?

I'm using sklearn to predict product groups from product titles. That is working very well, if the titles are similar to the ones I labeled. Simplified example: ...
0
votes
1answer
25 views

Combine Pandas DataFrame Rows Based on Matching Data and Boolean

I have a Pandas DataFrame with sales data and columns for year, ISO week, price, ...
1
vote
1answer
24 views

Tool for clustering and cleansing data set

I have a large-ish data set (400K records) composed of two fields (both strings). I am looking for a tool that will enable me to cluster the data e.g. around the first column, either using exact ...
0
votes
0answers
14 views

Formatting multi-level variable width data for model

I would like to calculate a value based on the number of lines on outage at a given time. An outage consists of the two endpoints and the voltage of the line: ...
1
vote
1answer
17 views

If categorical variable has some hierarchy, should I just label them or split into dummy variables (One-Hot encode)?

I have a column which has 5 unique categories. There's a hierarchy between these categories (Best > good > OK/Not Sure > Bad > Worst) In this case, should I label them based on hierarchy like: ...
1
vote
1answer
20 views

Input Normalization for Transfer Learning

If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train ...
1
vote
0answers
30 views

Cleaning Excel Data in Merged Cells

I need to use an Excel file that have many merged cells with no specific order. I guess that the file creator thought that it would be more readable to merge the cells that have the same value. ...
1
vote
1answer
24 views

Data duplication optimization

I am working in python3 cleaning data. I have a large number of midi files scraped from a variety of sources using beautiful soup...
0
votes
2answers
57 views

Understanding missing values in dataset

I have recently worked with a dataset of real estate transactions with missing entries for some features. For instance, GarageYrBlt (year when a garage was built) ...
0
votes
0answers
10 views

Dict with features and classification -> two seperate but aligned lists

For my ML project, feature set extraction is expensive, but I need to be able to retrain the model on a fairly regular basis. For this reason I'm extracting my features from all of my documents at ...
1
vote
3answers
48 views

What is the correct procedure when “joining” data takes ~6 hours?

I am dealing with bike-share data. I have 2 DataFrames: trips_df (subset shown), total entries = ...
0
votes
0answers
12 views

Removing ambiguous data: Same input variables with different class labels

Background: I'm working with a tree-based ensemble model on a large data set. The target variable y is a binary attribute that have two classes (True and False). I noticed that some of the ...
1
vote
0answers
13 views

Term for an identifier that has been superseded

Is there a 'proper' term for an ID (or IDs) that have been superseded by (or merged into) another ID? My use case: rsIDs are used by geneticists to refer to a SNP. These take the form of a string '...
0
votes
2answers
33 views

Tips for checking data integrity / data sanity?

I've read a few vague articles and watched a couple of YouTube videos on data integrity and data sanity, but none of them have mentioned ways to actually check these on datasets. I am interested in ...
0
votes
1answer
24 views

Script to convert non-numeric elements that are numbers to numeric

I'm self-teaching myself some scikit-learn and tensorflow right now, and trying to get better at cleaning data since the rest seems reasonably straightforward. I downloaded some Google Play Store ...
2
votes
0answers
101 views

Jaccard Similarity with Binary Data

I have 5400 rows of data and 3211 columns of attributes. The first 4 columns are ID/Name/ParentID/ObjectType - the rest of the 3207 columns are the attributes that are to be used for similarity ...
1
vote
2answers
19 views

Keeping part of a string in R [closed]

I have a dataframe with the following column city <- c("Sydney NSW", "Newcastle NSW", "Liverpool NSW", "Broken Hill NSW") I want to maintain everything prior ...
3
votes
1answer
177 views

Splitting train/test sets by an identifier?

I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, ...
0
votes
0answers
6 views

What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?

Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem ...
0
votes
0answers
25 views

Apache Nifi out of memory

I've set up a data pipeline using Apache nifi. It processes 3GB scale json data. When I try to flatten these Json files it always outputs: ...
1
vote
0answers
28 views

Keras Binary Classification val_acc won't go past ~67; Full data and code included

I'm working on a binary classification in Keras with a Tensorflow backend. No matter how much I tweak, I can't seem to get my model past a val_acc of 67%. Is there something I'm missing, or is this ...
1
vote
1answer
33 views

How to normalize data from multiple sources?

I am trying to model an individuals' purchasing behavior using different data sources (ex: Zalando, Otto, etc.,). When I combine data sources, I see that the data across these channels is very ...
1
vote
1answer
48 views

Is this a data issue, or a model issue? A Keras binary classification model

I've been trying to create a binary classification model that predicts wether there will be a train delay based on the train and time. Here is a link to the data The issue I'm having is that my ...