Skip to main content

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

206 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3 votes
0 answers
26 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
dmrak's user avatar
  • 31
3 votes
0 answers
196 views

How to use zero-inflated negative binomial regression for binary classification task?

I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly ...
Rohit Gavval's user avatar
3 votes
1 answer
64 views

Data Exploration Led Conversion to Ordinal Variable

[I encountered this question at an interview few weeks ago and I am still not clear.] If all the values in a categorical column fuel_mileage come from the set <...
ImranAli's user avatar
2 votes
0 answers
49 views

change target variable value to reflect better affordability

Context I am working on a regression problem trying to predict affordability. My dataset contains daily installments repaying a purchase in a form of contract. Essentially, a minimum daily rate the ...
user128337's user avatar
2 votes
0 answers
60 views

Shuffling data yields significantly worse performance

Edit: I've experimented a few times, shuffling the data at various steps. It seems that as long as I restart the python kernel and reset the dataframe indices, the performance is good. I'm still not ...
tensormoby's user avatar
2 votes
1 answer
41 views

Manual Data Cleanup Tools

I am writing an ETL pipeline for geospatial data of the form place_name,address,longitude,latitude,id_linking_to_other_dataset As the last step in the pipeline, I ...
Hayden's user avatar
  • 21
2 votes
0 answers
234 views

How to process categorical variable having lots of unique values in linear regression?

I have House Price dataset and I am using linear regression to predict the house price. while data preprocessing I found a variable called "Location" and it have around 342 unique value. For ...
Rahul Pawade's user avatar
2 votes
1 answer
44 views

Architectures that take inputs of mixed sampling rates

Let's say a model is trained on multiple datasets of 1D time series. These datasets have been gathered with different sampling rates. I plan to use a convolution neural network to process these time ...
Chris Liu's user avatar
2 votes
2 answers
240 views

Missing value Imputation in dataset

I have two separate files for Testing and Training. In the training data, I am dropping rows that contain too many missing values . But , In the test data , I cannot afford to drop the rows so I have ...
Bharathi A's user avatar
2 votes
0 answers
24 views

Correlation matrix on multi-time scales

I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams. For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (...
JungleDiff's user avatar
2 votes
0 answers
252 views

Tools for reading data from large, irregular csv files (aka excel file hell)

I have a csv file with 1000 columns and 50 rows that was collected over a few months. Embedded randomly in the file are ~1000 small datasets with semi-standard formats (some columns differ, so does ...
R Greg Stacey's user avatar
2 votes
0 answers
76 views

Keras Binary Classification val_acc won't go past ~67; Full data and code included

I'm working on a binary classification in Keras with a Tensorflow backend. No matter how much I tweak, I can't seem to get my model past a val_acc of 67%. Is there something I'm missing, or is this ...
PavlovsCat's user avatar
2 votes
0 answers
1k views

Issues with pandas chunk merge

I'm trying to solve a kaggle competition Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back to disk. This is the code I'm using: ...
Sany's user avatar
  • 21
2 votes
0 answers
126 views

Prediction questions related to the dataset

I have been self-learning data science from different sources. I have a dataset which was sent to me by my friend from one of her college courses. The work is to be done in R preferably. Description: ...
El Tikki's user avatar
2 votes
0 answers
409 views

Online noise removal techniques

There are lots of noise removal algorithms for offline dataset. I would like to ask if there are some suitable online implementations to remove noise of the data in real-time (similar to Kalman ...
Abdullah Nazir's user avatar
2 votes
0 answers
120 views

How to read data in DSX from Data hub object storage container?

I am creating a new notebook with DSX. I have a data set that I want to use. It is not CSV. I found that I could go to Data hub by clicking top-right tile near my logged in name, then object storage ...
Vinayak Agrawal's user avatar
2 votes
0 answers
78 views

What tools are available for semi-automated matching of dirty columnar data

Are there any automated or semi-automated tools for finding matching "similar" or data in two columnar data sets? The data I'm working with was collected (and handled) by different organizations. ...
D. Woods's user avatar
  • 121
2 votes
0 answers
167 views

Combining parameters for Douglas-Peucker Simplification

NOTE : I'm not sure if this is the right forum for this question. if not, please advice. Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect ...
ShahiM's user avatar
  • 121
2 votes
0 answers
199 views

Natural Language text categorization using RapidMiner

I'm new to data mining, so this might sound like a very simple task to some. I work in reliability engineering in aviation and have a set of data that is generated on a daily basis regarding system ...
Yousif Jawhar's user avatar
2 votes
0 answers
90 views

Cross-sell models and additional holders

I would like to pose a question about how to treat additional holders in the propensity-to-buy models of banking products. Up to now I was only taking into considerations the clients as first holders....
Elisa's user avatar
  • 81
2 votes
1 answer
33 views

How to make smaller categories with factor character variables

I have this data set with consist of ISO3166 Alpha-2 codes for countries. Example: DE, AD, AE etc They are coded as factor variables in R and there are about 173 observations. Now because there are ...
Beharrlich's user avatar
1 vote
1 answer
41 views

Should I drop duplicates over features but no target

I'm in a debate with someone about a problem where there are duplicates over features (i.e. $ X_1 = X_2 $ but $ Y_1 != Y_2 $). My point of view is that we should keep those datas, as they can be ...
EzrielS's user avatar
  • 323
1 vote
0 answers
74 views

How can I detect errors/irregularity in a dataset

I've been working on a pathfinding project using topographic data for my software development course, even though I've never really had calculus lessons. The algorithm works fine by itself, however, ...
Maximus's user avatar
  • 11
1 vote
1 answer
44 views

Handling missing value in column having textual data

I have been working on a supervised ML use case where dataset has Numerical (Price), Categorical(Category) and Textual data(Description) as features. Description feature has about 30% missing values. ...
shobhit kumar's user avatar
1 vote
0 answers
63 views

Data Leakage (specific case with standardization)

I read a few blog posts and found some books that caution against data leakage when adjusting the training data using any information from the held out datasets. For example, if I were to standardize ...
dzheng1887's user avatar
1 vote
0 answers
24 views

Data filtering framework

I have procurement data that needs to be labeled with product categories. It's tabular data, containing 700k rows and a mix of data types (dates, free text, floats, etc.) The product set we currently ...
Benjamin B.'s user avatar
1 vote
0 answers
52 views

supply chain analytics dataset?

I need to develop a model which enhances the supply chain and reduces the cost and wastage for perishable products like meat, seafood, dairy products and fruits and vegetables etc. Could someone ...
Abhishek Singh's user avatar
1 vote
1 answer
50 views

Is there a tokenizer to tokenize Swift language code in python

...
Shamsudeen McHalwai's user avatar
1 vote
1 answer
16 views

How to perform linear regression on a parameter that represents state/configuration of a machinery in a production process?

I am trying to perform linear regression on a manufacturing process in order to determine the influencing parameters on a particular product. The thing is there are several production parameters, and ...
Manas Pandey's user avatar
1 vote
0 answers
246 views

Is it possible to use structured(tabular) data as a reinforcement learning environment?

I want to do an RL project in which the agent will learn to drop duplicates in a tabular data. But I couldn't find any examples of RL being used that way - checked the RL based recommendation systems ...
aby's user avatar
  • 11
1 vote
0 answers
28 views

How can one ensure the quality of collectively created ground truth labels?

Here is the situation: I have a small tool to assign (one or multiple) labels to image data. In the tool, you can not only assign a label(s) to the images but also assign a score for the certainty of ...
metaclypse's user avatar
1 vote
0 answers
86 views

How to represent/aggregate data belonging to same group, and feed to model for training?

I have a problem in which there are couple of records that belong to the same group, like multiple observations. For example this data set has three groups, For each group, I have three observations....
abdul's user avatar
  • 113
1 vote
0 answers
442 views

How to combine data from multiple Google Trends queries effectively?

As you might know, Google Trends works by normalising a random sample of the search term data, with the sample changing at least once per day, from my experience. This is not an issue for western ...
PhilipTsv's user avatar
1 vote
0 answers
141 views

how to extract the following keywords containing cells from all columns of csv dataset and copy to new Analysis column

...
ayesha's user avatar
  • 11
1 vote
0 answers
24 views

Tokenizer returning incorrect values and losing a lot of data

(cross posted from main stackoverflow) This is a weird situation so I hope I can explain it correctly. My partner and I are working on a ML project where we create a model that predicts whether a ...
hoshii_tomato's user avatar
1 vote
0 answers
25 views

Missing data for time-series forecasting/classification

I want to train a model to predict some customer behavior. I have two types of data that I have to deal with. My measurements are daily. The first type is regular measurements without any missing data,...
kangaroobogaloo's user avatar
1 vote
0 answers
56 views

How to handle unclassifiable data in the dataset

Premise: Classification problem Input is three text fields Output classes are A, B, A&B (Note: A and B are not always exclusive though usually are, hence the 'A&B' class) Sci-Kit Learn is the ...
Chor Hatara Hud'u Keturi's user avatar
1 vote
0 answers
25 views

How to handle insufficient information about the collected data in the dataset?

I am trying to create a ML model to predict foodborne diseases. I found online a few datasets with the informations I need. The problem is that none of them explains how they got the medical measures. ...
FateMintTrio's user avatar
1 vote
0 answers
18 views

Problem with FFT in Python for smaller df

...
Subhadip Saha's user avatar
1 vote
1 answer
18 views

Full gaps in the curve

This is the data from Irelands vaccine program. There are these missing bits of data. Is there away to fill these gaps with proportional data to make a smooth curve?
GethLife's user avatar
1 vote
0 answers
57 views

Group points to reduce data set such that the linear regression stays the same

I have a very long dataset and I'm trying to reduce it by grouping the data in periods of 24 hours. In this way, there will be a single data point that represents that day, but they must yield the ...
Schroeder's user avatar
1 vote
0 answers
9 views

Identifying Mixed Label Duplicates In Large Data Set

I have a data set containing 250k documents and 400k labeled topics. Each document may have 200+ topics. These topics may significantly overlap in subject and create noise. I want to reduce the ...
micah's user avatar
  • 111
1 vote
0 answers
31 views

How does a misrepresented disproportionate data affects modelling?

Let's say I have a dataset of the occurrence of pregnancies each time is tried, the ground truth of success to failure rate is 30:70. But the dataset with me now is a 70:30 dataset. How would that be ...
user122977's user avatar
1 vote
0 answers
62 views

Dataset format for Transformer text-generation

I'm trying to find some tutorials on training Transformer for generating comments on articles. So far, I found an article showing how to train GPT2 as a chat-bot. Input files in that example are given ...
Bojan Vukasovic's user avatar
1 vote
0 answers
19 views

Does it make more sense to use central tendency methods on training and test sets separately?

I'll explain further, so I'm taking a data science course on cleaning and preparing data and I'm on the how to handle missing data section. So the question is essentially what you see above except it ...
Isaac Stong's user avatar
1 vote
1 answer
22 views

Should I apply a transformation to columns with INTEGERS, in case I want to reduce the skewness of that column?

I am performing EDA on a dataset of Hotel Reservations. Target is Categorical stating if a given customer will cancel the reservation or not. Dataset has 25 features, 30244 entries. I have two ...
leahnanno's user avatar
1 vote
1 answer
21 views

Handling Missing Inconsistent Educational Data

I'm an educational researcher learning about machine learning so I can further explore my data beyond the usual statistics. I currently have some assessment data but I am not sure how to appropriately ...
VBNub's user avatar
  • 11
1 vote
0 answers
731 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
Yogesh's user avatar
  • 111
1 vote
1 answer
24 views

Approach to labelling new data for training

Overview: Imagine an application that identifies cats and dogs from their phone camera. User's take a photo of their pet and it tells them if it is a dog or a cat. The data is then sent to the server. ...
GILO's user avatar
  • 111
1 vote
0 answers
36 views

How to take the keywords from the given dataset to train GPT-2 based chatbot?

I am working with a dataset that contains Questions on various Events conducted by a college and the corresponding answers for the queries. I am using this dataset to train a GPT-2 355M model to ...
Shivang Kohli's user avatar

1
2 3 4 5