Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0
votes
1answer
16 views

Create new column with rounded values

I am trying to create a new column in this dataset. I would like to have rating to be 1, 2, 3, 4 or 5 instead of 1.25, 2.00, 3.75, ..., 5.00. Can you help me? The only solution I found is this ...
2
votes
0answers
13 views

How to perform data scaling/standardization on dataset containing grouped values?

So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example: ...
2
votes
1answer
24 views

Data-preprocessing for Machine Learning model

I am confused about how to preprocess range based category such as age, tumor-size & inv-nodes. Should I take an average of the limits, as in - 14.5, 24.5 and so on or do one hot encoding of the ...
1
vote
0answers
17 views

Excluding “mislabeled” examples in training set based on out of time data

This question is specifically regarding imperfect labels. I'd like to understand the theoretical and practical implications of removing examples from the training set based on information obtained ...
1
vote
0answers
12 views

Is it necessary to transform data to normal distribution when removing outliers for xgboost?

sorry if this is statistics 101 but i cannot find a similar question. I am wanting to use xgboost to classify my data in two classifications. my data is numerical (financial statement data) and i can ...
0
votes
0answers
19 views

How to clean messy column and reshape data structure in pandas

I have a dataset that looks like this. There are 122 columns of coordinates excluding the date (Time) column and 7000 rows. What I want to do is split the lat and lon into two columns, and then ...
0
votes
0answers
13 views

Some doubts in sklearn.preprocessing.Normalizer ? Please explain in lucid manner without jagron

In sklearn.preprocessing.Normalizer it mentioned that “”Scaling inputs to unit norms is a common operation for text classification or clustering for instance. For instance the dot product of two l2-...
0
votes
0answers
19 views

Multiple Values for One Day

I have two questions. 1- I have weather data of 10 turbines and I know their collective production(Power).I also know maximum power a turbine can make. How can I forecast collective production if I ...
0
votes
1answer
12 views

Subdivide a numerical vector with a normal distribution

I have a numerical array of prices values. I'd like to do classification on this parameter, so I'd like to create a certain number of classes with the same granularity. I'd like to create a ...
1
vote
1answer
20 views

Filling missing values for Embedded List in Python3

I searched for a similar question but I didn't come across. And I'm new in this area, I hope I explained my question well enough. I have a dataset consist of text data. I store them in a list and ...
0
votes
0answers
9 views

How to identify corresponding record of a car from a semi-structured string?

I am trying to build an application that can take a record of a car from different websites, compare it to data i have in a CSV file and return me the matching row. Each website will present and ...
0
votes
1answer
21 views

How do I iterate over my images in dataset?

I am building an autoencoder with help from this site. There I was trying to build an autoencoder for my own custom data. My images are stored in a folder IMG and have names like ...
0
votes
0answers
17 views

How to use data provided in EEM20 forecasting competition?

I am new to competitions. I have gone through few Kaggle competitions and most of the time, they provide train data, test data and other supplementary data. Recently, I came across a new competition (...
0
votes
0answers
18 views

Flow System - Analysing Measurement Errors

I have a closed system with multiple flow measurement points as shown; where in theory (A1 + A2 + A3) = (B1 + B2 + B3) = C = (D1 + D2) and D1 = (E1 + E2 + E3 + E4) D2 = (F1 + F2 + F3 + F4) Some of ...
1
vote
1answer
41 views

How to fix spelling mistakes in data?

I have an input data file which contains list of drug names. I have more than 1000 unique drug names. However, the drug names has spelling mistakes and space character issues. For ex: we have ...
1
vote
1answer
44 views

Dummy Variable Trap

In my course about machine learning I'm studying multiple linear regression and we talked about dummy variable trap. I have a data set which contains country, height, weight, gender of every person ...
1
vote
0answers
39 views

Data Exploration Led Conversion to Ordinal Variable

[I encountered this question at an interview few weeks ago and I am still not clear.] If all the values in a categorical column fuel_mileage come from the set <...
0
votes
1answer
37 views

Remove all columns where the entire column is null

I have a very dirty csv where there are several columns with only null values. I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
0
votes
0answers
3 views

Manipulating the relative weight of tokens inside CountVectorizer

This is transparently a classic IMDB data recommender question. I'm trying to build a recommender system that suggests movies that a user is likely to enjoy. If I have my terminology correct I am ...
0
votes
0answers
17 views

How could you predict customer churn using transactions date?

I am new to machine learning and I would like to predict churn using dates of transactions. I tried to prepare my data and I couldn't obtain good results. I would like to predict unique Customers, ...
0
votes
0answers
36 views

Encoding with OrdinalEncoder : how to give levels as user input?

I am trying to do ordinal encoding using: from sklearn.preprocessing import OrdinalEncoder I will try to explain my problem with a simple dataset. ...
3
votes
5answers
219 views

How can I handle a column with list data?

I have a dataset which I processed and created six features: ['session_id', 'startTime', 'endTime', 'timeSpent', 'ProductList', 'totalProducts'] And the ...
1
vote
1answer
32 views

How do I change the location of Google BigQuery

So, I'm trying to use Google BigQuery for the first time for a project of mine, and I'm a bit confused. The documentation isn't helping much, and it looks like all the Google employees are gone thanks ...
0
votes
1answer
15 views

Is it acceptable to make new independent variables out of old ones?

I've recently been tasked with an Data Science interview assignment and looking over the variables, I wondered if it is professionally acceptable to make a new independent variable out of old ones (or ...
1
vote
1answer
19 views

What is the best way to match street addresses in a dataset in Python?

I have a dataset of land parcels owned by the government and I am attempting to match street addresses to an existing list of government agencies. I've used fuzzy matching and used a regex that ...
10
votes
5answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
0
votes
1answer
24 views

Standardizing features by one specific feature

I am working on a project with a dataset that looks something like the following: ...
0
votes
1answer
55 views

how to use feature engineering on a label

I found a dataset about DDOS attacks id like to use but the tool used to extract features from pcap files seems to have indented the columns a little weird as seen below there are indents in unusual ...
0
votes
0answers
6 views

Identifying data distribution for my data and model building strategy

i am a beginner for panel data econometrics. Need help of experts of the field deciding if it is panel or pooled data or should i use any other methodology. I have data from wind mill and we use wind ...
4
votes
2answers
115 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
1
vote
0answers
16 views

Find the mode value and frequency in R

I'm trying to come up with a function in R that gives the mode value of a column along with the number of times (or frequency) that the value occurs. I want it to exclude missing (or blank) values, ...
0
votes
1answer
17 views

How to transform and cummulate data in python

I would like to transform dataframe and cummulate them using pandas. year country value 1999 JAPAN 10 2000 KOREA 15 2000 USA 20 2001 USA 13 2002 JAPAN 30 * I want to transform ...
0
votes
0answers
19 views

conversion of any date to datetime in pandas dataframe

I want to define a function that takes in a pandas dateframe and iterates through its columns to find if there is any date field or a timestamp which can be an object. I want to convert those fields ...
0
votes
2answers
41 views

Data Cleaning, refining etc

I have data as below: Carpanter Carepnter Carpentor Labourer Labor Labour Housewife House Wife housewife. I want to clean data and rectify the spelling mistakes ...
14
votes
10answers
3k views

How can I appropriately handle cleaning of gender data?

I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform analysis of it. In this dataset, the gender field ...
0
votes
0answers
68 views

Since is not possible test all the possible combination, what is the correct procedure to follow on building Machine Learning?

Sorry, I'm a little confused and this is a general question. How can I be sure that the procedure that I am following is the correct one? Following the steps for building a machine learning model, we ...
1
vote
1answer
15 views

Data Under fitting and its handling

Regularization is used to handle over-fitting problem. Similarly can we have some methodology to overcome under fitting problem or merely adding new features or training data will help us in reducing ...
0
votes
0answers
22 views

One Hot Encoding where all sequences don't have all values

Is there a way (other than manually creating dictionaries) to one hot encode sequences in which not all values can be present in a sequence? sklearn's OneHotEncoder ...
0
votes
1answer
23 views

redundancy or functional dependency of two columns

Being a beginner in Data Analysis I ask you not to be much judgemental about this question. I was tring to find if there is a standard way (function?) in Pandas module to identify redundancy of a ...
0
votes
0answers
15 views

Trouble understanding how I could use multivariate time series to predict when an error will occur?

First off, I have very limited knowledge statistics-wise and am more of a coder. I was thrown into a large scale project and could use some guidance. I have a large multivariate time series dataset ...
0
votes
1answer
16 views

Why normalize function has a different result on a matrix vs single value?

I have a matrix like: B=[ 1.5035; 1.5728; 1.6485; 1.5369; 1.5467; 1.572; 1.5374; 1.787; 1.5825; 1.6905]; Using normalize function like normalize(B,'range') ...
2
votes
4answers
560 views

Is it good practice to convert columns with a number to a range between 0 and 1?

Relatively new to data science. I heard something about converting columns which contain integers into a range between 0 and 1. I think the reasoning was that so all the columns will be more similar ...
0
votes
0answers
23 views

Input missing data with interpolation [duplicate]

When dealing with missing values there are several strategies that can be used such as: imputing with the mean, median, a fixed value... I have read that you can do things such as interpolation. ...
0
votes
1answer
19 views

Python how to add a condition to the groupby while calculating the median of a field?

I need to add a new condition to the following line: train["Age"].fillna(train.groupby("Ticket")["Age"].transform("median"), inplace=True) Currently, its taking ...
0
votes
2answers
31 views

Replace Values with corresponding numbers in the same dataframe

Please help me to replace NR values with corresponding values as in the dataframe.
1
vote
1answer
103 views

PySpark: How do I specify dropna axis in PySpark transformation?

I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument ...
0
votes
0answers
16 views

box plot and the anomaly detection?

i'm doing a classification problem with 50000 rows × 5000 columns of dataset. calssification label is 300 labels 1. This is the few example of box plot of some feature. what can I inference from ...
0
votes
0answers
77 views

Normalizing Feature/Label with Negative Values

I am creating a neural network using tensorflow that predicts the energy consumption of a vehicle. Originally, I planned on normalizing all of the features from 0 to 1 using the scikit-learn object ...
-3
votes
1answer
32 views

How to do feature selection for classification problem? Which technique will work? [closed]

I have 200 variables with 200000 records. How to find best features from this variables? I have tried correlation technique via Heatmap but all the variables have near to same correlation score < 0....

1
2 3 4 5
10