Questions tagged [pandas]
pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.
1,216
questions
0
votes
0
answers
6
views
Can label encoder be used to deal with unordered category variables?
Many people say that one hot should be used to deal with unordered category variables, but I see that there is a high praise project on kaggle, which directly uses label encoder to deal with unordered ...
1
vote
2
answers
45
views
python compare 2 dataframes [closed]
in reference to this post, python-pandas-compare-columns-in-separate-dataframes-then-delete-non-matching, I can't get the isin function to work with my 2 example dataframes :
df 1 :
...
- 11
0
votes
2
answers
43
views
+50
Global feature importance for classification problem where some values are irrelevant
I have a binary classification problem where most of the features are categorical with 4 possible values: yes, no, irrelevant, nan. I am trying to find the modular global feature importance of those ...
- 51
0
votes
0
answers
11
views
need to get value based on condition in pandas [closed]
I have a excel where I have CustomerName and ID. I have a separate customer transaction in anther excel I need to assign coustomerID in the Transaction excel based on the name of the customername. ...
0
votes
0
answers
31
views
I'm iterating a dataframe with a for loop, does anybody know a faster way of doing this? [migrated]
I'm working on a dataframe where I need to add a new column which is based on other existing rows in the dataframe. Here is a simplified version of what I'm trying to do. Basically there is a ...
- 1
0
votes
1
answer
15
views
How do I not display rows that have an empty value when trying to output a dataframe with pandas [closed]
I have this table where there are missing values under the value2 column.
Value1
Value2
1000
1000
1000
500
1000
560
1000
560
What I would like to do is to display the above table but without ...
- 15
0
votes
0
answers
11
views
Constructing pandas DF with model trained categorical variable
I've trained a lightGBM model on a dataset X where X has a categorical (in the pandas sense) variable. This model trains fine and when I predict using it all looks good - I can even change the value ...
0
votes
0
answers
15
views
Changing a 0-1 column datatype from int64 to uint8 such as in pandas.get_dummies()
Is it advisable to change the datatype int64 of a 0-1's column to uint8 such as ...
1
vote
0
answers
16
views
Create a Nested Dictionary from a Pandas Dataframe [migrated]
I have a dataframe with columns like this:
Name, Education, DOB, ...
0
votes
0
answers
15
views
How to find location for stretch of values more than fixed number and their average in dataframe python3 [migrated]
I have a data-frame something like this:
And I wish to find the the location (start and end) and the average of the stretch of stretch where values in column 2 are more than 0.2. Something like this:
...
- 3
0
votes
1
answer
41
views
Merging two datasets with different features for machine learning prediction
I'm trying to create a model which predicts Real estate prices with xgboost in machine learning, my question is : Can i combine two datasets to do it ?
First dataset : 13 features
Second dataset : ...
0
votes
0
answers
8
views
Turning multiple binary columns into categorical (with less columns) with Python Pandas
I want to turn these categories into values of categorical columns. The values in each category are the current binary columns present in the data frame. We have : A11, A12.. is a detail of A1 so if ...
- 21
0
votes
1
answer
49
views
How to groupby and sum values of only one column based on value of another column
I have a dataset that has the following columns: Category, Product, Launch_Year, and columns named 2010, 2011 and 2012. These 13 columns contain sales of the product in that year. The goal is to ...
- 23
1
vote
1
answer
28
views
How to return the number of values that has a specific count
I would like to find how many occurrences of a specific value count a column contains. For example, based on the data frame below, I want to find how many values in the ID column are repeated twice
<...
- 15
0
votes
1
answer
14
views
How does Pandas store timezone information?
Timestamp columns are stored internally in UTC, regardless of whether they have a timezone or not. This is stated in the docs:
https://pandas.pydata.org/docs/user_guide/timeseries.html#working-with-...
- 101
1
vote
2
answers
175
views
Compare string entries of columns in different pandas dataframes
I have two dataframes, df1 and df2, both with different number of rows.
df1 has a column 'NAME', a short string; and df2 has a column 'LOCAL_NAME', a much longer string that may contain the exact ...
- 23
0
votes
0
answers
4
views
Identifing this dataset for sanitising
I am beginner here starting with data science for analytics. I am trying to figure out what data set this is and how to read it from python. I have an idea of the steps but not sure how to code it in ...
- 1
0
votes
0
answers
51
views
ValueError: continuous is not supported
I am working on a regression problem and building a model using Random
Forest Regressor but while trying to get the accuracy I am getting
ValueError: continuous is not supported.
...
-1
votes
2
answers
35
views
0
votes
2
answers
106
views
Exploratory data analysis (EDA) on large dataset
I am working with lots of data (we have a table that produces 30 million rows daily).
What is the best way to explore it (do on EDA)?
Take a frictional slicing of the data randomly (100000 rows)
or ...
- 25
0
votes
0
answers
19
views
How to optimize Pandas DataFrame reorganizing?
I have a DataFrame which looks like this:
date
person
value
2022-05-01
A
5
2022-05-01
B
4
2022-05-02
A
5
2022-05-02
B
9
I want to convert it to that form:
...
0
votes
1
answer
87
views
Python: calculate the weighted average correlation coefficient
I am calculating the volatility (standard deviation) of returns of a portfolio of assets using the variance-covariance approach. Correlation coefficients and asset volatilities have been estimated ...
- 101
0
votes
0
answers
26
views
write a dataframe using a column that contains a cycle (number of the weekin every year) in python
I want to write a dataframe with a column of the number of week and make it as a row that contains 53 columns (I mean weeks).
I really need your help.
NOTE THAT: each year has 53 week, and the column ...
- 1
0
votes
0
answers
18
views
NaNs in predictions with LSTM
I have an LSTM model that I have trained and tested it with a dataset. Now I want to test it to an other dataset and I use the following snippet:
...
- 101
0
votes
2
answers
34
views
Create new rows based on a value in a column
My dateset is generated like the example
df = {'event':['A','B','C','D'],
'budget':['123','433','1000','1299'],
'duration_days':['6','3','4','2']}
I need ...
2
votes
1
answer
23
views
The single CSV created by combining a large number of CSV files is too large to process. What options do I have?
The dataset I am currently working on has more than 100 csv files, with each of size more than 250MB. These are files containing time series data captured from different locations and all the files ...
- 21
1
vote
0
answers
43
views
0
votes
0
answers
23
views
How can i merge two datasets with similar words in python?
For instance i have a row value on the dataset_1 : "Entity" = Apple
dataset_2 : "Entity" = iCloud Apple
(Entity is the column) I need to merge one dataset to the other by the ...
- 1
0
votes
0
answers
23
views
How to sample a dataframe or numpy array with a particular interval?
I have the following dataframe :
...
- 1
0
votes
1
answer
54
views
Plot multiple time series from single dataframe
I have a dataframe with multiple time series and columns with labels. My goal is to plot all time series in a single plot, where the labels should be used in the legend of the plot. The important ...
2
votes
0
answers
23
views
Not able to encode multiple categorical columns at once
I have written the following code for encoding categorical features of the dataframe( named 't') -
...
- 21
2
votes
0
answers
24
views
How to perform Timeseries Forecasting on dataset with repeating dates?
I have a dataset, I have to perform timeseries forecasting on that dataset. The dataset has date column, the dates in date column are duplicated. We have 5 classes, a date will have a sales record for ...
0
votes
0
answers
8
views
Autocorrelation Chart showing different colours and no Shadow for checking the Orders to be used for ARIMA
I have been doing a project on Time Series data and one of the items was to test for AutoCorrelation and found the charts in a different color than what I expected. I also don't see the shaded part ...
0
votes
1
answer
14
views
What to do with missing origin city values
Hello fellow data scientists.
I am new in this field, and I face to a problem, for what I need an advice.
So I have data, where one column is product ID, and another which say from which city it ...
0
votes
0
answers
6
views
0
votes
0
answers
6
views
Prophet Weight Cloumn On Time Series (Python)
Can we add a weight column for times on Prophet? So I want the values of old dates to be less important while recent dates are more important. Is there a parameter or method for this? Or does the ...
- 113
0
votes
0
answers
6
views
Handling missing values in IP addresses and key-like features
I have a log dataset that contains +30 features. One group of these features are of the following type, for example, request_id, user_partyrole_id, authentication_id, user_login_key and such ip and ...
0
votes
1
answer
23
views
How to set a new numeric index
I have read data from a json file, and the dataframe looks like:
Col1 Col2 Col3
A: 1 Hi 7
B: 22 CC 1
I think it is because the file was a ...
0
votes
0
answers
12
views
Formatting df's multi index
I have a multi-index dataframe used in a block of code. It's index looks like this:
...
- 101
0
votes
0
answers
41
views
regression.fit(x_train, y_train) is not working on python
I try to deal with my homework. The Job is to take the Achievement data and perform a multi-linear regression on it.
The code is published here.
I am quite new to programming in Python and in data ...
0
votes
0
answers
135
views
ValueError: Input contains NaN, infinity or a value too large for dtype('float64') - fitting a model
I am attempting to train a model on a pandas DataFrame and am running into ValueError issue. I have verified that there are no ...
- 302
1
vote
1
answer
40
views
max temperature for every month given several months and several years
I'm not sure if this is the right place to put this.
So I have a dataframe consisting of just 4 columns but around 2000 rows.
It's in a csv format. A typical value of the first column which is titled ...
- 133
1
vote
0
answers
15
views
How should I OneHotEncod a column of (8128 rows and) 2058 nuniques?
The title, pretty much.
I just want to know the best and most efficient way to OneHotEncode a column with like 2058 nuniques. Doing a ...
- 111
0
votes
0
answers
24
views
0
votes
0
answers
96
views
pandas - Find rows where text in one column is in text in another column using vectorized operations
I wish to find instances where text in col2 is contained in col1; ultimately, I want to remove the text from col1 (replace the col2 text in col1 with '').
I want to achieve this using vectorized ...
- 121
1
vote
0
answers
30
views
How to solve this IndexError? [closed]
I have created a training dataframe Traindata as following:
dataFile='/content/drive/Colab Notebooks/.../Normal_Anomalous_8Digits.csv'
data8=pd.read_csv(dataFile)
...
- 21
0
votes
2
answers
80
views
How can I add a column indicating clusters on my original df? Python
I have clustered pitchers in baseball based off of averages of advanced metrics. I am working with statcast data, so every data point is a pitch thrown as follows:
pitcher_name
batter_name
spin_rate
...
0
votes
1
answer
46
views
what exactly is the "order"-parameter in pandas interpolation?
what exactly is the "order"-parameter in pandas interpolation?
it is mentioned in the doc:
...
- 11
1
vote
1
answer
139
views
how many rows does pandas' interpolate consider? [closed]
How does pandas' DataFrame.interpolate() work in relation to the amount of rows it considers:
is it just the row before the NaNs and the row right after?
Or is it the whole DataFrame (how does that ...
- 11
1
vote
1
answer
39
views
Natural language processing [closed]
I am new to NLP. I converted my JSON file to CSV with the Jupyter notebook. I am unsure how to proceed in pre-processing my data using techniques such as tokenization and lemmatization etc. I ...
- 11